OdysseusNet Getting Started

Overwiew

In this tutorial we will going to setup Odysseus Studio with OdysseusNet capabilities and start it as a master node. Then worker nodes are started with docker-compose and will be dynamically added to the master node with OdysseusNet in Odysseus Studio. At next queries will be remotely executed on the worker nodes from the master node.

Setup of Odysseus Studio with OdysseusNet

Setup Odysseus with Development with Odysseus, but change the following

Clone the odysseusnet-repository https://git.swl.informatik.uni-oldenburg.de/scm/oi/odysseusnet.git
Choose the development-branch
Choose the target_development_stable.target within Eclipse RCP

Now you have installed the target platform, before you launch the product, we are going to add the OdysseusNet monolithic target to the normal monolithic target (see for a general tutorial Adding features to products) :

Double click the de.uniol.inf.is.odysseus.monolithic.product.product under the directory de.uniol.inf.is.odysseus.monolithic.product.product in the Eclipse Project explorer
Go the the Contents-Tab
Add de.uniol.inf.is.odysseus.net.monolithic.feature to the list of features
Save with CTRL+S
Go to the Overview-Tab, synchronize and launch the eclipse application, this launches the Odysseus Studio (see Odysseus Studio)

The OdysseusNet monolithic target also includes the OdysseusNet server target.

If successfully started you can close Odysseus Studio for now for changing OdysseusNet config under ODYSSEUS_HOME (OdysseusNet Configuration)

With Linux the ODYSSEUS_HOME (ODYSSEUS_HOME) is ~/.odysseus, there you can change the odysseusNet.config to

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE properties SYSTEM "http://java.sun.com/dtd/properties.dtd">
<properties>
<comment>OdysseusNet Property File edit only if you know what you are doing</comment>
<entry key="net.networkoperatorgenerator">websocket</entry>
<entry key="net.node.default.username">System</entry>
<entry key="net.dd.local">true</entry>
<entry key="net.node.name">OdysseusNode_1554</entry>
<entry key="net.discoverer.name">BroadcastOdysseusNodeDiscoverer,IPListOdysseusNodeDiscoverer</entry>
<entry key="net.querydistribute.allocation">querycount</entry>
<entry key="net.source.lifetime">3600000</entry>
<entry key="net.connect.selector.name">GroupSelectorComponent</entry>
<entry key="net.nodemanager.storetype">filestore</entry>
<entry key="net.nodemanager.filename">/home/$USER/.odysseus/store/nodes.store</entry>
<entry key="net.dd.checkinterval">30000</entry>
<entry key="net.remoteUpdate">false</entry>
<entry key="net.node.communicator">rest</entry>
<entry key="net.node.preserveid">false</entry>
<entry key="net.querydistribute.minport">10000</entry>
<entry key="net.autostart">true</entry>
<entry key="net.querydistribute.partition">querycloud</entry>
<entry key="net.querydistribute.randomport">false</entry>
<entry key="net.discoverer.interval">5000</entry>
<entry key="net.node.default.password">manager</entry>
<entry key="net.node.group">OdysseusGroup</entry>
<entry key="net.querydistribute.maxport">20000</entry>
<entry key="net.logging.receive">false</entry>
</properties>

Replace $USER with your local user

Setup of Odysseus worker nodes

Now we start Nexmark (a data stream simulation tool, see Getting Started with Nexmark) and Odysseus worker nodes with docker-compose, therefore create a docker-compose.yml in a directory (it will be named $CURRENT_DIRECTORY here for reference, in practice it could be e. g. the odysseusnet-repo) and with the following content (adopted from OdysseusNet Docker Compose Example):

version: '3.3'

services:
    nexmark:
        image: odysseusol/nexmark
        ports:
           - 65440-65443:65440-65443
    worker01:
        image: odysseusol/odysseusnet
        stdin_open: true
        ports:
            - 18881:8888
        volumes:
            - ./worker01:/var/lib/odysseus

    worker02:
        image: odysseusol/odysseusnet
        stdin_open: true
        ports:
            - 18882:8888
        volumes:
            - ./worker02:/var/lib/odysseus

    worker03:
        image: odysseusol/odysseusnet
        stdin_open: true
        ports:
            - 18883:8888
        volumes:
            - ./worker03:/var/lib/odysseus

As of now a normal Ubuntu 20.04 only supports docker-compose yml syntax to version 3.3 instead of 3.8 in Windows as stated in other yml files, but most of them are downwards compatible, because newer features are not that widespread at the moment.

As a workaround make sure the worker01-03 directories have the 777-permissions with:

sudo chmod 777 -R worker0*

Otherwise the internal Odysseus worker node user cannot access the volume properly and fails.

All configuration files will be placed under the $CURRENT_DIRECTORY, e. g. there are the following files:

logs odysseus.conf odysseusNet.conf reloadlog.store scheduling.conf store

You can then start the docker stack with:

docker-compose up -d

Setup of Odysseus Webstudio

You can use the Odysseus Webstudio (WebStudio) for monitoring and interacting with the worker nodes. To install it, first clone the Webstudio-repo (https://git.swl.informatik.uni-oldenburg.de/projects/API_APPS/repos/webstudio/browse?at=refs%2Fheads%2Fdevelopment) and checkout the development-branch with

git clone https://git.swl.informatik.uni-oldenburg.de/scm/api_apps/webstudio.git
git checkout -b development

Then create and give the all permissions to everyone to the following directories as a workaround:

mkdir frontend odysseushome
chmod 777 -R frontend odysseushome

For starting the docker stack you can set the images accordingly in the docker-compose.yml file:

version: '3.3'

services:
  frontend:
    image: odysseusol/webstudio_frontend
    #build: frontend
    ports:
      - "4200:80"
    environment:
      NODE_ENV: ${NODE_ENV}
    restart: always
    depends_on:
      - backend

  backend:
    image: odysseusol/webstudio_backend
    #build: backend
    ports:
      - "3000:3000"
    environment:
      NODE_ENV: ${NODE_ENV}
      MONGODB_URI: ${MONGODB_URI}
      SECRET_TOKEN: ${SECRET_TOKEN}
      DOWNLOAD_TOKEN: ${DOWNLOAD_TOKEN}
      PROJECTS_FOLDER: ${PROJECTS_FOLDER}
    restart: always
    depends_on:
      - database
    volumes:
      - backenddata:/home/node/app/projects
      - ./backend/logs:/home/node/app/logs

  database:
    image: mongo
    restart: always
    ports:
      - "27017:27017"
    volumes:
      - mongodbdata:/data/db

  odysseus:
    image: odysseusol/odysseus
    restart: always
    ports:
      - "8889:8888"
    volumes:
      - ./odysseushome:/var/lib/odysseus

volumes:
  mongodbdata:
  backenddata:
  odysseushome:

At last you can start the stack with:

docker-compose up -d

Add the nodes in Odysseus Studio

Before adding the worker nodes to Odysseus Studio, stop the OdysseusNet feature in the nodes tab, it will be started automatically when adding the nodes dynamically with OdysseusScript.

Before you can install queries on the nodes, you have to add the worker nodes with:

#CONFIG DISTRIBUTE true
#PARSER PQL

#DEFINE HOST 192.168.188.36
#ADD_NODE (name=worker_1) (serverAddress=${HOST}) (serverPort=18881) (username=System) (password=manager)
#ADD_NODE (name=worker_2) (serverAddress=${HOST}) (serverPort=18882) (username=System) (password=manager)
#ADD_NODE (name=worker_3) (serverAddress=${HOST}) (serverPort=18883) (username=System) (password=manager)

Install the remote query

Run the following OdysseusScript query in Odysseus Studio.

#REMOTEQUERY (name=worker_1)
#PARSER PQL


#RUNQUERY
nexmark:person := ACCESS({source='nexmark:person',
    wrapper='GenericPush',
    transport='TCPClient',
    protocol='SizeByteBuffer',
    dataHandler='Tuple',
    options=[
        ['host', 'nexmark'],
        ['port', '65440'],
        ['ByteOrder', 'LittleEndian']
        ],
    schema=[
        ['timestamp', 'STARTTIMESTAMP'],
        ['id', 'INTEGER'],
        ['name', 'STRING'],
        ['email', 'STRING'],
        ['creditcard', 'STRING'],
        ['city', 'STRING'],
        ['state', 'STRING']
        ]
    })
    
    

#RUNQUERY
out = nexmark:person

Remark, that the host is 'nexmark' instead of 'localhost', because the remote worker node 'worker_01' is started in the default docker network and access the nexmark by the DNS resolved service name 'nexmark'

See the remote query output

Here is described how you can see the remote query output.

Odysseus Studio

You can the following query in Odysseus Studio to access the output (Reading Query Results via REST/Websocket):

#PARSER PQL
#RUNQUERY
in = ACCESS({
    transport = 'WebsocketClient', 
    wrapper = 'GenericPush',
    datahandler = 'Tuple',
    protocol = 'CSV',
    source = 'access',
    readMetadata = 'true',
    options = [
        ['uri','ws://localhost:18881/queries/0/a2e1a574-6c72-4961-b304-111c01708445/0/CSV/fif8htets00ql9q71okbuu7qs9']
    ],
    schema = [['timestamp','StartTimestamp'],['id','Integer'],['name','String'],['email','String'],['creditcard','String'],['city','String'],['state','String']]
    }
)

You have to adjust the optioins-uri in the query with the rootOperators-websockets-CSV uri from http://localhost:18881/queries/0

You can then show the last x elements:

Odysseus Webstudio

In Webstudio you can see the query output unter the queries tab:

Space shortcuts

Page tree