...

Odysseus has a wide range of operators build in and are explained here.

Base Operators

ACCESS

Description

The access operator can be used to integrate new sources into odysseus. Further information can be found in the Documentation to the "Access Operator Framework".

Parameter

• source: The name of the access operator
• wrapper: In Odysseus the default wrappers are GenericPush and GenericPull
• transport: The transport defines the transport protocol to use.
• protocol: The protocol parameter defines the application protocol to tansform the processing results.

• datahandler: This parameter defines the transform of the single attributes of the processing results.
• options: Transport protocol and application protocol depending options
• schema: The output schema of the access operator (may depend on the protocol handler)

Example

AGGREGATE

Description

This operator is used to aggregate and group the input stream.

Parameter

• group_by: An optional list of attributes over which the grouping should occur.
• aggregations: A list if elements where each element contains up to four string:
  
  • The name of the aggregate function, e.g. MAX
  • The input attribute over which the aggregation should be done
  • The name of the output attribute for this aggregation
  • The optional type of the output
• dumpAtValueCount: This parameter is used in the interval approach. Here a result is generated, each time an aggregation cannot be changed anymore. This leads to fewer results with a larger validity. With this parameter the production rate can be raised. A value of 1 means, that for every new element the aggregation operator receives new output elements are generated. This leads to more results with a shorter validity.

...

output = aggregateAGGREGATE({group_by = ['bidder'], aggregations=[ ['MAX', 'price', 'max_price', 'double'] ]}, input)

ASSUREHEARTBEAT

Description

This operator assures that there will be periodically a heartbeat to avoid blocking because of missing information about time progress. The operator garantees, that no element (heartbeat or streamobject) is send, that is older than the last send heartbeat (i.e. the generated heartbeats are in order and indicate time progress). Heartbeats can be send periodically (sendAlwaysHeartbeats = true) or only if no other stream elements indicate time progess (e.g. in out of order scenarios) independent if a new element has been received or not.

Parameter

• RealTimeDelay: How long should the operator wait in transaction time (real time) before it should send a punctuation

• ApplicationTimeDelay: How long is the realTimeDelay in terms of application time (typically this should be the same, but for simulations this could be adapted)

• timeUnit: What is the time unit (see Java TimeUnit). Minimum Time unit is milliseconds!

• sendAlwaysHeartbeat: If true, a heartbeat is send periodically for every realTimeDelay. This is usefull for out of order processing

• allowOutOfOrder: If set to true, the operator allows heartbeats to be send, that lie before the last send element. In other cases this is not allowed.

Example

out = AssureHeartbeat({realTimeDelay=5000, APPLICATIONTIMEDELAY=5000, SENDALWAYSHEARTBEAT='false', ALLOWOUTOFORDER='false'},nexmark:person2)

BUFFER

Description

Typically, Odysseus provides a buffer placement strategy to place buffers in the query plan. This operator allows adding buffers by hand. Buffers receives data stream elements and stores them in an internal elementbuffer. The scheduler stops the execution here for now. Later, the scheduler resumes to execution (e.g. with an another thread).

Parameter

• Type: The type of the buffer. The following types are currently part of Odysseus:
  
  • Normal: This is the standard type and will be used if the type parameter is absent
  • Currently, no further types are supported in the core Odysseus. In the future, there will be some buffers integrated to treat prioritized elements.
• MaxBufferSize: If this value is set, a buffer will block if it reaches its maximal capacity. No values will be thrown away; this is not a load shedding operator!

Example

BUFFEREDFILTER

Description

This operator can be used to reduce data rate. It buffers incoming elements on port 0 for bufferTime and evaluates a predicate over the elements on port 1. If the predicate for the current element e evaluates to true, all elements from port 0 that are younger than e.startTimeStamp()-bufferTime will be enriched with e and delivered for deliverTime. Each time the predicate evaluates to true, the deliverTime will be increased.

Parameter

• Predicate: The predicate that will be evaluated on element of port 1
• BufferTime: The time in history, the elements will be kept in history
• DeliverTime: The time in the future, the elements will be delivered

Example

BufferedFilter({predicate = RelationalPredicate('id == 2 || id == 4 || id == 10'), bufferTime = 5000, deliverTime = 20000}, nexmark:bid2, nexmark:person2)

CALCLATENCY

Description

Odysseus has some features to measure the latency of single stream elements. This latency information is modeled as an interval. An operator in Odysseus can modify the start point of this interval. This operator sets the endpoint and determines the place in the query plan, where the latency measurement finds place. There can be multiple operators in the plan, to measure latency at different places.

Parameter

Example

CHANGEDETECT

Description

This operator can be used to filter out equal elements and reduce data rate.

Parameter

• ATTR (List<String>): only these attribute are considered for the comparison
• Heartbeatrate (Integer): For each nth element that is filtered out, a heartbeat is generated to state the progress of time
• deliverFirstElement (Boolean): If true, the first element will be send to the next operator.

DIFFERENCE

Description

This operator calculates the difference between two input sets.

Parameter

None

Example

...

 

Code Block
theme Eclipse language javascript title Difference Operator linenumbers true
output = DIFFERENCE(left, right)

 

diff = difference(mapped, projected)

ENRICH

Description

This operator enriches tuples with information from the context store. Further Information: http://odysseus.offis.uni-oldenburg.de/twiki/bin/view/Main/ContextStore

Parameter

• ATTRIBUTES: The attributes from the store object, that should be used for enrichment
• STORE: The name of the store
• OUTER:

Example

EVENTTRIGGER

Description

Parameter

Example

EXISTENCE

Description

This operator tests an existence predicate and can be used with the type EXISTS (semi join) and NOT_EXISTS (anti semi join). The predicates can be evaluated against the element from the first input and the second input.
Semi join: All elements in the first input for which there are elements in the second input that fulfills the predicate are sent.
Semi anti join: All elements in the first input for which there is no element in the second input that fulfills the predicate are sent.

Parameter

• Predicate: The predicate

• Type: Either EXISTS or NOT_EXISTS


Example

...

 

Code Block
theme Eclipse language javascript title Difference Operator linenumbers true
output = EXISTENCE({type='EXISTS', predicate=RelationalPredicate('auction = auction_id')},

...

left,

...

right) output = EXISTENCE({type='NOT_EXISTS', predicate=RelationalPredicate('auction = auction_id')},

...

left,

...

right)

 

FILESINK

Description

The operator can be used to dump the results of an operator to a file.

Parameter

• FILE: The filename to dump. If the path does not exist it will be created.
• FILETYPE: The type of file, CSV: Print as CSV, if not given, the element will be dump to file a raw format (Java toString()-Method)

Example

JOIN

Description

Parameter

Example

joined = join({predicate = RelationalPredicate('auction_id = auction')}, renamed, nexmarkBid)

LEFTJOIN

Description

Parameter

Example

MAP

Description

Performs a mapping of incoming attributes to outcoming attributes using map functions. Odysseus also provides a wide range of mapping functions.

Parameter

• expressions: An array A list of expressions to map multiple incoming atttribute values to outcomping attributes

Example

output = mapMAP({expressions = ['auction_id * 5','sqrt(auction_id)']}, input)

SASE

Description

Dieser Operator ermöglicht es, Anfragen mit zeitlichen Mustern zu definieren. Zu diesem Zweck wird die SASE+ Anfragesprache verwendet und die Anfrage im Parameter query übergeben. Die angebenen Quellen müssen den passenden Typ haben (für das folgende Beispiel also s05 und s08), die Reihenfolge ist egal. Ggf. muss der Typ einer Quelle vorher noch mit Hilfe der Rename-Operation definiert werden. Der Parameter heartbeatrate legt fest, wie oft ein Hearbeat generiert werden soll, wenn ein Element verarbeitet wurde, welches aber nicht zu einem Ergebnis geführt hat.

Parameter

• heartbeatrate
• query
• OneMatchPerInstance

Example

s05 = RENAME({type='s05', aliases = ['ts', 'edge']},...)
PATTERNDETECT({heartbeatrate=1,query='PATTERN SEQ(s05 s1, s08 s2) where skip_till_any_match(s1,s2){ s1.edge=s2.edge } return s1.ts,s2.ts'}, s05, s08)

PROJECT

Description

Parameter

• attributes: A list of attribute names to project on

Example

...

 

Code Block
theme Eclipse language javascript title Map Operator linenumbers true
output = PROJECT({attributes = ['auction', 'bidder']},

...

input)

 

PUNCTUATION

Description

Der Punctuation Operator erlaubt das Einfügen von Punctuation in den Verarbeitungsstrom um so nachfolgende Operatoren vor einem möglichen Buffer-Overflow zu schützen. Hierfür hat der Operator zwei Eingänge I1 und I2. Der Eingang I1 stellt den Dateneingang dar und leitet eingehende Tupel direkt an den Ausgang. Der zweite Eingang I2 dient als Frequenzgeber. Sobald die Differenz der Eingangstupel zwischen dem ersten Eingang I1 und dem zweiten Eingang I2 über einen bestimmten Schwellwert steigt, wird in den linken Strom an I1 eine Punctuation eingefügt und ältere Tupel die eventuell danach eintreffen verworfen.
Der Punctuation Operator kann dabei ein nicht deterministisches Verhalten erzeugen, weil das Einfügen von Punctuations von der aktuellen Systemlast abhängen kann und sollte nur verwendet werden, wenn eine normale Verarbeitung der Daten aufgrund ihrer schwankenden Frequenzen nicht möglich ist.

Parameter

Example

punctuation({ratio = 10}, left, right)

RENAME

Description

Renames the attributes.

Parameter

• aliases: The list new attribute names to use from now on. If the flag pairs is set, aliases will be interpretet as pairs of (old_name, new_name). See the examples below.
• type: By default, each schema has the name of the source from which the data comes from (i.e. it is the name of the type that is processed by this operator). With this parameter the type (source) name can be changed. This is needed in the SASE operator. 
• pairs: Optional boolean value that flags, if the given list of aliases should be interpretet as pairs of (old_name, new_name). Default value is false.

Example

renamed = rename({aliases = ['auction_id', 'bidder_id', 'another_id']}, projected)
(Renames the first attribute to auction_id, the second to bidder_id and the last to another_id.) 

renamedPairs = rename({aliases = ['auction_id', 'auction', 'bidder_id', 'bidder'], pairs = 'true'}, projected)
(Due the set flag pairs, the rename operator renames the attribute auction_id to auction and bidder_id to bidder.)

ROUTE

Description

This operator can be used to route the elements in the stream to different further processing operators, depending on the predicate.

Parameter

Example

route({predicates=[RelationalPredicate('price > 200'), RelationalPredicate('price > 300')], RelationalPredicate('price > 400')}, nexmark:bid2)

SELECT

Description

Parameter

Example

selected = select({ predicate=RelationalPredicate('price > 100') }, nexmarkBid)

...

This operator can be used to publish processing results to multiple endpoints using different transport and application protocols.

Parameter

• wrapper: In Odysseus the default wrappers are GenericPush and GenericPull
• transport: The transport defines the transport protocol to use.
• protocol: The protocol parameter defines the application protocol to tansform the processing results.

• datahandler: This parameter defines the transform of the single attributes of the processing results.
• options: Transport protocol and application protocol depending options


Example

output = SENDER({
wrapper='GenericPush',
transport='TCPClient',
protocol='CSV',
dataHandler='Tuple',
options=[['host', 'example.com'],['port', '8081'],['read', '10240'],['write', '10240']]
}, input)

SINK

Description

Parameter

Example

SOCKETSINK

Description

This operator can be used to send/provide data from Odysseus via a tcp socket connection. (Remark: This operator will potentially change in future)

Parameter

Depending on the parameter push:Boolean (todo: change name!), the parameter have to following meaning:

• push = false (Default): On Odysseus side a server socket connection is opened
  
  • host:String typically 'localhost'
  • sinkport:Integer On which port should the socket be opened
• push = true: Odysseus connects to a server
  
  • host:String Server to connect to
  • port:Integer Port on host to connect to

Example

socketsink({host='localhost', push=false, sinkport=4712, sinkType='bytebuffer', sinkName='driverChannel'}, timestampToPayload(person))

STORE

Description

Parameter

Example

TIMESTAMPTOPAYLOAD

Description

This operator is needed before data is send to another system (e.g. via a socket sink) to keep the time meta information (i.e. start and end time stamp). The input object gets two new fields with start and end timestamp. If this output is read again by (another) Odysseus instance, the following needs to be attached to the schema:

['start', 'StartTimestamp'], ['end', 'EndTimestamp']

Parameter

The operator is parameterless.

Example

driverChannel = socketsink({sinkport=4712, sinkType='bytebuffer', sinkName='driverChannel'}, timestampToPayload(person))

UNION

Description

Parameter

Example

united = union(mapped, projected)

UNNEST

Description

The UnNest operator performs a split on the incoming tuple with a multi value attribute to create multiple tuples

Parameter

• attribute: The attribute to perform the unnest on.

Example

output = unnest({attribute='myAttribute'},input)

UDO

Description

The UDO operator calls a user defined operator.

Parameter

• class: The name of the user defined operator
• init: Additional parameter for the user defined operator

Example

output = udo({class='MyUDO', init='some parameter'}, input)

WINDOW

Description

Parameter

• size: The size of the window
• advance: The advance the window moves forward
• slide: The slide of the window
• type: The type of the window. The possible values are Time, Tuple, and Unbound
• partition: The partition attribute of the window
• startCondition: The start condition for a predicate window
• endCondition: The end condition for a predicate window

Example

//sliding time window
 output = window({size = 5, advance = 1, type = 'time'}, input)

 //sliding tuple window partioniert ueber bidder
 output = window({size = 5, advance = 1, type = 'tuple', partition=['bidder']}, input) 

 //unbounded window
 output = window({type = 'unbounded'}, input) 

 //now window (size = 1, advance = 1)
 output = window({type = 'time'}, input)

 //sliding delta window, reduces time granularity to value of slide
 output = window({size = 5, type = 'time', slide = 5}, input)

 // Predicate window
 output = window({startCondition=RelationalPredicate('a>10'), endCondition = RelationalPredicate('a<10')}, input)

Benchmark

BATCHPRODUCER

Description

Parameter

Example

BENCHMARK

Description

Parameter

Example

PRIOIDJOIN

Description

Parameter

Example

TESTPRODUCER

Description

Parameter

Example

Data Mining

CLASSIFY

Description

Parameter

Example

HOEFFDINGTREE

Description

Parameter

Example

LEADER

Description

Parameter

Example

SIMPLESINGLEPASSKMEANS

Description

Parameter

Example

FREQUENT_ITEM

Description

Parameter

Example

Storing

DATABASESOURCE

Description

Parameter

Example

DATABASESINK

Description

Parameter

Example

Extension of PQL – Make New Logical Operators Available

...

Since there are already a lot of parameter types, firstly check if there is already a suitable parameter. However, if you need a new parameter type, you have to implement the interface IParameter. We recommend using the existing abstract implementation AbstractParameter. We show how to create a new parameter type with the help of an example.
In our example we want to have a parameter that resolves a file from a given path. We first create a new class called FileParameter and inherit the AbstractParamter class. The generic type T that is provided by the AbstractParameter is used to declare the java class that is returned by the Parameter. In our case, this should be the class java.io.File.

You may also generate a serial version uid and create the method stub for the one needed method called internalAssignment(). This method is responsible for transforming the value that is given by the user into the output format (in our case File). We assume that the user provides a string that holds a path. We can access this value via the inputValue variable, so that we are casting this value to string:

Now, we can create a File object and set this object as the resulting value:

As you can see, the output-value is set via the setValue() method. That's all. Notice, that we assume a String so that the user should use a string-based declaration (using apostrophes) like fileparam='C:/example.csv'.
If you need further functionalities, it is possible to override some methods. For example, if you want to validate the input, you can overwrite internalValidateion(). In our example, we can, for example, check whether the file exists or not:

attributes = ['auction', 'bidder']