This document describes the basic concepts of the Procedural Query Language (PQL) of Odysseus and shows how to use the language. In contrast to languages SQL based languages like the Continuous Query Language (CQL) or StreamSQL, PQL is more procedural and functional than declarative. This document shows how to formulate queries with PQL.

Using PQL in Queries

PQL is an operator based language where an operator can be seen as a logical building block of the query. Thus, PQL is the connection of several operators. Since Odysseus differentiates between logical operators and their physical operators, which are the implementing counterpart, PQL is based upon logical operators. Therefore, it may happen that the query gets changed during the transformation from the logical query plan into the physical query plan. This includes also logical optimization techniques like the restructuring of the logical query plan. To avoid this, you can explicitly turn off the query optimization.

Define an Operator

An operator can be used in PQL via its name and some optional settings, which can be compared with a function and the variables for the function:

OPERATORNAME(parameter, operator, operator, ...)

The first variable (parameter) describes operator dependent parameters and is used for configuring the operator. Note, that there is only one parameter variable! The other variables (operator) are input operators, which are the preceding operators that push their data into this operator. The inputs of an operator can be directly defined by the definition of another operator:

OPERATOR1(parameter1, OPERATOR2(Parameter2, OPERATOR3(...)))

Except for source operators (usually the first operator of a query) each operator should have at least one input operator. Thus, the operator can only have parameters:

OPERATOR1(parameter1)

Accordingly, the operator may only have input operators but no parameters:

OPERATOR1(OPERATOR2(OPERATOR3(...)))

Alternatively, if the operator has neither parameters nor input operators, the operator only exists of its name (without any brackets!), so just:

OPERATORNAME

It is also possible to combine all kinds of definitions, for example:

OPERATOR1(OPERATOR2(Parameter2, OPERATOR3))

Intermediate Names, Views and Sources

Since the nesting of operators may lead to an unreadable code, it is possible to name operators to reuse intermediate result. This is done via the "=" symbol. Thus, we can temporary save parts of the query, for example (it is important to place blanks before and after the "=" symbol!) :

Result2 = OPERATOR2(Parameter2, OPERATOR3)

The defined names can be used like operators, so that we can insert them as the input for another operator, for example:

Result2 = OPERATOR2(Parameter2, OPERATOR3)OPERATOR1(Result2)

There could be also more than one intermediate result, if they have different names:

Result1 = OPERATOR1(Parameter1, …)Result2 = OPERATOR2(Parameter2, Result1)Result3 = OPERATOR3(Parameter3, Result2)

And you can use the intermediate name more than one time, e.g. if there are two or more operators that should get the same preceding operator:

Result1 = OPERATOR1(Parameter1, …)OPERATOR2(Parameter2, Result1)OPERATOR3(Parameter3, Result1)

All intermediate results that are defined via the "=" are only valid within the query. Thus, they are lost after the query is parsed and runs. This can be avoided with views.
A view is defined like the previous described intermediate results but uses ":=" instead of "=", e.g.:

Result2 := OPERATOR2(Parameter2, OPERATOR3)

Such a definition creates an entry into the data dictionary, so that the view is globally accessible and can be also used in other query languages like CQL.
Alternatively, the result of an operator can also be stored as a source into the data dictionary by using "::="

Result2 ::= OPERATOR2(Parameter2, OPERATOR3)

The difference between a view and a source is the kind of query plan that is saved into the data dictionary and is reused. If a view is defined, the result of the operator is saved as a logical query plan, which exists of logical operators. Thus, if another query uses the view, the logical operators are fetched from the data dictionary and build the lower part of the new operator plan or query. If an operator is saved as a source, the result of the operator is saved as a physical query plan, which exists of already transformed and maybe optimized physical operators. Thus, reusing a source is like a manually query sharing where parts of two or more different queries are used together. Additionally, the part of the source is not recognized if the new part of the query that uses the source is optimized. In contrast, the logical query plan that is used via the a view is recognized, but will not compulsorily lead to a query sharing.
Finally, all possibilities gives the following structure:

QUERY = (TEMPORARYSTREAM | VIEW | SHAREDSTREAM)+
TEMPORARYSTREAM = STROM "=" OPERATOR
VIEW = VIEWNAME ":=" OPERATOR
SHAREDSTREAM = SOURCENAME "::=" OPERATOR

Parameters – Configure an Operator

As mentioned before, the definition of an operator can contain a parameter. More precisely, the parameter is a list of parameters and is encapsulated via two curly brackets:

OPERATOR({parameter1, paramter2, …}, operatorinput)

A parameter itself exists of a name and a value that are defined via a "=". For example, if we have the parameter port and want to set this parameter to the 1234, we use the following definition:

OPERATOR({port=1234}, …)

The value can be one of the following simple types:

Integer or long: OPERATOR({port=1234}, …)

Double: OPERATOR({possibility=0.453}, …)

String: OPERATOR({host='localhost'}, …)

Furthermore, there are also some complex types:

Predicate: A predicate is normally an expression that can be evaluated and returns either true or false. In most cases a predicate is simple a string, e.g.:

OPERATOR({predicate='1<1234'}, …)

Hint: In some cases the predicate must be in this form PREDICATE_TYPE('1<1234'), where PREDICATE_TYPE can be something like RelationalPredicate.

List:It is also possible to pass a list of values. For that, the values have to be surrounded with squared brackets:

OPERATOR({color=['green', 'red', 'blue']}, …)
(Type of elements: integer, double, string, predicate, list, map).

Map: This one allows maps like the HashMap in Java. Thus, one parameter can have a list of key-value pairs, where the key and the value are one of the described type. So, you can use this, to define a set of pairs where the key and the value are strings using the "=" for separating the key from the value:

OPERATOR({def=['left'='green', 'right'='blue']}, …)
It is also possible that values are lists:
OPERATOR({def=['left'=['green','red'],'right'=['blue']]}, …)
Remember, although the key can be another data type than the value, all keys must have the same data type and all values must have the same data type
Notice, that all parameters and their types (string or integer or list or…) are defined by their operator. Therefore, maybe it is not guaranteed that the same parameters of different operators use the same parameter declaration – although we aim to uniform all parameters.

Ports – What if the Operator Has More Than One Output?

There are some operators that have more than one output. Each output is provided via a port. The default port is 0, the second one is 1 etc. The selection for example, pushes all elements that fulfill the predicate to output port 0 and all other to output port 1. So, if you want to use another port, you can prepend the port number with a colon in front of the operator. For example, if you want the second output (port 1) of the select:

PROJECT({…}, 1:SELECT({predicate='1<x'}, …))

The Full Grammar of PQL

QUERY = (TEMPORARYSTREAM | VIEW | SHAREDSTREAM)+
TEMPORARYSTREAM = STREAM "=" OPERATOR
VIEW = VIEWNAME ":=" OPERATOR
SHAREDSTREAM = SOURCENAME "::=" OPERATOR
OPERATOR = QUERY | [OUTPUTPORT ":"] OPERATORTYPE "(" (PARAMETERLIST [ "," OPERATORLIST ] | OPERATORLIST) ")"
OPERATORLIST = [ OPERATOR ("," OPERATOR)* ]
PARAMETERLIST = "{" PARAMETER ("," PARAMETER)* "}"
PARAMETER = NAME "=" PARAMETERVALUE
PARAMETERVALUE = LONG | DOUBLE | STRING | PREDICATE | LIST | MAP
LIST = "[" [PARAMETERVALUE ("," PARAMETERVALUE)*] "]"
MAP = "[" [MAPENTRY ("," MAPENTRY*] "]"
MAPENTRY = PARAMETERVALUE "=" PARAMETERVALUE
STRING = "'" [~']* "'"
PREDICATE = PREDICATETYPE "(" STRING ")"

List of available PQL Operators

Odysseus has a wide range of operators build in and are explained here.

Error rendering macro 'children'

null

Base Operators

Pattern Matching

TODO: TRANSLATE!

PATTERN

Description

This generic operator allows the definition of different kinds of pattern (e.g. all, any). For sequence based patterns see SASE operator (below). In the following the implemented pattern types are desribed.

Parameter

type: What kind of pattern should be detected. See below for all supported types and examples
eventTypes: Describes the types of the input ports
time: If there should be a temporal context (window) this states the time and
timeUnit is the unit of this time
size: For element based windows, this is the count of elements that are treated together, size and time can be used together
assertions: Predicate over the input data that must be fullfilled to create an output
outputmode: states, what the operator should deliver:
- EXPRESSION: use the return parameter to create the output
- INPUT: Deliver events from input port 0, can be changed with parameter inputPort
- TUPLE_CONTAINER: Deliver all events that are related to this matching
- SIMPLE
return: see outputmode/EXPRESSION
inputPort: see outputmode/INPUT

PATTERN({type = 'ABSENCE', eventTypes = ['bid'],
outputMode = 'SIMPLE', time = 400}, bid)

PATTERN({type = 'FUNCTOR', eventTypes = ['bid'],
assertions = ['avg_price < 140']},
AGGREGATE({aggregations=[['AVG', 'price', 'avg_price',
'double']]}, bid))

Subset Selection

RELATIVE-N-HIGHEST

Beschreibung

Die Anfrage wählt alle sechs Sekunden die drei höchsten Gebote aus.

Benötigte Parameter

attribute, count, time oder size

Mögliche Parameter

assertions, outputMode, return, inputPort, timeUnit

Besonderheiten

Der Ausgabemodus SIMPLE ist zwar möglich, macht aber nicht soviel Sinn.

PATTERN({type = 'RELATIVE_N_HIGHEST', eventTypes = ['bid'],
attribute = 'price', count = 3,
time = 6, timeUnit = 'SECONDS',
outputmode = 'expressions',
return = ['bid.timestamp', 'bid.bidder',
'bid.price']}, bid)

RELATIVE-N-LOWEST

Beschreibung

Die Anfrage wählt alle sechs Sekunden aus den Geboten, die höher als 80 sind, die drei niedrigsten Gebote aus.

Benötigte Parameter

attribute, count, time oder size

Mögliche Parameter

assertions, outputMode, return, inputPort, timeUnit

Besonderheiten

Der Ausgabemodus SIMPLE ist zwar möglich, macht aber in diesem Kontext normalerweise keinen Sinn.

PATTERN({type = 'RELATIVE_N_LOWEST', eventTypes = ['bid'],
assertions = ['price > 80'],
attribute = 'price', count = 3,
time = 10, timeUnit = 'SECONDS',
outputmode = 'TUPLE_CONTAINER'}, bid)

Modale

ALWAYS

Beschreibung

Wenn innerhalb dem festen Intervall von drei Sekunden alle Gebote größer als 140 sind, werden diese von dem Pattern ausgegeben.

Benötigte Parameter

time oder size

Mögliche Parameter

assertions, outputMode, return, inputPort, timeUnit

Besonderheiten

Ist der Ausgabemodus nicht SIMPLE, werden bei der Erfüllung des Patterns alle relevanten Events ausgegeben, die die Assertions erfüllen.

PATTERN({type = 'ALWAYS', eventTypes = ['bid'],
time = 3, timeUnit = 'SECONDS',
assertions = ['bid.price > 140'],
outputMode = 'INPUT'}, bid)

SOMETIMES

Beschreibung

Das Pattern ist erfüllt, wenn innerhalb dem festen Intervall von zehn Sekunden mindestens ein Gebot größer als 280 ist.

Benötigte Parameter

time oder size

Mögliche Parameter

assertions, outputMode, return, inputPort, timeUnit

Besonderheiten

Ist der Ausgabemodus nicht SIMPLE, werden bei der Erfüllung des Patterns alle relevanten Events ausgegeben, die die Assertions erfüllen.

PATTERN({type = 'SOMETIMES', eventTypes = ['bid'],
time = 10, timeUnit = 'SECONDS',
assertions = ['bid.price > 280']}, bid)

Temporal Order

SEQUENCE

Beschreibung

Die Anfrage wählt Attribute von Personen und den Geboten der jeweiligen Personen aus, bei denen die Person vor den Gebot auftritt und sein Gebot größer als 200 ist.

Besonderheiten

Die Anfrage basiert auf dem SASE-Operator. Der Parameter query erwartet eine Anfrage, die in der SASE-Anfragesprache formuliert ist. Vgl. SASE

SASE({query = 'PATTERN SEQ(person p, bid b)
WHERE skip_till_next_match(p,b)
{p.id = b.bidder, b.price > 200}
RETURN p.id, p.name, b.price', schema=[['id','Integer'],'name','String'], type='PersonEvent1'} , person, bid)

FIRST-N

Beschreibung

Die Anfrage wählt alle zehn Sekunden die ersten drei Gebote aus, die größer als 100 sind und gibt die angegebenen Attribute aus.

Benötigte Parameter

count, time oder size

Mögliche Parameter

assertions, outputMode, return, inputPort, timeUnit

Besonderheiten

Der Ausgabemodus SIMPLE ist zwar möglich, macht aber in diesem Kontext normalerweise keinen Sinn.

PATTERN({type = 'FIRST_N', eventTypes = ['bid'],
time = 10, timeUnit = 'SECONDS',
count = 3,
assertions = ['bid.price > 100'],
outputmode = 'EXPRESSIONS',
return = ['bid.timestamp', 'bid.bidder',
'bid.price']}, bid)

LAST-N

Beschreibung

Die Anfrage wählt alle zehn Sekunden die letzten drei relevanten Events aus. Dies können Gebote und Auktionen sein.

Benötigte Parameter

count, time oder size

Mögliche Parameter

assertions, outputMode, return, inputPort, timeUnit

Besonderheiten

Der Ausgabemodus SIMPLE ist zwar möglich, macht aber in diesem Kontext normalerweise keinen Sinn. Beinhaltet die Ausgabe verschiedene Event-Typen macht der Ausgabemodus TUPLE_CONTAINER Sinn, da bei dort das Schema der Daten keine Rolle spielt. Bei anderen Ausgabemodi entstehen unter Umständen null-Werte oder ähnliches.

PATTERN({type = 'LAST_N', eventTypes = ['person',
'auction'],
time = 10, timeUnit = 'SECONDS',
count = 3,
outputmode = 'TUPLE_CONTAINER'}, auction, person)

PATTERN({type = 'MIXED', eventTypes = ['bid'],
attribute = 'price',
time = 2, timeUnit = 'SECONDS'}, bid)

Spatial Pattern

MIN-DISTANCE

MAX-DISTANCE

AVERAGE-DISTANCE

RELATIVE-MIN-DISTANCE

RELATIVE-MAX-DISTANCE

RELATIVE-AVERAGE-DISTANCE

Spatial Temporal Pattern

MOVING-IN-A-CONSTANT-DIRECTION

MOVING-IN-A-MIXED-DIRECTION

STATIONARY

MOVING-TOWARD

SASE

Description

This operator allows to define temporal pattern to match against the stream. For this purpose we use the SASE+ query language. The query is expressed in the parameter query. The used source has to be of the correct type (for the example s05 and s08). The order is not important. If the type of the source is not set or wrong it can be set using the Rename Operator.

Dieser Operator ermöglicht es, Anfragen mit zeitlichen Mustern zu definieren. Zu diesem Zweck wird die SASE+ Anfragesprache verwendet und die Anfrage im Parameter query übergeben. Die angebenen Quellen müssen den passenden Typ haben (für das folgende Beispiel also s05 und s08), die Reihenfolge ist egal. Ggf. muss der Typ einer Quelle vorher noch mit Hilfe der Rename-Operation definiert werden. Der Parameter heartbeatrate legt fest, wie oft ein Hearbeat generiert werden soll, wenn ein Element verarbeitet wurde, welches aber nicht zu einem Ergebnis geführt hat.

Parameter

heartbeatrate: The rate to generate heartbeats if an element was processed without given a result.
query: The SASE+ query
OneMatchPerInstance

Example

SASE Operator

s05 = RENAME({type='s05', aliases = ['ts', 'edge']},...)
PATTERNDETECT({heartbeatrate=1,query='PATTERN SEQ(s05 s1, s08 s2) where skip_till_any_match(s1,s2){ s1.edge=s2.edge } return s1.ts,s2.ts'}, s05, s08)

Benchmark

PRIOIDJOIN

Description

Parameter

Example

TESTPRODUCER

Description

Parameter

Example

Machine Learning / Data Mining

Available mining or machine learning operators are described here: Machine Learning

Probabilistic Processing

LINEARREGRESSION

Description

This operator performs a linear regression on the given set of explanatory attributes to explain the given set of dependent attributes

Parameter

dependent: List of dependent attributes
explanatory: List of explanatory attributes

Example

LinearRegression Operator

output = linearRegression({dependent = ['x'], explanatory = ['y']}, input)

LINEARREGRESSIONMERGE

Description

Parameter

dependent: List of dependent attributes
explanatory: List of explanatory attributes

Example

LinearRegressionMerge Operator

output = linearRegressionMerge({dependent = ['x'], explanatory = ['y']}, input)

EM

Description

This operator fits gaussian mixtures model to the input stream.

Parameter

attributes: The attributes to fit
mixtures: Number of mixtures to fit the data

Example

EM Operator

output = em({attributes = ['x','y'], mixtures = 2}, input)

SampleFrom

Description

This operator samples from the given list of probabilistic continuous distributions.

Parameter

attributes: The attributes to sample from
samples: Number of samples

Example

SampleFrom Operator

output = sampleFrom({attributes = ['x','y'], samples = 50}, input)

ExistenceToPayload

Description

The input object gets one new field with tuple existence.

Parameter

The operator is parameterless.

Example

ExistenceToPayload Operator

output = existenceToPayload(input)

KalmanFilter (in progress)

Description

This operator uses a kalman filter to estimate the distribution of one or more attribute values.

Parameter

attributes: The attributes to perform the filter on
transition: The transition matrix
control: The control matrix
processnoise: The process noise matrix
measurement: The measurement matrix
measurementnoise: The measurement noise matrix

Example

Kalman Filter Operator

output = kalmanfilter({attributes = ['x','y'], transition=[], control=[], processnoise=[], measurement=[], measurementnoies=[]}, input)

Space shortcuts

Page tree

Procedural Query Language (PQL)

Using PQL in Queries

Define an Operator

Intermediate Names, Views and Sources

Parameters – Configure an Operator

Ports – What if the Operator Has More Than One Output?

The Full Grammar of PQL

List of available PQL Operators

Base Operators

Pattern Matching

TODO: TRANSLATE!

PATTERN

Description

Parameter

Logical

ALL

Beschreibung

Possible Parameter

Besonderheiten

ANY

Beschreibung

Mögliche Parameter

Besonderheiten

ABSENCE

Beschreibung

Mögliche Parameter

Besonderheiten

Threshold

COUNT

Beschreibung

Mögliche Parameter

Besonderheiten

VALUE-MAX

Beschreibung

Mögliche Parameter

Besonderheiten

VALUE-MIN

Beschreibung

Mögliche Parameter

Besonderheiten

VALUE-AVERAGE

Beschreibung

Mögliche Parameter

Besonderheiten

Subset Selection

RELATIVE-N-HIGHEST

Beschreibung

Benötigte Parameter

Mögliche Parameter

Besonderheiten

RELATIVE-N-LOWEST

Beschreibung

Benötigte Parameter

Mögliche Parameter

Besonderheiten

Modale

ALWAYS

Beschreibung

Benötigte Parameter

Mögliche Parameter

Besonderheiten

SOMETIMES

Beschreibung

Benötigte Parameter

Mögliche Parameter

Besonderheiten

Temporal Order

SEQUENCE

Beschreibung

Besonderheiten

FIRST-N

Beschreibung

Benötigte Parameter

Mögliche Parameter

Besonderheiten

LAST-N

Beschreibung

Benötigte Parameter