Required Feature: Parallelization
You are able to use Odysseus Script to parallelize an created script automatically. To use this functionality Odysseus Script provides three keywords and a UI for benchmarking.
...
- Parallelization degree: (mandatory) Defines the degree of parallelization that should be used. It is also possible to use the constant AUTO, to detect the available cores and use this value.
- Buffer-size: Defines the number of elements inside of the parallel operators. There is also the possibility to use the constant AUTO, to use an default value.
...
The following example shows the usage of this keyword. This example uses the inter-operator parallelization with an degree of 4 and an automatic buffersize.
...
| Info |
|---|
If this keyword is used, every operator of the query, which has an compatible parallelization strategy is transformed. If only one or subset of operators should parallelized, the following keyword need to be used in addition. |
...
#INTEROPERATOR keyword
The #INTEROPERATOR keyword is an addition to the #PARALLELIZATION keyword for inter-operator parallelization. With this keyword it is possible to select one or more operators, which should be parallelized. There is also the possibility to configure the parallelization for each operator. This keyword provides following parameters:
- Operator-Ids (mandatory): one or more operatorIds for operators the need to be parallelized. If more than one id is defined the ids need to be separated by commas (avoid blanks between ids and commas).
- Parallelization degree: (mandatory) Defines the degree of parallelization that should be used. It is also possible to use the constant AUTO to detect the available cores and use this value, or GLOBAL to use the value defined in the #PARALLELIZATION keyword.
- Buffer-size: (mandatory) Defines the number of elements inside of the used buffers. There is also the possibility to use the constant AUTO to use an default value, or GLOBAL to use the value defined in the #PARALLELIZATION keyword.
- Parallelization strategy (optional): the parallelization strategy that should be used for the given operatorIds. Otherwise the preferred strategy is automatically detected.
- Fragmentation type (optional): the fragmentation type that should be used inside of the strategy. Otherwise the preferred type is automatically selected.
- Use parallel Operators (optional): if this option is selected, parallel operators are used instead of normal sequential operators. If the given operator does not support parallel execution, this option is ignored. Default value is false.
...
The following code example shows the usage of this keyword. Only the aggregation is parallelized, because only this id is defined. The global parallelization degree is overwritten with the value of 2. With the constant GLOBAL the value for the buffersize is used from the global definition. In addition to this parameters, also the parallelization strategy is defined manually. In this case the AggregateMultithreadedTransformationStrategy is used. Note that the strategy need to be fit to the operator type defined with the id. In addition the strategy need to be compatible for the operator. In some cases it is not possible to use the selected strategy, e.g. an grouping inside the aggregation is needed. See the list below, for more informations. The last parameter in this example is the optional selection of an fragmentation type. Note that not every strategy supports all fragmentation types. See the list below for all possible combinations.
...
Strategies and supported fragmentation types
| Logical operator | Parallelization strategies | Description | Supported fragmentation types | Allows definition of endpoint |
|---|---|---|---|---|
| JoinAO | JoinTransformationStrategy | Uses an Hash-Fragmentation for both input streams. The fragmentation attributes are gathered from the join attributes. Note that only equals-predicates (which are concatenated with &&) are supported. The fragmented datastream is merged with an UNION Operator. | HashFragmentAO |  |
| AggregateAO | NonGroupedAggregateTransformationStrategy |
| Uses an RoundRobin or Shuffle Fragmentation for splitting the input datastream. This strategy works with partial aggregates and merges the datastream both with an union operator and an additional aggregate operator for merging the partial aggregates. This strategy works with and without grouping. Only aggregations with one input attribute are supported. | RoundRobinFragmentAO |  | |
ShuffleFragmentAO | |||
GroupedAggregateTransformationStrategy | Uses a Hash-Fragmentation for the input stream. The fragmentation attributes are gathered from the grouping attributes. So this strategy only works if the aggregate operator has an grouping. The fragmented datastream is merged with an UNION Operator. | HashFragmentAO | |
| Info |
|---|
Bold fragmentation types shows the preferred type if nothing is defined. |
...
#INTRAOPERATOR keyword
The #INTRAOPERATOR keyword is an addition to the #PARALLELIZATION keyword for intra-operator parallelization. With this keyword it is possible to select one or more operators, which should be parallelized. There is also the possibility to configure the parallelization for each operator. This keyword provides following parameter
...
| Code Block |
|---|
#PARSER PQL
#PARALLELIZATION (degree=8) (buffersize=10000) (type=INTRA_OPERATOR)
#INTRAOPERATOR (buffersize=10000) (id=joinId) (degree=8)
#RUNQUERY
windowBid = TIMEWINDOW({SIZE = [1, 'MINUTES'],
advance = [1, 'SECONDS']
}, bid)
windowAuction = TIMEWINDOW({SIZE = [10, 'MINUTES'],
advance = [1, 'SECONDS']
}, auction)
join = JOIN({id='joinId',PREDICATE = 'bid.auction == auction.id'}, windowBid, windowAuction) |
| Info | ||||
|---|---|---|---|---|
| ||||
It is possible to use both inter and intra operator parallelization at the same time. The following example shows how this works:
If you want to use inter and intra operator parallelization both for the same operator use the option useParallelOp=true
|
Parallelization Benchmarker
...
If configuration is done, hit the "Start Analysis" button and the benchmarker calculates all needed executions and starts them one after the other. This may take some time. If the benchmarking is done, the result is shown and there is the possibility to copy the resulting Odysseus script keywords in your query.
