Attention: ATM CPU and Memory are not correctly measured!
The evaluation feature offers some graphical user interface and instrumentation to measure the latency and throughput of a given query and to create some plots.
Create a new evaluation
Before the evaluation feature can be used the features monolithicrcp.evaluation.studio feature and latency have to be added to the odysseus Odysseus product.
An evaluation is maintained by an evaluation job, which encapsulates all necessary settings. A evaluation job is project relative so that it must be assigned to an Odysseus project.
- To create a new evaluation, go to an Odysseus project and use right click, browse to "new" and choose "Evaluation Job".
- Choose a project wide unique name for your evalation evaluation job and press next.
- Choose the Odysseus Script file from the project you want to evaluate.
- Now you should see the evaluation job in your editor view. You may change the settings, which are described in the following.
Settings for an evaluation
Query File: This is the query file you want to evaluate. It should deliver at least one executable, physical query plan. Remember that the query will be executed more than once, so you should design the script to be reexecutablere-executable. This means for example, the query file may not try to install new sources which are already installed. Hint: you may use something like "#DROPALLSOURCES" or "#IFSRCDEF/#IFSRCNDEF" in Odysseus Script.
New (2017.02.16): There is now the possibility to add a Setup File and a Tear Down file. Similar to JUnit these scripts are called directly before and after the run (e.g. to register sources). These scripts can ran before and after each run or onyl once before and after the whole runs (Setup/Tear down every Run).
Folder for processing results:This is a the folder where the evaluation dumps the results. For each new run of an evalationevaluation, a new subfolger subfolder (named by date and time) is created.
Folder for plots: If you want to create plots from the results, the created plots are saved here.
Parameters: These parameters can be used to iterate over certain settings (or values) in your query file. If you created a new evaluation job, you have two predefined examples: "var_1" and "var_2". "var_1" is the name of the variable and the values are given at the right (each line is a value). So the predifined predefined parameter "var_1" has the values "a", "b" and "c" and "var_2" the values "x", "y", and "z". If the parameters are active (the checkbox is checked), the evaluation will iterate over the values. It will produce each possible combination of all active parameters. For this example there will be 9 different runs:
...
You can add a new parameter using the "Add" button and remove a parameter by selectiing selecting the parameter and using the "Remove" button.
...
Measure Latency: If it is activated, the evaluation job adds the his pretransformation pre-transformation handler to measure the latency of the query. The pretransformation pre-transformation handler changes the query files (e.g. it activates the metadata latency by adding "#METADATA Latency" to the query. Furthermore, it adds the LatencyCalculationPipe and a FileSink to each root operator of the query. Therefore, if your query has different sinks, you will also have different latency results. TODO: This has changed. Currently your sources must contain valid metadata for processing!!
Create plots for latencies: If activated, the evaluation will use the merged files to create some plots, which are stored within the plot-folder (look above)
Measure Throughput: If this is active, the evaluiation evaluation job adds his pretransformation pre-transformation handler to measure the throughput of a query. The pretransformation pre-transformation handler changes the query by adding a throuput throughput operator after each source.
...
Create plots for throughput: If activated, the evaluation will use the merged files to create some plots, which are stored within the plot-folder (look above).
Measure Resources: If this is active, the evaluation job adds his pre-transformation handler to measure the CPU and memory consumption. The pre-transformation handler changes the query by adding the "#METADATA Systemload" directive to the query. To measure the systemload, the "Systemload Metadata Feature" is required.
Create plots for CPU: If activated, the evaluation will use the merged files to create a plot of the CPU consumption.
Create plots for memory: If activated, the evaluation will use the merged files to create a plot of the memory consumption.
Plotsettings: You can choose the export type: PDF, PNG, JPEG, or GnuPlotGNUPLOT. Furthermore you may define the dimension of the plot, its height and width.
If the export type is PDF, PNG, or JPEG the resulting plot will look like the figure above. If the export type is chosen to be GNUPLOT, the evaluation feature will generate a .dat file with the merged measurements and a .gnu file containing a Gnuplot script to generate a LaTeX file.
Run and stop the evaluation
...
To stop an evaluation you may click "Cancel" on the Progressmonitorprogressmonitor.
If you stop a query by hand (e.g. using the Queries view) - one evaluation run is canceled and the next run will be installed and started automatically.