To achieve completeness, which is needed for Rollback Recoveryand Precise Recovery, all elements that could not be processed due to a system failure have to be available after system recovery in order to process them. We call the application, that is responsible for the incoming data streams Backup of Data Streams (BaDaSt). BaDaSt can be downloaded via one of our download sites.
Attention: Currently, the downloadable BaDaSt application is not working. We try to fix it as soon as possible!
When you start BaDaSt, a Zookepper server and a Kafka server are started. Kafka handles the preservation of incoming data stream elements per stream (each stream is a Kafka topic and every stream element has a Kafka offset). Zookeeper, Kafka and BaDaSt are configurable by a properties file within the Odysseus HOME folder (.odysseus). Once BaDaSt is running, you can control it from Odysseus with an Odysseus Script command, called BADASTRECORDER type=<type> source=<source>. The field type specifies the recorder that means a transport handler. The following code block contains all available recorders. The field source defines which data stream from which external source should be preserved. Note that you have to (1) define the source within the same script file and before the recorder command and (2) use the same name for the source as you do in the source definition in Odysseus.
#BADASTRECORDER type=FileRecorder source=bid /// Use the FileRecorder, if the data source is a finite file (e.g., a CSV) #BADASTRECORDER type=TCPRecorder source bid /// Use the TCPRecorder, if the data source can be accessed by TCP. Host and port are taken from the source definition.
If Odysseus crashes and restarts, recovery operators will be inserted into the query plan. Those operators handle the access to BaDaSt, receive the preserved data stream elements and try to synchronize with the original data source.