Work in progress!
Although, there is already a large set of wrappers available for Odysseus, there is often the need to create a new specific adapter for a specific source. To ease the creation of new wrapper, we created an adapter framework.
...
- Transport: Die physical bridge between the external systems and Odysseus is given by this handler. It is responsible for the communication, this could be e.g. a file access, a tcp client/server or a message bus handler.
Hint: There are scenarios in which it is not feasible to separate transport and protocol layer. In such cases, one can implement the combination as a transport handler and use it in combination with the "None" protocol handler. - Protocol: While the transport handler handles the connections, the protocol handler is responsible for the interpretation of the given input coming from the transport handler. I.e. the protocol handler translates the incoming data into a format understandable by Odysseus
- DataHandler: Finally, the internal format can be defined. This is the data type, which is used by the internal operators of Odysseus. There are different kind of operators for different kinds of datatypes. DataHandler are e.g. Tuple or KeyValue.
...
The wrapper can also be used at sink side, i.e. to send data from Odysseus to other systems. In this case, the information is send from the datahandler via the protocol handler to the transport handler. Each handler can provide both ways, i.e. retrieving and sending of data but is not required to. So some handler may only be used in sources, while other only in sinks.
Here are two sequence diagramms that show via an example how the different handler interoperate:
This document explains how to write new wrappers using this generic wrappers.
To create a new TransportHandler, the Interface ITransportHandler must be implemented or the class AbstractTransportHandler be extended.
Depending on the way, the handler works, different methods need to be implemented.
Independent of Push/Pull
| Code Block | ||
|---|---|---|
| ||
public ITransportHandler createInstance(IProtocolHandler<?> protocolHandler, Map<String, String> options); |
This method must return a new initialized transport handler. Typically, the constructor is called.
| Code Block | ||||
|---|---|---|---|---|
| ||||
@Override
public ITransportHandler createInstance(
IProtocolHandler<?> protocolHandler, Map<String, String> options) {
return new FileHandler(protocolHandler, options);
} |
The methode getName() must deliver a global unique transport handler name.
| Code Block | ||
|---|---|---|
| ||
String getName(); |
Its a good was to use this a follows (again example of FileHandler):
| Code Block | ||
|---|---|---|
| ||
public static final String NAME = "File";
@Override
public String getName() {
return NAME;
} |
When implementing ITransportHandler, open and close need to be implemented.
Hint: In the following we will assume, that AbstractTransportHandler will be overwritten.
AbstractTransportHandler provides already default implementations that cannot be overwritten (its implementend in AbstractTransportHandlerDelegate):
| Code Block | ||
|---|---|---|
| ||
final synchronized public void open() throws UnknownHostException,
IOException {
if (openCounter == 0) {
if (getExchangePattern() != null
&& (getExchangePattern().equals(
ITransportExchangePattern.InOnly)
|| getExchangePattern().equals(
ITransportExchangePattern.InOptionalOut) || getExchangePattern()
.equals(ITransportExchangePattern.InOut))) {
callOnMe.processInOpen();
}
if (getExchangePattern() != null
&& (getExchangePattern().equals(
ITransportExchangePattern.OutOnly)
|| getExchangePattern().equals(
ITransportExchangePattern.OutOptionalIn) || getExchangePattern()
.equals(ITransportExchangePattern.InOut))) {
callOnMe.processOutOpen();
}
}
openCounter++;
}
final synchronized public void close() throws IOException {
openCounter--;
if (openCounter == 0) {
if (getExchangePattern() != null
&& (getExchangePattern().equals(
ITransportExchangePattern.InOnly)
|| getExchangePattern().equals(
ITransportExchangePattern.InOptionalOut) || getExchangePattern()
.equals(ITransportExchangePattern.InOut))) {
callOnMe.processInClose();
}
if (getExchangePattern() != null
&& (getExchangePattern().equals(
ITransportExchangePattern.OutOnly)
|| getExchangePattern().equals(
ITransportExchangePattern.OutOptionalIn) || getExchangePattern()
.equals(ITransportExchangePattern.InOut))) {
callOnMe.processOutClose();
}
}
} |
A TransportHandler can provide different exchange pattern. The handler must deliver the pattern when calling the following method:
| Code Block | ||
|---|---|---|
| ||
public ITransportExchangePattern getExchangePattern(); |
Currently, the following values are available (TODO: Extend description of remaining pattern):
- InOnly: The handler can only be used as source.
- RobustInOnly
- InOut: The handler can be used as source and as sink.
- InOptionalOut
- OutOnly: The handler can only be used as sink.
- RobustOutOnly
- OutIn
- OutOptionalIn
AbstractTransportHandler calls according to the exchange pattern the corresponding methods:
- processInOpen()
- processOutOpen()
- processInClose()
- processOutClose()
In this methods the TransportHandler must open or close the connections. The "IN"-methods are called for sources, the "OUT" for sinks. When starting or stopping a query, open and close are called respectively.
In the following again the implemenations for the FileHandler
| Code Block | ||
|---|---|---|
| ||
@Override
public void processInOpen() throws IOException {
if (!preload) {
final File file = new File(filename);
try {
in = new FileInputStream(file);
fireOnConnect();
} catch (Exception e) {
fireOnDisconnect();
throw e;
}
} else {
fis = new FileInputStream(filename);
FileChannel channel = fis.getChannel();
long size = channel.size();
double x = size / (double) Integer.MAX_VALUE;
int n = (int) Math.ceil(x);
ByteBuffer buffers[] = new ByteBuffer[n];
for (int i = 0; i < n; i++) {
buffers[i] = ByteBuffer.allocateDirect(Integer.MAX_VALUE);
channel.read(buffers[i]);
buffers[i].rewind();
}
in = createInputStream(buffers);
fireOnConnect();
}
}
@Override
public void processInClose() throws IOException {
super.processInClose();
if (fis != null) {
fis.close();
}
}
@Override
public void processOutOpen() throws IOException {
final File file = new File(filename);
try {
out = new FileOutputStream(file, append);
fireOnConnect();
} catch (Exception e) {
fireOnDisconnect();
throw e;
}
}
@Override
public void processOutClose() throws IOException {
fireOnDisconnect();
out.flush();
out.close();
} |
Generic Pull
After the connection is inialized, the framework tries to retrieve data from the TransportHandler. To be generic, we decided to use an InputStream for sources and an OutputStream for sinks. So the following methods need to be overwritten (Remark: It its not necessary to implement both methods, if the TransportHandler e.g. should only be used for sources):
| Code Block | ||
|---|---|---|
| ||
public InputStream getInputStream();
public OutputStream getOutputStream(); |
A typical implementation in FileHandler:
| Code Block | ||
|---|---|---|
| ||
@Override
public InputStream getInputStream() {
return in;
}
@Override
public OutputStream getOutputStream() {
return out;
} |
In some cases, sources deliver not an endless data stream. For such cases the method
| Code Block | ||
|---|---|---|
| ||
public boolean isDone(); |
can be overwrittten.
GenericPush
In generic push szenarios for sources there is no method that can be overwritten because it depends on the transport type and e.g. libararies that receive data from external sources. The information that is read must be send to the corresponding transport handler. To simplify the process, AbstractTransportHandler(Delegate) provides the following methods that should be used:
| Code Block | ||
|---|---|---|
| ||
public void fireProcess(ByteBuffer message) {
for (ITransportHandlerListener<T> l : transportHandlerListener) {
// TODO: flip() erases the contents of the message if
// it was already flipped or just created...
// In other words: This method expects that the byte buffer
// is not fully prepared
message.flip();
l.process(message);
}
}
public void fireProcess(T m) {
for (ITransportHandlerListener<T> l : transportHandlerListener) {
l.process(m);
}
}
public void fireProcess(String[] message) {
for (ITransportHandlerListener<T> l : transportHandlerListener) {
l.process(message);
}
} |
The fireProcess methods can be used with ByteBuffers and String-Arrays or with a Generic. In the latter case, the corresponding ProtocolHandler must read this type, else a class cast exception will be thrown.
An example to the use fireProcess-Methods can be found in the RabbitMQ transport handler:
| Code Block | ||
|---|---|---|
| ||
@Override
public void processInOpen() throws IOException {
internalOpen();
// Create Consumer
boolean autoAck = false;
channel.basicConsume(queueName, autoAck, consumerTag,
new DefaultConsumer(channel) {
@Override
public void handleDelivery(
String consumerTag,
com.rabbitmq.client.Envelope envelope,
com.rabbitmq.client.AMQP.BasicProperties properties,
byte[] body) throws IOException {
// String routingKey = envelope.getRoutingKey();
// String contentType = properties.getContentType();
long deliveryTag = envelope.getDeliveryTag();
try{
ByteBuffer wrapped = ByteBuffer.wrap(body);
wrapped.position(wrapped.limit());
fireProcess(wrapped);
}catch(Exception e){
e.printStackTrace();
}
channel.basicAck(deliveryTag, false);
};
});
} |
Here you can see, that every source type needs a special handling for sending. Here e.g. a callback object (DefaultConsumer) is defined in RabbitMQ that calls fireProcess.
Two additional methods are used to inform listener about connection states (connect and disconnect)
| Code Block | ||
|---|---|---|
| ||
public void fireOnConnect(ITransportHandler handler) {
for (ITransportHandlerListener<T> l : transportHandlerListener) {
l.onConnect(handler);
}
}
public void fireOnDisconnect(ITransportHandler handler) {
for (ITransportHandlerListener<T> l : transportHandlerListener) {
l.onDisonnect(handler);
}
} |
These methode are defined in the interface ITransportHandlerListener that is implemented by IProtocolHandler, the basic interface for ProtocolHandler.
Registering the handler
Odysseus is OSGi based and all the handler are implemented as declartive services.
For this, you have to create a xml file, typically placed under a folder called OSGI-INF, where you state the global unique name of the handler, the implemenation class and the interface that this handler provides. In the following is the example for the FacebookTransportHandler.
| Code Block | ||
|---|---|---|
| ||
<?xml version="1.0" encoding="UTF-8"?>
<scr:component xmlns:scr="http://www.osgi.org/xmlns/scr/v1.1.0" name="de.uniol.inf.is.odysseus.wrapper.facebook.physicaloperator.access.FacebookTransportHandler">
<implementation class="de.uniol.inf.is.odysseus.wrapper.facebook.physicaloperator.access.FacebookTransportHandler"/>
<service>
<provide interface="de.uniol.inf.is.odysseus.core.physicaloperator.access.transport.ITransportHandler"/>
</service>
</scr:component> |
The file MANIFEST.MF (typically provided in META-INF) must contain a hint to this new file
| Code Block |
|---|
Service-Component: OSGI-INF/FacebookTransportHandler.xml, OSGI-INF/FacebookProtocolHandler.xml |
Remark: An eclipse wizzard can be used to create this file and the reference inside the MANIFEST.MF: File/New/Component Definition
Creating a new Protocol Handler
...