Page tree
Skip to end of metadata
Go to start of metadata

The file transport handler allows the reading and writing to arbitrary files

Options

  • filename: The path to the file

  • append: append values to end of file. If set to false the values are overwritten. (default false)

  • createdir: When writing to a file, creates the file directory if it doesn't exist. (default false)

  • writedelaysize: Uses a buffer with specified size which has to be filled before data is written to the file (default 0)

  • preload: Loads the whole file into memory when the query is started (default false, incompatible with delayopenin)

  • delayopenin: When reading from a file, delays opening until reading the first tuple. Can be used to monitor log files which do not exist upon starting the query (default false)

  • delayopenout: When writing to a file, delays opening (and, if specified, creating the directory) until writing the first tuple (default false, incompatible with preload)

Example

PQL

File Transport Handler
input = ACCESS({source='Source',
wrapper='GenericPull',
transport='File',
protocol='CSV',
dataHandler='Tuple',
options=[['filename', '/src/odysseus/input.csv']],
schema=[
['id', 'Double'],
['data', 'String']]
})

output = SENDER({sink='Sink',
wrapper='GenericPush',
transport='File',
protocol='CSV',
dataHandler='Tuple',
options=[['filename', '/src/odysseus/output.csv']]
}, input)

CQL

File Transport Handler
CREATE STREAM source (id Double, data STRING)
    WRAPPER 'GenericPush'
    PROTOCOL 'CSV'
    TRANSPORT 'File'
    DATAHANDLER 'Tuple'
    OPTIONS ( 'filename' '/src/odysseus/input.csv')


CREATE SINK sink (id Double, data STRING)
    WRAPPER 'GenericPush'
    PROTOCOL 'CSV'
    TRANSPORT 'File'
    DATAHANDLER 'Tuple'
    OPTIONS ( 'filename' '/src/odysseus/output.csv')
  • No labels