File reference to a Hadoop Filesystem (wrapper feature must be installed)

Options

  • fs.default.name: The name of the Hadoop filesystem
  • Filename: The path to the file
  • append: append values to end of file. If set to false (default) the values are overwritten.


#PARSER PQL
#RUNQUERY
input = ACCESS({source='Source',
wrapper='GenericPull',
transport='hdfs',
protocol='csv',
dataHandler='Tuple',
Internship 2015
options=[['fs.default.name','hdfs://master:9000'],
['username','hduser'],['Filename', '/user/sorted_data.csv'],
['append','false']],
schema=[
['1', 'String'],
['2', 'String'],})
output = SENDER({sink='Sink',
wrapper='GenericPush',
transport='hdfs',
protocol='CSV',
dataHandler='Tuple',
options=[['fs.default.name','hdfs://master:9000'],
['username','hduser'],['Filename', '/user/sorted_data.csv'],
['append','false']]
}, input)
  • No labels