Store sensor data in HDFS

The SAP Data Hub, developer edition per default comes with an Apache Hadoop installation. Open http://localhost:50070 and check that HDFS is up and running.

Navigate to Utilities | Browse the file system (1) and display the /tmp/hdfsManager directory (when you install SAP Data Hub, developer edition initially, this directory is empty).

Open the pipeline which you have created during the previous tutorials (test.myFirstPipeline) in the modelling environment http://localhost:8090. For this, select the Graphs tab in the tab bar on the left side and search for test.myFirstPiepline.

Remove the connection between the Kafka Consumer2 operator and the ToString Converter operator.

Add a HDFS Producer operator to the pipeline by drag & drop. Then connect the message port of the Kafka Consumer2 operator to the inFile port of the HDFS Producer operator.

Configure the HDFS Producer operator. You need to maintain the following properties:

The HDFS Producer will write the received data to files in the /tmp/hdfsManager directory. The files follow the scheme test_<counter>.txt (whereas <counter> is an incremental integer).

Add a HDFS Consumer operator to the pipeline by drag & drop. Then connect the outFile port of the HDFS Consumer operator to the ininterface port of the ToString Converter operator.

Configure the HDFS Consumer operator. You need to maintain the following properties:

Afterwards press the Save button.

Press the Run button to execute the pipeline.

When the Status tab indicates that the pipeline is running, use the context menu Open UI of the Terminal operator to see the generated sensor data.

In contrast to the previous tutorial, this time the generated sensor data is not sent from the Kafka Receiver operator to the Terminal operator directly, but via HDFS. Hence the Terminal also shows you information about the created files.

Open http://localhost:50070 and display the /tmp/hdfsManager directory. The longer the pipeline runs, the more files you will find there.

Stop the pipeline by pressing the Stop button.