Running the Apache Beam samples With the Beam Direct Runner

The Direct Runner executes pipelines on your machine and is designed to validate that pipelines adhere to the Apache Beam model as closely as possible. Instead of focusing on efficient pipeline execution, the Direct Runner performs additional checks to ensure that users do not rely on semantics that are not guaranteed by the model.

Using the Direct Runner for testing and development helps ensure that pipelines are robust across different Beam runners. In addition, debugging failed runs can be a non-trivial task when a pipeline executes on a remote cluster. Instead, it is often faster and simpler to perform local unit testing on your pipeline code. Unit testing your pipeline locally also allows you to use your preferred local debugging tools.

check the Apache Beam Direct Runner docs for more details.

The default settings for the direct runner should work by default, all you need to do is specify your fat jar location.

Contrary to the Flink and Spark runners, the direct runner can be started directly from Hop Gui.

The output from beam/pipelines/generate-synthetic-data.hpl (with a reduced number of rows) is shown below.