Local Pipeline Engine

Local

The local runner runs on the local Hop engine. This is the runtime configuration you’ll use when testing pipelines on your local machine while developing.

Options

Option Description Default

Option	Description	Default
Row set size	The row set buffer size.	10.000
Safe mode	Checks every row passed through your pipeline and ensure all layouts are identical. If a row does not have the same layout as the first row, an error is generated and reported.	false
Collect metrics	Collects metrics to monitor the performance of your pipeline.	false
Sort transforms	Performs a topological sort of the transforms in the pipeline before execution.	false
Log rows feedback	Enables log rows feedback, showing a log line every after processing multiple of 50.000 (default) rows.	false
Feedback size in rows	The number of rows to return as feedback.	50.000
Wait time buffer check (ms)	This indicates the polling frequency when there are no rows in the input buffer of a transform, a lower value will cause higher CPU load when the pipeline has many idle transforms.	20
Sample type while running in the GUI	Here you can specify which rows you want to see when you click on the little grid icons on the transform icons during a pipeline execution. You can choose: `None`: do not show, `First` : show the first rows, `Last` : show the last rows or `Random`: a random selection of rows (reservoir sampling)	Last
Number of rows to sample in the GUI	The number of rows which will be collected	100
Make this pipeline transactional	If you enable this there will always be just one connection used per database. At the end of the pipeline there will be a commit or rollback (in case of error) on all named connections at the same time. Note that child pipelines or workflows are also automatically run as a transaction. Also be aware that while you can commit and rollback across multiple databases that this still means that you can have a successful commit on one database and a failure on another. This is not a two-phase-commit system.	false

Row set size

The row set buffer size.

10.000

Safe mode

Checks every row passed through your pipeline and ensure all layouts are identical. If a row does not have the same layout as the first row, an error is generated and reported.

false

Collect metrics

Collects metrics to monitor the performance of your pipeline.

false

Sort transforms

Performs a topological sort of the transforms in the pipeline before execution.

false

Log rows feedback

Enables log rows feedback, showing a log line every after processing multiple of 50.000 (default) rows.

false

Feedback size in rows

The number of rows to return as feedback.

50.000

Wait time buffer check (ms)

This indicates the polling frequency when there are no rows in the input buffer of a transform, a lower value will cause higher CPU load when the pipeline has many idle transforms.

Sample type while running in the GUI

Here you can specify which rows you want to see when you click on the little grid icons on the transform icons during a pipeline execution. You can choose: None: do not show, First : show the first rows, Last : show the last rows or Random: a random selection of rows (reservoir sampling)

Last

Number of rows to sample in the GUI

The number of rows which will be collected

100

Make this pipeline transactional

If you enable this there will always be just one connection used per database. At the end of the pipeline there will be a commit or rollback (in case of error) on all named connections at the same time. Note that child pipelines or workflows are also automatically run as a transaction. Also be aware that while you can commit and rollback across multiple databases that this still means that you can have a successful commit on one database and a failure on another. This is not a two-phase-commit system.

false