Run, Preview and Debug a Pipeline

Running a pipeline to see how it performs can be done by completing one of the following tasks:

In the pipeline run dialog, hit the 'New' button in the upper right corner to create a new 'Pipeline run configuration'.

In the dialog that pops up, add 'Local Pipeline' as the pipeline configuration name and choose the 'Local pipeline engine' as engine type.

Click 'Ok' to return to the pipeline run dialog.

Choose the logging level as described below.

If the pipeline has parameters, give them a value on the Parameters tab of this dialog, or leave the value empty to use the default value. See Working with parameters.

Make sure your configuration is selected and hit 'Launch'.

When a pipeline has run successfully, a check mark is displayed in the upper right corner of each transform. With more than one copy, that check is shown only when every copy of the transform has finished. Until then the corner shows how many copies are done. See Specify copies.

The transforms also show a little table icon, giving you access to a preview of the results of that transform.

When a pipeline fails, a red triangle is shown in the upper right corner of the failed transform. Hover your mouse over the red error triangle to take a quick peek at the error message. The full stacktrace is available in the logs. Check pipeline error handling to learn about how you can handle errors in your pipeline gracefully (which is not necessarily what you want).

After each run the execution results are displayed in the panel on the bottom of your window. The panel has these tabs:

The same tabs are shown for a workflow, see Run and Debug a Workflow.

To inspect pipeline data while controlling how execution pauses, use Preview & debug (toolbar, Run menu, or transform actions). This opens a single dialog for preview rows and optional breakpoint behavior.

You can start it in any of these ways:

In the dialog, choose the transform whose rows you want to inspect. Set how many rows to retrieve (Retrieve first rows), and optionally enable Pause on breakpoint and define a condition so the pipeline pauses when that condition matches a row. When finished press Quick Launch. To change the pipeline run configuration, click Configure.

The collected rows are shown in the preview window when execution pauses or finishes.

Debug starts a local session and parks every transform before the first row. A Debug mode badge stays in the top left of the canvas. Each hop shows cursor at stored: how far the next transform has read, and how many rows are kept on that hop. A green border means that transform is running. A blue border means it is parked. Several transforms can run at the same time. Click Debug again, or Exit debug, to leave the session and return to the editor.

The Debugger tab in the execution results has the transforms, the hops, and the rows on the selected hop.

Keyboard shortcuts, when the pipeline graph has focus:

Debug does not stop a pipeline that is already running. Stop that run first. Leaving the session with Exit debug, closing the graph, or starting a normal run removes the hop buffers and the spill files. A normal run does not keep the yellow step marker.

The toolbar above preview windows and other table grids in Hop Gui can find a value or a column, and export the rows.

The Find (search) icon opens a menu with two items:

The two export icons write the rows of the table to a file:

Both ask which columns to export, with all of them selected, and also write the # row number column first. In Hop Gui the file is written to the temporary folder and opened with the program that handles the file type. Hop Web asks where to save it.

Preview windows (and other table grids in Hop Gui) can contain a large number of columns. A table view is a named subset of those columns, in the order you want to see them. Hidden columns stay in the data and are still exported, copied or saved. A view can also carry filters on its columns: rows that do not match are hidden while the view is applied, and come back with Show all columns. Row filters are offered when browsing data, such as Show rows. A grid that edits configuration leaves Add filter and Remove filter disabled and does not hide rows for a saved filter, so confirming the dialog still keeps every row.

Open the Table views dialog from the table toolbar icon, from the row context menu, or from the column header context menu.

In the dialog:

Views are stored per project in the Hop audit folder and are matched by column name. A view such as customer-bk-fields can be applied to any table that has those field names, not only the preview where you created it.

The column header context menu can hide that column, add a filter on it, or remove the filter it already has. Remove filter stays disabled until that column has a filter.

Add filter lists the unique sorted values of that column, the same way the dialog does. Select the values to keep and press OK.

The filter is applied straight away. The column header shows a filter icon, and only the matching rows stay visible.

You can also drag headers to reorder them (desktop Hop Gui). Those one-off changes can be captured with Use current layout and then saved as a named view.

The table toolbar itself can be turned on or off with Show a toolbar above tables in the Configuration perspective.