Pipeline sources
Every transform in a pipeline starts at the same time. Most of them wait for rows from a previous hop. A pipeline source does not need that hop: it produces rows on its own (from a file, a database, generated data, a stream, and so on).
That is the beginner question behind this page: can this transform be the first icon on the canvas?
How to spot a source in Hop Gui
These labels all use the same setting on the transform (canStartWithoutInput()), so they stay in sync with how the pipeline actually runs.
-
In the add-transform dialog (click the canvas) or the palette, search for
pipeline source. Source transforms also show Can start without incoming hops in the tooltip. -
Hover a transform icon on the canvas: sources show the same tooltip line.
-
Verify the pipeline: each source gets a comment This transform can start without incoming hops (pipeline source).
Two independent behaviours
Starting without a hop and reading hops are not the same thing.
| Kind | Starts without a hop? | Reads incoming hops? |
|---|---|---|
Generate Rows, Data Grid, Cube Input, Kafka Consumer, \ldots |
Yes |
No. Hop Gui refuses a hop into these, and Verify reports an error if one is already there. |
Get Variables, Get System Info, Random Value, Table Input |
Yes |
Yes. Incoming rows are drained and the extra fields are added. |
JSON Input, Execute SQL script, Mail Input, \ldots |
Yes until you enable the "from field" / "each row" option |
Yes only in that configuration. File/static mode refuses hops. |
REST Client, Dummy, Calculator, \ldots |
No. Put a source in front. |
Yes |
Third-party plugins that do not override these flags are treated as stream transforms: hops into them stay allowed.
Typical sources
Use these as starting points; the search keyword above is the complete, up-to-date list for your installation.
- Always a source (no incoming hop)
-
Generate Rows, Data Grid, De-serialize from file, file readers in file mode (CSV, Text, Excel, JSON, XML, YAML, \ldots), stream consumers (Kafka, JMS, Beam/Spark inputs).
- Source that still reads hops
-
Get Variables, Get System Info, Generate random values, Table Input (incoming rows parameterize SQL).
- Dual-mode (option on the dialog)
-
JSON Input (Source is from a previous transform), Execute SQL script (Execute for each row), Mail Input (Get folder from field), and other file/query inputs with a "from field" or "each incoming row" checkbox.
- Needs a hop
-
REST Client, lookups, calculators, Dummy, most "do something to the row" transforms.
Hops into a source that does not read
If you connect a hop into Generate Rows (or JSON Input with "source from field" off), the rowset fills and the pipeline never finishes.
Hop Gui therefore refuses that hop when you draw it, offers to remove existing ones when you close the dialog after changing the option, and Verify reports an error. Enable the option named in that error if you really want the transform to read the previous stream.
Error hops (red) and info hops are not treated as main input: they stay allowed.
Pipelines that already have such a hop and whose upstream produces no rows used to finish. They now fail at init (and leftover rows stop the pipeline). To keep those files running, set '${HOP_ALLOW_UNCONSUMED_MAIN_INPUT}' to Y on the run (Verify still reports the error).