Fields, parameters, and variables

Apache Hop moves information in three related but different ways:

  • Fields are the typed columns on a data row that flow between transforms.

  • Parameters are named inputs declared on a pipeline or workflow. They become variables of the same name when that pipeline or workflow starts.

  • Variables are named string values in a scope (JVM, environment, project, run configuration, workflow, or pipeline).

They can look similar in the UI, especially when a dialog column is titled Parameters/Variables, but they do not behave the same way.

Fields, parameters, and variables are available downstream when they are in scope. You can pass them again through as many nesting levels as you need. The Get Parameters / Get Variables buttons only look one level up: they read the pipeline or workflow the action or transform points at, not the whole ancestor chain.

At a glance

Fields Parameters Variables

What they are

Named, typed columns on a data row

Named inputs declared on a pipeline or workflow

Named string values in a scope

Typical analogy

Columns in a result set

Function arguments

Environment or configuration

Type

Hop data types (String, Integer, Date, and so on)

Always a string; activated as a variable of the same name

Always a string

Scope

The stream between transforms, and result rows between a parent and a child

The pipeline or workflow that declared them

JVM, environment, project, run configuration, workflow, or pipeline

Direction

Flow with the data. A child can return rows to its parent

Passed downstream when the child starts

Inherited downstream. Can be set "upward" only with a wider scope

When they become visible

After a row has been produced

After the child pipeline or workflow has started

After the pipeline or workflow that should read them has started

For the full variable reference (how to write ${NAME}, hierarchy, resolvers, and the built-in names), see Variables.

Fields

A field is a column on a data row. Transforms read incoming fields, add or change fields, and send the row on.

After a pipeline run, click the small table icon on a transform to preview the cached rows and see the fields that were in scope. That preview is described in Run, Preview and Debug a Pipeline.

Fields do not travel as variables. To use a field value as a variable in a child pipeline or workflow, map it on the Parameters tab of a Pipeline Executor, Workflow Executor, Pipeline action, or Workflow action. To turn a variable back into a field, use Get variables.

A child can send fields back to its parent. Typical pattern:

  1. In the child pipeline, end with Copy rows to result.

  2. On the parent Pipeline Executor, open the Result rows tab and declare the field names and types you expect back.

  3. Use the executor hop This output will contain the result rows after execution.

The samples project includes this pattern in samples/loops/pipeline-executor.hpl.

Copy rows to result as a way to build loops from a workflow is deprecated. Prefer a pipeline or workflow executor. Returning result rows from an executor child is still the supported way to send fields back to the parent.

Parameters

Think of parameters as function arguments. Declaring them on a pipeline is like writing MyPipeline(parameter1, parameter2, …​). Each parameter has a name, an optional default, and an optional description (pipeline or workflow properties, Parameters tab).

When the pipeline or workflow starts, each parameter is activated as a variable of the same name. From that point you can write ${MY_PARAM} anywhere a variable is accepted. Use Get variables only when you need that value as a field in the stream.

Rules that matter in practice:

  • Declare the parameter on the receiving pipeline or workflow. If a Pipeline Executor (or Pipeline action) sets CUSTOMER_ID, the child must list CUSTOMER_ID in its own properties. Leave the child’s default empty when you want the caller to supply the value.

  • Parameters travel downstream when the child is started. They are not sent back to the parent. To return information, send fields (result rows, above) or set a variable with a wider scope.

  • A value you pass on the caller’s Parameters tab wins over the child’s default. A column titled Parameters/Variables means: set this name in the child, overriding whatever was there before. That is how you override a child default for one execution.

  • Pass parent values to matching parameters decides what happens to a parameter the child declares but the caller’s Parameters tab does not list. With the option on, the caller’s value of that name is passed down. With it off, the child keeps its own default, so a parameter it declares shields it from an unrelated value of the same name in the caller’s scope.

  • The option does not control variables. A child pipeline or workflow always inherits the variables of whatever started it. The option only decides whether a declared parameter is seeded from the caller or keeps its default.

  • Every action and transform that runs a child resolves parameters this way: the Pipeline and Workflow actions, the Pipeline and Workflow Executor transforms, the Mapping transforms, and Metadata Injection.

  • Get Parameters fills the tab from the child you selected. It does not walk further up the call stack.

See Create a Pipeline for the properties dialog.

Variables

Variables are more global than fields, and you choose the scope: the Java virtual machine, the root workflow, the parent, and so on. They are always strings.

Two practical kinds:

  • Configuration variables are set before execution: system properties, environment files, project variables, run configurations, or hop-run -p. Any downstream workflow or pipeline can use ${myVariable} directly. You do not need Get variables unless the value must become a field.

  • Runtime variables are set while a pipeline or workflow is running, usually with Set Variables or the Set Variables action. A running pipeline cannot pick up a new value that another transform in the same pipeline just set: all transforms start together. Set the variable, then start another pipeline or workflow (or let a Pipeline Executor start the child once per row). A nested pipeline that was already running is the same execution; it will not see the new value either.

A variable can be visible "upstream" only when it was set with a wider scope than the child (for example Valid in the parent or Valid in the root). Set Variables also crawls nested execution engines so a value set in a child can reach a mixed workflow/pipeline hierarchy when the chosen scope allows it.

The places you can define variables, and which level inherits from which, are listed in Variables: How can I define variables and Variables: Hierarchy.

Passing values between pipelines and workflows

You want to…​ Do this

Pass a field into a child as a parameter

Map the field on the caller’s Parameters tab. Declare the same name on the child.

Pass a configuration value into everything

Set an environment, project, or run-configuration variable. Refer to ${NAME} in any descendant.

Use a variable as a field

Get variables in the pipeline that needs the field.

Return fields from a child to the parent

Copy rows to result in the child, and Result rows on the parent executor.

Return a single value toward a parent

Set Variables with a parent, grand-parent, or root scope.

Loop with a new parameter value each time

Pipeline Executor or Workflow Executor (once per row, or grouped). Each child start sees the new parameter. See Loops in Apache Hop.

Common pitfalls

  • Same name, different thing. A field customer_id, a parameter CUSTOMER_ID, and a variable CUSTOMER_ID can all exist at once. The field is not the variable. Map it explicitly when you need the value in the other form.

  • Declaring a parameter shields the child from a same-named variable. A parameter is an input the child declares, so the child’s default applies whenever nothing was passed to it - a project or environment variable of the same name does not fill it in by itself. If the child should pick up HOSTNAME from the environment, either tick Pass parent values to matching parameters on the caller, or list HOSTNAME on the caller’s Parameters tab. Where a value is genuinely environment-wide, not declaring it as a parameter at all and just reading ${HOSTNAME} is simpler.

  • Do not set and read a variable in the same pipeline. Transforms run in parallel. There is no guaranteed order, so the reader can see a previous run’s value or nothing. This is the most common variable surprise.

  • A pipeline that is already running will not see new variables. Start a new pipeline (or a new executor iteration) after Set Variables. Parameters are the alternative when you need a value at start-up of each child.

  • Get Parameters only sees one level. If the name is declared two levels up, type it or copy it yourself.

  • Workflows have no data stream. You cannot read fields in a workflow action the way a transform does. Pass parameters into the next pipeline, or use Get variables there.

See also