Simple Mapping transform Icon Simple Mapping

Description

The Simple Mapping transform allows you to re-use a series of transforms in the form of a "re-usable pipeline" also known as a "Mapping".

Supported Engines

Hop Engine

Supported

Spark

Supported

Flink

Supported

Dataflow

Supported

Think of it as a re-usable function in programming – you pass some data into the function and it returns data back. In the Simple Mapping transform you can specify one Mapping Input transform to pass data to and one Mapping Output transform to retrieve rows of data from, as well as parameters to pass to the mapping.

Use this transform if you find yourself repeating the same logic multiple times over several different pipelines.

Options

The options are fairly self-explanatory:

  • Transform name: a unique name in your pipeline

  • Pipeline: The name of the mapping (sub-) pipeline to embed. This pipeline is executed separately and is considered a separate entity. It is always executed with a standard local Hop pipeline run configuration since that is currently the only one capable of streaming data into and out of transforms.

  • Parameters tab: specify the parameters you want to pass to the mapping pipeline

  • Input tab: specify which fields to pass to the Mapping Input transform in the mapping. You can map (hence the name) field names.

  • Output tab: specify how to rename output fields retrieved from the Mapping Output transform.

Sample

In the samples project you can find two examples of the Simple Mapping transform, the transforms/files/simple-mapping-child.hpl and simple-mapping-parent.hpl pipelines.

In the sample files discussed above, the parent pipeline sends first and last name fields to the child, a calculation is performed and then a field is populated with the calculation and sent back.