Sorted Schema Merge transform Icon Sorted Schema Merge

Description

The Sorted Schema Merge transform combines the behavior of Sorted Merge and Stream Schema Merge:

  • It merges multiple pre-sorted input streams into one globally sorted output stream.

  • Input streams may have different field layouts. The transform builds a unified output schema that is the union of all input fields.

  • Fields that are not present on a given input stream are left null (sparse row layout) on rows that come from that stream.

A typical use case is combining data from multiple Data Vault satellites that share business keys and load timestamps, then using Repeat Fields to fill previously known attribute values when building an SCD2 table.

Each input stream must already be sorted on the configured sort keys (for example with ORDER BY in Table Input, or with Sort Rows). This transform does not re-sort within each stream; it only merges already-sorted streams.

Supported Engines

Hop Engine

Supported

Spark

Not Supported

Flink

Not Supported

Dataflow

Not Supported

Options

Option Description

Transform name

Name of the transform; must be unique in a single pipeline.

Sort keys

The key fields used to compare rows across input streams, including ascending or descending direction. Every sort key must exist on every input stream.

Input transforms

The list of upstream transforms that feed this transform. Use Get to fill the list from connected hops. At least two input streams are required.

Notes

  • When the same field name appears on more than one stream with different data types, the merged field type is coerced to String (same behavior as Stream Schema Merge).

  • Rows with equal sort keys keep a stable order based on the input stream index.

  • Safe mode row layout checks are not applied across streams with different schemas (excludeFromRowLayoutVerification).