Spark Lake Table Merge transform Icon Spark Lake Table Merge

Description

Spark Lake Table Merge applies a Spark SQL MERGE INTO against a Delta Lake or Apache Iceberg target table on the native Spark pipeline engine.

The single main-stream upstream hop is the merge source. The transform builds (or accepts) MERGE SQL, executes it as a Spark action, and registers an empty leaf Dataset.

Connectors ship with the native Spark plugin. Session rules: Lakehouse tables on the native Spark engine.

Supported Engines

Hop Engine

Not Supported

Single Threaded

Not Supported

Native Spark

Supported

Beam Spark

Supported

Beam Flink

Not Supported

Beam Dataflow

Not Supported
Spark here means Hop’s native Spark pipeline engine only (not Beam Spark).

Options

Option Description

Transform name

Unique name of the transform in the pipeline.

Format

delta or iceberg.

Identifier mode

PATH or TABLE for the target table.

Table path / Table identifier / Spark Catalog

Same semantics as Lake Table Input/Output for the merge target.

Merge condition

ON clause body. Use aliases t (target) and s (source), e.g. t.id = s.id. Required unless raw SQL is set.

When matched

UPDATE_ALL — WHEN MATCHED THEN UPDATE SET *
DELETE — delete matched target rows
NONE — no matched clause

When not matched

INSERT_ALL — WHEN NOT MATCHED THEN INSERT *
NONE — no not-matched clause

When not matched by source

NONE (default) or DELETE (Delta; Iceberg support depends on engine/version). Removes target rows with no source match.

Raw MERGE SQL

Advanced override. When non-empty, replaces structured fields. Empty by default. Operator is trusted to supply valid SQL.

Behaviour

  • Source rows are registered as a temporary view; structured mode emits SQL similar to:

    MERGE INTO delta.`/path/to/table` AS t
USING hop_merge_src_… AS s
ON t.id = s.id
WHEN MATCHED THEN UPDATE SET *
WHEN NOT MATCHED THEN INSERT *

  • Target SQL ids: Delta PATH → delta.\`path`; Iceberg PATH → hop_iceberg.\`uri\`; TABLE → multi-part identifier.

  • At least one action (matched and/or not-matched) is required.

  • Metrics are log-only (merge completion); GUI in/out row counts for the sink may stay at zero.

Metadata injection

Supports metadata injection.

Keys: FORMAT, IDENTIFIER_MODE, TABLE_PATH, TABLE_IDENTIFIER, CATALOG_METADATA_NAME, MERGE_CONDITION, MATCHED_ACTION, NOT_MATCHED_ACTION, NOT_MATCHED_BY_SOURCE_ACTION, RAW_MERGE_SQL.

Useful for multi-table CDC/upsert templates where merge keys and target tables vary per run.

Notes

  • Exactly one main-stream input hop.

  • Prefer structured fields for common upserts; use raw SQL only when you need full MERGE control.

  • Packaging and PATH vs TABLE: lakehouse guide.

Related pages