Spark Lake Table Input transform Icon Spark Lake Table Input

Description

Spark Lake Table Input reads an open table format table (Delta Lake or Apache Iceberg) as a Spark SQL Dataset on the native Spark pipeline engine.

It supports PATH and TABLE identifiers, optional field projection, and time travel by version/snapshot or timestamp.

This transform is not available on the local Hop engine or on Beam engines. Connector JARs ship with the native Spark plugin; see Lakehouse tables on the native Spark engine.

Supported Engines

Hop Engine

Not Supported

Single Threaded

Not Supported

Native Spark

Supported

Beam Spark

Supported

Beam Flink

Not Supported

Beam Dataflow

Not Supported
Spark here means Hop’s native Spark pipeline engine only (not Beam Spark).

Options

Option Description

Transform name

Unique name of the transform in the pipeline.

Format

delta or iceberg.

Identifier mode

PATH — table root path/URI.
TABLE — catalog-qualified name (e.g. lake.db.orders).

Table path

Used when mode is PATH. Must be visible to Spark workers (local path on local[*]; shared URI on cluster).

Table identifier

Used when mode is TABLE. Multi-part name: catalog.namespace.table.

Spark Catalog metadata name

Optional Hop Spark Catalog entry to register for TABLE mode (Iceberg Hadoop/REST, …). Leave empty if the catalog is already configured on the session.

Time travel type

NONE (current), VERSION, or TIMESTAMP.

Time travel version

Delta table version number or Iceberg snapshot id (when type is VERSION). Supports variables.

Time travel timestamp

Spark-parseable timestamp string when type is TIMESTAMP, e.g. 2024-01-15 10:30:00.

Extra options

Optional Spark read options as key=value lines (advanced).

Fields

Optional projection. When empty, all columns are returned. When set, columns are selected/cast by name (same idea as Spark File Input with a schema).

Time travel

Type Delta Iceberg

NONE

Current table version

Current snapshot

VERSION

Read option versionAsOf

SQL VERSION AS OF <snapshot_id>

TIMESTAMP

Read option timestampAsOf

SQL TIMESTAMP AS OF TIMESTAMP '…'

Tip: After two Overwrite writes, Delta version 0 is typically the first commit. For Iceberg, inspect {table}.snapshots (or catalog metadata) for snapshot ids.

Metadata injection

All options support metadata injection (ETL Metadata Injection).

Common keys: FORMAT, IDENTIFIER_MODE, TABLE_PATH, TABLE_IDENTIFIER, CATALOG_METADATA_NAME, TIME_TRAVEL_TYPE, TIME_TRAVEL_VERSION, TIME_TRAVEL_TIMESTAMP, EXTRA_OPTIONS.

Field projection group FIELDS: NAME, TYPE, LENGTH, PRECISION, FORMAT_MASK.

Typical pattern: template pipeline with Lake Table Input; injecting pipeline supplies table paths / identifiers (and optional field list) per supplier or table.

Notes

  • Prefer Lake Table Input over Spark File Input for delta / iceberg so session extensions, hop_iceberg PATH catalog, and probes stay consistent.

  • See the lakehouse guide for packaging, PATH vs TABLE, and troubleshooting.

Related pages