Spark Catalog

Description

Spark Catalog is a reusable Hop metadata type (category Connections) that describes a Spark SQL catalog for open table formats on the native Spark pipeline engine.

Lake Table transforms in TABLE mode can reference a Spark Catalog by name. Hop expands the entry into spark.sql.catalog.<catalogName>.* configuration when building or preparing the session (LakeSessionPlan / SparkCatalogApplier).

PATH-mode Iceberg tables do not require this metadata type — the engine auto-registers a built-in Hadoop catalog named hop_iceberg. Use Spark Catalog for production Iceberg warehouses, REST catalogs, and other named catalogs.

Connector packaging and session matrix: Lakehouse tables on the native Spark engine.

Load catalog template

Use Load catalog template at the top of the form to fill fields from a named scenario. Placeholders (bucket, host, URI) must still be adjusted for your environment. Secrets are never stored in templates — put tokens in Credential.

Every template writes a commented documentation link as the first line of Extra conf, for example:

# docs: https://iceberg.apache.org/docs/latest/spark-configuration/

# lines are ignored when Hop builds the Spark session, so the link is guidance only.

Template Type What it sets

Iceberg Hadoop (local)

hadoop

Catalog lake, warehouse file:///tmp/hop-warehouse

Iceberg Hadoop (object store)

hadoop

Catalog lake, warehouse s3a://bucket/warehouse, S3 io-impl in conf extra

Iceberg REST

rest

Catalog lake, URI https://catalog.example.com/v1

Iceberg REST (authenticated)

rest

Same as REST; conf extra notes that Credential maps to catalog .token

Hive Metastore (advanced)

hive

Catalog hive, thrift URI placeholder (extra deps required)

AWS Glue (advanced)

glue

Catalog glue, S3 warehouse + IO placeholders (extra deps required)

Nessie (advanced)

custom

Catalog nessie, Nessie catalog-impl / URI / ref placeholders

Databricks Unity Catalog (advanced)

custom

Catalog unity, commented workspace URI/warehouse examples

Delta named catalog (advanced)

custom

DeltaCatalog implementation; prefer spark_catalog when coexisting with Iceberg

The Hop metadata Name (how transforms select this entry) is not changed by a template — only catalog fields are.

Options

Option Description

Name

Hop metadata name (how transforms reference this entry).

Load catalog template

Button that applies a named scenario (see above). Confirms before replacing non-empty fields.

Catalog name

Spark SQL catalog name — first segment of catalog.namespace.table (e.g. lake).

Catalog type

hadoop — Iceberg Hadoop warehouse (default).
rest — Iceberg REST catalog.
custom — supply implementation + conf extra.
hive / glue — advanced; conf-extra oriented, not packaged by default.

Implementation

Spark catalog class. Default for Iceberg is org.apache.iceberg.spark.SparkCatalog.

Warehouse

Warehouse path for Hadoop (and similar) catalogs, e.g. file:///data/warehouse or an object-store URI.

URI

Catalog service URI (REST and similar).

Credential

Optional secret (password field). For REST catalogs, applied as spark.sql.catalog.<name>.token when set.

Conf extra

Additional key=value lines (one per line; # comments and blank lines ignored).
Short keys (e.g. io-impl) become spark.sql.catalog.<catalogName>.io-impl.
Keys that already start with spark. are applied as full Spark conf keys.

How it is applied

For a catalog named lake of type hadoop with warehouse file:///tmp/wh, Hop effectively sets:

spark.sql.catalog.lake=org.apache.iceberg.spark.SparkCatalog
spark.sql.catalog.lake.type=hadoop
spark.sql.catalog.lake.warehouse=file:///tmp/wh

Lake Table transforms then use identifiers such as lake.db.orders.

When both Delta and Iceberg are used in one session, keep Delta on spark_catalog (DeltaCatalog) and put Iceberg under a different catalog name (e.g. lake or hop_iceberg).

spark-submit

Export project metadata JSON that includes every Spark Catalog entry referenced by the pipeline (and any used only via run configuration). Alternatively, pass equivalent --conf spark.sql.catalog.… flags on the submit command. Paths and warehouses must be reachable from the driver and executors.

Related pages