Native Spark Pipeline Engine

Overview

The native Spark pipeline engine executes batch Hop pipelines on Apache Spark 4.1.x without Apache Beam. It builds a Spark SQL Dataset graph from the pipeline and runs transforms via native Dataset operators or generic mapPartitions mini-pipelines.

For a full introduction — concepts, local[*], fat jar, spark-submit, Spark File I/O, ${Internal.Transform.ID}, metrics, limitations, and integration-test examples — see Getting started with the native Spark pipeline engine.

Paths on Spark File and Lake Table transforms are Spark/Hadoop URIs, not Hop VFS schemes — see Paths and file systems on native Spark.

This page is not the same as Beam Spark, which runs Hop over the Apache Beam Spark runner (Spark 3.5.x line).

Native Spark pipeline run configuration editor

Requirements

  • Java 21

  • Spark 4.1.x (bundled under plugins/engines/spark/lib for local[*])

Options

Load configuration template

Use Load configuration template to fill the fields below from a common deployment style:

  • Local execution — local[*], Spark UI off (embedded Spark in Hop)

  • Local (capped cores) — local[4]

  • Spark standalone server — spark://host:7077 (replace host), sample driver/executor memory

  • spark-submit (cluster) — master empty so spark-submit --master wins; use a native-provided fat jar

  • Databricks — submit-oriented placeholders (master empty); set workspace-specific values yourself

  • YARN client — master yarn, client deploy mode

  • Lakehouse local — same as local; Delta/Iceberg connectors live on the plugin classpath

Templates replace the current engine options after confirmation when the form already has custom values. The form fields update immediately after you apply a template. Edit values afterward as needed.

For named Iceberg/Delta catalogs used by Lake Table transforms in TABLE mode, see Spark Catalog (Load catalog template).

Option Description Default

Spark master

Spark master URL: local[*], local[4], or spark://host:7077. Leave empty under spark-submit so the CLI master is used.

local[*]

Application name

Spark application name.

Apache Hop

Fat jar file location

Optional Hop fat jar staged for executors when Hop starts the Spark session. Leave empty for pure spark-submit with a native-provided fat jar.

Spark config (key=value lines)

Additional SparkConf entries, one key=value per line.

Driver memory

spark.driver.memory (e.g. 2g).

Executor memory

spark.executor.memory.

Executor cores

spark.executor.cores.

Temp location

Directory for temporary files.

JVM temp dir

Plugins to stage

Optional comma-separated plugin folders to stage on the cluster.

Path scheme map (from=to lines)

Optional URI scheme rewrite for Spark File / Lake PATH only (e.g. s3=s3a, minio=s3a). Does not apply to classic Hop VFS transforms. Configure target FS (e.g. spark.hadoop.fs.s3a.*) separately. See Paths and file systems.

Generic transform run mode

Default for Hop transforms that run as mapPartitions mini-pipelines (Workflow Executor, most classic transforms). DISTRIBUTED fans work out across Spark partitions/nodes. DRIVER_ONLY materializes the transform input on the Spark driver and runs the mini-pipeline once there, so nested workflows do not execute on multiple nodes. Override per transform with the Spark Run Mode context action on the canvas.

DISTRIBUTED

Also configure Execution information location and Execution data profile on the run configuration main tab when you want GUI metrics history and row samples.

Generic transform run mode and per-transform override

Most classic Hop transforms (including Workflow Executor, Pipeline Executor, and mappings) execute on Native Spark as generic mapPartitions work. When the input Dataset has many partitions, Spark can run that nested work on multiple executors at once. That is often desirable for pure row transforms, but nested workflows or other non-reentrant side effects may need a single place of execution.

  • Run configuration default — Generic transform run mode: DISTRIBUTED (default) or DRIVER_ONLY.

  • Per-transform override — right-click a transform → Spark Run Mode:

    • Inherit — use the run configuration default

    • Force distributed — always use mapPartitions across partitions

    • Force Driver Only — always run this transform’s mini-pipeline on the Spark driver

      The override is stored on the transform (attributes group spark / key run_mode) and is saved with the pipeline. When a force override is set, a small badge is drawn at the bottom-left of the transform icon.

Transform with Force Driver Only Spark run-mode badge

Native Dataset handlers (Spark File / Lake Table, Memory Group By, Sort, Unique, Merge Join) ignore this setting.

DRIVER_ONLY collects that transform’s input rows on the driver. Prefer DISTRIBUTED for high-volume data; use driver-only for nested execution control (for example a Workflow Executor over a modest set of control rows).

Nested Native Spark pipelines (orchestrator pattern)

A parent Native Spark pipeline can start child Native Spark pipelines via Pipeline Executor (run configuration = Native Spark):

  • Put Force Driver Only (or run-config DRIVER_ONLY) on the Pipeline Executor so children start on the driver.

  • Each child reuses the parent SparkSession (same Spark application) — it does not start a second spark-submit.

  • Parameterize the child with control keys (country, month, partition) for paths and filters.

  • Nested engines must not stop the shared session when they finish.

Do not run nested Native Spark executors under DISTRIBUTED mapPartitions on workers (no safe nested SparkContext). For multi-application isolation (separate Spark apps per key), use an external job launcher instead.

Demo: sample project spark-demo pipeline pipelines/03-run-pipelines.hpl (see Cluster demo walk-through).

Transforms not supported on Native Spark

Most classic Hop transforms run on Native Spark via native Dataset handlers or generic mapPartitions. The list below is taken from each transform’s Supported Engines table (Native Spark = Not Supported). When in doubt, open the transform page — that table is the source of truth.

For broader design limits (copy/distribute, streaming, Spark SQL, MLlib), see Getting started — Limitations and design rules.

Beam-only transforms

Use a Beam engine instead.

Barriers, stream ordering, and merge-style transforms

These rely on waiting for other transforms, full-stream ordering, or multi-stream merge semantics that do not map cleanly to a Spark Dataset graph.

Sorted aggregation and hash-set unique

Workflow / pipeline result and control transforms

Result-row and result-file hops, injectors, and metadata injection are not supported as Native Spark graph nodes.

Other

  • Kafka Consumer — no native Spark structured-streaming / Kafka Dataset I/O; use Beam Kafka transforms on a Beam engine for Kafka pipelines

Related