OpenLineage
OpenLineage is an open standard for lineage metadata collection.
The OpenLineage technology plugin is a Hop lineage sink: it subscribes to the in-engine
lineage observation hub and maps Hop’s neutral lineage events
to spec-correct OpenLineage RunEvent JSON, which it posts to any OpenLineage-compatible collector.
The primary target is Marquez.
Events are generated by the spec’s official openlineage-java client and delivered asynchronously off
the pipeline thread, with bounded retries on transient HTTP failures. Delivery is bounded in memory
and, with a spool configured, survives a collector outage or a restart — see Durability.
Enabling the sink
The simplest way is a Lineage Sink of type OpenLineage, selected in a pipeline or workflow run configuration: the runs of that configuration report their lineage to it, with the options of the sink, and no variables are needed.
The sink can also be configured with variables, for every run of a Hop process. The lineage hub is disabled by default. Enable it and select the OpenLineage sink with these engine-scoped variables (see the lineage hub for the hub-wide variables):
| Variable | Default | Description |
|---|---|---|
|
|
Set to |
|
(empty) |
Set to |
The sink itself reads the following variables when it initializes:
| Variable | Default | Description |
|---|---|---|
|
(required) |
Collector URL, e.g. |
|
|
OpenLineage job namespace: the orchestrator instance that produces the events, not the environment. See Job namespace and environments. |
|
(plugin URL) |
OpenLineage |
|
|
|
|
|
Parse |
|
|
|
|
|
Emit a |
|
(empty) |
Optional bearer token sent as |
|
|
HTTP connect timeout. |
|
|
HTTP read/response timeout. |
|
|
Async outbound queue capacity. |
|
|
Retries per event on transient HTTP failure (exponential backoff). |
|
|
Initial retry backoff; doubles each attempt. |
|
|
What to do when the outbound buffer is full. |
|
|
Under |
|
|
Maximum time to drain the queue on shutdown. |
|
|
If > 0, periodically log |
|
(empty) |
Trust/key store for HTTPS collectors with a custom CA/cert. |
|
(empty) |
Store password (used with the trust store path). |
|
|
Trust store type (e.g. |
|
(empty) |
Folder that holds undelivered events. Empty = no spool (see Durability). |
|
|
Ceiling on the spool’s size on disk (64 MiB), per process. |
There is no proxy setting — route via the JVM -Dhttps.proxyHost properties if needed.
Durability
Events are sent in the order they were produced. When a delivery fails, the sink stops at that
event, pauses (one second, doubling up to a minute) and tries again from there, so a collector that
is down costs one attempt per pause rather than one per event, and a run’s COMPLETE never
overtakes its START. Meanwhile new events wait: in memory up to
HOP_LINEAGE_OPENLINEAGE_BUFFER_SIZE, after which the overflow policy applies.
By default that is all: an event still waiting when the process ends is lost. Set
HOP_LINEAGE_OPENLINEAGE_SPOOL_DIR to a writable folder to change that into at least once:
-
every event is written to its own file in the spool before it is posted. A thread of the sink’s own writes it, so the pipeline thread never waits on the disk, and the spool keeps filling while the sink waits on a collector that does not answer;
-
the file is deleted when the collector answers 2xx, or refuses the event (a 4xx other than 401, 403, 404 and 407, which point at configuration and are kept); a refused event is logged and not sent again;
-
when the collector is down at shutdown, the sink does not wait for it: the events stay on disk and the next process that starts sends them, oldest first, before its own.
A collector outage, a GMS restart, or a process that dies between "pipeline finished" and "HTTP 200" therefore delays lineage instead of losing it. A process that dies after a successful POST but before the delete sends one event again; OpenLineage events set aspects on entities rather than append records, so the replay writes the same state again.
The spool is bounded by HOP_LINEAGE_OPENLINEAGE_SPOOL_MAX_BYTES. At the cap, new events are still
sent but no longer spooled (counted, and reported once as an error), and the oldest files are kept:
those are the ones no later run will re-emit. dropped then counts only events that could not be
written to disk and did not fit in memory either.
Several processes can share a spool folder, for example a Hop Server and the hop-run commands on
the same machine. Each process writes to a folder of its own inside it and renews a heartbeat file
there every 30 seconds. A process only sends the events of a folder whose owner is gone: one that
shut down (it removes its heartbeat), or one whose heartbeat is three minutes old (it was killed).
It moves those files into its own folder first, so no event is sent by two processes.
Event mapping
| Hop lineage event | OpenLineage output |
|---|---|
|
|
|
The run’s hop graph. With |
|
The workflow’s hop graph. On their |
|
Physical file datasets ( |
|
Relational table datasets — see Database lineage. A written table carries a |
|
Skipped (internal row shapes between transforms, not physical datasets). |
|
Skipped. |
OpenLineage has no event of its own for a dataset appearing on its own — a collector attaches a
dataset to the run that reports it — so FILE_IO and RELATIONAL_IO observations are held against
the run they were made in and travel on that run’s COMPLETE or FAIL event. A pipeline therefore
reports the files and tables it read and wrote on its own completion, and the collection of a run
never has to guess which event a dataset belongs to. An observation whose run has already reported its
end, or one still pending when the sink shuts down, is sent as a standalone dataset event instead of
being dropped.
Every run event carries a processing_engine run facet (name hop and the Hop version) and says
what its job is: a jobType facet (PIPELINE, WORKFLOW, TRANSFORM or ACTION) and a hop
job facet with the subject type and the pipeline, workflow, transform or action name and file name.
A job name such as nightly/Load orders cannot be split back reliably, so a collector that models
pipelines and transforms separately reads those names from the facet.
A transform that ran exactly one SQL statement (Table Input, Execute SQL) has it in the standard
sql job facet. A Formula or Calculator lists the expressions it calculates its fields with in a
hopTransformLogic job facet: fields, each with its field, its language (formula or
calculator) and its expression, such as [price] * [qty] or MULTIPLY(total, rate).
Row counts
A transform’s COMPLETE or FAIL event carries the row counters of that transform copy in a
hopRowCounts run facet: linesRead and linesWritten (rows from and to other transforms),
linesInput and linesOutput (rows read from and written to files and tables), linesUpdated and
linesRejected. These are the transform’s own counters, the numbers it logs as I=, O=, R=,
W= and U=; Text File Output counts its header and footer lines as output too, so a file with a
header shows one line more than it has rows. A pipeline’s event carries the sum over all its
transforms and copies of the counters that add up: linesInput, linesOutput, linesUpdated and
linesRejected. That sum is not the pipeline result Hop logs, which takes the highest value of each
counter, and a Metadata Injection transform counts the rows of the pipeline it runs too. Only the
local pipeline engine sends counters; Beam, Spark and remote runs don’t have final ones when their
run ends.
A dataset gets the standard inputStatistics or outputStatistics facet with its rowCount when
the count is known to be its own: the transforms that read (or wrote) it read (or wrote) nothing
else. The rows written are the rows output plus the rows updated. A transform that writes several
files or tables, such as Text File Output with a file per copy or Table Output with the table in a
field, gives them no row count rather than a wrong one. On a pipeline’s event a dataset’s count is
the sum over its transforms and their copies; with transform jobs on, each copy’s event counts the
rows of that copy. Rows a transform deleted from a table (Delete, an Execute SQL with a single
DELETE statement) are not rows written; a table emptied and reloaded in one run counts the rows
loaded. An Execute SQL script of several statements gives no count, as its counters add all of them
up, and neither does a transform that counts 0 while it does handle rows (not every transform counts
what it reads or writes).
A pipeline’s COMPLETE or FAIL event also lists every dataset the run wrote, once, in a
hopWrites run facet: its namespace and name, its operationType (INSERT, UPDATE,
DELETE, CREATE, DROP or ALTER) and, when known, its rows. The DataHub sink turns these
into operations. A pipeline that was stopped (Stop, Abort, or a failing parent pipeline stopping
it) says so with stopped in its hopRowCounts facet and lists no writes.
Apache Beam and Spark run the transforms of a pipeline in small local pipelines on their workers. With lineage on in the JVM of those workers, as with the Beam DirectRunner or Spark in local mode, each of those reports a run of its own, with the counts and writes of its bundle or partition.
The run id is the executing subject’s log-channel id. By default file I/O is correlated to
the parent pipeline run, so inputs and outputs appear on the pipeline job in Marquez. Every run also
names the run that started it: a transform’s ParentRunFacet points at its pipeline, an action’s at
its workflow, and a pipeline’s at the action that ran it (or at the parent pipeline, for a
sub-pipeline). Actions are jobs of their own, named <workflow>/<action>, so a workflow with several
actions no longer files them all under the workflow’s job.
Job hierarchy
Jobs are named <owner>/<job>: a transform is <pipeline>/<transform>, an action is
<workflow>/<action>. With the parent facets on, a collector that understands parent runs draws the
orchestrated run as a tree:
workflow ──▶ action ──▶ pipeline ──▶ transform
A pipeline or workflow that nothing started (a hop-run invocation, the GUI) has no parent.
Set HOP_LINEAGE_OPENLINEAGE_PARENT_RUNS=N to leave the facets out when the collector builds the
hierarchy from something else.
Job granularity
OpenLineage derives dataset-to-dataset lineage through jobs: a job’s inputs are paired all-to-all
with its outputs. Filing every dataset of a pipeline under the pipeline’s job is what makes Marquez
show them on the job a user opens, but it also means an n-input/m-output pipeline produces the
full n×m product: reading a and b and writing x and y yields a→x, a→y, b→x and b→y.
Set HOP_LINEAGE_OPENLINEAGE_TRANSFORM_JOBS=Y to make the job mean the transformation. Each
transform’s job then carries:
-
the datasets it read itself, plus
-
everything its upstream transforms produced — what they wrote if they wrote anything, otherwise what they read; a transform that does no I/O of its own (a filter, a select) passes what it received through to its downstream transforms.
That gives a writer’s job both ends of a real pair (a→x and b→y above), and lineage still
crosses transforms that do not touch a file or table. The pipeline job remains the run-level entry
point, without datasets. The trade-off is the one the default was chosen for: with the flag on, the
datasets sit on the transform jobs in Marquez, so a user who opens the pipeline has to click through
to a transform to see them.
Forward propagation needs the pipeline’s hop graph. The engine emits it as a PIPELINE_GRAPH event
next to the run’s START, and this sink uses it only when the flag is on. A run without a graph — an
engine that predates the event, or an event stream that lost it — keeps the pipeline-granular
mapping, so the flag can stay on across an upgrade.
What a transform consumed is only known once its pipeline is over: an upstream writer reports its
table when it finishes, which can be after a downstream transform finished. With the flag on, the
transforms' COMPLETE and FAIL events are therefore sent when the pipeline ends, just ahead of the
pipeline’s own.
Database lineage
RELATIONAL_IO events record the relational tables a transform reads from or writes to. The physical
dataset is identified by a database namespace (e.g. postgres://host:5432) and a fully-qualified
database.schema.table name (see Dataset identity), so a Hop write and, for example, a dbt read
of the same table land on one node in the lineage graph.
The following transforms emit relational lineage:
-
Reads — Table Input (its
SELECTis parsed to recover the source tables and column lineage). -
Writes — Table Output, Insert / Update, Update, Dimension Lookup/Update, Combination Lookup/Update, and the database bulk loaders (CrateDB, MonetDB, MySQL, Oracle, PostgreSQL, Vertica).
-
Deletes — Delete reports the affected table only. A delete produces no columns, so the event carries neither a column schema nor column lineage.
-
Reads and writes — Execute SQL (its statements are parsed for both input and output tables).
SQL parsing
For transforms that only provide a SQL statement (Table Input, Execute SQL), the sink recovers the
tables — and, for an INSERT … SELECT, the column-level lineage — by parsing the statement with the
OpenLineage SQL parser. Parsing is on by default and controlled by HOP_LINEAGE_OPENLINEAGE_SQL_PARSE.
Tables named without a schema or catalog are qualified using the connection’s default database and
preferred schema, so a bare SELECT … FROM orders shares one identity with a schema-qualified write
of the same table.
An INSERT INTO summary (order_id, customer) SELECT id, cust FROM … names the output columns after
the target list, not after the SELECT expressions: the sink reads the two lists from the statement
and renames each parsed edge positionally. The rename is only applied when both lists can be read
unambiguously (no *, a plain VALUES insert, or two statements), because attaching column lineage
to the wrong column is worse than leaving the parser’s name in place.
Column-level lineage
The sink emits an OpenLineage columnLineage facet on the output dataset in two cases:
-
SQL-derived — an
INSERT … SELECT(Execute SQL) maps each target column to the source columns the parser resolves it from. -
Stream path — a relational write transform whose columns are traced back through the pipeline stream to the read (or lookup) that produced each field, giving
target.column ← source.columnedges. Covered writers: Table Output, Insert / Update, Update, Dimension Lookup/Update (natural keys and argument-bound attributes), Combination Lookup/Update (keys), and the database bulk loaders. Generated columns — a dimension’s technical key, version and validity dates — have no stream source and carry no column lineage.
Writes that replace their target (a truncate, or the target of an overwrite) additionally carry a
lifecycleStateChange facet.
Column types in the schema facet use the lowercase OpenLineage names — string, int, double,
decimal, date, timestamp, boolean, binary, and struct for JSON — rather than Hop’s own
type descriptions. A type with no honest OpenLineage equivalent (Serializable, or a type a plugin
invented) is left out instead of guessed.
Job namespace and environments
The job namespace identifies the orchestrator instance that produced the events, not the
environment: hop on a single deployment, hop-prod and hop-dev when one collector holds several.
It is part of a job’s identity, so two environments that share it merge into one job in the collector’s
UI even though they wrote different datasets. A collector may name the orchestrator from the
processing_engine facet instead (DataHub’s OpenLineage endpoint does), in which case the namespace
does not appear in its job identity at all.
The environment itself is a property of the collector’s deployment: the same collector marks every dataset it receives with one environment, whichever engine sent it, which is what lets a Hop file dataset and a Spark file dataset join up. Two environments that must not merge therefore need two collector deployments, not two namespaces.
Unit test results
A pipeline unit test that runs with lineage reports each golden data set as an OTHER event with a hopUnitTest run facet: the test, the data set, whether it passed, error when the rows could not be compared at all, what the comparison found, and the datasets the transform declares it writes. Those datasets are not the event’s outputs: the test run didn’t write them. The event belongs to the tested transform’s run with HOP_LINEAGE_OPENLINEAGE_TRANSFORM_JOBS=Y, to the pipeline’s run otherwise.
A unit test runs a changed copy of the pipeline. A lineage sink selected in a run configuration only gets the test’s results, not the lineage of that copy.
Dataset identity
The (namespace, name) rules the sink uses to identify datasets — and the naming another producer
must match to stitch into Hop’s lineage graph — are a stable, versioned contract documented in
OpenLineage dataset identity. Read it before
integrating a second lineage producer against the same collector.
Verifying
-
Set
HOP_LINEAGE_ENABLED=Y,HOP_LINEAGE_SINK_IDS=openlineage, andHOP_LINEAGE_OPENLINEAGE_URL. -
Run a pipeline that reads a file (e.g. a CSV input).
-
Open the Marquez UI and confirm a job run (
START/COMPLETE) and at least one file dataset. -
For database lineage, run a pipeline that reads and writes a table (e.g. Table Input → Table Output) and confirm the input/output table datasets — and, for an
INSERT … SELECT, thecolumnLineagefacet — in Marquez.
On startup the sink logs OpenLineage lineage sink initialized: url=…, namespace=….
Tools > Lineage sinks in Hop GUI, and the status page of a Hop Server, show what the sink sent, what waits and the last error; see Sink health.