Change Data Capture

Apache Hop can load a data warehouse or another target from changes in a source, not only from full extracts. That is Change Data Capture (CDC): detect inserts, updates and deletes, then apply only those rows downstream.

Hop does not ship a single “CDC transform”. There is also no built-in Debezium or GoldenGate connector. CDC in Hop is a choice of approach, depending on what the source can already tell you:

  • a change log produced by another product

  • a column that only moves forward (a timestamp or an increasing ID)

  • nothing but two full snapshots you can compare

The chat-level picture of “a queue plus a schedule that reads deltas” is only one of those three. The pages below treat each pattern separately.

The three approaches

Approach Source of truth Typical Hop entry point Detects deletes? Latency Cost

Log sniffing

Redo / WAL / trail consumed by Debezium, Oracle GoldenGate, and similar tools

Kafka consumer, JMS consumer, or Table Input

Yes, when the log emits delete (or tombstone) events

Near real-time, or a scheduled drain of the topic

Whatever the log product costs to run; Hop only consumes the events

Filtered selection

A monotonic timestamp or an always-increasing ID

A watermark you wrote on the last successful run, then Table Input with a WHERE clause

Only if the source marks deletes (a flag or a delete table)

The batch interval of the workflow

One cheap range query

Comparing snapshots

Two full extracts, sorted on a key

Merge Rows (diff), or a stored hash key (MD5 / SHA)

Yes

Batch, after both extracts are available

Read (and usually sort) both full sets

How to choose

Use log sniffing when a CDC product already writes change events to Kafka, JMS, files, or a staging table. Hop’s job is to parse those records and apply them. Hop is not a log miner.

Use filtered selection when every insert or update you care about touches a timestamp or an always-increasing ID, and you can answer “what is the last time this workflow or pipeline ran successfully?”. The simplest answer is to write that watermark yourself at the end of the success path. Execution information can look the same fact up from Hop’s own execution store.

Use snapshot comparison when there is no change log and no reliable watermark: daily files, legacy systems, or “today’s extract versus yesterday”. Merge Rows (diff) compares two sorted streams of millions of rows in one pass and flags each row as new, changed, identical, or deleted.

Applying the changes

All three approaches end the same way: you have rows that should be inserted, updated, or deleted on a target. Common apply transforms are:

Slowly changing dimensions are a target pattern. They do not detect source changes. Pair them with one of the three CDC approaches above.

Further reading