Execution Information Location

Description

The Execution Information Location determines where Apache Hop can send information execution information to.

After execution this information can be consulted from the Execution Information Perspective.

Old executions can be removed from that perspective, or on a schedule with the Delete execution information action. The action deletes stored executions and leaves this metadata object in place.

Options

Option	Description	Default
Name	The name to use for this Execution Information Location	-
Description	The description to use for this Execution Information Location	-
Data logging delay (ms)		2000
Data logging interval (ms)		5000
Data logging size (rows)		1000
Location type	The location plugin type to write to. Depending on the type you can enter more plugin specific options.

Option

Description

Default

Name

The name to use for this Execution Information Location

Description

The description to use for this Execution Information Location

Data logging delay (ms)

2000

Data logging interval (ms)

5000

Data logging size (rows)

1000

Location type

The location plugin type to write to. Depending on the type you can enter more plugin specific options.

Location type specific options

File location

With a file location you can write execution information to files in a path on the local file system.

Options:

Root folder: the folder to write the execution information to.
Create folder?: automatically create the root folder if it doesn’t exist.

Like the file location, execution information is written to a folder. The difference is that information is written per pipeline or workflow. All the information needed is aggregated into these top level JSON files. The way this happens is by keeping the execution information in a memory cache for a period that can be configured. The upside is that there is a lot less stress on the file system. The downside is that there is a bit of extra memory being used. Certain navigation operations in the GUI, for example to find related executions, will need to read a lot more data.

Options:

Root folder: the folder in which execution information is stored in JSON files
Create folder?: automatically create the root folder if it doesn’t exist.
Persistence delay: This is the maximum time to wait before execution information is written to disk, expressed in milliseconds.
Maximum cache age: This is the maximum time to keep execution information around in memory, before clearing it out, expressed in milliseconds. It’s advised to keep this below the expected duration of your workflow or pipeline.

The root folder can be a local path or a VFS URI. For example, on Databricks you can store execution JSON on a Unity Catalog Volume via a Databricks VFS Connection (db-volume:///executions) so a native Spark job on the cluster and the laptop GUI share the same location. Full walkthrough: Execution information on a UC Volume.

Remote location

A remote location allows you to write execution information to a Hop Server.

Options:

Hop Server: the remote Hop Server to write the execution information to.
Execution Information Location: the Execution Information Location to be used on the remote Hop Server.

Graph database location

With this location type (previously Neo4j location) you can store execution information in a graph database which speaks Cypher: Neo4j, Memgraph, Amazon Neptune, FalkorDB or Apache AGE. For more information on this plugin, see the graph database location type documentation.

Elastic location

The Elastic location type is a caching location that stores one JSON document per execution in an Elasticsearch index. Authenticate with an API key. See the Elastic location type.

OpenSearch location

The OpenSearch location type is the same idea as Elastic — a caching location, one JSON document per execution — for an OpenSearch cluster. Authenticate with a username and password. Listing uses OpenSearch SQL (/_plugins/_sql) rather than Elasticsearch _search. See the OpenSearch location type for the options and a comparison with Elastic.

Caching Database location

Like the Caching File location, execution information is aggregated in memory and then persisted as one document per pipeline or workflow. Instead of writing JSON files, this location stores the same JSON payload in a single relational table, together with a small set of filter columns so the Execution Information perspective can query efficiently without parsing the CLOB on every list request.

This deliberately avoids the old multi-table 3NF logging schema used in Kettle: one table, dialect-agnostic DDL generated from Hop’s database layer, and no per-dialect schema maintenance for nested execution details.

Options:

Relational database connection: the Hop relational connection to use
Schema name (optional): optional schema for the table
Table name: defaults to hop_executions
Persistence delay: maximum time to wait before persisting dirty cache entries (ms)
Maximum cache age: maximum time to keep an entry in memory without read/write (ms)
Show table and indexes DDL: generates dialect-specific CREATE TABLE and CREATE INDEX statements for the selected connection (does not execute them). Run the DDL on your database before writing execution information.

Table layout (logical):

Primary key id (log channel / execution id)
Filter columns: name, execution_type, parent_id, registration_date, execution_start_date, execution_end_date, failed, status_description, duration_ms, project_id
json: CacheEntry JSON written on insert (execution, children, samples, logging text, …)
state_json: later execution state, child states, and samples for a local run

project_id is optional and nullable. It is filled from the active project’s id (HOP_PROJECT_ID, stored as projectId in project-config.json). When that variable is empty, Hop does not filter this table, which is the same behavior as 2.19. A table created before the column existed keeps working: writes use the old column list and no project filter is applied until you add the column. The Show table and indexes DDL button prints CREATE TABLE for a new table (project_id and state_json are included) and, below a comment, ALTER TABLE plus an index that adds project_id to an older table. Run one of those blocks, not both. Hop does not execute the DDL.

state_json lets a long run update execution state without rewriting json. A table that does not have that column yet keeps working: every save rewrites json, and Hop logs an ALTER TABLE statement at startup. Hop does not change the table by itself.

When a project id is set and the column exists, the list shows rows for that id and rows whose project_id is null or empty. Those untagged rows are executions from 2.19 or from a project that still has no id. Rows tagged with a different id are hidden, including from "last execution" lookups.

Indexes are recommended on the filter columns used by the GUI (start date, name, type, failed, parent id, status, project id). Use the Show table and indexes DDL button to obtain statements for your database dialect.

Project id on Elastic and OpenSearch

Elastic and OpenSearch documents can carry the same value as a root projectId keyword. The create-index action adds that field. If the index already exists, the same action adds the field to the mapping and does not recreate the index. Listing uses that keyword when the active project has an id, and lists every document when it does not. Documents written without a project id stay visible to every project.

The field is omitted from the JSON document when the project has no id. A 2.19 runtime reads those documents. A document that includes projectId is not readable by 2.19, because that version rejects unknown JSON properties in execution documents.

Cleanup

If you want to automate removal of execution information from your graph database you can use a pipeline or workflow to do so.