Hop Marketplace

Some Hop plugins are optional: they are not bundled in the default client zip so the download sizes stay within reason. The marketplace installs those plugins into the running Hop client’s plugins/ tree from a Maven repository.

Restore the full plugin set

The default hop-client zip is intentionally smaller: we made large plugins optional since version 2.19.0. To install all marketplace-optional plugins that used to ship in the download (before version 2.19), use the generated environment file at the installation root:

cd /path/to/hop
./hop marketplace apply -f full-client-env.yaml

# Please restart Hop server or GUI instances that might still be open

Notes:

  • Requires network access to Apache Repository / Maven Central (or a mirror configured in hop-config).

  • For SNAPSHOT builds when developing you can point the marketplace at a local Nexus that has the plugin zip files published. See folder docker/marketplace-nexus in the source code to run your own Nexus locally.

  • The download is large (hundreds of MB); that is expected.

  • Do NOT pass --prune unless you also want other marketplace installs removed.

  • full-client-env.yaml is generated at build time from marketplace plugin source file optional-plugins.yaml (source of truth).

Beam SDKs/runners live under plugins/engines/beam/lib-beam (installed with the Beam plugin), not under lib/beam. Launch scripts add that folder to the classpath only when it exists. Developers who want Beam baked into a local hop-client build can use Maven profile -Pbeam (for example ./mvnw -pl assemblies/client -am package -Pbeam).

Defaults

Without extra configuration, Hop uses:

  1. asf (primary) — https://repository.apache.org/content/groups/public/

  2. central (fallback) — https://repo1.maven.org/maven2/

Install tries the primary repository first, then other enabled repositories, until the plugin zip is found.

Settings are stored under the marketplace key in

${HOP_CONFIG_FOLDER}/hop-config.json
config/hop-config.json in your client distribution is only provided to get started but will be not survive a Hop version upgrade. Configure your own configuration.

CLI overview

From the Hop install directory:

./hop marketplace install hop-tech-parquet
./hop marketplace install hop-engines-spark:2.19.0
./hop marketplace install hop-engines-spark:2.19.0 --repo asf
./hop marketplace list
./hop marketplace query parquet
./hop marketplace query "spark engine"
./hop marketplace uninstall hop-tech-parquet

# Repositories
./hop marketplace repo list
./hop marketplace repo add --id corporate --url https://nexus.example.com/repository/hop/ --primary
./hop marketplace repo set-primary asf
./hop marketplace repo reset-defaults

# Declarative environment (CI / Docker)
./hop marketplace apply -f hop-env.yaml
./hop marketplace apply -f hop-env.yaml --prune
./hop marketplace validate -f hop-env.yaml
./hop marketplace validate -f hop-env.yaml --strict

After install, apply, or uninstall, restart Hop so the plugin registry reloads.

Coordinates for install: artifactId, artifactId:version, or groupId:artifactId:version. Optional --repo <id> forces a single repository (skips the fallback chain).

query searches the local optional-plugin catalog (not Maven Central). The filter is a case-insensitive substring matched against artifact id, name, category, description, and install path. Omit the filter (or pass an empty string) to list every optional plugin. Matches may show [installed] when the plugin folder is present under the Hop install.

GUI

The marketplace GUI is the primary way to install optional plugins, manage Maven repositories, and edit declarative environment files without using the CLI.

Opening the marketplace

You can open the marketplace dialog from:

  • Tools → Marketplace…

  • The Hop Gui toolbar marketplace icon

Hop Gui toolbar marketplace icon

On first use you may also see the welcome page that introduces the marketplace:

Marketplace welcome dialog

The main Plugin Marketplace dialog has three tabs:

  • Plugins — browse, search, install, and uninstall optional plugins

  • Environment — edit hop-env.yaml / JSON environments and validate or apply them

  • Repositories — configure Maven repositories (same settings as the CLI / hop-config)

After install, apply, or uninstall, restart Hop so the plugin registry reloads.

Plugins tab

Marketplace dialog Plugins tab

Use this tab to work with the local optional-plugin catalog:

  • Search — type three or more characters to filter by name, artifact id, category, description, or install path. Clear the filter with the clear button.

  • Table columns: Name, Artifact, Category, Status, Description.

  • Status values:

    • Installed — present on disk with a marketplace install receipt (version shown when known)

    • Present on disk — plugin folder exists but no marketplace receipt

    • Not installed — not found under plugins/

  • Install / Uninstall / Refresh — act on the selected plugin (uninstall requires a marketplace receipt)

Install uses the primary repository first, then other enabled repositories (see Repositories tab).

Environment tab

Marketplace dialog Environment tab

This tab embeds the full environment editor (no separate “Edit…” window). It is the GUI equivalent of hop marketplace validate / apply and of editing hop-env.yaml by hand.

File toolbar

  • New — empty environment specification

  • Open… — open a hop-env.yaml / .yml / .json file

  • Save / Save As… — write YAML or JSON (by extension)

  • Filename combo — current path plus recently used environment files (remembered via the Hop audit list for this namespace). Selecting an entry loads that file.

  • On open, Hop prefers the last file you edited in this dialog; otherwise it may pre-select full-client-env.yaml at the install root when that file exists.

Nested editor tabs

Tab Contents

General

Spec version, hopVersion, and enforceOnRun (when true, hop-run can fail if the install drifts from this file; also controlled by -Dhop.env.enforce=true).

Repositories

Ordered list of Maven repositories for this environment file (first row is primary). Optional username/password. Import from hop-config… copies enabled repositories from hop-config (one-shot; env-file repos stay independent of the main Repositories tab).

Plugins

Plugin GAVs to install. From catalog… multi-selects from the optional-plugin catalog. Leave Group ID blank to use org.apache.hop. Version can default from Hop version on General when applying.

Dependencies

Optional jars (for example JDBC) with target directory under Hop home (default lib/jdbc).

Validate and Apply

At the bottom of the Environment tab:

  • Validate — compare the local install to the saved environment file without downloading.

  • Apply — install missing plugins and dependencies declared in the file.

  • Prune extras on apply — same as CLI apply --prune (confirm before running).

  • Strict validate — also fail when extra marketplace-installed plugins (receipts under plugins/.marketplace/) are not listed in the file.

If validation finds drift, Hop shows a report. Use Apply to fix, then restart Hop.

Marketplace environment drift report

Repositories tab

Marketplace dialog Repositories tab

Configure the global marketplace repository list (stored under marketplace in ${HOP_CONFIG_FOLDER}/hop-config.json — the same config the CLI uses):

  • Table columns: Primary, Enabled, Id, Name, URL, Auth

  • Add / Edit / Remove — maintain repository entries

  • Set primary — install tries this repository first

  • Up / Down — order fallbacks after the primary

  • Reset defaults — Apache Repository (primary) + Maven Central

  • Save — write hop-config (required to keep changes)

Add or edit a repository with id, name, URL, optional credentials, and primary/enabled flags:

Add or edit marketplace repository

Projects lifecycle environments

When the Projects plugin is installed, each lifecycle environment can store marketplace-specific attributes (namespaced under the marketplace group). On the environment properties dialog, open the Marketplace plugins tab:

Lifecycle environment Marketplace plugins configuration

Typical options:

  • Environment file — path to a hop-env.yaml (variables allowed, for example under the project home)

  • On enable — off (no check), warn (log or dialog on drift), or enforce (fail enabling the environment if the install does not match)

  • Strict — treat extra marketplace plugins as drift

  • Auto-apply missing plugins on enable — optional; off by default (use only on controlled machines)

If On enable is unset, purpose-based defaults apply: Production → enforce, Testing / Acceptance → warn, Development / CI-style purposes → off.

This complements hop-env enforceOnRun (checked at hop-run) and the Environment tab Validate / Apply actions (manual checks from the marketplace dialog).

Declarative environments (hop-env.yaml)

For CI/CD, Docker, and reproducible servers you can declare the required plugins (and optional jars) in a file, commit it to Git, and apply it without using the GUI.

Supported file names / formats:

  • hop-env.yaml / hop-env.yml (YAML)

  • hop-env.json (JSON — same field names)

hop marketplace apply

Installs missing plugins and dependencies declared in the environment file. Already-satisfied plugins are skipped.

Option Description

-f, --file (required)

Path to hop-env.yaml or hop-env.json.

--prune

After installing what the file requires, uninstall marketplace-managed plugins that have an install receipt but are not listed under plugins. Core/bundled plugins and plugins present on disk without a marketplace receipt are not removed by prune.

Examples:

# Install everything listed (idempotent)
./hop marketplace apply -f hop-env.yaml

# Also remove marketplace plugins that are no longer in the file
./hop marketplace apply -f ./deploy/hop-env.yaml --prune

Behaviour notes:

  • Repositories listed in the file override hop-config for this apply (first entry is primary; later entries are fallbacks). If the file omits repositories, hop-config repositories are used.

  • Plugin version defaults to hopVersion in the file, then the marketplace/hop version.

  • Plugins already on disk without a marketplace receipt (for example installed offline with tools/install-wave1-plugins.sh) count as satisfied and are not re-downloaded.

  • Dependencies are downloaded as jars into target (default lib/jdbc).

Always restart Hop after a successful apply when new plugins were installed.

hop marketplace validate

Compares the local install to the environment file without downloading or installing anything. Useful in CI as a pre-flight check.

Option Description

-f, --file

Path to the environment file. If omitted, Hop discovers a file (see How the environment file is discovered).

--strict

Also fail when extra marketplace-installed plugins (receipts under plugins/.marketplace/) are present that are not listed in the file. Without --strict, extras are ignored.

Exit codes:

  • 0 — environment matches (prints OK: environment matches …)

  • non-zero — drift detected or error (prints a drift report and suggests hop marketplace apply -f …)

What counts as drift (always hard-fail):

  • Missing plugin — not on disk and no install receipt

  • Version mismatch — receipt version differs from the version required in the file

  • Missing dependency — jar not found under the dependency target directory

Examples:

./hop marketplace validate -f hop-env.yaml
./hop marketplace validate -f hop-env.yaml --strict

# Discover hop-env under PROJECT_HOME or the Hop install
export PROJECT_HOME=/path/to/my-project
./hop marketplace validate

hop-run enforcement

When a hop-env file is found and enforcement is on, hop-run validates before execution and aborts on missing plugins/deps or version mismatches (same checks as validate without --strict).

Enable enforcement either:

  • in the file: enforceOnRun: true, or

  • with a system property: -Dhop.env.enforce=true (or Y)

If a hop-env file is present but enforceOnRun is false and the property is unset, hop-run only logs that the file was found and skips the check.

How the environment file is discovered

Used by validate (when -f is omitted) and by hop-run:

  1. Explicit path: -f / --file (validate only), or variable / property HOP_ENV_FILE / hop.env.file

  2. Under PROJECT_HOME (system property or env): hop-env.yaml, then .yml, then .json

  3. Under the Hop install root: same file names

hop-env.yaml format

Full example:

# Schema version of this file format (optional; default "1.0")
version: "1.0"

# Default plugin version when a plugin entry omits "version"
hopVersion: "2.19.0"

# If true, hop-run fails when the install drifts from this file
enforceOnRun: false

# Maven repositories for apply (optional).
# First entry is primary; later entries are fallbacks for that apply only.
# If omitted, hop-config marketplace.repositories are used.
repositories:
  - id: asf
    url: "https://repository.apache.org/content/groups/public/"
  - id: central
    url: "https://repo1.maven.org/maven2/"
  # Private mirror example:
  # - id: corporate
  #   url: "https://nexus.example.com/repository/hop-plugins/"
  #   username: reader
  #   password: "…"   # prefer env HOP_MARKETPLACE_PASSWORD for secrets

# Required optional plugins (installed into plugins/)
plugins:
  - artifactId: hop-tech-parquet
    version: "2.19.0"          # optional if hopVersion is set
  - groupId: org.apache.hop    # optional; default org.apache.hop
    artifactId: hop-engines-spark
    version: "2.19.0"
  # Beam engine plugin only (Beam SDKs ship under plugins/engines/beam/lib-beam with the plugin):
  # - artifactId: hop-engines-beam
  #   version: "2.19.0"

# Optional non-plugin jars (e.g. JDBC drivers)
dependencies:
  - groupId: org.postgresql
    artifactId: postgresql
    version: "42.7.3"
    target: lib/jdbc           # optional; default lib/jdbc

Top-level fields

Field Required Description

version

No

Format version of the hop-env file itself (default 1.0).

hopVersion

Recommended

Default version for plugin entries that omit version. Also used when aligning plugins with a Hop release.

enforceOnRun

No

When true, hop-run validates this file before running and fails on missing plugins/deps or version mismatches.

repositories

No

List of Maven base URLs used when applying this file. Order: first = primary, rest = fallbacks. Each item: id, url, optional username / password.

plugins

No

List of optional Hop plugins to install. Each item: artifactId (required), optional groupId (default org.apache.hop), optional version (default hopVersion / Hop version).

dependencies

No

Extra jars (not Hop plugin zips). Each item: groupId, artifactId, version (all required), optional target directory under the Hop install (default lib/jdbc). Downloaded as artifactId-version.jar.

Minimal examples

Only plugins, ASF defaults from hop-config:

version: "1.0"
hopVersion: "2.19.0"
plugins:
  - artifactId: hop-tech-parquet
  - artifactId: hop-tech-cassandra

CI gate (no install):

./hop marketplace validate -f hop-env.yaml --strict || exit 1

Docker / pipeline install then run:

./hop marketplace apply -f /files/hop-env.yaml
# restart container or new process so plugins load
./hop-run.sh -f /files/main.hwf

A longer sample ships with the marketplace plugin sources (plugins/misc/marketplace/src/main/samples/hop-env.example.yaml) and may appear under samples after the plugin is installed.

Enterprise / air-gapped

Point marketplace repositories at your internal Nexus or Artifactory mirror of org.apache.hop plugin zips. Prefer anonymous read where possible; for private repos set Basic auth on the repository entry (in hop-config or in hop-env repositories) or use HOP_MARKETPLACE_USERNAME / HOP_MARKETPLACE_PASSWORD. Wrong credentials can cause HTTP 401 even when anonymous access would work.

For plugin authors

If you develop a third-party plugin (outside the Hop repository) and want to publish installable zips to your own Nexus, see the developer guide Publishing a Hop plugin to your own Maven repository in the Hop Development Manual (marketplace-publish-plugin.adoc). It covers packaging, Hop version requirements, other plugin dependencies, and a sample release script.