Deploying Hop Server

Hop Server is a long-lived process that executes pipelines and workflows. It is not a scheduler: something else (Hop Gui, hop-run, Apache Airflow, cron, Jenkins, a Kubernetes CronJob, …) has to submit the work. See Scheduling workflows and pipelines.

This page is the deployment story: after you develop a project, how does a running Hop Server get the files and the configuration it needs?

The same four patterns are how a project reaches any runtime, not only a server. For the wider picture — containers, schedulers, secrets, object storage and run history on a cloud platform — see Running Apache Hop in the cloud.

There are four patterns that cover almost every production setup:

Project in a Docker image — CI builds an image that contains the project. You can roll that image as a long-lived Hop Server, but you often do not need to: a scheduler can start a short-lived copy, run one pipeline or workflow, and exit.
Remote run with export resources — the client owns the project and sends a ZIP to a "dumb" server.
Git checkout on the server — the host has a clone; updating is git pull.
Standard apache/hop container — the published image, unchanged. JDBC drivers and marketplace plugins are installed when the container starts, and the pipelines and workflows are read from git through a named VFS connection.

What you deploy

Two things make a Hop project runnable.

The project lives in git: project-config.json, the metadata/ folder, pipelines, workflows, and any other files the project itself owns. See Projects and environments.

The lifecycle environment is per runtime: one or more JSON files with hostnames, credentials, inbound paths, and a purpose (Development, Test, Production). Those values change between laptops, test and production, and they usually must not live in the project repository.

Hop Server then needs the project in one of two ways:

On disk, registered and enabled. The official Docker image does this from HOP_PROJECT_* / HOP_ENVIRONMENT_* variables. On a host you register the project and environment once with hop-conf, then start the server with -e. The environment already references its project, so you do not pass -j as well. This is how the image and the git checkout work. The standard container is the same shape with a smaller folder: the enabled project holds the Git connection and the metadata the run needs, and HOP_FILE_PATH points at a file inside the repository. Environment configuration files must be present on the server.
As a payload from the client. A Remote pipeline or Remote workflow run configuration with Export linked resources to server sends a ZIP (the files Hop can pack, plus serialized metadata, plus the execution configuration). This is export resources. Environment configuration is applied on the client; variables travel with the execution.

<metadata_folder> in hop-server.xml is the older, lower-level hook. Prefer enabling a project so ${PROJECT_HOME}, metadata inheritance and environment variables all work. When both a project and <metadata_folder> are set, both providers sit on the metadata stack.

Choose a pattern

Project in a Docker image Export resources from a client Git checkout on the server Standard apache/hop container

	Project in a Docker image	Export resources from a client	Git checkout on the server	Standard `apache/hop` container
Who owns the project files	The image (immutable per build)	The client (Airflow, hop-run, Hop Gui)	The server host (working tree)	The git remote. The container holds a small project (the connection and the metadata) and checks the repository out when a file is opened
How you update	Rebuild and roll the container	Change the client project; the next run sends a new ZIP	`git pull` (restart only for environment or server config)	Change the revision on the connection. A moving branch is fetched again when the connection says so
Environment configuration	Mounted or injected on the server	Applied on the client	Files on the server, usually outside the clone	Mounted or injected on the server
Extra files (JDBC, plugins, `.properties`, JSON schemas)	`COPY` into the image or a volume	Not sent. Must already exist on the server, or be rewritten with the named-resource folder mapping	Present in the clone or installed on the host	Downloaded at start, or mounted. See Standard apache/hop container
File paths	`${PROJECT_HOME}` is real on the server	Paths are rewritten; see the export resources caveats	`${PROJECT_HOME}` is the clone directory	The file you run is a Git VFS URL such as `ops:///workflows/daily.hwf`. `${PROJECT_HOME}` is the small local project, not the repository
Best for	Enterprise CI/CD, Kubernetes, OpenShift; the same image as a short-lived job or a long-lived server	Orchestrators that already have the project; a server that should not own files	Smaller or ops-friendly hosts with git access	A platform where building and rolling a custom image is the expensive step
Poor fit	You cannot rebuild images. Use Standard apache/hop container instead	Heavy file I/O, extra config files, large data files	You cannot (or must not) run `git pull` on production hosts	Projects that open their files through `${PROJECT_HOME}`, or metadata that has to be read from inside the repository

Who owns the project files

The image (immutable per build)

The client (Airflow, hop-run, Hop Gui)

The server host (working tree)

The git remote. The container holds a small project (the connection and the metadata) and checks the repository out when a file is opened

How you update

Rebuild and roll the container

Change the client project; the next run sends a new ZIP

git pull (restart only for environment or server config)

Change the revision on the connection. A moving branch is fetched again when the connection says so

Environment configuration

Mounted or injected on the server

Applied on the client

Files on the server, usually outside the clone

Mounted or injected on the server

Extra files (JDBC, plugins, .properties, JSON schemas)

COPY into the image or a volume

Not sent. Must already exist on the server, or be rewritten with the named-resource folder mapping

Present in the clone or installed on the host

Downloaded at start, or mounted. See Standard apache/hop container

File paths

${PROJECT_HOME} is real on the server

Paths are rewritten; see the export resources caveats

${PROJECT_HOME} is the clone directory

The file you run is a Git VFS URL such as ops:///workflows/daily.hwf. ${PROJECT_HOME} is the small local project, not the repository

Best for

Enterprise CI/CD, Kubernetes, OpenShift; the same image as a short-lived job or a long-lived server

Orchestrators that already have the project; a server that should not own files

Smaller or ops-friendly hosts with git access

A platform where building and rolling a custom image is the expensive step

Poor fit

You cannot rebuild images. Use Standard apache/hop container instead

Heavy file I/O, extra config files, large data files

You cannot (or must not) run git pull on production hosts

Projects that open their files through ${PROJECT_HOME}, or metadata that has to be read from inside the repository

Standard apache/hop container

The published apache/hop image can take the place of a derived image. A Jenkins, GitHub Actions or GitLab CI job usually builds that derived image to add JDBC drivers, install a few marketplace plugins, and clone the project. Here the entrypoint installs the drivers and the plugins before Hop starts, and a Git repository connection checks the project out when Hop opens a file.

The checkout is read-only by default. A write changes the working copy on disk, and nothing is committed or pushed.

This needs Hop 2.20, which is where HOP_PLUGINS_DOWNLOAD and the Git VFS plugin arrive together. HOP_DRIVERS_DOWNLOAD is already in 2.19.

Drivers and plugins

Set HOP_DRIVERS_DOWNLOAD to a comma-separated list of driver ids, and HOP_DRIVERS_ACCEPT_LICENSE=true when one of them is a restricted driver (Oracle, MySQL, MariaDB, DB2, …). HOP_DRIVERS_MAVEN_REPO points the download at an internal Nexus or Artifactory. See Downloading JDBC drivers. A jar that hop driver list does not know is still a file you mount.

Set HOP_PLUGINS_DOWNLOAD to the marketplace plugins the run needs. The Git driver is one of them: hop-tech-git-vfs. The stock image does not contain it. A private plugin and that Apache plugin can be listed together; HOP_PLUGINS_MAVEN_REPO is tried first and the other configured repositories are still used. See Downloading Marketplace plugins.

A later start skips a plugin whose receipt version is already on disk. Drivers are downloaded on every start. lib/jdbc and the plugins directory have to be writable. On a read-only root filesystem, point HOP_SHARED_JDBC_FOLDERS and HOP_PLUGIN_BASE_FOLDERS at a volume.

The project you mount

Set HOP_PROJECT_FOLDER to a real directory. Hop resolves that home before it loads named VFS connections, and the Git scheme exists only after the connection has been read from a folder Hop can already open.

Mount a small project that holds the connection and the metadata the run needs. The pipelines and workflows stay in the remote repository. The mount in the example is read-only, so project-config.json has to be in it already. The entrypoint keeps that file (--project-keep-config-file). A missing file would be created with Enforce executions in project home turned on, and hop-run would then refuse ops:///workflows/daily.hwf because that path is outside /bootstrap.

json

{
  "metadataBaseFolder" : "${PROJECT_HOME}/metadata",
  "parentProjectName" : "default",
  "enforcingExecutionInHome" : false
}

parentProjectName set to default picks up the local run configuration shipped in the image. The metadata/ directory inside the git repository is a different folder. Hop reads metadata from the bootstrap project (and from that default parent), which is where database connections the pipeline uses belong.

bash

docker run --rm \
  -e HOP_DRIVERS_DOWNLOAD="oracle,mariadb:3.4.1" \
  -e HOP_DRIVERS_ACCEPT_LICENSE=true \
  -e HOP_PLUGINS_DOWNLOAD="hop-tech-git-vfs" \
  -e HOP_PROJECT_FOLDER=/bootstrap \
  -e HOP_PROJECT_NAME=bootstrap \
  -e HOP_ENVIRONMENT_NAME=prod \
  -e HOP_ENVIRONMENT_CONFIG_FILE_NAME_PATHS=/config/prod.json \
  -e HOP_FILE_PATH='ops:///workflows/daily.hwf' \
  -e HOP_RUN_CONFIG=local \
  -v /etc/hop/bootstrap:/bootstrap:ro \
  -v /etc/hop/environments/prod.json:/config/prod.json:ro \
  -v /etc/hop/secrets/deploy-key:/run/secrets/deploy-key:ro \
  -v /etc/hop/secrets/known_hosts:/run/secrets/known_hosts:ro \
  apache/hop:<version>

/bootstrap also needs metadata/git-vfs-connection/ops.json. HOP_RUN_CONFIG=local resolves through the default parent, so you only add a run configuration file when the run should use a different one.

Omit HOP_FILE_PATH and HOP_RUN_CONFIG and the same container starts Hop Server instead of hop-run. The server still needs the bootstrap project, the connection and the key.

The connection for an SSH deploy key, GitHub or GitLab, looks like this. authType is the name of the enum value. name is the VFS scheme.

json

{
  "name" : "ops",
  "repositoryUrl" : "git@git.example.com:data/your-project.git",
  "revision" : "a1b2c3d4e5f6789012345678901234567890abcd",
  "authType" : "DEPLOY_KEY",
  "privateKeyFile" : "/run/secrets/deploy-key",
  "sshUser" : "git",
  "knownHostsFile" : "/run/secrets/known_hosts",
  "readOnly" : true
}

A GitLab deploy token is the same file with authType set to USERNAME_PASSWORD, an https:// repository URL, the token’s user name, and the token as the password. The token needs the read_repository scope. Leave the token out of the URL.

Pin revision to a commit id. Leave the checkout folder empty on a short-lived container: the working copy goes under the temporary directory and disappears with the container. A long-lived server should set a checkout folder and a maximum checkout age, so a later commit on a branch is picked up without a restart. See Git repository.

Mount a known_hosts file, as in the example. acceptUnknownHosts set to true is the alternative when the host key cannot be supplied. A fresh container has no ~/.ssh/known_hosts, and the connection refuses the server until one of those two is set.

What the paths mean

ops:///workflows/daily.hwf is that file in the repository the connection named ops checked out.

${PROJECT_HOME} is /bootstrap, the folder you mounted. A pipeline stored as ${PROJECT_HOME}/pipelines/load.hpl resolves against that folder.

A path relative to the file that is running stays on the git scheme. For a workflow opened as ops:///workflows/daily.hwf, ${Internal.Entry.Current.Folder} is the workflows directory inside the checkout, so a workflow that refers to its neighbours that way keeps reading the repository.

A project that reaches its files through ${PROJECT_HOME} still wants an image or a checkout.

The container needs the git host on every start. The connection names the revision, and the image tag names the Hop version.

You often do not need a Hop Server

If the project is already in an image, a scheduler can start that image as a short-lived container (locally, or on Docker, Kubernetes, OpenShift, …), run one pipeline or workflow, and exit. There is no long-lived Hop Server to keep alive, upgrade or authenticate.

The official apache/hop image does both jobs. Set HOP_FILE_PATH and HOP_RUN_CONFIG and the entrypoint runs hop-run and exits. Omit them and it starts Hop Server. On the stock image, HOP_FILE_PATH can be a Git VFS URL.

Because the container disappears when the run finishes, send execution information somewhere that outlives it: a mounted volume (file or caching-file location), a relational database, OpenSearch, Neo4j or Elastic. You can inspect those runs later from the Execution Information perspective in Hop Gui.

This is the usual production pattern when a scheduler (Airflow, Jenkins, cron, a Kubernetes Job or CronJob) already owns the calendar. Keep a long-lived Hop Server when you need web services, or when many clients submit work to a shared always-on engine.

See Skip Hop Server: short-lived containers, Scheduling workflows and pipelines, and the Airflow how-to.

Environment configuration

Use project variables for values that are stable and safe to commit (folder names relative to ${PROJECT_HOME}). Use environment variables for infrastructure: ${DB_HOSTNAME}, credentials, inbound paths.

Keep environment JSON outside the project folder when it contains secrets. Typical places:

a second, private repository
files dropped onto the host by the deployment pipeline
a Kubernetes Secret or ConfigMap mounted as a file
a variable resolver (HashiCorp Vault, Azure Key Vault, Google Secret Manager) so the committed file only holds resolver expressions

An environment can list several configuration files. In Docker that is the comma-separated HOP_ENVIRONMENT_CONFIG_FILE_NAME_PATHS variable.

A production file looks like this:

json

{
  "variables" : [ {
    "name" : "DB_HOSTNAME",
    "value" : "db.prod.example.com",
    "description" : "Warehouse host"
  }, {
    "name" : "DB_PASSWORD",
    "value" : "#{vault:hop/data/some-db:password}",
    "description" : "Resolved at runtime"
  } ]
}

Default metadata password protection is obfuscation, not encryption. For production use the AES2 encoder or a secrets resolver, change the default Hop Server credential (cluster / cluster), and enable TLS. See Passwords and the project’s SECURITY.md.

For export resources the same environment files are used on the client (hop-run.sh -e prod …). The server does not read them.

What Hop Server does not do

It does not schedule work. Use an external scheduler: Airflow, cron, Jenkins, a Kubernetes CronJob, or hop-run from a wrapper.
It does not keep a durable work queue. A restart drops in-flight executions unless you set --shutdown-timeout / HOP_SERVER_SHUTDOWN_TIMEOUT and the orchestrator waits.
A short-lived apache/hop container that runs hop-run and exits is not Hop Server. For a project baked into an image that is often the better production pattern — see You often do not need a Hop Server.

How you start, stop and query a server is still on the Hop Server reference page.