Running Apache Hop in the cloud

A Hop project is a folder: project-config.json, a metadata/ folder, pipelines, workflows and whatever else the project owns. See Projects and environments.

Something has to put that folder inside the runtime. The three patterns on Deploying Hop Server are not Hop Server specific — they are the three ways a project reaches any runtime, server or not:

Baking the project into an image is the optimal choice on a container platform, because it is the only one of the three where the running unit is fully described by an image tag. Fetching at start-up is attractive when many small projects share one image, at the cost of a network dependency on every start.

Data files are a separate question from project files. Read and write them through Apache Hop VFS so a path is a configuration value rather than something baked into a transform — see Rules that hold on every cloud.

Environment configuration is per runtime: hostnames, credentials, inbound and outbound paths, and a purpose (Development, Test, Production). They live in a lifecycle environment, which is one or more JSON files that must not be in the project repository when they contain secrets.

On a cloud runtime, those environment files usually arrive as a mounted configuration object or a file written by the deployment pipeline. An environment can list several files; in the container image that is the comma-separated HOP_ENVIRONMENT_CONFIG_FILE_NAME_PATHS variable.

For secrets, prefer not to place the value in the file at all. A variable resolver lets the committed file hold an expression that is resolved at runtime against your secret store:

The expression format is #{name:key:element}, where name is a resolver you configured. Every major cloud secret store, and Vault and OpenBAO, have a resolver — see Where your cloud provider plugs in. This is what keeps the same project and the same image running unchanged in Development, Test and Production: only the resolved values differ.

The default password protection in Hop metadata is obfuscation, not encryption. For anything running outside your laptop, use the AES2 encoder or a secrets resolver (also check Passwords).

A short-lived container. The apache/hop image runs hop-run and exits when HOP_FILE_PATH and HOP_RUN_CONFIG are set. On a container platform this is a job or task that starts, runs and disappears, and it is the default production pattern that can be used with any scheduler.

A long-lived Hop Server. The same image starts Hop Server when those two variables are omitted. Choose this when you need web services or the REST API, or when many clients submit work to a shared always-on engine. It is a normal long-running HTTP service: give it a readiness and liveness probe on hop/status, put it behind whatever ingress you already use, and enable TLS and non-default credentials before exposing it.

A distributed engine. For volumes that do not fit one container, the pipeline can be executed on a cluster and only the run configuration changes: the project, the metadata and the pipeline stay the same. Native Spark compiles the pipeline into a Spark job and submits it to your own cluster or to Databricks, where a Databricks Connection carries the workspace and its token. Apache Beam runs the same pipeline on Google Cloud Dataflow, Apache Spark, Apache Flink or another Beam runner. A workflow can also hand work to the platform rather than run it: Databricks job run triggers a job that already exists in the workspace, and Databricks job wait polls a run started fire-and-forget, or started by something else entirely, until it reaches a terminal state.

Hop Web is a fourth, different thing: a browser-based Hop Gui for development, not an execution runtime. Running it on a cloud platform requires you to manage its persistence and upgrades — see Hop Web in Docker.

Hop Server is not a scheduler and Hop has no calendar of its own. Something outside Hop decides when a pipeline or workflow runs: a scheduler, a cron service, an orchestration service, a CI pipeline, or a message that arrives.

That is a feature on a cloud platform, not a gap. Whatever already schedules the rest of your workload can start a Hop container or call a Hop Server, and you keep one place where dependencies, retries, alerting and calendars live.

Two things are worth deciding explicitly:

The Apache Airflow how-to is a worked example of the pattern; the same principle applies to any scheduler.

A container is ephemeral. When a short-lived run finishes, the container and everything written inside it is gone — including the execution history you would want to look at when someone asks why last night’s load was empty.

Configure an execution information location that outlives the container. The available types cover the usual cloud building blocks: a file or caching-file location on a mounted volume or object storage, a relational database, Neo4j, Elastic, OpenSearch, or a remote Hop Server. Runs recorded there are readable afterwards from the Execution Information perspective in Hop Gui, whichever runtime produced them.

For log lines rather than execution metadata, containers write to stdout and your platform’s log collector picks them up; set the verbosity with HOP_LOG_LEVEL. When you need logging as data — durable, queryable, joined to your own tables — use the pipeline log and workflow log metadata types to write it to a database of your choice.

These are provider-independent and worth checking before the first production run.

Nothing written inside the container survives it. Temporary files, ${java.io.tmpdir}, sort and group-by spill files, and anything a transform writes to a relative path are all gone when the container exits. Anything that must outlive the run goes to a mounted volume, object storage or a database.

Object storage is not a filesystem. There are no real folders, no atomic rename, no append, and listing is a paid API call that returns objects rather than a directory tree. Assert existence with a check on the object itself rather than on its "folder", write once rather than appending, and expect eventual consistency in listings. Supported operations lists what each file system can actually do.

Environment configuration must reach the runtime, and must not be in the project repository. The project’s default image is the same in every environment; the environment files are not part of it.

JDBC drivers and extra plugins never travel with the project. They are baked into the image, downloaded at start-up with HOP_DRIVERS_DOWNLOAD (pointing HOP_DRIVERS_MAVEN_REPO at an internal repository for restricted networks), or mounted. Notably they are not included when a client sends export resources.

Give the JVM the memory the container was granted. The image defaults HOP_OPTIONS to -XX:+AggressiveHeap so the JVM uses the memory assigned to the container. If you override HOP_OPTIONS, set the heap deliberately — a container memory limit is not a JVM heap setting, and the platform will kill a container that exceeds its limit rather than let the JVM handle it.

Identity beats long-lived keys. Where your platform can grant the running container an identity that carries permissions to storage, databases and secrets, use it and keep static credentials out of the environment files entirely.

Egress and network policy are part of the deployment. A pipeline that reaches a database, an API or object storage needs that path open from wherever the container runs, which is rarely the same place development happened.

Scale out, not up. A pipeline on the local engine is a single JVM. More throughput means more containers each running a smaller unit of work, or a distributed engine — and splitting the workload is a design decision in the workflow, not a runtime setting.

Pin the image tag. latest may make a run irreproducible and can turn an unrelated pull into an unplanned upgrade. Pin a release tag and upgrade on purpose.

Almost every cloud deployment is one of these, or a mix.

Everything above is the same on every cloud. The provider-specific part is narrow, and it is already documented per provider:

If your provider is not in these lists, the question to ask is which standard it speaks — S3-compatible storage, a JDBC-compatible database, an OCI container runtime — rather than whether Hop supports it by name.