Scheduling workflows and pipelines

Hop has no calendar of its own. Hop Server does not queue work for later, and the Start action’s repeat option is not a scheduler.

Something outside Hop decides when a pipeline or workflow runs: a crontab, a systemd timer, Windows Task Scheduler, Jenkins, a Kubernetes CronJob, Apache Airflow, or any other orchestrator you already use. That outside process starts hop-run (or a short-lived container that runs hop-run) and treats the exit code as the result.

That is a feature, not a gap. Calendars, retries, overlapping-run policy, alerting and dependencies then live in one place — the scheduler you already operate — instead of inside a long-lived Java process.

The Apache Airflow how-to is a full worked example of one of these options. This page is the catalogue: the command the scheduler should run, then a short recipe for each common scheduler.

Choose how the run starts

How the scheduler starts the work When to use it Hop side

hop-run on a host

The host already has a Hop client, a registered project and an environment

The scheduler calls a small wrapper script. Exit code 0 is success.

A short-lived apache/hop (or project-baked) container

The project is already an image, or you do not want Hop installed on the scheduler host

The scheduler runs docker run --rm …​ or a Kubernetes Job. The container exits when hop-run exits.

Submit to a long-lived Hop Server

Many clients share one always-on engine, or you need web services

The scheduler still calls hop-run, but with a remote run configuration. See Deploying Hop Server.

For batch work, prefer the first or the second. A long-lived Hop Server is extra machinery if a scheduler already owns the calendar — see Skip Hop Server: short-lived containers.

Scheduler Repeats Typical fit

cron

Yes

A Linux or macOS host that already has a Hop client

at

Once

A single future run on that same host (at 22:00, at now + 2 hours)

systemd timer

Yes

A Linux host that already uses systemd (the usual replacement for an ad-hoc crontab)

Windows Task Scheduler

Yes

A Windows host that already has a Hop client

Jenkins

Yes

Jenkins already runs your CI; a scheduled job can run hop-run or docker run

Kubernetes CronJob

Yes

The project is already a container image on a cluster

Apache Airflow

Yes

You need DAG dependencies, sensors, retries or a shared data platform calendar

GitHub Actions on.schedule, GitLab scheduled pipelines and similar CI timers are the same idea as Jenkins: they start hop-run or a container on a calendar. They are CI systems first; use them when that is already where the work is triggered.

Prerequisites

For hop-run on a host:

  • A Hop client on that host (the same archive you unzip for Hop Gui), and Java 21 on PATH or in HOP_JAVA_HOME / JAVA_HOME. See Installation and configuration.

  • The project and a lifecycle environment registered in HOP_CONFIG_FOLDER, so -e prod is enough. See Projects and environments and the git checkout setup.

  • A run configuration that exists in the project’s metadata (usually local).

For a short-lived container:

In both cases, send execution information somewhere that outlives the process. A container (and often a cron job’s working directory) is gone when the run finishes.

A wrapper around hop-run

Point the scheduler at a small script, not at hop-run.sh directly. The script is where you set HOP_JAVA_HOME, HOP_CONFIG_FOLDER and the log file.

That avoids three quoting traps:

  • cron treats % as a newline, so a hop-run command with date formats or Windows-style +%Y does not belong in the crontab itself.

  • systemd expands ${NAME} in ExecStart= from the unit environment. A literal ${PROJECT_HOME} for hop-run has to be escaped as $${PROJECT_HOME} ($$ becomes $) — or you call a script and never think about it.

  • The shell expands ${PROJECT_HOME} if you double-quote it. hop-run resolves that variable after it enables the project, so the wrapper must pass it in single quotes.

 

  • Linux, macOS

  • Windows

/usr/local/bin/run-hop-nightly.sh:

#!/usr/bin/env bash
set -euo pipefail

export HOP_JAVA_HOME=/usr/lib/jvm/java-21-openjdk
export HOP_CONFIG_FOLDER=/etc/hop/config

HOP=/opt/hop
mkdir -p /var/log/hop
LOG=/var/log/hop/nightly-$(date +%Y%m%d-%H%M%S).log

"${HOP}/hop-run.sh" \
  -e prod \
  -r local \
  -f '{openvar}PROJECT_HOME{closevar}/main.hwf' \
  -l Basic \
  -lf "${LOG}" \
  >/dev/null 2>&1

chmod +x /usr/local/bin/run-hop-nightly.sh and run it once by hand before you schedule it. --logfile (-lf) overwrites the file, which is why the name includes a timestamp. It adds a file listener alongside the console logger, so hop-run still writes to stdout unless you redirect it as above. cron mails on any output, not on a non-zero exit, and without that redirect a successful run would mail the whole log.

On Linux, put flock -n /tmp/hop-nightly.lock in front of the script so a second start fails instead of running the same load twice (flock is not on macOS by default). Otherwise use the scheduler’s own "do not start a second instance" option, shown in each recipe below.

C:\hop-jobs\run-hop-nightly.cmd:

@echo off
setlocal

set "HOP_JAVA_HOME=C:\Program Files\Microsoft\jdk-21"
set "HOP_CONFIG_FOLDER=C:\hop-config"
set "HOP=C:\hop"

cd /d "%HOP%"
call hop-run.bat -e prod -r local -f {openvar}PROJECT_HOME{closevar}/main.hwf -l Basic
exit /b %ERRORLEVEL%

call is required: without it, control never returns from hop-run.bat and any lines after it (cleanup, exit /b) are skipped.

Task Scheduler’s setting If the task is already running: Do not start a new instance is the overlap policy on Windows.

Use -e (the lifecycle environment) rather than -j (the project). The environment already points at the project, and it is what differs between Development, Test and Production.

hop-run exit codes: 0 success, 1 the workflow or pipeline failed, 2 hop-run itself failed, 9 bad arguments. Every scheduler below treats a non-zero exit as a failed run.

Recipes

cron

# 02:00 every day, host local time.
# Linux: flock refuses a second start while the previous run still holds the lock.
# cron mails on any stdout/stderr, not on exit code; the wrapper keeps those quiet.
0 2 * * * /usr/bin/flock -n /tmp/hop-nightly.lock /usr/local/bin/run-hop-nightly.sh

Install with crontab -e as the user that should run Hop (not root, unless that user owns the project files). cron’s environment is minimal: PATH is often just /usr/bin:/bin, and it does not load ~/.bashrc. That is why HOP_JAVA_HOME and HOP_CONFIG_FOLDER live in the wrapper, not in the crontab.

cron uses the host’s local timezone. Put the real command in the script; keep % out of the crontab line.

at

at runs a command once at a future time. It is the one-shot counterpart of cron, and it is what you want for "run this load tonight at 22:00" or "run it two hours from now". The package is often not installed by default (apt install at / dnf install at).

echo /usr/local/bin/run-hop-nightly.sh | at 22:00
echo /usr/local/bin/run-hop-nightly.sh | at now + 2 hours
echo /usr/local/bin/run-hop-nightly.sh | at 02:00 tomorrow

atq lists pending jobs, atrm <job> cancels one. The job runs with the environment at captured when you submitted it, which is closer to your interactive shell than cron is — still prefer the wrapper so Java and HOP_CONFIG_FOLDER are explicit.

systemd timer

A Type=oneshot service plus a timer is the systemd equivalent of a crontab line. systemd will not start a second instance of the same service while the first is still running.

/etc/systemd/system/hop-nightly.service:

[Unit]
Description=Nightly Hop load
After=network.target

[Service]
Type=oneshot
User=hop
Group=hop
ExecStart=/usr/local/bin/run-hop-nightly.sh

/etc/systemd/system/hop-nightly.timer:

[Unit]
Description=Run the nightly Hop load at 02:00

[Timer]
OnCalendar=*-*-* 02:00:00
Persistent=true
RandomizedDelaySec=90

[Install]
WantedBy=timers.target

Persistent=true runs a missed job after a reboot. RandomizedDelaySec spreads the start if many timers fire at midnight.

sudo systemctl daemon-reload
sudo systemctl enable --now hop-nightly.timer
systemctl list-timers hop-nightly.timer
journalctl -u hop-nightly.service

A long-lived Hop Server is a different unit (Type=simple, Restart=on-failure). A sample is in the repository at docs/hop-user-manual/modules/ROOT/assets/files/hop-server/hop-server.service.

Windows Task Scheduler

  1. Open Task Scheduler and choose Create Task (not Create Basic Task — you need the overlap setting).

  2. General: name it, choose Run whether user is logged on or not, and run as the account that can read the project and HOP_CONFIG_FOLDER.

  3. Triggers: New → Daily at 02:00, or One time for the equivalent of at.

  4. Actions: Start a program → C:\hop-jobs\run-hop-nightly.cmd. Set Start in to C:\hop so relative paths inside hop-run.bat resolve.

  5. Settings: If the task is already running, then the following rule applies: Do not start a new instance.

From a command prompt the same daily task is:

schtasks /create /tn "Hop nightly" /sc daily /st 02:00 /ru hop /rp * /tr "C:\hop-jobs\run-hop-nightly.cmd"

schtasks /create /sc once /st 22:00 …​ is the one-shot form. schtasks /run /tn "Hop nightly" runs it immediately; schtasks /query /tn "Hop nightly" /v shows the last result.

Jenkins

Jenkins as a scheduler is a job with a cron trigger that runs hop-run or docker run. That is a different use of Jenkins from building a project image, which is CI.

A Freestyle project is enough: Build Triggers → Build periodically with a cron string (0 2 * * ), then an *Execute shell (or Execute Windows batch command) step that calls the wrapper. Check Do not allow concurrent builds so two ticks cannot overlap.

The same job as a Pipeline:

pipeline {
  agent any
  triggers { cron('0 2 * * *') }
  options { disableConcurrentBuilds() }
  stages {
    stage('Hop') {
      steps {
        sh '/usr/local/bin/run-hop-nightly.sh'
      }
    }
  }
}

If the agent should not have a Hop client, start a short-lived container instead (see Short-lived container below) from that sh step. Pin the image tag; do not use latest on a calendar.

Short-lived container

The official image runs hop-run and exits when HOP_FILE_PATH and HOP_RUN_CONFIG are set. Full variable list: Hop in Docker.

docker run --rm \
  -e HOP_LOG_LEVEL=Basic \
  -e HOP_PROJECT_FOLDER=/files \
  -e HOP_PROJECT_NAME=your-project \
  -e HOP_ENVIRONMENT_NAME=prod \
  -e HOP_ENVIRONMENT_CONFIG_FILE_NAME_PATHS=/config/prod.json \
  -e HOP_FILE_PATH='{openvar}PROJECT_HOME{closevar}/main.hwf' \
  -e HOP_RUN_CONFIG=local \
  -v /opt/hop-projects/your-project:/files:ro \
  -v /etc/hop/environments/prod.json:/config/prod.json:ro \
  apache/hop:<tag>

Replace <tag> with a release tag (not latest). Replace the two bind mounts with nothing extra when CI already baked the project into the image; then HOP_PROJECT_FOLDER is a path inside the image such as /your-project.

cron, systemd, Jenkins and Airflow can all run that docker run --rm line. On a cluster the same contract is a Job.

Kubernetes CronJob

apiVersion: batch/v1
kind: CronJob
metadata:
  name: hop-nightly
spec:
  schedule: "0 2 * * *"
  timeZone: "Europe/Brussels"
  concurrencyPolicy: Forbid
  successfulJobsHistoryLimit: 3
  failedJobsHistoryLimit: 3
  jobTemplate:
    spec:
      backoffLimit: 1
      template:
        spec:
          restartPolicy: Never
          containers:
            - name: hop
              image: registry.example.com/your-project-hop:1.2.3
              env:
                - name: HOP_LOG_LEVEL
                  value: Basic
                - name: HOP_PROJECT_FOLDER
                  value: /your-project
                - name: HOP_PROJECT_NAME
                  value: your-project
                - name: HOP_ENVIRONMENT_NAME
                  value: prod
                - name: HOP_ENVIRONMENT_CONFIG_FILE_NAME_PATHS
                  value: /config/prod.json
                - name: HOP_FILE_PATH
                  value: '{openvar}PROJECT_HOME{closevar}/main.hwf'
                - name: HOP_RUN_CONFIG
                  value: local
              volumeMounts:
                - name: env
                  mountPath: /config
                  readOnly: true
          volumes:
            - name: env
              secret:
                secretName: hop-prod-env

concurrencyPolicy: Forbid is the overlap lock. backoffLimit: 1 retries once immediately; further retries belong to the next calendar tick (or to Airflow). timeZone needs a current enough Kubernetes; without it the schedule is UTC. The project-in-an-image page is the image this CronJob runs.

Apache Airflow

Airflow is the right scheduler when the Hop run is one task among dependencies, sensors and retries. The Airflow how-to uses the DockerOperator against a short-lived apache/hop container — the same contract as Short-lived container.

Two other honest options, covered there and on Remote run with export resources:

  • BashOperator calling hop-run on a worker that has a Hop client (the wrapper in this page).

  • hop-run with a remote run configuration, so Airflow submits work to a Hop Server instead of executing it on the worker.

A cron expression on the DAG (schedule='0 2 * * *', or schedule_interval on older Airflow) is Airflow’s equivalent of the crontab line above. Prefer Airflow’s own DAG schedule over also wrapping the DAG in cron.

What not to use as a scheduler

The Start action can repeat a workflow on an interval or at a clock time. That option exists for historical reasons. It keeps the Java process running for as long as the workflow is "scheduled", holds memory, and has no overlap policy, no durable calendar and no alerting.

Use it only as a last resort on a workstation. In production, start the workflow once from hop-run and let the scheduler start it again next time.

Things every scheduler has to get right

  • The process environment. cron in particular will not see the JAVA_HOME or PATH from your login shell. Set HOP_JAVA_HOME (or JAVA_HOME) and HOP_CONFIG_FOLDER in the wrapper or the unit, not by hoping they are inherited.

  • Overlapping runs. A load that still runs at 02:00 the next day will corrupt results if a second copy starts. flock, systemd’s default, Task Scheduler’s Do not start a new instance, Jenkins disableConcurrentBuilds(), Kubernetes concurrencyPolicy: Forbid, Airflow max_active_runs=1 — pick one and make it explicit.

  • Logs that outlive the process. hop-run writes to stdout (which systemd, Jenkins and Kubernetes already collect) and optionally to -lf. -lf does not stop the console logger: cron mails on any leftover stdout, not on a non-zero exit, so the wrapper above discards it. Configure an execution information location on the run configuration so you can inspect the run later from the Execution Information perspective.

  • Timezones. cron and systemd use the host timezone; Kubernetes CronJob defaults to UTC; Airflow uses the DAG timezone. Write the intended zone down next to the expression.

  • Image tags and Hop versions. Pin them. latest on a calendar turns an unrelated pull into an unplanned production upgrade.

See also