The Hop toolbox
You’ll spend most of your time in Hop Gui, but a project that only runs from a desktop GUI is not much use in production.
The split is roughly this: designing pipelines and workflows and creating metadata is what Hop Gui is for. Configuring and running them is where the command line takes over, and that is the half a scheduler, a container or a CI/CD pipeline needs.
One command to start them all
Since Hop 2.x there is a single hop launcher in the installation folder.
It replaces the individual hop-run, hop-conf, hop-gui, … scripts, and plugins can add their own commands to it.
-
Windows
-
Linux, macOS
hop.bat help
./hop help
help lists the commands available in your installation, which depends on the plugins you have installed:
Commands:
help Display help information about the specified command.
conf Configure Hop
driver Manage JDBC drivers (list, install)
encrypt Encrypt secrets
export Export pipeline, workflow, or model diagrams to various formats
(SVG, Mermaid, etc.)
gui The Hop GUI
import Import metadata
marketplace Install and manage optional Hop plugins from a Maven repository
naming-check Validate names against project Naming Schemes
run Run a pipeline or workflow
search Search in Hop metadata
server Run a Hop server
setup Configure Hop launcher environment variables
(HOP_CONFIG_FOLDER, HOP_AUDIT_FOLDER, HOP_JAVA_HOME,
HOP_OPTIONS, HOP_SHARED_JDBC_FOLDERS). Distinct from hop
conf, which edits hop-config.json.
Two options are worth knowing before anything else, because nearly every command takes them:
| Option | What it does |
|---|---|
|
enable a project before the command runs |
|
enable an environment before the command runs |
So running your workflow from the previous chapter looks like this:
./hop run -j getting-started -r local -f '${PROJECT_HOME}/first-workflow.hwf'
the older hop-run.sh, hop-conf.sh and friends still ship and still work. New examples in this manual use hop <command>.
|
The tools
| Tool | What you use it for | Read more |
|---|---|---|
|
The visual development environment: build, preview, run and debug pipelines and workflows. |
|
|
Run a pipeline or workflow. This is what a scheduler, a CI job or a container calls. |
|
|
Manage projects, environments and cloud configuration from a script. |
|
|
Point Hop at a user-level config folder so your projects and settings survive an upgrade. |
|
|
Run a Hop Server to execute pipelines and workflows remotely, over REST. |
|
|
Search across all the metadata in a project: which pipeline uses this connection, where is this field set. |
|
|
Obfuscate a password so it isn’t stored in clear text in a configuration file. |
|
|
Convert a Kettle/PDI repository into a Hop project. |
|
|
List the JDBC drivers Hop can download, and install them. |
|
|
Export pipeline and workflow diagrams to SVG or Mermaid, for one file or a whole folder. |
|
|
Install optional plugins into a Hop client from a Maven repository. |
Some tools are optional plugins rather than part of the standard client.
Install them with hop marketplace:
| Tool | What you use it for | Read more |
|---|---|---|
|
Static analysis over a project: check pipelines, workflows and metadata without running them. Useful as a build step. |
|
|
Generate documentation for your own Hop project, in Markdown or HTML. |
|
PyHop |
Drive Hop from Python. |
|
Hop Arrow |
Expose pipeline output over an Apache Arrow Flight server. |
| Hop Gui also runs in a browser. See Hop Web and Hop Web in Docker. |
Try one
The quickest way to see the command line in action is to run something you already built:
./hop run -j getting-started -r local -f '${PROJECT_HOME}/first-pipeline.hpl'
You get the same log you saw in the Logging tab of Hop Gui, on stdout. That is the whole deployment story for a lot of Hop projects: the same file, the same run configuration, started by cron, a Kubernetes CronJob or Airflow instead of by you.
See Scheduling workflows and pipelines for the details.