Hop Server

Hop Server is a lightweight server to run workflows and pipelines with the Remote pipeline or Remote workflow run configurations.

Additionally, a Hop Server can also be accessed directly through a number of web services and in combination with the Web Service and Asynchronous Web Service metadata types.

Starting and Stopping Hop Server

General Use

Hop Server is available as a script in your Hop installation directory.

Run Hop Server without any parameters to display its usage options. On Windows, this is hop-server.bat, on Mac and Linux, run hop-server.sh.

Usage: <main class> [-k] [-gs] [-e=<environmentOption>] [-id=<id>]
                    [-j=<projectOption>] [-l=<level>] [-p=<password>]
                    [-ps=<pipelineName>] [-u=<username>] [-ws=<workflowName>]
                    [-s=<systemProperties>[,<systemProperties>...]]...
                    <parameters>...
      <parameters>...   One XML configuration file or a hostname and port
  -e, --environment=<environmentOption>
                        The name of the lifecycle environment to use
      -gs, --general-status
                        List the general status of the server
      -id=<id>          Specify the ID of the pipeline or workflow to query
  -j, --project=<projectOption>
                        The name of the project to use
  -k, --kill            Stop the running hopServer server.  This is only allowed
                          when using the hostname/port form of the command. Use the
                          -s and -u options to authenticate
  -l, --level=<level>   The debug level, one of NONE, MINIMAL, BASIC, DETAILED,
                          DEBUG, ROWLEVEL
  -p, --password=<password>
                        The server password.  Required for administrative operations
                          only, not for starting the server.
      -ps, --pipeline-status=<pipelineName>
                        List the status of the pipeline with this name (also specify
                          the -id option)
  -s, --system-properties=<systemProperties>[,<systemProperties>...]
                        A comma separated list of KEY=VALUE pairs
  -u, --userName=<username>
                        The server user name.  Required for administrative
                          operations only, not for starting the server.
      -ws, --workflow-status=<workflowName>
                        List the status of the workflow with this name (also specify
                          the -id option)

Example: hop-server.sh 0.0.0.0 8080
Example: hop-server.sh 192.168.1.221 8081

Example: hop-server.sh -e aura-gcp gs://apachehop/hop-server-config.xml
Example: hop-server.sh 127.0.0.1 8080 --kill --userName cluster --password cluster

The available Hop Server options are:

Short	Extended	Description
-h	--help	This help text
-p	--password	The server password. Required for administrative operations only, not for starting the server.
-k	--kill	Stop the running hop server.
-u	--userName	The server password. Required for administrative operations only, not for starting the server.
-s	--system-properties	Manually set system environment variables. Specify a comma separated list of KEY=VALUE pairs.
-e	--environment	The name of the project lifecycle environment to enable before startup. This is provided for by the 'projects' plugin.
-j	--project	The name of the project to enable before startup. This is provided for by the 'projects' plugin.
-gs	--general-status	List the general status of the server.
-ps	--pipeline-status	List the status of the pipeline with this name (also specify the -id option)
-ws	--workflow-status	List the status of the workflow with this name (also specify the -id option)
-id		Specify the ID of the pipeline or workflow to query

Short

Extended

Description

-h

--help

This help text

-p

--password

The server password. Required for administrative operations only, not for starting the server.

-k

--kill

Stop the running hop server.

-u

--userName

The server password. Required for administrative operations only, not for starting the server.

-s

--system-properties

Manually set system environment variables. Specify a comma separated list of KEY=VALUE pairs.

-e

--environment

The name of the project lifecycle environment to enable before startup. This is provided for by the 'projects' plugin.

-j

--project

The name of the project to enable before startup. This is provided for by the 'projects' plugin.

-gs

--general-status

List the general status of the server.

-ps

--pipeline-status

List the status of the pipeline with this name (also specify the -id option)

-ws

--workflow-status

List the status of the workflow with this name (also specify the -id option)

-id

Specify the ID of the pipeline or workflow to query

Start Hop Server with command line parameters

Hop Server can be started with hostname or ip address and port number as unnamed arguments:

hop-server <Interface address> <Port> [-h] [-p <arg>] [-s] [-u <arg>]

Example startup commands are:

hop-server.sh 127.0.0.1 8080

hop-server.sh 192.168.1.221 8081

Listen to all interfaces on the server:

hop-server.sh 0.0.0.0 8080

Start Hop Server with a configuration file

Specify the xml configuration file as the only argument:

hop-server <Configuration File>

The syntax of this configuration file is fairly simple:

<hop-server-config>

  <hop-server>
    <name>server-8181</name>
    <hostname>localhost</hostname>
    <port>8181</port>
  </hop-server>

  <!-- Join the web server thread and wait until it's finished.
       The default is true
  -->
  <joining>true</joining>

  <!-- The maximum number of log lines kept in memory by the server.
       The default is 0 which means: keep all lines
   -->
  <max_log_lines>0</max_log_lines>

  <!-- The time (in minutes) it takes for a log line to be cleaned up in memory.
       The default is 0 which means: never clean up log lines
  -->
  <max_log_timeout_minutes>1440</max_log_timeout_minutes>

  <!-- The time (in minutes) it takes for a pipeline or workflow execution to be removed from the server status.
       The default is 0 which means: never clean executions
  -->
  <object_timeout_minutes>1440</object_timeout_minutes>

  <!-- The folder to read metadata objects from so that web services and database connections for sequences can be found.
       The default is that no metadata is configured: remotely executed pipelines and workflows will have their own metadata.
  -->
  <metadata_folder></metadata_folder>

</hop-server-config>

Example startup commands with a configuration file are:

hop-server.sh /foo/bar/hop-server-config.xml

Or with a remote configuration file:

hop-server.sh http://www.example.com/hop-server-config.xml

You can also enable a project lifecyfle environment for the Hop server:

hop-server.sh -e graph-aws hop-server.xml

In the sample above the environment contains configuration files with variables which are loaded. With the environment the server also knows the project home folder. The server configuration file is found in the home folder automatically with the implicit relative path.

SSL configuration

To protect traffic between a Hop Server and its clients (Hop Run, Hop GUI, a browser, Hop Server command line querying, …) it’s possible to use encrypt data using a secure socket layer (SSL) connection on top of the regular web traffic with a hyper treading transfer protocol (HTTP). The combination of both is called HTTPS. To run a Hop server using this https:// protocol you can add an sslConfig section in the hop-server-config/hop-server path.

The 3 main options are:

keyStore : the path to the java keystore file, created with keytool
keyStorePassword : the password to the keystore file
keyPassword : the key password. If this is the same as the keystore password you can omit this option.

The HTTP protocol used is version 1.1 or HTTP/1.1. The type of keystore read is a Java Keystore or type: JKS. Let’s take a look at how we can generate a sample keystore:

# Generate a new key
#
openssl genrsa -des3 -out hop.key

# Make a new certificate
#
openssl req -new -x509 -key hop.key -out hop.crt

# Create a PKCS12 keystore and import it into a JKS keystore
# The resulting file is: keystore
#
keytool -keystore keystore -import -alias hop -file hop.crt -trustcacerts
openssl req -new -key hop.key -out hop.csr
openssl pkcs12 -inkey hop.key -in hop.crt -export -out hop.pkcs12
keytool -importkeystore -srckeystore hop.pkcs12 -srcstoretype PKCS12 -destkeystore keystore

Here is an example of the information to include in your server XML:

<hop-server-config>
<hop-server>
...

    <sslConfig>
      <keyStore>/path/to/keystore</keyStore>
      <keyStorePassword>password</keyStorePassword>
      <keyPassword>keyPassword</keyPassword>
    </sslConfig>

    <!-- Add the following line to support querying over https -->
    <sslMode>Y</sslMode>
  </hop-server>
  ...
</hop-server-config>

Start Hop Server with docker

It’s often very convenient to run a Hop docker container since it has all the required software automatically delivered. For a complete description of the standard Hop docker container see the full reference in the technical documentation. Here is an example of how you would start a "long-lived" docker container:

docker run \
  -p 8080:8080 \
  -e HOP_SERVER_PORT=8080 \
  -e HOP_SERVER_USER=username \
  -e HOP_SERVER_USER=password \
  apache/hop

Stopping Hop Server

In a testing setup where Hop Server was started from a terminal, the process can be terminated through CTRL-C.

In headless environments, the same hop-server command used to start the server can be used to stop it:

hop-server.sh 127.0.0.1 8080 -s -u cluster -p cluster

Verify startup

Starting a Hop Server on the local machine e.g. on port 8081 will only take 1 or 2 seconds.

The console output will look similar to what is listed below:

2020/06/20 18:35:12 - HopServer - Installing timer to purge stale objects after 1440 minutes.
2020/06/20 18:35:12 - HopServer - Created listener for webserver @ address : localhost:8081

Query a server from the command line

You can query the new server with another hop-server command:

sh hop-server.sh -gs -u cluster -p cluster 127.0.0.1 8080
Pipelines: 0 found.

Workflows: 0 found.

Query a pipeline from the command line

sh hop-server.sh -id 375c9113-b538-4559-8e98-ee02a435fbb9 -u cluster -p cluster -ps service-example -j my-project hop-server.xml
2021/10/01 13:27:04 - HopServer - Enabling project 'my-project'
  ID: 375c9113-b538-4559-8e98-ee02a435fbb9
      Name:     service-example
      Status:   Finished
      Start:    2021/10/01 13:26:45.128
      End:      2021/10/01 13:26:45.220
      Log date: 2021/10/01 13:27:04.363
      Errors:   0
      Transforms: 4 found.
        1
          Name:      a,b
          Copy:      0
          Status:    Finished
          Input:     0
          Output:    0
          Read:      1
          Written:   1
          Rejected:  0
          Updated:   0
          Errors:    0
        2
...
        3
...
        4
...
      Logging:
          2021/10/01 13:26:45 - service-example - Executing this pipeline using the Local Pipeline Engine with run configuration 'local'
          2021/10/01 13:26:45 - service-example - Execution started for pipeline [service-example]
          2021/10/01 13:26:45 - a,b.0 - Finished processing (I=0, O=0, R=1, W=1, U=0, E=0)
          2021/10/01 13:26:45 - c,d.0 - Finished processing (I=0, O=0, R=1, W=1, U=0, E=0)
          2021/10/01 13:26:45 - build JSON.0 - Finished processing (I=0, O=1, R=1, W=1, U=0, E=0)
          2021/10/01 13:26:45 - OUTPUT.0 - Finished processing (I=0, O=0, R=1, W=1, U=0, E=0)
          2021/10/01 13:26:45 - service-example - Pipeline duration : 0.092 seconds [  0.092" ]
          2021/10/01 13:26:45 - service-example - Execution finished on a local pipeline engine with run configuration 'local'

Query a workflow from the command line

sh hop-server.sh -ws test-workflow -id e24b4549-edf0-4d77-987e-f103b630b4cc -u cluster -p cluster localhost 8181
  ID: e24b4549-edf0-4d77-987e-f103b630b4cc
      Name:     test-workflow
      Status:   Finished
      Log date: 2021/10/01 14:27:45.891
      Result:   true
      Errors:   0
      Logging:
          2021/10/01 14:27:45 - test-workflow - Start of workflow execution
          2021/10/01 14:27:46 - test-workflow - Starting action [sample]
          2021/10/01 14:27:46 - sample - Using run configuration [remote-8181]
          2021/10/01 14:27:46 - sample - Executing this pipeline using the Remote Pipeline Engine with run configuration 'remote-8181'
          2021/10/01 14:27:46 - sample - 2021/10/01 14:27:46 - sample - Executing this pipeline using the Local Pipeline Engine with run configuration 'local'
          2021/10/01 14:27:46 - sample - 2021/10/01 14:27:46 - sample - Execution started for pipeline [sample]
          2021/10/01 14:27:47 - sample - 2021/10/01 14:27:47 - 1M.0 - Finished processing (I=0, O=0, R=0, W=1000000, U=0, E=0)
          2021/10/01 14:27:47 - sample - 2021/10/01 14:27:47 - someString,someInt.0 - Finished processing (I=0, O=0, R=1000000, W=1000000, U=0, E=0)
          2021/10/01 14:27:47 - sample - 2021/10/01 14:27:47 - id.0 - Finished processing (I=0, O=0, R=1000000, W=1000000, U=0, E=0)
          2021/10/01 14:27:47 - sample - 2021/10/01 14:27:47 - sample - Pipeline duration : 0.977 seconds [  0.977" ]
          2021/10/01 14:27:47 - sample - 2021/10/01 14:27:47 - sample - Execution finished on a local pipeline engine with run configuration 'local'
          2021/10/01 14:27:47 - sample - Execution finished on a remote pipeline engine with run configuration 'remote-8181'
          2021/10/01 14:27:48 - test-workflow - Starting action [true]
          2021/10/01 14:27:48 - test-workflow - Starting action [false]
          2021/10/01 14:27:48 - test-workflow - Starting action [log-something]
          2021/10/01 14:27:48 - Subject - Message
          2021/10/01 14:27:48 - test-workflow - Starting action [Success]
          2021/10/01 14:27:48 - test-workflow - Finished action [Success] (result=[true])
          2021/10/01 14:27:48 - test-workflow - Finished action [log-something] (result=[true])
          2021/10/01 14:27:48 - test-workflow - Finished action [false] (result=[true])
          2021/10/01 14:27:48 - test-workflow - Finished action [true] (result=[true])
          2021/10/01 14:27:48 - test-workflow - Finished action [sample] (result=[true])
          2021/10/01 14:27:48 - test-workflow - Workflow execution finished
          2021/10/01 14:27:48 - test-workflow - Workflow duration : 2.715 seconds [  2.714" ]

Connect to the Hop Server UI

To connect to the previously started server, point your browser to http://localhost:8081.

You’ll be prompted for your username and password. The default is cluster for both the username and password. The defaults obviously should be changed in any environment that goes beyond a simple local developer setup.

on startup, the pipeline and workflow lists shown below will be empty. Run a workflow or pipeline through the Hop Remote pipeline engine run configuration or through the REST api. When pipelines or workflows are executed on the server, you’ll be able to follow the logging output either from the terminal or a log file (e.g. piped from the startup command).

For each of the options in the pipeline and workflow dialogs described below, select a pipeline and workflow from the list and select the desired option.

The header bar for workflows and pipelines is almost identical (from left to right).

Run
Stop the running pipeline/workflow
Cleanup pipeline	Cleanup a pipeline: close remote sockets etc
View pipeline/workflow details
Remove pipeline/workflow from list

Run

Stop the running pipeline/workflow

Cleanup pipeline

Cleanup a pipeline: close remote sockets etc

View pipeline/workflow details

Remove pipeline/workflow from list