Hop REST Web Application

Description

The Hop REST web application is deployed as a WAR file hop-rest-<version>.war. You can put this archive in the webapps folder of your favorite server (e.g. Apache Tomcat). We suggest that you rename it to hop.war or that you unzip it in a hop folder.

The base URL

When deployed under a webapps/hop folder you will get the services, detailed below, under the base URL of /hop/api/v1/.

Configuration

When the web application starts it will look for an environment variable called HOP_REST_CONFIG_FOLDER to determine where it will look for a file called hop-rest.properties. This file can contain the following properties to determine the configuration of the application:

Property Description

logLevel

The logging level to use, one of NOTHING, ERROR, MINIMAL, BASIC, DETAILED, DEBUG, or ROWLEVEL.

metadataExportFile

The metadata to use for the web application in the form of a single JSON 'export' file. Use the GUI tools menu or sh hop-conf.sh --export-metadata to export metadata to a single file.

environmentName

In case the projects plugin is available it will enable this environment and the underlying project (to be configured on the system, point to a configuration with HOP_CONFIG_FOLDER) before starting the application.

projectName

In case the projects plugin is available it will enable this project (to be configured on the system, point to a configuration with HOP_CONFIG_FOLDER) before starting the application.

Metadata services

The metadata services are deployed under sub-path metadata/.

List the metadata type keys

Type

GET

Path

metadata/types

Produces

application/json

Example call

curl -X GET http://localhost:8080/hop/api/v1/metadata/types

Example output

["dataset","workflow-log","async-web-service","execution-info-location","web-service","cassandra-connection","workflow-run-configuration","unit-test","rdbms","mongodb-connection","file-definition","splunk","pipeline-run-configuration","pipeline-log","server","partition","execution-data-profile","neo4j-connection","pipeline-probe","neo4j-graph-model"]

List the metadata elements for a key

Type

GET

Path

metadata/list/{key}

Arguments

key : the key of the metadata type

Produces

application/json

Example call

curl -X GET http://localhost:8080/hop/api/v1/metadata/list/pipeline-run-configuration/

Example output

["local"]

Get a metadata element

Type

GET

Path

metadata/{key}/{name}

Arguments

key : the key of the metadata type, {name} : the name of the metadata element to get.

Produces

application/json

Example call

curl -X GET http://localhost:8080/hop/api/v1/metadata/pipeline-run-configuration/local/

Example output

{"engineRunConfiguration":{"Local":{"feedback_size":"50000","sample_size":"1000","sample_type_in_gui":"None","wait_time":"2","rowset_size":"10000","safe_mode":false,"show_feedback":false,"topo_sort":false,"gather_metrics":false,"transactional":false}},"defaultSelection":false,"configurationVariables":[],"name":"local","description":"Runs your pipelines locally with the standard local Hop pipeline engine","dataProfile":"first-last","executionInfoLocationName":"local"}

Save a metadata element

Type

POST

Path

metadata/{key}

Arguments

key : the key of the metadata type to save

Consumes

application/json

Produces

application/json

Example call

curl -X POST http://localhost:8080/hop/api/v1/metadata/pipeline-run-configuration/ -d '{"engineRunConfiguration":{"Local":{"feedback_size":"50000","sample_size":"1000","sample_type_in_gui":"None","wait_time":"2","rowset_size":"10000","safe_mode":false,"show_feedback":false,"topo_sort":false,"gather_metrics":false,"transactional":false}},"defaultSelection":false,"configurationVariables":[],"name":"local","description":"Runs your pipelines locally with the standard local Hop pipeline engine","dataProfile":"first-last","executionInfoLocationName":"local"}'

Delete a metadata element

Type

DELETE

Path

metadata/{key}/{name}

Arguments

key : the key of the metadata type to save. name : the name of the metadata element to delete.

Produces

application/json

Example call

curl -X DELETE http://localhost:8080/hop/api/v1/metadata/pipeline-run-configuration/Flink/

Output

Flink

Plugins services

The plugins services are deployed under sub-path plugins/.

List all the plugin type classes

Type

GET

Path

plugins/types

Produces

application/json

Example call

curl -X GET http://localhost:8080/hop/api/v1/plugins/types

Example output

["org.apache.hop.core.compress.CompressionPluginType","org.apache.hop.core.row.value.ValueMetaPluginType","org.apache.hop.core.database.DatabasePluginType","org.apache.hop.core.plugins.TransformPluginType","org.apache.hop.core.auth.AuthenticationConsumerPluginType","org.apache.hop.imp.ImportPluginType","org.apache.hop.core.encryption.TwoWayPasswordEncoderPluginType","org.apache.hop.core.auth.AuthenticationProviderPluginType","org.apache.hop.core.plugins.PartitionerPluginType","org.apache.hop.core.logging.LoggingPluginType","org.apache.hop.execution.plugin.ExecutionInfoLocationPluginType","org.apache.hop.core.config.plugin.ConfigPluginType","org.apache.hop.core.plugins.HopServerPluginType","org.apache.hop.core.plugins.ActionPluginType","org.apache.hop.core.extension.ExtensionPointPluginType","org.apache.hop.pipeline.engine.PipelineEnginePluginType","org.apache.hop.pipeline.transform.RowDistributionPluginType","org.apache.hop.workflow.engine.WorkflowEnginePluginType","org.apache.hop.core.vfs.plugin.VfsPluginType","org.apache.hop.execution.sampler.ExecutionDataSamplerPluginType","org.apache.hop.metadata.plugin.MetadataPluginType"]

List all the plugins for a given type class

Type

GET

Path

plugins/list/{typeClassName}/

Arguments

key : the class name of the plugin type

Produces

application/json

Example call

curl -X GET http://localhost:8080/hop/api/v1/plugins/list/org.apache.hop.pipeline.engine.PipelineEnginePluginType

This call retrieves all the information about all the pipeline engine plugins.

Example output

[{"category":null,"name":"Beam DataFlow pipeline engine","description":"This allows you to run your pipeline on Google Cloud Platform DataFlow, provided by the Apache Beam community","ids":["BeamDataFlowPipelineEngine"],"pluginType":"org.apache.hop.pipeline.engine.PipelineEnginePluginType","imageFile":null,"separateClassLoaderNeeded":false,"classLoaderGroup":null,"nativePlugin":false, …​ }

Execution services

The 'Execution' services are deployed under sub-path execution/.

Execute a pipeline synchronously

Type

POST

Path

execution/sync

Produces

application/json

Example call

curl -X POST http://localhost:8080/hop/api/v1/execution/sync/ -H 'Content-Type: application/json' -d '{ "service" : "test", "runConfig" : "local", "variables" : { "VAR1" : "value1", "VAR2" : "value2" }, "bodyContent" : "This is body content" }' `

Example output

The executing pipeline can produce output like this:

{ "rows" : [ { "id" : 1, "uuid" : "93b534f0-31ef-4992-8012-ffbfd3d7d489", "var1" : "value1", "var2" : "value2", "bodyContent" : "This is body content" }, …​ ] }

The body to post can contain the following options (see also: the example above)

  • execute: the name of the Web Service metadata element to use

  • runConfig: the name of the pipeline run configuration to use

  • variables: a map with variables (or parameters) with their names and values

  • bodyContent: this will be set as a variable using the body content variable option in the Web Service metadata.

Execution Information Location services

The execution information location services are deployed under sub-path location/.

Get execution IDs

Type

POST

Path

location/executions/{locationName}

Arguments

locationName : the name of the execution information location to query.

Consumes

application/json

Produces

application/json

Example call

curl -X GET http://localhost:8080/hop/api/v1/location/executions/local/ -H 'Content-Type: application/json' -d '{ "includeChildren" : "true", "limit" : 100 }' `

Example output

The list of execution IDs looks like this:

["af84cbc2-0166-4dea-956f-72b73cf66d0d", "bf84cbc2-0166-4dea-956f-72b73cf66d0e", "cf84cbc2-0166-4dea-956f-72b73cf66d0f"]

The body to post can contain the following options (see also: the example above)

  • includeChildren: Set this to true if you want to see child executions of workflows and pipelines

  • limit: the maximum number of IDs to retrieve or a value ⇐0 to get all IDs

Get execution

Type

GET

Path

location/executions/{locationName}/{executionId}

Arguments

locationName: the name of the execution information location to query. executionId: the ID of the execution to retrieve.

Produces

application/json

Example call

curl -X GET http://localhost:8080/hop/api/v1/location/executions/local/df84cbc2-0166-4dea-956f-72b73cf66d0d/ `

Example output

The execution details look like this:

{"name":"pipeline-name","filename":"/path/filename.hpl","id":"df84cbc2-0166-4dea-956f-72b73cf66d0d","parentId":null,"executionType":"Pipeline", …​ }

Get execution state

Type

GET

Path

location/state/{locationName}/{executionId}

Arguments

locationName: the name of the execution information location to query. executionId: the ID of the execution state to retrieve.

Produces

application/json

Example call

curl -X GET http://localhost:8080/hop/api/v1/location/state/local/df84cbc2-0166-4dea-956f-72b73cf66d0d/ `

Example output

The execution state looks like this:

{"executionType":"Pipeline","parentId":null,"id":"df84cbc2-0166-4dea-956f-72b73cf66d0d","name":"test-service","copyNr":null,"loggingText":"logging-text","lastLogLineNr":14,"metrics":[{"componentName":"id","componentCopy":"0","metrics":{"Read":5,"Buffers Output":0,"Errors":0,"Input":0,"Written":5,"Updated":0,"Output":0,"Rejected":0,"Buffers Input":0}},{"componentName":"uuid","componentCopy":"0","metrics":{"Read":5,"Buffers Output":0,"Errors":0,"Input":0,"Written":5,"Updated":0,"Output":0,"Rejected":0,"Buffers Input":0}},{"componentName":"Enhanced JSON Output","componentCopy":"0","metrics":{"Read":5,"Buffers Output":0,"Errors":0,"Input":0,"Written":1,"Updated":0,"Output":1,"Rejected":0,"Buffers Input":0}},{"componentName":"OUTPUT","componentCopy":"0","metrics":{"Read":1,"Buffers Output":0,"Errors":0,"Input":0,"Written":1,"Updated":0,"Output":0,"Rejected":0,"Buffers Input":0}},{"componentName":"5 rows","componentCopy":"0","metrics":{"Read":0,"Buffers Output":0,"Errors":0,"Input":0,"Written":5,"Updated":0,"Output":0,"Rejected":0,"Buffers Input":0}},{"componentName":"Get variables","componentCopy":"0","metrics":{"Read":5,"Buffers Output":0,"Errors":0,"Input":0,"Written":5,"Updated":0,"Output":0,"Rejected":0,"Buffers Input":0}}],"statusDescription":"Finished","updateTime":1678191016156,"childIds":["46690405-4b06-4353-973a-06aff689afe0","ec974060-e228-4438-8408-b049803eb316","f7344a76-06b0-4499-95a1-1b30bd987561","32837620-df15-4602-b32c-69111689767f","db2e3feb-3ab1-4491-a4dc-7036cc0c3a3f","4653fb73-ac94-4ac3-adef-a14e6129aa14"],"details":{},"failed":false,"containerId":"c7c90ed5-6684-408a-be11-d1fd1c65164a"}