Hop Import

Hop Import is a command line tool to import and convert third party metadata to an Apache Hop format.

The same import runs from the Hop GUI through File / Import from Kettle/PDI. See Import Kettle (PDI) projects for that walkthrough and the options in the import dialog.

Usage

Run hop import -h to see the options, also when other options are given. hop-import.sh -h and hop-import.bat -h show the same list:

Usage

Usage: hop import [-efhlpv] [--no-apply-naming-schemes]
                  [-c=<targetConfigFilename>] [-i=<inputFolderName>]
                  [-j=<jdbcPropertiesFilename>] [-k=<kettlePropertiesFilename>]
                  [-n=<namingSchemeName>] [-o=<outputFolderName>]
                  [-P=<projectName>] [-r=<defaultPipelineRunConfiguration>]
                  [-s=<sharedXmlFilename>] [-t=<type>]
                  [-w=<defaultWorkflowRunConfiguration>]
Import metadata
  -c, --target-config-file=<targetConfigFilename>
                        The target config file to write variable to
  -e, --skip-existing   Skip existing files in the target folders
  -f, --skip-folders    Skip import of sub-folders
  -h, --help            Displays this help message and quits.
  -i, --input=<inputFolderName>
                        The input folder to read from
  -j, --jdbc-properties=<jdbcPropertiesFilename>
                        The jdbc.properties file to read from
  -k, --kettle-properties=<kettlePropertiesFilename>
                        The kettle.properties file to read from
  -l, --list-plugins    List the available import plugins
  -n, --naming-scheme=<namingSchemeName>
                        Naming scheme metadata name in the target folder to
                          apply to relational connections
      --no-apply-naming-schemes
                        Do not apply a naming scheme; connection names are
                          still aligned to one spelling
  -o, --output=<outputFolderName>
                        The output folder to write to
  -p, --skip-hidden     Skip import of hidden files and folders
  -P, --project=<projectName>
                        Import into this project. An existing project's home
                          folder is used as the target folder, an unknown
                          project is registered at the target folder.
  -r, --pipeline-run-configuration=<defaultPipelineRunConfiguration>
                        The run configuration to set on every imported
                          pipeline. Without it the name found in the source
                          file is kept.
  -s, --shared-xml=<sharedXmlFilename>
                        The shared.xml file to read from
  -t, --type=<type>     The type of import plugin to use (e.g. kettle)
  -v, --version         Print version information and exit
  -w, --workflow-run-configuration=<defaultWorkflowRunConfiguration>
                        The run configuration to set on every imported
                          workflow. Without it the name found in the source
                          file is kept.

Place naming-scheme JSON files in <output>/metadata/naming-scheme/ before running the command (or create them in the target project in Hop Gui). The unique Hop metadata scheme, or a unique General scheme, is applied to imported relational connection names. Use --naming-scheme to pick one by name.

Run configurations and projects

The job entries in a Kettle job can name a run configuration, for example Pentaho local, that doesn’t exist in Hop.

-r sets the run configuration of every imported Pipeline action, and of the Simple mapping and Metadata injection transforms.
-w sets the run configuration of every imported Workflow action.

Without them, a Pipeline or Workflow action keeps the name in the Kettle file, or gets local when it has none.

-P imports into a project:

When the project exists, its home folder is the output folder, and -o is ignored.
Otherwise the project is registered with the -o folder as its home, after both folders are checked.

shell

hop import \
  --type kettle \
  --input /projects/kettle/inventory \
  --output /projects/hop/inventory \
  --project inventory \
  --pipeline-run-configuration local \
  --workflow-run-configuration local

Examples

Import a set of Kettle files and folders into a project stored on Amazon AWS S3:

Windows
Linux, macOS

shell

hop-import.bat \
  --type kettle \
  --input /projects/kettle/inventory \
  --output s3:///apache-hop/s3project \
  --target-config-file imported-env-conf.json \
  --kettle-properties /home/etl/.kettle/kettle.properties \
  --shared-xml /home/matt/.kettle/shared.xml

shell

sh hop-import.sh \
  --type kettle \
  --input /projects/kettle/inventory \
  --output s3:///apache-hop/s3project \
  --target-config-file imported-env-conf.json \
  --kettle-properties /home/etl/.kettle/kettle.properties \
  --shared-xml /home/matt/.kettle/shared.xml

The logging will show what is going on and will print a report at the end:

shell

2021/06/22 16:39:29 - HopImport - Import is skipping existing target files
2021/06/22 16:39:29 - HopImport - Import is skipping hidden files and folders
2021/06/22 16:39:29 - HopImport - Import is not skipping sub-folders
2021/06/22 16:39:29 - HopImport - Finding files to import
2021/06/22 16:39:29 - HopImport - We found 84 kettle files.
2021/06/22 16:39:29 - HopImport - Importing files
2021/06/22 16:39:29 - HopImport -   - Saving file s3:///apache-hop/s3project/load-nodes/sample-model.json
2021/06/22 16:39:29 - HopImport -   - Saving file s3:///apache-hop/s3project/scaleable-file-processing/input/file02.csv
2021/06/22 16:39:29 - HopImport -   - Saving file s3:///apache-hop/s3project/graph-output/READ.me
2021/06/22 16:39:29 - HopImport -   - Saving file s3:///apache-hop/s3project/scaleable-file-processing/PDI/Check slave server.hpl
2021/06/22 16:39:29 - HopImport -   - Saving file s3:///apache-hop/s3project/remove-everything/remove all in database.hwf
...
2021/06/22 16:39:37 - HopImport -   - Saving file s3:///apache-hop/s3project/scaleable-file-processing/input/file01.csv
2021/06/22 16:39:37 - HopImport - Importing connections
2021/06/22 16:39:41 - HopImport - Importing variables
Creating new default Hop configuration file: s3:///apache-hop/s3project/imported-env-conf.json

2021/06/22 16:39:43 - HopImport - Imported:
2021/06/22 16:39:43 - HopImport - 10 jobs
2021/06/22 16:39:43 - HopImport - 31 transformations
2021/06/22 16:39:43 - HopImport - 43 other files
2021/06/22 16:39:43 - HopImport - 86 variables were imported into environment config file s3:///apache-hop/s3project/imported-env-conf.json
2021/06/22 16:39:43 - HopImport - You can use this as a configuration file in an environment.
2021/06/22 16:39:43 - HopImport - 3 database connections where saved in metadata folder s3:///apache-hop/s3project/metadata
2021/06/22 16:39:43 - HopImport -
2021/06/22 16:39:43 - HopImport - Connections with the same name and different configurations have only been saved once.
2021/06/22 16:39:43 - HopImport - Check the following file for a list of connections that might need extra attention: s3:///apache-hop/s3project/connections.csv

Functionality

In general the plugin copies Kettle files and non-Kettle files.

Kettle files

File Conversion

File	Conversion
Transformations	Kettle transformation files (.ktr extension) are converted to Hop pipelines. During this process the XML metadata is converted to the appropriate Hop format.
Jobs	Kettle job files (.kjb extension) are converted to Hop workflows. During this process the XML metadata is converted to the appropriate Hop format.
kettle.properties	The Kettle properties file in your .kettle folder (typically found in the home directory or `${KETTLE_HOME}`) often contains variables and values regarding your environment. These variables and values are converted into an environment configuration file if you specified the `-c` or `--target-config-file` option. When you create an environment in Hop you can simply add this file to it to make everything work. If the configuration file already exists it will be updated, not overwritten. The description of the newly imported variables is set to `Imported from Kettle` to indicate that it’s new. Values of existing variables are overwritten and the existing description is kept.
shared.xml	The shared.xml file in your .kettle folder (typically found in the home directory or `${KETTLE_HOME}`) often contains connections which are shared across many transformations and jobs. These connections are imported as Relational Database Connection metadata stored in the target folder `metadata/rdbms` folder.
jdbc.properties	A jdbc.properties file sometimes contains relational database metadata which is converted into generic RDBMS connections metadata.

Transformations

Kettle transformation files (.ktr extension) are converted to Hop pipelines. During this process the XML metadata is converted to the appropriate Hop format.

Jobs

Kettle job files (.kjb extension) are converted to Hop workflows. During this process the XML metadata is converted to the appropriate Hop format.

kettle.properties

The Kettle properties file in your .kettle folder (typically found in the home directory or ${KETTLE_HOME}) often contains variables and values regarding your environment. These variables and values are converted into an environment configuration file if you specified the -c or --target-config-file option. When you create an environment in Hop you can simply add this file to it to make everything work. If the configuration file already exists it will be updated, not overwritten. The description of the newly imported variables is set to Imported from Kettle to indicate that it’s new. Values of existing variables are overwritten and the existing description is kept.

shared.xml

The shared.xml file in your .kettle folder (typically found in the home directory or ${KETTLE_HOME}) often contains connections which are shared across many transformations and jobs. These connections are imported as Relational Database Connection metadata stored in the target folder metadata/rdbms folder.

jdbc.properties

A jdbc.properties file sometimes contains relational database metadata which is converted into generic RDBMS connections metadata.

Non-Kettle files

Non-Kettle files like text files, JSON, XML, … they are all copied over by default, including the sub-folders. You can change this behavior with the options:

--skip-folders : don’t copy sub-folders from the input folder
--skip-hidden : Don’t copy hidden files and folders like .git, .gitignore and so on
--skip-existing : Only copy files which haven’t been copied yet