Hop Import

Hop Import is a command line tool to import and convert third party metadata to an Apache Hop format.

The same import runs from the Hop GUI through File / Import from Kettle/PDI. See Import Kettle (PDI) projects for that walkthrough and the options in the import dialog.

Usage

Usage help can be obtained by running hop-import with the -h option, even when other options are specified:

Usage

Usage: <main class> [-efhlp] [-c=<targetConfigFilename>] [-i=<inputFolderName>]
                    [-j=<jdbcPropertiesFilename>]
                    [-k=<kettlePropertiesFilename>] [-o=<outputFolderName>]
                    [-s=<sharedXmlFilename>] [-t=<type>]
  -c, --target-config-file=<targetConfigFilename>
                        The target config file to write variables to
  -e, --skip-existing   Skip existing files in the target folders
  -f, --skip-folders    Skip import of sub-folders
  -h, --help            Displays this help message and quits
  -i, --input=<inputFolderName>
                        The input folder to read from
  -j, --jdbc-properties=<jdbcPropertiesFilename>
                        The jdbc.properties file to read from
  -k, --kettle-properties=<kettlePropertiesFilename>
                        The kettle.properties file to read from
  -l, --list-plugins    List the available import plugins
  -o, --output=<outputFolderName>
                        The output folder to write to
  -p, --skip-hidden     Skip import of hidden files and folders
  -s, --shared-xml=<sharedXmlFilename>
                        The shared.xml file to read from
  -t, --type=<type>     The type of import plugin to use (e.g. kettle)
  -v, --version         Print version information and exit

Examples

Import a set of Kettle files and folders into a project stored on Amazon AWS S3:

Windows
Linux, macOS

hop-import.bat \
  --type kettle \
  --input /projects/kettle/inventory \
  --output s3:///apache-hop/s3project \
  --target-config-file imported-env-conf.json \
  --kettle-properties /home/etl/.kettle/kettle.properties \
  --shared-xml /home/matt/.kettle/shared.xml

sh hop-import.sh \
  --type kettle \
  --input /projects/kettle/inventory \
  --output s3:///apache-hop/s3project \
  --target-config-file imported-env-conf.json \
  --kettle-properties /home/etl/.kettle/kettle.properties \
  --shared-xml /home/matt/.kettle/shared.xml

The logging will show what is going on and will print a report at the end:

2021/06/22 16:39:29 - HopImport - Import is skipping existing target files
2021/06/22 16:39:29 - HopImport - Import is skipping hidden files and folders
2021/06/22 16:39:29 - HopImport - Import is not skipping sub-folders
2021/06/22 16:39:29 - HopImport - Finding files to import
2021/06/22 16:39:29 - HopImport - We found 84 kettle files.
2021/06/22 16:39:29 - HopImport - Importing files
2021/06/22 16:39:29 - HopImport -   - Saving file s3:///apache-hop/s3project/load-nodes/sample-model.json
2021/06/22 16:39:29 - HopImport -   - Saving file s3:///apache-hop/s3project/scaleable-file-processing/input/file02.csv
2021/06/22 16:39:29 - HopImport -   - Saving file s3:///apache-hop/s3project/graph-output/READ.me
2021/06/22 16:39:29 - HopImport -   - Saving file s3:///apache-hop/s3project/scaleable-file-processing/PDI/Check slave server.hpl
2021/06/22 16:39:29 - HopImport -   - Saving file s3:///apache-hop/s3project/remove-everything/remove all in database.hwf
...
2021/06/22 16:39:37 - HopImport -   - Saving file s3:///apache-hop/s3project/scaleable-file-processing/input/file01.csv
2021/06/22 16:39:37 - HopImport - Importing connections
2021/06/22 16:39:41 - HopImport - Importing variables
Creating new default Hop configuration file: s3:///apache-hop/s3project/imported-env-conf.json

2021/06/22 16:39:43 - HopImport - Imported:
2021/06/22 16:39:43 - HopImport - 10 jobs
2021/06/22 16:39:43 - HopImport - 31 transformations
2021/06/22 16:39:43 - HopImport - 43 other files
2021/06/22 16:39:43 - HopImport - 86 variables were imported into environment config file s3:///apache-hop/s3project/imported-env-conf.json
2021/06/22 16:39:43 - HopImport - You can use this as a configuration file in an environment.
2021/06/22 16:39:43 - HopImport - 3 database connections where saved in metadata folder s3:///apache-hop/s3project/metadata
2021/06/22 16:39:43 - HopImport -
2021/06/22 16:39:43 - HopImport - Connections with the same name and different configurations have only been saved once.
2021/06/22 16:39:43 - HopImport - Check the following file for a list of connections that might need extra attention: s3:///apache-hop/s3project/connections.csv

Functionality

In general the plugin copies Kettle files and non-Kettle files.

Kettle files

File Conversion

File	Conversion
Transformations	Kettle transformation files (.ktr extension) are converted to Hop pipelines. During this process the XML metadata is converted to the appropriate Hop format.
Jobs	Kettle job files (.kjb extension) are converted to Hop workflows. During this process the XML metadata is converted to the appropriate Hop format.
kettle.properties	The Kettle properties file in your .kettle folder (typically found in the home directory or `${KETTLE_HOME}`) often contains variables and values regarding your environment. These variables and values are converted into an environment configuration file if you specified the `-c` or `--target-config-file` option. When you create an environment in Hop you can simply add this file to it to make everything work. If the configuration file already exists it will be updated, not overwritten. The description of the newly imported variables is set to `Imported from Kettle` to indicate that it’s new. Values of existing variables are overwritten and the existing description is kept.
shared.xml	The shared.xml file in your .kettle folder (typically found in the home directory or `${KETTLE_HOME}`) often contains connections which are shared across many transformations and jobs. These connections are imported as Relational Database Connection metadata stored in the target folder `metadata/rdbms` folder.
jdbc.properties	A jdbc.properties file sometimes contains relational database metadata which is converted into generic RDBMS connections metadata.

Transformations

Kettle transformation files (.ktr extension) are converted to Hop pipelines. During this process the XML metadata is converted to the appropriate Hop format.

Jobs

Kettle job files (.kjb extension) are converted to Hop workflows. During this process the XML metadata is converted to the appropriate Hop format.

kettle.properties

The Kettle properties file in your .kettle folder (typically found in the home directory or ${KETTLE_HOME}) often contains variables and values regarding your environment. These variables and values are converted into an environment configuration file if you specified the -c or --target-config-file option. When you create an environment in Hop you can simply add this file to it to make everything work. If the configuration file already exists it will be updated, not overwritten. The description of the newly imported variables is set to Imported from Kettle to indicate that it’s new. Values of existing variables are overwritten and the existing description is kept.

shared.xml

The shared.xml file in your .kettle folder (typically found in the home directory or ${KETTLE_HOME}) often contains connections which are shared across many transformations and jobs. These connections are imported as Relational Database Connection metadata stored in the target folder metadata/rdbms folder.

jdbc.properties

A jdbc.properties file sometimes contains relational database metadata which is converted into generic RDBMS connections metadata.

Non-Kettle files

Non-Kettle files like text files, JSON, XML, … they are all copied over by default, including the sub-folders. You can change this behavior with the options:

--skip-folders : don’t copy sub-folders from the input folder
--skip-hidden : Don’t copy hidden files and folders like .git, .gitignore and so on
--skip-existing : Only copy files which haven’t been copied yet