Environments

Managing projects and environments

Most developers who design and manage data processing on a daily basis work on a multitude of projects and modules. Different sets of workflows and pipelines require management for at least development, acceptance and production environments. Every project or environment comes with its own set of variables and configurations for databases, file paths, ..

How does Hop manage projects and environments

Hop allows developers to manage different projects and environments with their corresponding configurations and variables. Hierarchically each project can have multiple environments with different purposes. Each project will remember opened files, zoom levels and other UI settings.

Creating a project

To create a new project click the Add a new project button.

Add a new project

This button opens the following dialogue:

Create a project

Table 1. Project Properties
Property	Description	Variables Supported	Mandatory	Default
Name	The project name	Yes	No
Home folder	The folder where the project is located	Yes	No
Configuration file (relative path)	The folder where the project’s configuration json is located.	Yes	Yes
Description	A description for this environment	No	No
Company	The company this environment belongs to	No	No
Department	The department this environment is created for	No	No
Metadata base folder	The folder where this environment’s metadata will be stored	Yes	Yes	${ENVIRONMENT_HOME}/metadata
Unit test base path	The folder where this environment’s unit tests will be stored	Yes	Yes	${ENVIRONMENT_HOME}
Data Sets CSV Folder	The folder where this environment’s data files will be stored	Yes	Yes	${ENVIRONMENT_HOME}/datasets
Enforce execution in environment home	Give an error when trying to execute a pipeline or workflow which is not located in the environment home directory or in a sub-directory	Yes	Yes	checked
Project variables to set	A list of variable names, values and variable descriptions to use with this project	No	Yes

After creating a project the user interface will switch to it and ask if you want to create an environment.

Creating an environment

To create a new environment click the Add a new environment button.

Environment

This button opens the following dialogue:

Create a project

Table 2. Environment Properties
Property	Description	Variables Supported	Mandatory	Default
Name	The environment name	No	No	The last created project
Purpose	The purpose of this environment Development Testing Acceptance Production Continuous integration Common build …	No	No
Project	The project to which this environment belongs	No	No	The last created project
Configuration files	Files to use as configuration for this environment	No	No

After creating an environment the user interface will switch to it.

Switching between projects and environments

To switch between projects the projects list can be used from the main Hop Gui toolbar.

After switching to a project the environments list will be updated to the environments which belong to this project. All open files will be restored, including their zoom level and other UI settings.

Editing and deleting projects

To edit an existing project click the Edit the selected project button.

Edit the selected project

Clicking this button will re-open the project dialogue where changes can be made as described in Project Properties.

To delete a project click the Delete the selected project button.

Delete the selected project

Editing and deleting environments

To edit an existing environment click the Edit the selected environment button.

Edit the selected environment

Clicking this button will re-open the project dialogue where changes can be made as described in Environment Properties.

To delete a project click the Delete the selected environment button.

Delete the selected environment

Advanced project and environment management

Configuration files

Configuration files will be stored in the chosen project home directory as a json file. An environment can be set to point to (multiple) different configuration files as is needed. These config files contain the following:

{
  "metadataBaseFolder" : "${PROJECT_HOME}/metadata",
  "unitTestsBasePath" : "${PROJECT_HOME}",
  "dataSetsCsvFolder" : "${PROJECT_HOME}/datasets",
  "enforcingExecutionInHome" : true,
  "config" : {
    "variables" : [ {
      "name" : "foo"
    } ]
  }
}%

Advanced Hop environment configuration

Hop enviroments and their home folders are stored in the hop configuration file 'hop-config.json'. That file lives by default in the config folder of the Hop installation. System property 'HOP_CONFIG_FOLDER' can also be used to point to a different folder

{
 "environmentConfig" : {
  "enabled" : true,
  "openingLastEnvironmentAtStartup" : true,
  "environmentConfigFilename" : "environment.json",
  "environmentFolders" : {
   "Project 1 - DEV" : "/projects/one/dev/",
   "Project 1 - UAT" : "/projects/one/uat/",
   "Project 1 - PRD" : "/projects/one/prd/",
   "Project 2 - DEV" : "/projects/two/dev/",
   "Project 2 - UAT" : "/projects/two/uat/",
   "Project 2 - PRD" : "/projects/two/prd/",
  }
}

The Hop command line configuration tool

Hop Conf is a command line tool to manage environments. The hop-conf.sh script can be run with the -h flag (./hop-conf.sh -h) to display available options.

The available options are listed below:

Table 3. Hop-conf Options
Short Option	Extended Option	Description
-h	--help	Displays this help message and quits.
-ec	--environment-create	Create an environment. Also specify the name and its home
-ed	--environment-delete	Delete an environment
-el	--environment-list	List the defined environments
-em	--environment-modify	Modify an environment
-pc	--project-create	Create a new project. Also specify the name and its home
-pd	--prject-delete	Delete a project
-pl	--project-list	List the defined projects
-pm	--project-modify	Modify a project
-dv	--describe-variable=<describeVariable>	Describe a variable
-e	-environment=<environmentName>	The name of the environment to manage
-ep	--environment-project=<environmentProject>	The project for the environment
-eu	--environment-purpose=<environmentPurpose>	The purpose of the environment: Development, Testing, Production, CI, …
-fj	--generate-fat-jar=<fatJarFilename>	Specify the filename of the fat jar to generate from your current software installation
-p	--project=<projectName>	The project name
-pa	--project-metadata-base=<projectMetadataBaseFolder>	The metadata base folder (relative to home)
-pb	--project-datasets-base-base=<projectDataSetsCsvFolder>	The data sets CSV folder (relative to home)
-pf	--project-config-file=<projectConfigFile>	The configuration file relative to the home folder. The default value is project-config.json
-ph	--project-home=<projectHome>	The home directory of the project
-pp	--project-company=<projectCompany>	The company
-ps	--project-description=<projectDescription>	The description of the project
-pt	--project-department=<projectDepartment>	The department
-pu	--project-unit-tests-base=<projectUnitTestsBasePath>	The unit tests base folder (relative to home)
-px	--project-enforce-execution=<projectEnforceExecutionInHome>	Validate before execution that a workflow or pipeline islocated in the project home folder or a sub-folder (true/false)
-sv	--set-variable=<setVariable>	Set a variable, use format VAR=Value
-cfg	--config-file=<configFile>	Specify the configuration JSON file to manage
-cfd	--config-file-describe-variables=<configDescribeVariables>[,<configDescribeVariables>…]	A list of variable=description combinations separated by a comma
-cfv	--config-file-set-variables=<configSetVariables> ,<configSetVariables>…]	A list of variable=value combinations separated by a comma
-eg	--environment-config-files=<environmentConfigFiles>[, <environmentConfigFiles>…]	A list of configuration files for this lifecycle environment, comma separated
-pv	--project-variables=<projectVariables>[,<projectVariables>…]	A list of variable=value combinations separated by a comma