Project Metadata

A project’s metadata typically contains a number of database connections (relational or NoSQL), logging configurations and so on. This metadata is stored on a project level and is available through the metadata perspective.

The way metadata typically is handled is:

  • metadata objects (connections, logging, …​) are defined on a project level. E.g. a CRM database connection is created in the project, using a number of variables for hostname (${CRM_HOST}) and database name (${CRM_DBNAME}), with corresponding variables for the port, username, password and so on.

  • the metadata object configuration is defined on an environment level, where a number of variable values are created the contain the ip address for ${CRM_HOST} etc.

A project’s metadata is stored in ${PROJECT_HOME}/metadata, but can be overruled by setting the project’s metadata base folder ${HOP_METADATA_FOLDER} property in the project configuration dialog or directly in the project-config.json file.

Auto-export metadata to a single JSON file

Some integrations (for example Apache Airflow calling Hop Server, or Beam-style packaging) need the project metadata as a single JSON file rather than the per-type folder tree under metadata/.

You can already create that file manually:

  • GUI: Tools → Export metadata to JSON (Beam plugin)

  • CLI: hop-conf --project=<name> --export-metadata=<path/to/metadata.json>

Optionally, a project can keep that file up to date automatically. In the project properties dialog enable Auto-export metadata and set a filename (default metadata.json, relative to the project home). When enabled, Hop rewrites the file when:

  • the project is enabled / switched to, and

  • a metadata object is created, updated, or deleted in the GUI.

The setting is stored in project-config.json:

{
  "autoExportMetadata" : true,
  "autoExportMetadataFilename" : "metadata.json"
}

Read-only projects never write the export file. Export failures are logged and do not block saving metadata.

A basic project metadata folder will look similar to the one below:

.
├── cassandra-connection
├── dataset
├── file-definition
├── git
├── mongodb-connection
├── neo4j-connection
├── neo4j-graph-model
├── partition
├── pipeline-log
├── pipeline-probe
├── pipeline-run-configuration
│   ├── local.json
│   └── Spark.json
├── rdbms
│   └── crm.json
├── schema-definition
├── server
├── splunk
├── unit-test
├── web-service
├── workflow-log
└── workflow-run-configuration
    └── local.json
18 directories, 4 files