Data Set

dataset

A Data Set is a static pre-defined collection of rows, used in unit testing^, where a data set is used as an alternative input data set. This input set is used to be processed in a unit testing pipeline. After processing the result of the pipeline is compared to a data set that has been specified as the 'golden data set' for the test.

When a data set is attached as input, the marked transform is replaced with an Injector and the static data set rows are injected (so the original transform does not read from a file or a database). When a data set is attached as golden, the marked transform is replaced with a Dummy and the rows flowing into it are compared to the data set (so the original transform does not write to a file or a database).

See Pipeline Unit Testing for details.

Related Plugins

Data set input, Data set output

Options

Option	Description
Name	The name to be used for this data set
Description	The description to be used for this data set
Set Folder (or use HOP_DATASETS_FOLDER)	project folder where data sets can be found and are saved
Base file name	the data set default name
data set fields and their column names in the file	List of field name, type, format, length and precision. Describes the file layout for this data set.

Option

Description

Name

The name to be used for this data set

Description

The description to be used for this data set

Set Folder (or use HOP_DATASETS_FOLDER)

project folder where data sets can be found and are saved

Base file name

the data set default name

data set fields and their column names in the file

List of field name, type, format, length and precision. Describes the file layout for this data set.

New data set

Creating a data set suggests values you can still change. An existing data set is not rewritten.

Name is ds-<pipeline file>-<transform>. <pipeline file> is the file name of the pipeline open in the explorer, without its directory or extension. <transform> is included when exactly one transform is selected. Creating a data set from a transform uses that transform.
Base file name uses the same text with a .csv extension.
Set folder is '${HOP_DATASETS_FOLDER}' when that variable is set, otherwise it stays empty, so no machine-specific path is saved. An empty folder is resolved when the data set is used.
Browse… opens in that folder so you can pick it or a subfolder.

If a data set with the suggested name already exists, the suggestion ends with ` 2`, ` 3`, and so on. The base file name follows that name.

Samples

None