Apache Hop
Download
User manual 2.20.0 (pre-release)

Development Documentation

Edit this page

Spark file input transform Icon Spark file input

Description

Spark file input reads a file, directory or glob with spark.read.format(…​).load(path) on the native Spark pipeline engine and hands the resulting Dataset to the next transform. It is the bulk input of the native Spark engine: Spark reads the files in parallel on the executors, so use it instead of the classic Text File Input, CSV File Input or Parquet File Input transforms for large data.

This transform is not available on the local Hop engine or on the Beam engines.

The path is a Spark/Hadoop FileSystem URI that must be visible to every executor, not a Hop VFS path: a local path on local[], hdfs://… or s3a://… on a cluster. Hop-only schemes such as s3://, azure:// or a named MinIO connection do not work here; see Paths and file systems for the mapping and for the *Path scheme map on the run configuration.

Options

Option Description

Transform name

Unique name of the transform in the pipeline.

Spark path (Hadoop URI)

File, directory or glob to read. Supports variables. Avoid ${PROJECT_HOME} when running with a Spark project package: that variable points at the extracted definition files, not at data. Use a data variable such as HOP_DATA or an explicit s3a:// / hdfs:// URI instead.

Format

csv (default), parquet, json, orc or text. Any other Spark data source name is passed to spark.read.format() as is.

First line is header

CSV and text only. When checked, the first line names the columns and the field list is matched to them by name.

Separator

CSV and text only. Field separator, default ,. Supports variables.

Quote character

CSV and text only. Default ". Supports variables.

Infer schema (CSV without field list)

CSV and text only. Lets Spark run an extra pass over the data to guess column types. Only applied when the field list is empty; an explicit field list is safer and faster for production.

Multi-line

CSV and text only. Allows quoted values to span lines (Spark option multiLine). Disables header-based schema typing, so columns are read as strings and cast afterwards.

Extra options

Additional Spark read options as key=value lines, one per line. They are applied before the options above, so a key you set here wins over the dialog fields (for example header, sep, quote, mode, dateFormat, timestampFormat, recursiveFileLookup).

Fields (optional schema)

The columns to read, with Hop type, length, precision and format mask. See Fields below.

Fields

When the field list is empty every column in the file is passed on with the type Spark gives it (strings for CSV unless Infer schema is checked).

When fields are listed they are a projection: only the named columns are returned, in the listed order, cast to the Hop types you give.

  • CSV/text with a header: columns are matched to the file header by name, case-insensitive, in any order. Numeric and date fields are typed by the CSV reader itself; other columns are read as strings and cast.

  • CSV/text without a header: Spark names the columns _c0, _c1, … and the field list is mapped by position.

  • Parquet, ORC and JSON: the file already carries a schema, so the fields select and cast columns by name.

Date and timestamp fields without a format mask are parsed with a set of common masks (yyyy/MM/dd, yyyy-MM-dd, MM/dd/yyyy, dd/MM/yyyy, yyyyMMdd, and the same with a time part). Set a format mask on the field when your data uses something else.

Delimited files are read in Spark’s PERMISSIVE mode with leading and trailing whitespace ignored, so a value that does not fit its type becomes null instead of failing the job. Set mode=FAILFAST in Extra options if you want the job to fail instead.

Metadata injection

All options support metadata injection.

Field group FIELDS: NAME, TYPE, LENGTH, PRECISION, FORMAT_MASK.

Related pages