Beam Elasticsearch input

Description

A bounded source that reads JSON documents with Beam’s Elasticsearch scroll IO. This transform starts a Beam pipeline without incoming rows and provides a single String column to downstream transforms. It is not an incremental change feed and does not parse document properties into columns; use downstream JSON transforms for that.

Options

All String settings support Hop variables, for example ${ES_HOSTS} and ${ES_PASSWORD}. Variables are resolved while constructing the Beam graph. Empty required settings, unresolved variables and invalid configuration fail before submission; credentials must also be available on workers through the resolved graph.

Option Description

Option	Description
Transform name	Unique name within the pipeline.
Hosts	Required comma-separated HTTP(S) origins, for example `https://first.example:9200,https://second.example:9200`. Paths, URL credentials, query strings and fragments are rejected.
Index	Required index. Input can use an index expression understood by Elasticsearch; output needs a writable target index.
Document type (legacy)	Optional legacy type. See the version-specific limits below.
Username / Password	Optional basic authentication; supply both or neither. Passwords are stored with Hop’s password encryption convention; variable expressions remain expressions. This is not a substitute for restricting pipeline and runner access.
JSON field	Required name of the single output String field, default `json`. Each row contains the document `_source` JSON object, not the hit metadata such as `_id`.
Query JSON	Optional JSON request body, for example `{"query":{"match_all":{}}}`. Blank reads all documents. The body is also sent to the count endpoint by Beam, so it must be accepted by that endpoint. Elasticsearch performs query semantics validation.
Scroll keepalive	Positive scroll context duration, default `5m`; units `ms`, `s`, `m`, `h` and `d` are accepted.

Transform name

Unique name within the pipeline.

Hosts

Required comma-separated HTTP(S) origins, for example https://first.example:9200,https://second.example:9200. Paths, URL credentials, query strings and fragments are rejected.

Index

Required index. Input can use an index expression understood by Elasticsearch; output needs a writable target index.

Document type (legacy)

Optional legacy type. See the version-specific limits below.

Username / Password

Optional basic authentication; supply both or neither. Passwords are stored with Hop’s password encryption convention; variable expressions remain expressions. This is not a substitute for restricting pipeline and runner access.

JSON field

Required name of the single output String field, default json. Each row contains the document _source JSON object, not the hit metadata such as _id.

Query JSON

Optional JSON request body, for example {"query":{"match_all":{}}}. Blank reads all documents. The body is also sent to the count endpoint by Beam, so it must be accepted by that endpoint. Elasticsearch performs query semantics validation.

Scroll keepalive

Positive scroll context duration, default 5m; units ms, s, m, h and d are accepted.

Reading and permissions

The connector needs access to the root version response, index statistics, count (when a query is supplied), search and scroll APIs, including scroll cleanup. Workers must be able to reach the configured hosts. Runner splitting uses index statistics and Elasticsearch’s sliced scroll. In Beam 2.74.0 the scroll request implementation does not apply the SDK withBatchSize() option to the request, so Hop does not expose a misleading read-batch control. There is no point-in-time or continuous polling mode in this transform.

Elasticsearch and OpenSearch compatibility

These transforms use Apache Beam 2.74.0 ElasticsearchIO and its transitive Elasticsearch low-level REST client 7.17.22. The Beam implementation discovers the backend version from GET / and accepts Elasticsearch major versions 5, 6, 7, 8 and 9; support for 5 and 6 is deprecated upstream. This is Beam’s version gate, not a certification of every server version, distribution, security configuration or runner.

For Elasticsearch 5/6 output, supply the legacy document type required by your index. Input may use a type to restrict the search.
For Elasticsearch 7 and later, leave document type blank for typeless endpoints; typed paths are deprecated in 7 and removed in 8 and later.
OpenSearch is not a supported native backend for this connector. Native OpenSearch version responses with major versions 1, 2 or 3 are rejected by Beam’s version detection before reading documents.
A server or proxy that reports an Elasticsearch-compatible version can bypass that check, but this does not establish OpenSearch compatibility. Scroll, index statistics, bulk request/response formats, privileges and authentication must be verified for that exact deployment. Hop does not override or spoof the backend version.
Only basic username/password authentication is exposed. AWS SigV4 signing, API keys, bearer tokens, custom HTTP headers and mutual TLS configuration are not provided by these dialogs.
HTTPS uses the REST client’s normal certificate and hostname verification. There is no trust-all switch. Supply the necessary trusted CA through the worker JVM trust configuration on every worker.

The automated tests exercise the real Beam Direct runner and ElasticsearchIO against an in-process HTTP protocol fixture, including version discovery, sliced scroll requests, bulk payloads and item errors. They do not use a live Elasticsearch/OpenSearch cluster and do not certify those products or remote runners.