Beam Elasticsearch output
Description
A sink that writes JSON documents with Beam’s Elasticsearch bulk IO. It requires incoming rows and has no downstream row output. It is available only in Beam run configurations, not in the local Hop engine. The sink does not create mappings or manage index lifecycle. Ensure the target index and privileges are appropriate before running.
Options
All String settings support Hop variables, for example ${ES_HOSTS} and ${ES_PASSWORD}.
Variables are resolved while constructing the Beam graph. Empty required settings, unresolved variables and invalid configuration fail before submission; credentials must also be available on workers through the resolved graph.
| Option | Description |
|---|---|
Transform name |
Unique name within the pipeline. |
Hosts |
Required comma-separated HTTP(S) origins, for example |
Index |
Required index. Input can use an index expression understood by Elasticsearch; output needs a writable target index. |
Document type (legacy) |
Optional legacy type. See the version-specific limits below. |
Username / Password |
Optional basic authentication; supply both or neither. Passwords are stored with Hop’s password encryption convention; variable expressions remain expressions. This is not a substitute for restricting pipeline and runner access. |
JSON field |
Required incoming String field containing one JSON object per row, default |
Maximum documents per batch |
Positive maximum bulk batch document count, default |
Maximum batch bytes |
Positive bulk batching byte threshold, default |
Delivery and errors
Beam generates the bulk index directives; the payload field is the document, not a user-supplied bulk directive. Document IDs are generated by Elasticsearch. There is no explicit ID, upsert, delete or exactly-once setting in this transform. Runner retries or reruns can create duplicates, so account for that in your pipeline design. Bulk item failures fail the pipeline rather than silently discarding records. A failure can occur after other documents have already been persisted; there is no transaction rollback. The Hop written metric counts successful bulk responses. Read/output metrics count rows extracted and submitted to Beam IO, not an exactly-once delivery guarantee.
Elasticsearch and OpenSearch compatibility
These transforms use Apache Beam 2.74.0 ElasticsearchIO and its transitive Elasticsearch low-level REST client 7.17.22.
The Beam implementation discovers the backend version from GET / and accepts Elasticsearch major versions 5, 6, 7, 8 and 9; support for 5 and 6 is deprecated upstream.
This is Beam’s version gate, not a certification of every server version, distribution, security configuration or runner.
-
For Elasticsearch 5/6 output, supply the legacy document type required by your index. Input may use a type to restrict the search.
-
For Elasticsearch 7 and later, leave document type blank for typeless endpoints; typed paths are deprecated in 7 and removed in 8 and later.
-
OpenSearch is not a supported native backend for this connector. Native OpenSearch version responses with major versions 1, 2 or 3 are rejected by Beam’s version detection before reading documents.
-
A server or proxy that reports an Elasticsearch-compatible version can bypass that check, but this does not establish OpenSearch compatibility. Scroll, index statistics, bulk request/response formats, privileges and authentication must be verified for that exact deployment. Hop does not override or spoof the backend version.
-
Only basic username/password authentication is exposed. AWS SigV4 signing, API keys, bearer tokens, custom HTTP headers and mutual TLS configuration are not provided by these dialogs.
-
HTTPS uses the REST client’s normal certificate and hostname verification. There is no trust-all switch. Supply the necessary trusted CA through the worker JVM trust configuration on every worker.
The automated tests exercise the real Beam Direct runner and ElasticsearchIO against an in-process HTTP protocol fixture, including version discovery, sliced scroll requests, bulk payloads and item errors. They do not use a live Elasticsearch/OpenSearch cluster and do not certify those products or remote runners.