Beam Snowflake output

Description

Write one bounded Hop collection to a Snowflake table with Apache Beam SnowflakeIO 2.74.0. Hop formats each row as CSV values, SnowflakeIO stages those files in GCS, and Snowflake copies them into the table. Connect exactly one incoming transform. This is a terminal sink; it does not produce downstream rows.

Snowpipe is not used. An unbounded collection is rejected. Declared Beam compatibility is not a remote-runner certification. No test in this release runs a pipeline against Snowflake or GCS.

Options

Option Description

Option	Description
Transform name	Unique name in the pipeline.
Server	Required account host. It must look like `account.snowflakecomputing.com`. The value is not repeated in an error.
Username	Required Snowflake user.
Password	Used when no private key is set. Masked in the dialog and encrypted with Hop password conventions. Prefer a variable such as `${SNOWFLAKE_PASSWORD}`.
Private key	PEM private key. When this is set, Hop uses key-pair authentication and does not send the password. A passphrase is optional. Key files and OAuth are not offered.
Database	Optional database identifier.
Warehouse	Optional warehouse identifier.
Schema	Optional schema identifier.
Role	Optional role identifier.
Port	Optional port from 1 to 65535. Empty uses the driver default.
Staging bucket	Required `gs://` path ending with `/`. S3, Azure, and internal stages are not accepted.
Storage integration	Required Snowflake storage integration that can write the bucket.
Quotation mark	Optional single CSV quotation character. Empty uses the Beam default apostrophe. A backslash or whitespace is rejected.
Table	Required destination, as name, `schema.name`, or `database.schema.name`.
Write disposition	`APPEND` (default), `TRUNCATE`, or `EMPTY`.
Create disposition	`CREATE_NEVER` (default) or `CREATE_IF_NEEDED`. Incoming fields are checked either way. A Snowflake column list is sent only when the table may be created. Columns are nullable.

Transform name

Unique name in the pipeline.

Server

Required account host. It must look like account.snowflakecomputing.com. The value is not repeated in an error.

Username

Required Snowflake user.

Password

Used when no private key is set. Masked in the dialog and encrypted with Hop password conventions. Prefer a variable such as ${SNOWFLAKE_PASSWORD}.

Private key

PEM private key. When this is set, Hop uses key-pair authentication and does not send the password. A passphrase is optional. Key files and OAuth are not offered.

Database

Optional database identifier.

Warehouse

Optional warehouse identifier.

Schema

Optional schema identifier.

Role

Optional role identifier.

Port

Optional port from 1 to 65535. Empty uses the driver default.

Staging bucket

Required gs:// path ending with /. S3, Azure, and internal stages are not accepted.

Storage integration

Required Snowflake storage integration that can write the bucket.

Quotation mark

Optional single CSV quotation character. Empty uses the Beam default apostrophe. A backslash or whitespace is rejected.

Table

Required destination, as name, schema.name, or database.schema.name.

Write disposition

APPEND (default), TRUNCATE, or EMPTY.

Create disposition

CREATE_NEVER (default) or CREATE_IF_NEEDED. Incoming fields are checked either way. A Snowflake column list is sent only when the table may be created. Columns are nullable.

The column types follow the incoming row. String becomes VARCHAR (with its length when set). Integer becomes NUMBER(38,0), which is the SQL Beam emits for an integer. Number becomes FLOAT, which is an IEEE double and is not a lossless decimal. BigNumber becomes NUMBER, using the Hop length as precision and the Hop precision as scale when the length is set. Boolean becomes BOOLEAN. Date and Timestamp become TIMESTAMP_NTZ in UTC yyyy-MM-dd HH:mm:ss.SSS. Binary becomes BINARY hex. Other Hop types are rejected.

Execution and security

Hop’s lines-read and lines-output counts are conversion: a row was read, or it was formatted for SnowflakeIO. Lines written stays empty. A formatted row is not a confirmed COPY. The transform does not add a TLS switch. The Snowflake JDBC driver negotiates the connection.

Passwords, private keys, passphrases, bucket names, and cell values are not copied into Hop error text. A conversion error names the field only.

Limitations

GCS external stage only. No S3, Azure, or internal stage.
No Snowpipe and no streaming write. The input collection must be bounded.
Number values can lose precision. Use BigNumber for a decimal that must round-trip as text.
There is no controlled Snowflake or GCS round trip in the tests. Remote runners have not been certified.
A second connector review has not covered this transform.