Beam Kinesis Consume

Description

The Beam Kinesis Consume transform consumes (reads) messages from an Amazon Kinesis stream using the Beam pipeline execution engine.

Supported Engines

Hop Engine

Not Supported

Spark

Supported

Flink

Supported

Dataflow

Supported

Limitations

The main limitation of the Kinesis consumer is that it currently only supports reading String messages. Please create a request in GitHub if you want to see support for Avro messages and other formats.

Options

Option	Description
Transform name	Name of the transform, this name has to be unique in a single pipeline.
AWS access key	The Amazon Web Services access key to authenticate with.
AWS secret key	The Amazon Web Services secret key to authenticate with.
Stream name	The name of the Kinesis stream to consume from.
The name of the unique ID output field	This is the name of the field in the output of the transform which will contain a unique ID for the data (message) received.
The name of the data output field	This is the name of the field that will contain the actual data (message)
The type of data retrieved	Only String data (messages) are supported for now.
Partition key field name	Optional: the name of the field in the output which will contain the partition key
Sequence number field name	Optional: the name of the field in the output which will contain the sequence number of the message
Sub-sequence number field name	Optional: the name of the field in the output which will contain the sub-sequence number of the message
Shard ID field name	Optional: the name of the field in the output which will contain the Shard ID of the message
Stream name field name	Optional: the name of the field in the output which will contain the name of the stream being read
Maximum number of records	Optional: the maximum number of records this transform will read before stopping
Maximum read time (ms)	Optional: the maximum time in ms this transform will read before stopping
Request record limit	Optional: the number of records which will be read at once in every request (micro-batch)
Arrival time watermark policy?	Watermark messages at the time of arrival in the Beam pipeline.
Arrival watermark idle time (ms)	Denotes the duration for which the watermark can be idle
Processing time watermark?	Watermark message at the time of processing in the Beam pipeline.
Fixed delay rate limit policy?	Specifies a fixed delay rate limit policy with the default delay of 1s
Fixed delay rate limit policy delay (ms)	Specifies a fixed delay rate limit policy with the given delay.
Maximum capacity per shard	Specifies the maximum number of messages per one shard.

Option

Description

Transform name

Name of the transform, this name has to be unique in a single pipeline.

AWS access key

The Amazon Web Services access key to authenticate with.

AWS secret key

The Amazon Web Services secret key to authenticate with.

Stream name

The name of the Kinesis stream to consume from.

The name of the unique ID output field

This is the name of the field in the output of the transform which will contain a unique ID for the data (message) received.

The name of the data output field

This is the name of the field that will contain the actual data (message)

The type of data retrieved

Only String data (messages) are supported for now.

Partition key field name

Optional: the name of the field in the output which will contain the partition key

Sequence number field name

Optional: the name of the field in the output which will contain the sequence number of the message

Sub-sequence number field name

Optional: the name of the field in the output which will contain the sub-sequence number of the message

Shard ID field name

Optional: the name of the field in the output which will contain the Shard ID of the message

Stream name field name

Optional: the name of the field in the output which will contain the name of the stream being read

Maximum number of records

Optional: the maximum number of records this transform will read before stopping

Maximum read time (ms)

Optional: the maximum time in ms this transform will read before stopping

Request record limit

Optional: the number of records which will be read at once in every request (micro-batch)

Arrival time watermark policy?

Watermark messages at the time of arrival in the Beam pipeline.

Arrival watermark idle time (ms)

Denotes the duration for which the watermark can be idle

Processing time watermark?

Watermark message at the time of processing in the Beam pipeline.

Fixed delay rate limit policy?

Specifies a fixed delay rate limit policy with the default delay of 1s

Fixed delay rate limit policy delay (ms)

Specifies a fixed delay rate limit policy with the given delay.

Maximum capacity per shard

Specifies the maximum number of messages per one shard.