AWS SQS Reader AWS SQS Reader

Description

The AWS SQS Reader transform enables you to receive messages from Amazon Web Services Simple Queue Service within an Apache Hop pipeline.

Supported Engines

Hop Engine

Supported

Spark

Supported

Flink

Supported

Dataflow

Supported

Preconditions

Before the first execution you need to create an IAM-Role (e.g. for usage on EC2/ECS) or an IAM-User with an AWS Key and Secret and attach the required policies for pushing notifications via SNS.

You also need to create one ore more subscription topics you want to push the messages to.

Options

AWS Settings tab

option description

Use AWS Credentials chain

Apache Hop tries to pick up the AWS credentials from the host environment. For more information, check the Credentials docs.

AWS Access Key

your AWS Access Key (AWS_ACCESS_KEY_ID)

AWS Secret Access Key

the secret for your AWS Access Key (AWS_SECRET_ACCESS_KEY)

AWS Region

the AWS-Region the service is running in.

SQS Queueu URL

the URL of the SQS Queue (starting with https:// - not the ARN!)

Output definition

On the Output Definition Tab you can define the output fields for the informations read from the SQS message as well as some initial settings for receiving the messages.

Output Settings

Option Description

Delete Message(s)

delete the message(s) be from the queue after receiving?

Maximum messages to retrieve

finish after reaching this maximum number of messages. Zero (0) will receive all messages in the queue.

Output Settings

Option Description

MessageID

For each message an unique ID is retrieved from SQS. This can be written to an output field defined here.

MessageBody

Complete content of the message.

ReceiptHandle

Unique identifier for receiving a message.

MD5 Hash

MD5 hash of the message body

Message Node from SNS

if the message was sent from SNS this field will contain the content of the JSON node "Message" within the Message body.

Metadata Injection Support

All fields of this transform support metadata injection.