AWS Secrets Manager Variable Resolver

Description

The AWS Secrets Manager Variable Resolver looks up secrets in AWS Secrets Manager and makes them available as variables in your Apache Hop pipelines and workflows. Database passwords, API keys and other credentials stay in AWS and out of your project files.

Unlike Azure Key Vault or Hashicorp Vault there is nothing to install or provision: Secrets Manager is available in every region of every AWS account. You only need a secret and an identity that is allowed to read it.

Options

The editor only shows the options that apply to the authentication type you picked, so switching type changes which fields are on screen. Every option accepts variables, which is what lets one resolver serve several environments.

AWS region

The region that holds your secrets. The dropdown lists each region with the name AWS gives it, for example eu-west-1 - Europe (Ireland), but only the region code is stored and used, so the metadata file simply reads eu-west-1. You can also type a code yourself, or set a variable such as ${AWS_REGION}.

Secrets are regional and are not replicated between regions unless you explicitly configure replication, so this has to match the region the secret was created in. Leave the option empty to let Hop pick up the region the way the AWS CLI does, from the AWS_REGION environment variable or the active profile.

Authentication type

How Hop proves to AWS who it is. The three types have no options in common, so the editor shows only the fields belonging to the one you pick.

AUTOMATIC

Let the AWS SDK find credentials by itself. This is the default, and the one to prefer.

The SDK works through its standard provider chain and takes the first credentials it finds: Java system properties, then environment variables (AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY), then a web identity token (how EKS IRSA works), then the shared credentials file, then container credentials (the ECS task role), and finally the EC2 instance profile.

On EC2, ECS or EKS this means the surrounding infrastructure supplies a role automatically and no credential is stored in the Hop project at all. It also picks up whatever you already use locally, so if aws sts get-caller-identity works in the shell you start Hop from, this option works too.

Needs no further options.

ACCESS_KEYS

Use an access key and secret key you enter yourself, with an optional session token.

Use this when Hop runs outside AWS and there is no profile to read, or when you deliberately want a separate identity for secret lookups. Uses the Access key, Secret key and Session token options.

CREDENTIALS_FILE

Read credentials from an AWS credentials file, optionally choosing a profile from it.

Use this when you already keep several AWS identities side by side and want this resolver to use a specific one rather than whichever the chain happens to find first. Uses the Credentials file and Profile name options.

Prefer AUTOMATIC wherever the runtime environment can provide credentials. Long lived access keys stored in a project are exactly the kind of thing Secrets Manager exists to get rid of, and AUTOMATIC keeps the Hop project identical between a laptop, a server and a container.

Access key / Secret key / Session token

The AWS access key ID and secret access key, used only with authentication type ACCESS_KEYS.

The session token is only needed for temporary credentials, the kind handed out by sts assume-role or by AWS SSO. Leave it empty for a normal long lived key pair. Temporary credentials expire, so a resolver configured with them stops working when they do.

All three values are stored encrypted in the metadata file.

Credentials file / Profile name

Used with authentication type CREDENTIALS_FILE.

The credentials file is a file in the format the AWS CLI writes, holding one or more named profiles. Leave it empty to use the standard location, ~/.aws/credentials (%USERPROFILE%\.aws\credentials on Windows).

The profile name picks which entry in that file to use, and defaults to default.

Endpoint override

An alternative Secrets Manager endpoint. Use this for a VPC endpoint, or to point Hop at a local AWS emulator during development. Leave it empty to use the regular regional AWS endpoint.

Secret name prefix

A prefix placed in front of every secret name before the lookup. With a prefix of production/, the expression #{aws-secrets:database} reads the secret production/database.

This keeps expressions short and lets the same pipeline read from development/ or production/ by pointing the resolver at a different prefix per environment. The prefix itself may contain variables, for example ${ENVIRONMENT}/.

Version stage

Which version of the secret to read. AWSCURRENT by default, which is what you want unless you have a specific reason not to.

Secrets Manager does not overwrite a secret when its value changes: it stores a new version and keeps the old ones. A staging label is a moving pointer to one of those versions, so that something reading "the current value" does not need to know any version IDs. AWS maintains three labels for you:

AWSCURRENT

The value in use right now. This is what an application should read, and what this resolver reads unless you say otherwise.

AWSPREVIOUS

The value that was current before the most recent change.

AWSPENDING

A new value created by a rotation that has not been promoted yet. It exists only while a rotation is in progress.

When a secret rotates, AWS creates the new version with AWSPENDING, tests it, and then moves AWSCURRENT onto it while the version it replaced picks up AWSPREVIOUS. Because the labels move rather than the values, an expression reading AWSCURRENT keeps working across a rotation without any change in Hop.

Reasons to change this option are rare and usually temporary:

  • A rotation broke something and you need the previous credential back while you sort it out: AWSPREVIOUS.

  • You are testing a rotation before it is promoted: AWSPENDING.

Do not leave the option on AWSPREVIOUS. It is a pointer, not a snapshot: the next rotation moves it to what is current today, so a resolver left pointing at it silently starts serving a different secret.

Cache TTL (seconds)

How long a retrieved secret is reused before Hop asks AWS for it again. 0, the default, disables caching and looks the secret up every time.

Secrets Manager bills per API call and Hop resolves variables often, so a small value such as 60 can noticeably reduce both cost and latency. The trade-off is that a rotated secret takes up to this long to be picked up, so keep the value well below your rotation window.

Expression syntax

Secrets are addressed with the standard variable resolver syntax:

#{name:secret-name:value-key}

  • name: the name of the variable resolver metadata element

  • secret-name: the name of the secret in AWS Secrets Manager

  • value-key: when the secret holds JSON, the key to pick out of it (OPTIONAL)

A secret can hold a plain value or JSON: SecretString is free-form text and AWS makes no distinction. The console nudges you towards JSON because its create-secret screen opens on a Key/value tab, but the Plaintext tab next to it stores a bare value just as well.

For a plain secret, two parts are enough:

  • #{aws-secrets:my-api-key} : the value as stored

For a JSON secret, the third part picks one value out of it. Given a secret called some-db holding:

{"dbname":"test","host":"localhost","password":"some-password","port":3306,"username":"john"}

a resolver named aws-secrets gives you:

  • #{aws-secrets:some-db:host} : localhost

  • #{aws-secrets:some-db:username} : john

  • #{aws-secrets:some-db} : the entire JSON document

Keys in database credential secrets

Secrets created with one of the console’s database types (RDS, DocumentDB, Redshift) always use the same fixed key names, because the AWS rotation functions expect them:

username, password, engine, host, port, dbname, dbInstanceIdentifier

Note host, not hostname. Asking for a key that is not in the secret yields an empty value rather than an error, so if a lookup comes back empty, resolve the secret without the third part first and check the key names in the JSON you get back.

Use the name of the secret, not its ARN. Resolver expressions are split on the : character and an ARN contains colons of its own, so #{aws-secrets:arn:aws:secretsmanager:…​} cannot work. When your secret names share a common path, put that path in the Secret name prefix option instead of typing it in every expression.

Setting up AWS

Create a secret

# A plain value, resolved with a two part expression
aws secretsmanager create-secret \
  --name production/api-key \
  --secret-string "my-api-key"

# A JSON secret, so a single value can be picked out of it
aws secretsmanager create-secret \
  --name production/some-db \
  --secret-string '{"hostname":"localhost","port":"3306","username":"john","password":"some-password"}'

Grant read access

The identity Hop runs as needs secretsmanager:GetSecretValue on the secrets it reads:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "secretsmanager:GetSecretValue",
      "Resource": "arn:aws:secretsmanager:eu-west-1:123456789012:secret:production/*"
    }
  ]
}

Secrets encrypted with a customer managed KMS key also need kms:Decrypt on that key. Secrets using the AWS managed aws/secretsmanager key need no extra permission.

Usage

In a database connection

  1. Create or edit a relational database connection

  2. Set the Password field to #{aws-secrets:production/some-db:password}

  3. The password is retrieved from Secrets Manager when the connection is used

Per environment

Set a variable in each of your environments and let it point at the resolver expression in production only:

  • development: PASSWORD = dev-password

  • production: PASSWORD = #{aws-secrets:production/some-db:password}

Pipelines and workflows simply use ${PASSWORD} and need no changes between environments.

Troubleshooting

Errors are not shown in the user interface while you type an expression, since that would make expressions impossible to enter. Check the log in the Hop GUI execution pane to see what the resolver reports.

Common causes:

  • An empty value came back: the lookup worked but the JSON key does not exist. Resolve the secret without the third part to see the real key names.

  • Secret not found: the secret does not exist in the configured region, or the name prefix is missing. Remember that secrets are regional.

  • An ARN was used instead of a name: see the note in the expression syntax section above.

  • Access denied: the identity is missing secretsmanager:GetSecretValue, or kms:Decrypt for a customer managed key.

  • No region: no region is configured on the resolver and none could be found in the environment.

A secret holding a binary value is returned base64 encoded, since a variable value has to be text.