AWS, Vault Integration

#aws #vault

Managing secrets securely in cloud pipelines is a common challenge for DevOps teams.

Recently, I worked on automating AWS resource provisioning with Terraform while fetching sensitive data from Vault via GitHub Actions. Along the way, I ran into some interesting pitfalls that are worth sharing for anyone building similar pipelines.

AWS Credential Management

Our workflows involved multiple AWS accounts. The pipeline used the aws-actions / configure-aws-credentials action to assume IAM roles for different accounts.
A few best practices emerged:
Use IAM roles instead of static keys wherever possible.
Always reset AWS environment variables after assuming a role to avoid credential leakage across jobs.
Validate access early with aws sts get-caller-identity to catch misconfigurations.
This ensures Terraform operations like plan and apply run in the correct AWS context without accidentally using wrong credentials.

Vault Integration with GitHub Actions

We used Vault to manage sensitive secrets like API keys and database credentials. GitHub Actions provides OIDC tokens automatically, which Terraform Vault provider can use directly for authentication.

Lessons learned:

Avoid manual JWT fetching: Terraform can automatically fetch OIDC tokens via ACTIONS_ID_TOKEN_REQUEST_TOKEN and ACTIONS_ID_TOKEN_REQUEST_URL.
Match Vault auth methods to CI/CD provider: For GitHub Actions, use auth/github-jwt instead of gitlab-jwt. Using the wrong method causes JWT validation errors.
Network access matters: Even with correct credentials, the pipeline fails if the runner cannot resolve the Vault host. Ensure your GitHub Actions runner has network/DNS access to Vault endpoints.

Terraform Workflow Tips

When working with AWS + Vault + GitHub Actions, a few workflow adjustments make life easier:
Use backend configs for state in S3 rather than hard coding credentials in Terraform.
Separate jobs for validation and plan: This makes debugging easier when multiple AWS accounts and Vault secrets are involved.
Skip Vault temporarily for validation: Useful when testing network or AWS configurations before the Vault integration is ready.

Common Pitfalls

Some errors we ran into:
dial tcp: lookup review.vault.internal… no such host → Runner couldn’t reach internal Vault host.
error validating token: error verifying token signature… 404 Not Found → Vault auth method mismatched the CI/CD provider.
Both issues are easy to overlook but critical for a smooth pipeline.

Key Takeaways

Network first, credentials second: Ensure CI/CD runners can reach secret management endpoints.
Use built-in OIDC wherever possible: Reduces manual token handling and failure points.
Verify AWS role assumptions early: Misconfigured roles can silently cause Terraform to operate in the wrong account.
Keep workflows modular: Separate validation, planning, and secret fetching to isolate issues quickly.
Integrating AWS, Terraform, and Vault in CI/CD pipelines can be tricky, but following these practices makes it predictable, secure, and maintainable.