From Vulnerabilities to Vault: How We Stopped Hardcoding Secrets and Started Using Hashicorp Vault

We recently migrated our infrastructure from Kubernetes to HashiCorp Nomad.

Soon after, we encountered service discovery issues and integrated Consul to address them.

At this point, we were feeling a bit more relaxed, knowing that we could dedicate less time to infrastructure and focus more on product development, as we don’t have a separate resource dedicated to DevOps.

However, we encountered another challenge—a Nomad port was open on the Node, and a friendly person helped us identify this vulnerability.

They were able to access it directly using the server's IP address, without needing a domain.

As a result, we had to quickly rotate all our payment keys, AWS credentials, and other secrets.

The Problem: Hardcoded Secrets and Security Risks

In some areas, our team had hardcoded keys directly into the codebase.

This meant there wasn’t a single place to manage these secrets, and updating them across our systems was a manual, error-prone process.

It was clear we needed a solution that would allow us to change a secret in one place, and have it propagate automatically throughout the system.

Initially, we considered using AWS Secrets Manager alongside our GitLab CI/CD pipeline to inject environment variables during container deployments.

However, this approach required passing a large number of variables as arguments to nomad job run from the project's .gitlab-ci.yml, which cluttered our configuration and reduced system readability.

That’s when we thought, Why not go full HashiStack? We decided to explore Vault.

Exploring Secret Management Options

AWS Secrets Manager

AWS Secrets Manager offers capabilities for managing and rotating secrets such as database credentials, API keys, and other sensitive information.

• Access Management: AWS Secrets Manager relies on AWS IAM (Identity and Access Management) for managing permissions. For granular control, additional tools like Chamber maybe required. Chamber leverages AWS IAM roles and policies to restrict access by namespace, simplifying the process of partitioning secrets across different environments.
• Cross-Cloud Limitations: While effective within AWS (our existing infra), use in multi-cloud environments.
• Cost of service: $0.40 per secret per month.

HashiCorp Vault

Hashicorp Vault provides a robust, identity-based security solution designed to authenticate and authorize access to secrets automatically, supporting a wide range of infrastructures.

• Access Control: Vault offers RBAC (Role-Based Access Control) for user logins, enabling administrators to manage human access to secrets. For machine access, Vault provides policy-driven access with leases, allowing for precise control over which applications can access secrets and for how long. This includes options like automatic key rotation and expiration.
• Dynamic Secrets Generation: Vault can dynamically generate AWS IAM credentials, providing temporary access that’s tightly controlled by policies and easily revoked if needed.
• Cross-Environment Compatibility: Vault supports cross-region, cross-cloud, and cross-datacenter replication, making it a seamless choice for organizations with multi-cloud setups.
• Cost of service: Free.

Why We Chose HashiCorp Vault

Vault offered the ideal solution: it allowed Nomad to request sensitive information, such as payment secrets or AWS keys, directly from Vault.

Vault would verify if Nomad was authorized to access those secrets and promptly deliver the data.

This way, we could centralize key management, automate secret updates, and improve security by avoiding hardcoded keys.

We developed a proof of concept, where we demonstrated Vault's ability to manage key rotation and securely store secrets in one place, accessible across all our projects.

This not only simplified secret management but also ensured consistency and security for our entire infrastructure stack.

Continue reading full article...

Comments

[Anis Mer_]

I use Terraform extensively and rely on AWS Secrets Manager, but I’ve noticed that their billing can be misleading. For example, I have nearly 35 secrets stored on AWS, yet my bill for October shows the following breakdown:

---

Secrets Manager Billing (October 1 - October 31)

Total Charge: $0.54

Region: US East (N. Virginia)

Breakdown

1. Secrets Stored:

   • Charge: $0.53
   • Rate: $0.40 per secret/month
   • Usage: 1.319 secret-months

2. API Requests:

   • Charge: $0.01
   • Rate: $0.05 per 10,000 API requests
   • Usage: 1,581 API requests

---

Based on AWS documentation and my research, there are several factors AWS considers when calculating charges:

• Prorated Billing: Secrets are billed hourly, so if secrets were added, rotated, or deleted mid-month, the charges are adjusted accordingly.
• Active Secrets Only: AWS might only charge for secrets that were active and accessed during the billing cycle.
• Versioning: If you’re using secret rotation, AWS could charge based on the number of versions stored.

Even with these factors in mind, the discrepancy between my 35 secrets and the reported 1.319 secret-months usage still feels unclear. AWS doesn’t make the billing logic transparent, which complicates cost management and can deter usage.

For several months now, I haven’t paid more than 60 cents for nearly 35 secrets.

Have others faced similar confusion with Secrets Manager billing? I’d love to hear how others interpret or manage these charges.