From 3rd party API keys, to database configs and other secrets alike, a need often arises for secrets management and security of these credentials.
Without the right tools and knowledge, developers usually fail to keep the secrets safe.
In this article, we'll briefly explore the methods for secrets management in serverless applications that run on AWS.
In the end, you’ll get to learn the best tool/platform available for managing secrets – Doppler.
If you are a developer of serverless applications, then I bet you are already handling stuff like database connection strings, API tokens, passwords, and many other private keys.
These are secrets that need to be managed separately to keep your applications more secure and scalable.
The best practice to manage secrets is never to store them within your codebase (hardcoding). Neither should you commit them directly to a shared git repository.
Well, if you're into building scalable and secure serverless applications with AWS, taking care of the secrets may be a difficult task. So here are some of the ways to provide security for your functions or applications.
Lambda environment variables.
It is a common practice (not recommended, though) to have secret keys stored in an environment variable that is managed outside a source code or version control. The secrets in the environment variables can then be deployed to your lambda functions at runtime securely.
An optimal secret management solution for AWS serverless apps should provide a seamless secret lifecycle, fine-grained access control, scalability, and encryption for secrets in transit and at rest.
Let's see some of the standard options you can get started with as offered by AWS.
With just a simple call to AWS secrets manager API, developers can manage numerous API keys, database URLs, passwords, and other secrets that power serverless applications.
AWS Secrets manager features integrations with other AWS services like databases to provide seamless management and retrieval of secrets from a central point.
AWS secrets manager also features privilege level and policy-based access (fine-grained access control) for the secrets sent and retrieved by lambda functions. Additionally, the secrets stored are encrypted with the AWS KMS.
Secrets in serverless apps are kept secure by fetching them from the secrets manager at runtime and storing them in local variables while they are in use for every session.
AWS Secrets manager uses the lambda rotation function to automatically rotate and update credentials throughout their lifecycle to prevent secret leakages or consequences.
AWS Parameter store provides a central storage point for secrets and config data, usually key-value.
Secrets are stored as parameters where the name and value are supplied as details – (Key-Value pair).
SSM provides the option of supplying the secret values as simple strings (unencrypted) and Secure strings (KMS encrypted).
Below is a simple workflow of storing serverless secrets in parameter store:
• Creating a new parameter.
• Specifying the parameter type and the values to be stored.
• Referencing the parameters into your lambda functions.
AWS Secrets Manager vs. AWS Parameter Store (SSM)
Several factors come into play when choosing an option between Secrets Manager and SSM to manage your secrets.
While Secret Manager is explicitly designed for Secrets management, Parameter Store is designed to cater for a more comprehensive use case – ( secrets, configuration variables, and other custom settings required in applications development).
Secrets manager, therefore, comes out as a suitable management solution compared to SSM.
AWS Parameter Store, however, is flexible in terms of cost and comes with no additional charges for standard storage and throughput.
On the other hand, the AWS Secrets manager charges every secret stored/every 10000 API calls - often a concern when several secrets are stored and accessed.
Generally, Secrets Manager has the edge over SSM regarding secrets rotation, cross-account access, and multi-region replication for all your secrets.
The decision to manage your secrets using the native AWS secrets management offerings comes with challenges and drawbacks that hamper developers' productivity. Some of the limitations on secrets management using AWS services include:
o Restrictions on names and namespaces for your secrets
o Complex CLI and dashboard for managing secrets
o Rate quotas
o Maximum quotas.
I bet you are now wondering, is there a solution where it's easy to manage secrets but still share secrets with AWS?
Yes, the solution is Doppler, an amazing developer experience platform for managing your secrets.
In this section, you’ll learn how to manage secrets using Doppler, a Universal Secrets Manager, through the Doppler dashboard or Doppler CLI.
Doppler provides a single dashboard environment for managing secrets to eliminate the need for .env files, hard coding secrets, or using complex management vaults.
With Doppler's single point of reference, it is possible for teams to centrally manage app configurations, API keys, and database URLs secrets for any application, platform, and cloud provider.
The Doppler CLI is a lightweight binary that is installed in commonly used operating systems through package managers. (Scoop for this case).
Doppler CLI supports local development, CI/CD, staging, and production environments.
Running the script below in a command prompt will install Doppler CLI in the Windows platform.
Add Doppler's scoop repo
scoop bucket add doppler https://github.com/DopplerHQ/scoop-doppler.git
Install latest Doppler CLI
scoop install doppler
Once Doppler CLI is installed, an access token is needed for authentication to complete the local setup. To initiate the authentication, run doppler login within your code editor terminal or cmd to prompt authorization where you can supply account password and authentication code,
The authentication will give Doppler CLI access to secrets within your projects once configured for use with a project.
Doppler is a browser-based interactive platform that organizes secrets in projects and environments. The dashboard provides quick-start options for managing secrets - having fine-grained access control, logging, versioning, and scaled integrations with other platforms.
Like Doppler CLI, the Doppler Dashboard has access to secrets in every environment, keeping devices, projects, and teams in sync.
Developers are at liberty to create as many projects as they want -considering the teams involved and the application architectures being used.
The projects can also have many environments – Dev, Staging, CI/CD, Production, and more that are scaled easily.
To create a new project, head over to the Doppler dashboard and click on the + button next to the Projects workspace.
Any created project comes with three default environments for defining configuration at the root level.
Developers can customize their project environments to have configs responsible for vaulting app config and secrets such as API keys, database URLs, certs, etc…
Within your created project, you can supply secrets singly or import secrets - creating bulk secrets.
The add secrets option injects a secret key-value into Doppler once the secret name and value are saved.
Once you’ve supplied your secrets as required, you can set up your project with the local development environment using the Doppler CLI.
Running the doppler setup command locally within the project directory fetches your doppler projects and configs to sync your environments.
Dopplers offers automatic syncing of secrets with AWS services such as AWS Lambda, AWS Secrets Manager, and AWS Parameter Store.
Integrating these services requires a privileged account with AWS- (IAM user) and an existing Doppler project.
As a privileged user, you'll also need Access Key ID and Secret Access Key to get started with the integration.
As an example, we'll integrate our project in the dashboard with AWS Secrets Manager.
First, we’ll head over to the dashboard and select our project.
Next, we'll click on the integrations to display a list of Cloud Providers and other services.
Selecting AWS Secrets Managers prompts the authorization page to supply the Access Key ID and Secret Access Key.
After the authentication, we proceed to set up integration by selecting our project environment and the project itself. We also select the region for our integration.
To complete the integration, we click Set Up Integration to let Doppler sync the secrets in our test project to AWS Secrets Manager instantly.
Now every time, we’ll be working with Doppler and integrated AWS Secrets Manager, Doppler will automatically push secrets to the AWS Secrets Manager.
Doppler provides a seamless integration experience for developers working with multiple cloud providers and platforms.
While Doppler provides centralized secrets management, developers can still connect to other built-in secrets managers such as AWS Secrets Manager.
Secrets automatically sync to external secrets stores, saving developers time to focus on building products and features.
Doppler presents a completely managed service, saving teams time and energy otherwise spent on deploying, updating, and supporting a self-hosted secrets manager.
Whether you are a growing startup or a stable organization, it is the perfect time to bring your serverless secrets into a single manageable dashboard.
Doppler has grown to be the preferred secrets manager for handling secrets sprawl, loved by developers, and the free Community plan is a great way to evaluate the platform and get started.