Mastering Key Vaults - Securing Quella’s Sensitive Assets Like a Pro

Introduction

Managing secrets securely is a challenge every developer and security engineer faces at some point. Whether it’s API keys, passwords, or sensitive configurations, ensuring they don’t fall into the wrong hands is critical. Enter key vaults — a robust solution for securing sensitive assets. In this article, we’ll explore what key vaults are, how they function and how we at Quella integrated HashiCorp Vault for better secret management.

Why key vaults matter

Quella’s architecture, like many modern applications, relies on sensitive data to function. Before, secrets were stored in environment variables (like many other applications do as well), which, while convenient, came with some risks:

1. No secure storage: Environment variables are plain text, meaning they aren't stored in a secure way.
2. No auditing: Tracking who accessed or modified environment variables is nearly impossible.
3. Replication across systems: Environment variables are tied to a single location, meaning that if you needed the same secrets in multiple systems, you would have manually copy/paste them across those systems.
4. Vulnerable to command injection: Environment variables will be shown with the env command, meaning that if a hacker is every able to execute (remote) command injection, they will be able to see all your secrets.

Key vaults address these problems by providing a centralised, secure storage solution. They offer fine-grained access control, audit logs, and features like automatic key rotation.

Evaluating key vault solutions

The market offers a variety of key vault options, each with unique strengths. Notable solutions include:

• Microsoft Azure Vault: Ideal for organizations already using Azure.
• AWS Key Management Service (KMS): Integrates seamlessly with AWS ecosystems.
• HashiCorp Vault: Known for flexibility and support for both cloud and self-hosted deployments. Has the ability to create dynamic-secrets.

Besides that they generally over a similar set of features:

• They offer an SDK or API that another system can use to interact with the vault.
• A user interface to manage the vault and its assets.
• A CLI tool
• They offer fine-grained access control with policy's and roles.
• Build in key rotation mechanics.
• The ability to perform cryptographic operations (encrypting/decrypting data).

For Quella we chose HashiCorp Vault as our key vault solution, due to its robust feature set and self-hosting capabilities. It’s not tied to any specific cloud provider, making it ideal for our setup.

What technical mechanics are used by Hashicorp vault to ensure security

HasiCorp vault uses various technical mechanisms to make sure that secrets are stored in a secure storage location and can only be accessed by the allowed party's.

They do that with a few mechanics:

• All data is server by the API over TLS.
• Sensitive assets (secrets and encryption keys) are encrypted both in transit and at rest. Meaning that the storage backend that was chosen will never see the data from the secret in plain text.
• Hashicorp has a build-in fine-grained access policy system, meaning that only trusted party's have access to specific actions and data. In quella's system the backend would only have read access to a specific area of the vault, everything else stays hidden.
• All party's need to be authenticated with one of the many available authentication options offered by Vault, this could be basic token-based login or github based or one of the other offered integrations. During the vault's setup you are able to enable/disable specific authentication modules.

Furthermore, Hashicorp Vaults source code is open source: https://github.com/hashicorp/vault as part of their believes in the Kerckhoffs's principle. This means they do not believe in security by obscurity. That is why they leverage Golang crypto and x/crypto libraries to handle the heavy lifting associated with encrypting and decrypting data.

Those libraries contain the methods and functions that are the implementations of various algorithms like AES256 that are used within Hashicorp Vault, for both internally encrypting data and decrypting data and as part of our transit backend allowing developers to leverage cryptography without having to deploy our own cryptographic infrastructure. It then manages the encryption keys using their own internal keyring, however developers do have the option to implement their own Hardware security module if they would like to.

Lastly, as mentioned before, HC vault has auditing capabilities, there by being able to show exactly what party read what secret and/or did what action at what time.

Implementing HashiCorp Vault

Integration Challenges

While using Hashicorp Vault has some great benefits, integrating it isn’t plug-and-play. Some challenges that need to be solved and taken care of are:

1. Infrastructure Setup: Establishing a secure environment for self-hosting. We set up an internal network with firewalls to isolate the vault.
2. Access Dependency: Systems relying on the vault must handle high availability. If the vault is down, secrets cannot be accessed.
3. Token Management: The access token for the vault itself needs secure storage.

Practical Example: Vault Setup for Quella

Here’s how we integrated HashiCorp Vault into Quella:

1. We started by creating a Docker container using the hashicorp/vault image, initialised it, and unsealed it.

2. After starting the vault, We configured two key-value secret engines to store Quella-specific secrets—one for the demo and another for production.
   

3. We then stored the configuration secrets (e.g., Google Client Secret, JWT secrets) for our backend under a single entry for efficient retrieval.
   
   

4. Then in de codebase of our backend, we integrated VaultSharp and VaultSharp.Extensions.Configuration into the ASP.NET Core configuration provider, which enabled seamless use of sensitive configuration from the vault and non-sensitive configuration from our environment variables.
   

5. Now to only allow the backend API to see specific information, we defined a policy granting read-only access to the necessary secrets and restricted access to other paths.
   

# Allow reading the ConfigurationSecrets secret itself (not the entire path)
path "demo/data/ConfigurationSecrets" {
  capabilities = ["read"]
}

# Deny access to cubbyhole
path "cubbyhole/*" {
  capabilities = ["deny"]
}

# Allow reading and listing individual fields within ConfigurationSecrets
path "demo/data/ConfigurationSecrets/*" {
  capabilities = ["read", "list"]
}

1. Lastly we then generated a dedicated authentication token for our backend API and we attached our policy. As you can see, this only allows us to see the demo vault:

Now that the configuration and setup is done, we quickly tested if the secret was now available in our backend, so after putting down a breakpoint you can see that the secret variable does indeed have a value, hooray!

The JwtAuthSettings:AccessTokenSecret does indeed have a value, which it obtained from the key vault.

Now since we also wanted to use the encryption-as-a-service offered by hashicorp, we created a new VaultTransitService, which allows use to easily encrypt and decrypt data on the application level using an encryption key that was never exposed to us, this was done with the following two method implementations:

Encrypting data using encryption key from vault:

public async Task<string> EncryptAsync(string keyName, string plaintext) {
    var secret = await vaultClient.V1.Secrets.Transit.EncryptAsync(keyName, new EncryptRequestOptions {
        Base64EncodedPlainText = Convert.ToBase64String(Encoding.UTF8.GetBytes(plaintext))
    });
    return secret?.Data?.CipherText ?? throw new InvalidOperationException("Encryption failed. Ciphertext not returned from Vault.");
}

Decrypting data using encryption key from vault:

public async Task<string> DecryptAsync(string keyName, string ciphertext) {
    var secret = await vaultClient.V1.Secrets.Transit.DecryptAsync(keyName, new DecryptRequestOptions {
        CipherText = ciphertext
    });
    return Encoding.UTF8.GetString(Convert.FromBase64String(secret?.Data?.Base64EncodedPlainText));
}

Now we decided to use this to improve the security of our multi-factor authentication secret, which is used by the system to verify if a given OTP code is valid. In our codebase during the 2fa setup, we called the vaultTransitService to encrypt our TwoFactorKey:

Then when we need to validate the given TOTP code during login, we use the vaultTransitService to decrypt the TwoFactorKey:

Changes to Quella's architecture

The introduction of HashiCorp Vault required some changes to Quella’s architecture and infrastructure:

• The backend now retrieves secrets at startup, making it dependent on a seamless connectivity to the vault.
• We setup a dedicated internal network that ensures secure communication with the vault.
• We made sure that fields in our database support the encrypted format for the database fields that are being encrypted.

Things to keep in mind

Our journey with HashiCorp Vault highlighted some key takeaways:

1. The vault should have a high availability, since downtime in the vault affects all the related systems too, as they rely on Vault for essential configuration secrets.
2. The access tokens that are being used to access the vault by related systems, should be stored in a safe-way to stop the introduction of additional vulnerabilities.

Conclusion

Key vaults, like HashiCorp Vault, play an important role in securing modern applications. For Quella, the integration has not only strengthened our security posture but also streamlined secret management. By embracing tools like these, developers and security engineers can focus on building great applications, knowing their sensitive assets are in safe hands.

If you’re considering enhancing your application’s security, a key vault might be your next best investment.

If you would like to read the full report of our research, it can be found right here.