Hashicorp Vault: High-Level Architecture, Components, and Key Concepts

Image Source: Hashicorp, Architecture, https://developer.hashicorp.com/vault/docs/internals/architecture.

Hashicorp Vault is a flexibility and robustness secrets management tool. Installable as a simple binary that starts a single server or joins others to create a server cluster, it offers token-based, policy-controlled access to encrypted data. Incorporating Vault into applications can be done directly via the exposed REST-like API interface, by running the Vault binary in an agent mode that fetches secrets in the context of a server or containers, or by installing operator abstractions directly in the container orchestrating software Kubernetes.

Vaults operational flexibility and feature sets builds upon a complex architecture. This blog article is a lightweight introduction to this topic, providing the essential knowledge to understand operating and using a Hashicorp Vault server.

The background material for this article stems from the official Hashicorp Vault documentation about Vault internals, Concepts, and subsequent pages.

This article originally appeared on my blog admantium.com.

Hashicorp Vault Components

The official Hashicorp Vault documentation depicts the principal layers of Hashicorp Vault:

Source: Hashicorp, Architecture, https://developer.hashicorp.com/vault/docs/internals/architecture.

This high-level overview separates Vault into three layers of system interaction and design.

• External Access: An HTTPS API exposes functional access to a Vaults internal configuration, general access managements, and functionality covering the complete lifecycle of secrets.
• Internal Components: This layer houses various logical components that provide the rich feature set of secrets management. Components consolidate and expose dedicated functions, and they store data relevant for their respective operations, such as configured policies, tokens, or operational log files.
• Storage: This layer contains the actual secrets payload, the stored data such as access credentials or certificates. It is strongly encrypted and can only be accessed via higher-level functions provided by the internal components.

The shown internal components are grouped as follows:

• Token Management: Tokens are a foundational component of Hashicorp Vault interactions. The authentication mechanism effectively creates access tokens, and these tokens embody access rights to functions and secrets likewise. Technically, each token has a duration, a number of renewals, and several types of associated policies. To each token, a token accessor is provided, which governs access and functions to the Tokens data itself.
• Policy Management: Policies are another core security concept of Hashicorp Vault. Policies govern the access of all interactions with a Vault server, from its system configuration, complete component lifecycle management, and access to secrets. During the initial setup and configuration, the root access policy, attached to the root token, provides access to all features. And from here, fine-grained access controls become manageable by crafting policy objects, RBAC definitions written in the Hashicorp Configuration Language (HCL). Each policy is effectively a list of endpoints and capabilities specifying read or write access.
• Routing: External requests are intercepted and processed by this layer, inspecting parameters, payload and paths. Based on the internal configuration, these requests are validated with authentication and policy data, and either forwarded to the target component or rejected.
• Backends: Backends are functional components governing core features of Vault system functionality. Each backend exposes a dedicated path on its external API with which it can be accessed. Since core functionality is exposed as well as extended when additional components are activated, a configuration bootstrap is required - from a root access level account, the desired configuration is enabled, which adds additional callable access paths.
  • Authentication: Every external client needs to be authenticated with Vault before any further requests are processed. This component manages several external authentication providers, like GitHub or LDAP, as well as Vault native access methods. Conceptually this component issues access tokens to the clients, which are governed by a policy, which in turn determines which function the token is allowed to access. The top-level access path to this component is prefixed with /auth.
  • System Backend: This component provides exposes core functionality and configuration of a Vault server, requiring a user with elevated access rights. All functions of this backend can be accessed from the /sys path. Among its functions are the management of audit devices, authentication methods, token capability checking, access to the system configuration, internal metrics, health checks and much more.
  • Secret Engine: Secrets engines components wrap an internal or external provider with a uniform feature set. Engines, and by extension the secrets they manage, are governed by a lifecycle consisting of activation, tuning, moving and deactivation. Each engine provides different feature sets: storing encrypted data, generating data by invoking external third parties, or creating short-lived single-access secrets. Once an engine is enabled, it can be accessed via the /secrets prefix and its configured path.
• Core: The official documentation does not detail this component, but browsing the opensource code of Vault on Github shows that it provides low-level interfaces and technical constructs enabling and integrating all other components.

Hashicorp Vault Key Concepts

Components realize the implementation of several key concepts that are integral to Vaults functionality. This section explains them in a logical order reflecting the typical secrets lifecycle.

• Identity: Technical or personal users interacting with Hashicorp Vault are represented as entities with a name and internal ID. Each entity is mapped to one or more aliases, which encapsulate account information and authentication with an identity provider. This flexibility supports different authentication workflows when applications use Vault to access secrets. Interestingly, Vault itself can be configured as an OIDC identity provider. Technically, identity itself is a type of secrets engine. Supported identity providers are separated into three groups:
  • Generic: AppRole, JWT, OIDC, TLS, Userpass
  • Cloud: AliCloud, AWS, Azure, Google Cloud, GitHub
  • Infra: Kubernetes, LDAP, OKTA, Radius
• Tokens: Every interaction with a Hashicorp Vault server requires a token. They encapsulate access restrictions with a defined lease time, renewal capability, and one or more access policies that detail to which API endpoints the token user has access by metadata about their lease time, renewal, and policies. Different types of tokens are distinguished. Tokens that are used mainly from external applications to access secrets are a) service tokens, b) batch tokens, and c) periodic tokens. Tokens eligible for system management are a) service tokens, b) root token, and c) recovery token. As a practical example, here is the complete token information of a generated root token:

Key                  Value
--                   ----
token                hvs.HBpPkixg32mSSkSavWpRpOil
token_accessor       Nu0uaILfDAwaE08J5z23B6wl
token_duration       ∞
token_renewable      false
token_policies       ["root"]
identity_policies    []
policies             ["root"]

Technically, the token-based authentication is handled by an authentication provider, and it cannot be disabled because it’s the sole option to start and bootstrap a Vault server.

• Policy: Every Vault functionality is exposed as a URL path, accessible with a REST-Like API. Policies are technical constructs that configure accessible paths and the associated actions. Policies are written in the Hashicorp Configuration Language (HCL). As a practical example, the following policy allows read and write access to a secrets engine configured at path kv2:

path "kv2/data/*" {
  capabilities = ["create", "update"]
}

Conclusion

Hashicorp Vault is a flexible secrets management tool. This blog article explored the high-level architecture, components and key concepts encountered when working with Vault. The effective architecture encompasses three layers: external access, internal components, and storage. The internal components can be imagined as a barrier, and they implement lifecycle aspects of secret management: a) routing requests, b) backends for authentication, secrets, and system management, c) policy management, d) token management and the e) core that ties all components together. Key concepts of Vault are identity, token, and policy. Every system or personal account authenticated with a supported identity provider is mapped to an identity, and an access token with one or many access policies is created. These tokens are passed as part of the CLI interaction or with an API request, and enable any Vault operation or secrets management. Understanding these architectural and conceptual aspects helps to configure and operate Hashicorp Vault.