Automated Secret Retrieval Using Hashicorp Vault

John Lund-Molfese — Sun, 22 May 2022 14:21:55 +0000

Prerequisites:

A Vault path, with your secrets your project can use
A username and password to access your Vault secrets (or an alternative method of authentication)
A basic Spring Boot project.

There's a lot of good documentation on using the Vault CLI to automatically retrieve secrets and inject them into a Spring application at runtime. This saves time, because you don't have to manually enter them into a Spring properties file, and improves security by making it harder to accidentally commit credentials. But what if you aren't able to install the Vault CLI, or want to minimize the number of external dependencies your application requires users to install?

Another option is to use Vault's HTTP API, which provides similar functionality. In this example, we'll pretend we have an application that calls another API using a username and password we retrieve from Vault.

This guide will first detail how to retrieve secrets from your Vault path, then how to make them available while your application is running. This code was tested using Java 8, but should work for earlier and later versions.

Retrieving Secrets at Runtime

Let's start by creating a helper class to retrieve all the secrets in a Vault path as a java.util.Properties object we can inject into the application later. You can create this file in any package, for example, main/java/org.john.example/config/VaultRetrievalUtil.java .

package org.john.example.config;

import org.springframework.http.*;
import com.fasterxml.jackson.databind.JsonNode;
import org.springframework.web.client.RestTemplate;

import java.net.URI;
import java.util.*;

public class VaultRetrievalUtil{
    RestTemplate restTemplate = new RestTemplate();
}

The first method we'll write will consume our Vault credentials to retrieve a client token, which is required to access the Vault API. We need to escape quotes in the JSON request body.

    private String retrieveVaultClientToken(String vaultUsername, String vaultPassword) {
        String requestBody = "{\\"password\":\\"" + vaultPassword + "\"}";
        URI vaultLoginUri = URI.create("https://vault.mywebsite.com:8200/v1/auth/ldap/login/" + vaultUsername);
        RequestEntity<String> requestEntity = new RequestEntity<>(requestBody, HttpMethod.POST, vaultLoginUri);
        ResponseEntity<JsonNode> responseEntity = restTemplate.exchange(requestEntity, JsonNode.class);
        return Objects.requireNonNull(respone.getBody()).get("auth").get("client_token").asText();
    }

Once we have the token, we can write a method to retrieve the JSON object containing all our secrets in our Vault path.

    private JsonNode retrieveVaultSecrets(String vaultClientToken) {
        URI vaultSecretsPath = URI.create("https://vault.mywebsite.com:8200/v1/mySecretsPath")
        HttpHeaders httpHeaders = new HttpHeaders();
        headers.put("X-Vault-Token", Collections.singletonList(vaultToken));
        HttpEntity<String> requestEntity = new HttpEntity<>(null, httpHeaders);
        ResponseEntity<JsonNode> responseEntity = restTemplate.exchange(vaultSecretsPath, HttpMethod.GET, request, JsonNode.class);
        return Objects.requireNonNull(response.getBody()).get("data");
    }

We're halfway there! But we can't use this JsonNode directly, we'll need to convert it to a java.util.Properties object next.

    private Properties convertSecretsToSpringProperties(JsonNode vaultSecrets) {
        Properties properties = new Properties();
        for(Iterator<Map.Entry<String, JsonNode>> iterator = vaultSecrets.fields(); iterator.hasNext();){
            Map.Entry<String, JsonNode> secret = iterator.next();
            properties.put(secret.getKey(), secret.getValue().textValue());
        }
        return properties;
    }

Let's put it all together now and write a public method for our other classes to use. In the code below, we're assuming we've set the environment variables myVaultUsername and myVaultPassword with our vault username and password.

package org.john.example.config;

import org.springframework.http.*;
import com.fasterxml.jackson.databind.JsonNode;
import org.springframework.web.client.RestTemplate;

import java.net.URI;
import java.util.*;

public class VaultRetrievalUtil{
    RestTemplate restTemplate = new RestTemplate();

    public Properties retrieveVaultProperties(){
        String vaultUsername = System.getenv("myVaultUsername");
        String vaultPassword = System.getenv("myVaultPassword");

        if(vaultUsername != null && vaultPassword != null){
            String vaultClientToken = retrieveVaultClientToken(vaultUsername, vaultPassword);
            JsonNode vaultSecrets = retrieveVaultSecrets(vaultClientToken);
            return convertSecretsToSpringProperties(vaultSecrets);
        }
        else{
            /* Assuming if these credentials are not set, the user doesn't need credentials from Vault. 
            Alternatively, we could throw an exception if they're required for the application to work. */
            return new Properties();
        }
    }   

    private String retrieveVaultClientToken(String vaultUsername, String vaultPassword) {
        String requestBody = "{\\"password\":\\"" + vaultPassword + "\"}";
        URI vaultLoginUri = URI.create("https://vault.mywebsite.com:8200/v1/auth/ldap/login/" + vaultUsername);
        RequestEntity<String> requestEntity = new RequestEntity<>(requestBody, HttpMethod.POST, vaultLoginUri);
        ResponseEntity<JsonNode> responseEntity = restTemplate.exchange(requestEntity, JsonNode.class);
        return Objects.requireNonNull(respone.getBody()).get("auth").get("client_token").asText();
    }

    private JsonNode retrieveVaultSecrets(String vaultClientToken) {
        URI vaultSecretsPath = URI.create("https://vault.mywebsite.com:8200/v1/mySecretsPath")
        HttpHeaders httpHeaders = new HttpHeaders();
        headers.put("X-Vault-Token", Collections.singletonList(vaultToken));
        HttpEntity<String> requestEntity = new HttpEntity<>(null, httpHeaders);
        ResponseEntity<JsonNode> responseEntity = restTemplate.exchange(vaultSecretsPath, HttpMethod.GET, request, JsonNode.class);
        return Objects.requireNonNull(response.getBody()).get("data");
    }

    private Properties convertSecretsToSpringProperties(JsonNode vaultSecrets) {
        Properties properties = new Properties();
        for(Iterator<Map.Entry<String, JsonNode>> iterator = vaultSecrets.fields(); iterator.hasNext();){
            Map.Entry<String, JsonNode> secret = iterator.next();
            properties.put(secret.getKey(), secret.getValue().textValue());
        }
        return properties;
    }
}

That's it for now! Next we'll inject these properties into our application

Injecting Spring properties into our application

This part is simple, we'll create an instance of our VaultRetrievalUtil, then set the properties we retrieve from that class as the defaults in our main class. This means they can still be overridden if the user specifically chooses to define them elsewhere.

package org.john.example;

import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.cache.annotation.EnableCaching;
import org.john.example.config.VaultRetrievalUtil;

@SpringBootApplication
public class Application{
    public static void main(String[] args){
        VaultRetrievalUtil vaultRetrievalUtil = new VaultRetrievalUtil();
        SpringApplication application = new SpringApplication(Application.class);
        application.setDefaultProperties(vaultRetrievalUtil.retrieveVaultProperties());

        application.run(args);
    }
}

That's it! All of the properties in our Vault path are now accessible while the application is running. You can retrieve them like any Spring properties now, for example:

public class consumingClass{
    @Value("usernameToCallAnotherApi")
    private String usernameToCallAnotherApi;

    @Value("passwordToCallAnotherApi")
    private String passwordToCallAnotherApi;

    ...
}

Conclusion

You probably don't want to use this solution exactly in a production environment because it's insecure, especially if anyone else has access to your server. But for use locally, it should be fine, and avoids the possibility of accidentally committing credentials to source control.

Video Calls on Distributed Teams

John Lund-Molfese — Sun, 22 May 2022 14:20:25 +0000

Sometimes we Need Video Calls

On a distributed team, it's generally best to default to written communication
when discussing matters of business value for a few reasons:

Most forms of written communication (even chat!) externalize information. If you aren't able to attend at the time a video meeting is held, you're out of luck. Your voice won't be heard, and you won't be able to find what was discussed after the fact.
Many decisions require information which isn't available at the time of a video meeting. Asynchronous communication empowers individuals to learn and contribute on their schedule and at their own pace, rather than worrying about holding everyone else up.
Meetings have an outsized cost to software engineers and other people on the maker schedule.

Of course, there are exceptions. A public 1:1 meeting conducted on your company's internal wiki would be an interesting
experiment, but probably not the most comfortable or productive experience for anyone. Design thinking workshops with
stakeholders provide opportunities for iterating and clearing up misunderstandings rapidly when conducted over a video
call. Interviews with external candidates should probably be conducted over video calls, and the same thing for recurring
team meetings. There are many scenarios where a video call truly does make the most sense.

Also, useful written communication is a skill like any other. Especially if you're new to the industry, or have recently
made the switch to a distributed team, you might be more comfortable with the face to face conversations which are more
typical of in-person jobs. If a new member of your team asks for a video call, it makes sense to meet them where they
are, and worry about best practices later on.

DEV Community: John Lund-Molfese