How to "Reset" a HashiCorp Vault back to Zero State using Consul

There are two different methods to clear a Vault cluster in an emergency.

This assumes that your Vault Cluster already uses Consul as its Storage and HA backend. This process of giving Vault amnesia also applies to Vault Clusters with other storage backends, but I have only tested this on Consul.

Anyway, here are some scenarios where you might want to do this:

1. You want to clear all of the data in a HashiCorp Vault cluster without going through the process of re-deploying its infrastructure. This is useful for testing.
2. For users upgrading to Enterprise, there are fewer use cases for this, but it may come in handy if you want to ensure that no data exists in a target cluster before it is added to a replication set.

Resetting Vault using Consul is a short method for disaster recovery/backup.

It's valuable to read about, because this emergency recovery method is not in the publicly available documentation. At least, I haven't found it yet.

You can reset a Vault Cluster by giving Consul a little amnesia. Just delete Consul's data, and voila, Vault is reset, ready to initialize.

Vault Cluster Reset Method 1: Delete Consul's State by removing the contents of Consul's "Data Folder".

In your Consul configuration file, you should find a line like the following:

  "data_dir": "/opt/consul/data",

The part after "data_dir": is the folder whose contents you will want to delete to "clear" a given Consul node.

Example (on line 11): https://github.com/v6/super-duper-vault-train/blob/develop/configureconsul.sh#L11

This line in the Consul configuration file corresponds to the -data-dir command line flag when using Consul's CLI.

https://www.consul.io/docs/agent/options.html#_data_dir

If you want to reset the stored state of the whole cluster, just delete the contents of this folder on all of your Consul nodes. Before doing that, though, make sure to shut down Vault, then shut down Consul!

After you've deleted the contents of that folder, you can start Consul, then Vault.

Vault Cluster Reset Method 2: Delete Consul's state by removing Vault's directory in the Consul KV API.

Look in your Vault configuration file (the default location for it is /etc/vault/vault.hcl for a backend section. Under there, if you're using Consul as the backend, you should see a line defining a path. That's the path in the Consul KV store to delete if you want to reset Vault's data using the Consul KV API.

  path    = "vault/"

Example: https://github.com/v6/super-duper-vault-train/blob/develop/configurevault.sh#L6

For the above, I would run the following command to reset Consul:

consul kv delete -recurse vault/

If you get a 403 error, you can try checking to see whether a Consul ACL token is needed.

Error deleting key vault/: Unexpected response code: 403 (rpc error making call: Permission denied)

Here's an example of deleting the vault/ section of the KV store on a Vault Cluster, using an ACL token:

[vagrant@instance5 vagrant]$ export CONSUL_HTTP_TOKEN=a4c878e5-a0eb-48ef-b6b4-00e18a146bf2; consul kv delete vault/

Success! Deleted key: vault/

For a discussion on the Vault mailing list, with an example use case, you can check the following: https://groups.google.com/d/msg/vault-tool/kRFNlZYiyxc/d5sOBTwHBgAJ

Vault Reset Method 3: Delete Consul's state by removing Vault's directory in the Consul GUI.

Technically, this is very similar to Method 2, but it deserves some mention for those of us who aren't as familiar with Consul's CLI and API.

First, you'll need to get to the Consul GUI. Add your Consul's IP address to your address bar in a browser, then put :8500 after it, as in the following example:

Once you hit Enter to go to that IP address and port, you'll see the Consul web interface:

If you then navigate to the Key/Value page, assuming your default path in the Consul KV Store is defined as vault/, you'll see something like the following:

Delete that data by clicking the ... under Actions, and selecting Delete.

Here's an example of some steps you can work through using a local cluster:

1. Create a cluster
2. Initialize the cluster
3. Back up Consul (consul snapshot save mysnapshot.dat)
4. Shut down the cluster and its Consul backend
5. Delete the /opt/consul/data/raft folder of Consul
6. Restart Consul
7. Restart Vault
8. Check that Vault has been "wiped" by checking that Vault's UI prompts for initialization again
9. Restore Vault from a Consul Snapshot (consul snapshot restore mysnapshot.dat)

Article Shortlink: https://v.gd/NL3GLH

Comments

[Dmitry Kankalovich]

If you go with the default Helm installation of Vault then you should be aware that it seems like Vault caches its init state, even though you've deleted it in Consul.

What helped me:

• Delete init state in Consul: ssh to Consul server pod and run consul kv delete -recurse vault/
• Delete Vault pods: kubectl delete pods -l app.kubernetes.io/instance=vault -n your_namespace
• Observe Vault pods are re-created
• Initialize Vault
• Unseal Vault

[Bart D.]

The command to delete vault should be updated to :
consul kv delete -recurse vault/

Otherwise it doesn't delete vault/, even though it says it's successful.

[🦄N B🛡]

// , Updated, thanks for testing it, @bartdzkan .

[Alireza Nasri]

in case your storage backend is mysql with juju , you can connect to mysql database through mysqlclient and drop the table.

[Alireza Nasri]

what if the storage backend is mysql?
how can I delete that?