This post will show some recipes for working with the jq utility in Bash to get data from HashiCorp Vault Enterprise via its ReST API.
You can install jq from here:
And if you don't have a "Bash" environment, you can get one from here: https://git-scm.com (make sure to select the "Git Bash" option when given the choice in the installation process)
If you're on Windows you might have some troubles or need to use some different name for jq-64.exe.
You also need to download and unzip Vault from https://vaultproject.io, and run it in a Git Bash like this:
$ cd ~/Downloads
$ echo 'alias vault="~/Downloads/vault.exe"' >> ~/.bash_profile ##(Only do this if you're on Windows)
$ source ~/.bash_profile ## Again, needed only if you're on Windows
$ ./vault server -dev -dev-root-token-id=s.JzcMzvFCAVHvxE4xusglvoHF
==> Vault server configuration:
Api Address: http://127.0.0.1:8200
Cgo: disabled
Cluster Address: https://127.0.0.1:8201
Go Version: go1.14.7
Listener 1: tcp (addr: "127.0.0.1:8200", cluster address: "127.0.0.1:8201", max_request_duration: "1m30s", max_request_size: "33554432", tls
: "disabled")
Log Level: info
Mlock: supported: false, enabled: false
Recovery Mode: false
Storage: inmem
Version: Vault v1.5.3
Version Sha: 9fcd81405feb320390b9d71e15a691c3bc1daeef
WARNING! dev mode is enabled! In this mode, Vault runs entirely in-memory
and starts unsealed with a single unseal key. The root token is already
authenticated to the CLI, so you can immediately begin using Vault.
You may need to set the following environment variable:
PowerShell:
$env:VAULT_ADDR="http://127.0.0.1:8200"
cmd.exe:
set VAULT_ADDR=http://127.0.0.1:8200
The unseal key and root token are displayed below in case you want to
seal/unseal the Vault or re-authenticate.
Unseal Key: t/5BTQzZ5LbD0JIYkpQ9dY3fLEqZEMyena3aCh4+sWw=
Root Token: s.JzcMzvFCAVHvxE4xusglvoHF
Development mode should NOT be used in production installations!
==> Vault server started! Log data will stream in below:
2020-10-07T15:14:47.286-0700 [INFO] proxy environment: http_proxy= https_proxy= no_proxy=
2020-10-07T15:14:47.289-0700 [WARN] no `api_addr` value specified in config or in VAULT_API_ADDR; falling back to detection if possible, but this value s
hould be manually set
2020-10-07T15:14:47.327-0700 [INFO] core: security barrier not initialized
2020-10-07T15:14:47.327-0700 [INFO] core: security barrier initialized: stored=1 shares=1 threshold=1
2020-10-07T15:14:47.332-0700 [INFO] core: post-unseal setup starting
2020-10-07T15:14:47.349-0700 [INFO] core: loaded wrapping token key
2020-10-07T15:14:47.349-0700 [INFO] core: successfully setup plugin catalog: plugin-directory=
2020-10-07T15:14:47.356-0700 [INFO] core: no mounts; adding default mount table
2020-10-07T15:14:47.360-0700 [INFO] core: successfully mounted backend: type=cubbyhole path=cubbyhole/
2020-10-07T15:14:47.363-0700 [INFO] core: successfully mounted backend: type=system path=sys/
2020-10-07T15:14:47.365-0700 [INFO] core: successfully mounted backend: type=identity path=identity/
2020-10-07T15:14:47.370-0700 [INFO] core: successfully enabled credential backend: type=token path=token/
2020-10-07T15:14:47.370-0700 [INFO] rollback: starting rollback manager
2020-10-07T15:14:47.370-0700 [INFO] core: restoring leases
2020-10-07T15:14:47.376-0700 [INFO] expiration: lease restore complete
2020-10-07T15:14:47.376-0700 [INFO] identity: entities restored
2020-10-07T15:14:47.376-0700 [INFO] identity: groups restored
2020-10-07T15:14:47.377-0700 [INFO] core: post-unseal setup complete
2020-10-07T15:14:47.380-0700 [INFO] core: root token generated
2020-10-07T15:14:47.380-0700 [INFO] core: pre-seal teardown starting
2020-10-07T15:14:47.380-0700 [INFO] rollback: stopping rollback manager
2020-10-07T15:14:47.380-0700 [INFO] core: pre-seal teardown complete
2020-10-07T15:14:47.381-0700 [INFO] core.cluster-listener.tcp: starting listener: listener_address=127.0.0.1:8201
2020-10-07T15:14:47.381-0700 [INFO] core.cluster-listener: serving cluster requests: cluster_listen_address=127.0.0.1:8201
2020-10-07T15:14:47.381-0700 [INFO] core: post-unseal setup starting
2020-10-07T15:14:47.381-0700 [INFO] core: loaded wrapping token key
2020-10-07T15:14:47.381-0700 [INFO] core: successfully setup plugin catalog: plugin-directory=
2020-10-07T15:14:47.382-0700 [INFO] core: successfully mounted backend: type=system path=sys/
2020-10-07T15:14:47.383-0700 [INFO] core: successfully mounted backend: type=identity path=identity/
2020-10-07T15:14:47.383-0700 [INFO] core: successfully mounted backend: type=cubbyhole path=cubbyhole/
2020-10-07T15:14:47.384-0700 [INFO] core: successfully enabled credential backend: type=token path=token/
2020-10-07T15:14:47.384-0700 [INFO] rollback: starting rollback manager
2020-10-07T15:14:47.384-0700 [INFO] core: restoring leases
2020-10-07T15:14:47.384-0700 [INFO] identity: entities restored
2020-10-07T15:14:47.384-0700 [INFO] expiration: lease restore complete
2020-10-07T15:14:47.384-0700 [INFO] identity: groups restored
2020-10-07T15:14:47.384-0700 [INFO] core: post-unseal setup complete
2020-10-07T15:14:47.384-0700 [INFO] core: vault is unsealed
2020-10-07T15:14:47.401-0700 [INFO] core: successful mount: namespace= path=secret/ type=kv
2020-10-07T15:14:47.411-0700 [INFO] secrets.kv.kv_8bdaeac5: collecting keys to upgrade
2020-10-07T15:14:47.411-0700 [INFO] secrets.kv.kv_8bdaeac5: done collecting keys: num_keys=1
2020-10-07T15:14:47.411-0700 [INFO] secrets.kv.kv_8bdaeac5: upgrading keys finished
Leave Vault running in that terminal window, and open another one to run the following stuff in:
export VAULT_ADDR=http://127.0.0.1:8200
export VAULT_TOKEN=s.JzcMzvFCAVHvxE4xusglvoHF
Now, to prepare further, create some auth methods:
./vault auth enable kubernetes
./vault auth enable userpass
./vault auth enable okta
./vault auth enable aws
Example 1: Get a list of all of the Auth Methods that are of type okta
Shameless theft from this SO dude: https://stackoverflow.com/questions/18592173/select-objects-based-on-value-of-variable-in-object-using-jq/31911811#comment112920282_18608100
First, look at the response without jq:
curl --request GET --header "X-Vault-Token: $VAULT_TOKEN" --header "X-Vault-Namespace: $VAULT_NAMESPACE" $VAULT_ADDR/v1/sys/auth
Here's the output from that command:
$ curl --request GET --header "X-Vault-Token: $VAULT_TOKEN" --header "X-Vault-Namespace: $VAULT_NAMESPACE" $VAULT_ADDR/v1/sys/auth
{"kubernetes/":{"accessor":"auth_kubernetes_9bc9df5a","config":{"default_lease_t
tl":0,"force_no_cache":false,"max_lease_ttl":0,"token_type":"default-service"},"
description":"","external_entropy_access":false,"local":false,"options":null,"se
al_wrap":false,"type":"kubernetes","uuid":"72df2827-2ac1-1fb1-7d5a-c6262a4f5300"
},"token/":{"accessor":"auth_token_332e9370","config":{"default_lease_ttl":0,"fo
rce_no_cache":false,"max_lease_ttl":0,"token_type":"default-service"},"descripti
on":"token based credentials","external_entropy_access":false,"local":false,"opt
ions":null,"seal_wrap":false,"type":"token","uuid":"ba81c552-e8aa-9ba9-5243-0580
59545ccc"},"userpass/":{"accessor":"auth_userpass_1dd44d0b","config":{"default_l
ease_ttl":0,"force_no_cache":false,"max_lease_ttl":0,"token_type":"default-servi
ce"},"description":"","external_entropy_access":false,"local":false,"options":nu
ll,"seal_wrap":false,"type":"userpass","uuid":"e02a55d0-bd07-efeb-f5de-37fb5dcd5
84a"},"okta/":{"accessor":"auth_okta_fb869c57","config":{"default_lease_ttl":0,"
force_no_cache":false,"max_lease_ttl":0,"token_type":"default-service"},"descrip
tion":"","external_entropy_access":false,"local":false,"options":null,"seal_wrap
":false,"type":"okta","uuid":"1a445f3a-d302-ac6b-4508-def7a289894a"},"request_id
":"dc3dbab5-e001-498b-514c-05665baf09d2","lease_id":"","renewable":false,"lease_
duration":0,"data":{"kubernetes/":{"accessor":"auth_kubernetes_9bc9df5a","config
":{"default_lease_ttl":0,"force_no_cache":false,"max_lease_ttl":0,"token_type":"
default-service"},"description":"","external_entropy_access":false,"local":false
,"options":null,"seal_wrap":false,"type":"kubernetes","uuid":"72df2827-2ac1-1fb1
-7d5a-c6262a4f5300"},"okta/":{"accessor":"auth_okta_fb869c57","config":{"default
_lease_ttl":0,"force_no_cache":false,"max_lease_ttl":0,"token_type":"default-ser
vice"},"description":"","external_entropy_access":false,"local":false,"options":
null,"seal_wrap":false,"type":"okta","uuid":"1a445f3a-d302-ac6b-4508-def7a289894
a"},"token/":{"accessor":"auth_token_332e9370","config":{"default_lease_ttl":0,"
force_no_cache":false,"max_lease_ttl":0,"token_type":"default-service"},"descrip
tion":"token based credentials","external_entropy_access":false,"local":false,"o
ptions":null,"seal_wrap":false,"type":"token","uuid":"ba81c552-e8aa-9ba9-5243-05
8059545ccc"},"userpass/":{"accessor":"auth_userpass_1dd44d0b","config":{"default
_lease_ttl":0,"force_no_cache":false,"max_lease_ttl":0,"token_type":"default-ser
vice"},"description":"","external_entropy_access":false,"local":false,"options":
null,"seal_wrap":false,"type":"userpass","uuid":"e02a55d0-bd07-efeb-f5de-37fb5dc
d584a"}},"wrap_info":null,"warnings":null,"auth":null}
Pretty messy, right? Let's clean it up a bit:
curl --request GET --header "X-Vault-Token $VAULT_TOKEN" --header "X-Vault-Namespace $VAULT_NAMESPACE" $VAULT_ADDR/v1/sys/auth | jq
The above command will show the output in more of a structured format with indentation to indicate deeper levels of the object.
Here's the output from the above command:
$ curl --request GET --header "X-Vault-Token: $VAULT_TOKEN" --header "X-Vault-Namespace: $VAULT_NAMESPACE" $VAULT_ADDR/v1/sys/auth | jq
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 2695 0 2695 0 0 105k 0 --:--:-- --:--:-- --:--:-- 105k
{
"token/": {
"accessor": "auth_token_332e9370",
"config": {
"default_lease_ttl": 0,
"force_no_cache": false,
"max_lease_ttl": 0,
"token_type": "default-service"
},
"description": "token based credentials",
"external_entropy_access": false,
"local": false,
"options": null,
"seal_wrap": false,
"type": "token",
"uuid": "ba81c552-e8aa-9ba9-5243-058059545ccc"
},
"userpass/": {
"accessor": "auth_userpass_1dd44d0b",
"config": {
"default_lease_ttl": 0,
"force_no_cache": false,
"max_lease_ttl": 0,
"token_type": "default-service"
},
"description": "",
"external_entropy_access": false,
"local": false,
"options": null,
"seal_wrap": false,
"type": "userpass",
"uuid": "e02a55d0-bd07-efeb-f5de-37fb5dcd584a"
},
"okta/": {
"accessor": "auth_okta_fb869c57",
"config": {
"default_lease_ttl": 0,
"force_no_cache": false,
"max_lease_ttl": 0,
"token_type": "default-service"
},
"description": "",
"external_entropy_access": false,
"local": false,
"options": null,
"seal_wrap": false,
"type": "okta",
"uuid": "1a445f3a-d302-ac6b-4508-def7a289894a"
},
"kubernetes/": {
"accessor": "auth_kubernetes_9bc9df5a",
"config": {
"default_lease_ttl": 0,
"force_no_cache": false,
"max_lease_ttl": 0,
"token_type": "default-service"
},
"description": "",
"external_entropy_access": false,
"local": false,
"options": null,
"seal_wrap": false,
"type": "kubernetes",
"uuid": "72df2827-2ac1-1fb1-7d5a-c6262a4f5300"
},
"request_id": "633adc4e-bd04-e579-58d4-34bde3dc91ee",
"lease_id": "",
"renewable": false,
"lease_duration": 0,
"data": {
"kubernetes/": {
"accessor": "auth_kubernetes_9bc9df5a",
"config": {
"default_lease_ttl": 0,
"force_no_cache": false,
"max_lease_ttl": 0,
"token_type": "default-service"
},
"description": "",
"external_entropy_access": false,
"local": false,
"options": null,
"seal_wrap": false,
"type": "kubernetes",
"uuid": "72df2827-2ac1-1fb1-7d5a-c6262a4f5300"
},
"okta/": {
"accessor": "auth_okta_fb869c57",
"config": {
"default_lease_ttl": 0,
"force_no_cache": false,
"max_lease_ttl": 0,
"token_type": "default-service"
},
"description": "",
"external_entropy_access": false,
"local": false,
"options": null,
"seal_wrap": false,
"type": "okta",
"uuid": "1a445f3a-d302-ac6b-4508-def7a289894a"
},
"token/": {
"accessor": "auth_token_332e9370",
"config": {
"default_lease_ttl": 0,
"force_no_cache": false,
"max_lease_ttl": 0,
"token_type": "default-service"
},
"description": "token based credentials",
"external_entropy_access": false,
"local": false,
"options": null,
"seal_wrap": false,
"type": "token",
"uuid": "ba81c552-e8aa-9ba9-5243-058059545ccc"
},
"userpass/": {
"accessor": "auth_userpass_1dd44d0b",
"config": {
"default_lease_ttl": 0,
"force_no_cache": false,
"max_lease_ttl": 0,
"token_type": "default-service"
},
"description": "",
"external_entropy_access": false,
"local": false,
"options": null,
"seal_wrap": false,
"type": "userpass",
"uuid": "e02a55d0-bd07-efeb-f5de-37fb5dcd584a"
}
},
"wrap_info": null,
"warnings": null,
"auth": null
}
Now, let's get all of the Auth Methods that are of type okta:
curl --request GET --header "X-Vault-Token: $VAULT_TOKEN" --header "X-Vault-Namespace: $VAULT_NAMESPACE" $VAULT_ADDR/v1/sys/auth | jq .data | jq 'to_entries[] | select(.value.type=="okta") | .key'
The first part gets you the contents of the "data" part of the response. The second part, with the 'to_entries[], gets you the data in a format where you can read out the key names separately. The | select(.value.type=="okta") part gets you the auth methods that have the value "okta" under the key type. The | .key' part gets you the key of the auth method, usually ending in a /.
Here's the output for my example (should just be "okta/"):
$ curl --request GET --header "X-Vault-Token: $VAULT_TOKEN" --header "X-Vault-Namespace: $VAULT_NAMESPACE" $VAULT_ADDR/v1/sys/auth | jq .data | jq 'to_entries[] | select(.value.type=="okta") | .key'
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 2695 0 2695 0 0 125k 0 --:--:-- --:--:-- --:--:-- 125k
"okta/"
Example 2: How to get an accessor based on a Bash variable:
export OKTAAUTH='okta/'
cat << EOF > jqparams.txt
."data"."${OKTAAUTH}/"."accessor"
EOF
curl --request GET --header "X-Vault-Token: $VAULT_TOKEN" --header "X-Vault-Namespace: $VAULT_NAMESPACE" $VAULT_ADDR/v1/sys/auth | jq -f jqparams.txt
Here's an example:
$ export OKTAAUTH='okta/'
$ cat << EOF > jqparams.txt
> ."data"."${OKTAAUTH}"."accessor"
> EOF
$ curl --request GET --header "X-Vault-Token: $VAULT_TOKEN" --header "X-Vault-Namespace: $VAULT_NAMESPACE" $VAULT_ADDR/v1/sys/auth | jq -f jqparams.txt
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 2695 0 2695 0 0 125k 0 --:--:-- --:--:-- --:--:-- 125k
"auth_okta_fb869c57"
$
Example 3: How to get an Accessor Without a Bash Variable
In this case, it's the same as Example 2, except that you needn't bother with a jqparams.txt file.
."data"."okta/"."accessor"
Further reading for the nerds:
https://remysharp.com/drafts/jq-recipes
Top comments (0)