DEV Community

Cover image for JQ what I learned today
Austin Cunningham
Austin Cunningham

Posted on • Updated on

JQ what I learned today

What's jq ?

A command line parser for json that can be downloaded here or you can use the playground. I will use this local json file as an beginner example hello-world.json

{
  "hello": "world",
  "Water":{
    "Atlantic":"Ocean",
    "Pacific":"Ocean",
    "Mediterranean":{
      "Ocean": false,
      "Sea": true
    }
  },
  "Land":{
    "Europe":[
      "Ireland",
      "France",
      "Germany"
    ],
    "Asia":[
      "India",
      "China"
    ]
  }
}
Enter fullscreen mode Exit fullscreen mode

Using jq

Simplest use case is to render the whole json file and pretty print it

# You can pass filters and json files as arguments
jq '.' hello-world.json
# You can cat the file and pipe it to jq
cat hello-world.json | jq
# You can curl remote json and pipe it to jq e.g. from https://stedolan.github.io/jq/tutorial/
curl 'https://api.github.com/repos/stedolan/jq/commits?per_page=5' | jq '.'
Enter fullscreen mode Exit fullscreen mode

Alt Text

Filtering with jq

The manual is full of options for filtering and can be a lot to take in. We have seen already the most basic filter '.' which can be used with either method of invoking jq to return all data in the json.

# We can filter to specific keys
$ jq '.hello' hello-world.json                                               
"world"
#  you can create a new object and create a new json files
$ cat hello-world.json | jq .Water > water.json
{
  "Atlantic": "Ocean",
  "Pacific": "Ocean",
  "Mediterranean": {
    "Ocean": false,
    "Sea": true
  }
}
# We can drill down
$ cat hello-world.json | jq '.Water.Atlantic'
"Ocean"
$ cat hello-world.json | jq '.Water.Mediterranean'
{
  "Ocean": false,
  "Sea": true
}
$ cat hello-world.json | jq '.Water.Mediterranean.Ocean'
false
# arrays syntax is a little different
$ cat hello-world.json | jq '.Land.Europe[]'
"Ireland"
"France"
"Germany"
$ cat hello-world.json | jq '.Land.Europe'  
[
  "Ireland",
  "France",
  "Germany"
]
# Arrays can be accessed by indexes or splits [0:2]
$ cat hello-world.json | jq '.Land.Europe[0]'
"Ireland"
$ cat hello-world.json| jq '.Land.Europe[0:2]'
[
  "Ireland",
  "France"
]
# array symbol can be used to iterate over values
$ cat hello-world.json | jq '.Land[]'
[
  "Ireland",
  "France",
  "Germany"
]
[
  "India",
  "China"
]
$ cat hello-world.json | jq '.Water[]'
"Ocean"
"Ocean"
{
  "Ocean": false,
  "Sea": true
}
# Lots of other array slicing methods available in the manual this is only scratching the surface
Enter fullscreen mode Exit fullscreen mode

You can pipe multiple filters together and reassign values to create new json

$ cat hello-world.json | jq '.Land | .Europe | {EU:[{country:.[]}]}'
{
  "EU": [
    {
      "country": "Ireland"
    },
    {
      "country": "France"
    },
    {
      "country": "Germany"
    }
  ]
}

Enter fullscreen mode Exit fullscreen mode

Using jq files to do more advance stuff

You can use jq files as filters with the -f flag to create more complex filters with supporting functions. The file Node-metrics.json has some cpu and memory usages stats from a Kubernetes Openshift cluster. I am looking to generate a report from this json.

I created the following file filterGenerateCsv.jq to filter the data in the json and creates an csv.

# function definition filter the data
def getNodeMetrics:
    [.items[]|
    {
        name:.metadata.name,
        cpu:.usage.cpu, 
        memory:.usage.memory
    }
    ];


# use geNodeUsages to generate csv
.[] |
getNodeMetrics |
group_by(.name) |
map({
  "Node Name": .[].name ,
  "CPU Node - Real": [.[].cpu] | add, 
  "Memory Node - Real": [.[].memory] | add
})| (.[0] | to_entries | map(.key)), (.[] | [.[]]) | @csv

#  (.[0] | to_entries | map(.key)) this filter sets the first array as follows
# [
#  "Node Name",
#  "CPU Node - Real",
#  "Memory Node - Real"
# ]
# (.[] | [.[]]) filter generates the arrays of values and @csv pipes to csv format
Enter fullscreen mode Exit fullscreen mode

I can then call the jq file as my filter argument with the -f flag

# Curl the file and use the -f flag for file to point at filterGenerateCsv.jq
$ curl https://gist.githubusercontent.com/austincunningham/eca562b35651b9a6dd214d0023c19cdf/raw | jq -f filterGenerateCsv.jq 
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  5594  100  5594    0     0   9497      0 --:--:-- --:--:-- --:--:--  9481
"\"Node Name\",\"CPU Node - Real\",\"Memory Node - Real\""
"\"ip-10-0-130-91.eu-west-1.compute.internal\",\"130m\",\"1596380Ki\""
"\"ip-10-0-131-137.eu-west-1.compute.internal\",\"115m\",\"1257300Ki\""
"\"ip-10-0-133-196.eu-west-1.compute.internal\",\"248m\",\"1787868Ki\""
"\"ip-10-0-134-141.eu-west-1.compute.internal\",\"473m\",\"3330440Ki\""
"\"ip-10-0-137-148.eu-west-1.compute.internal\",\"127m\",\"1485252Ki\""
"\"ip-10-0-139-72.eu-west-1.compute.internal\",\"172m\",\"1524604Ki\""
"\"ip-10-0-140-152.eu-west-1.compute.internal\",\"113m\",\"1276292Ki\""
"\"ip-10-0-140-180.eu-west-1.compute.internal\",\"609m\",\"4152784Ki\""
"\"ip-10-0-142-230.eu-west-1.compute.internal\",\"667m\",\"4110252Ki\""
"\"ip-10-0-142-51.eu-west-1.compute.internal\",\"727m\",\"4080132Ki\""

Enter fullscreen mode Exit fullscreen mode

More to learn here but I think jq will become something I will use more in the future. The files used in this blog are on this github repo

Top comments (1)

Collapse
 
savagepixie profile image
SavagePixie

jq is wonderful. It's saved me a lot of trouble when I need to fetch data using curl on the terminal.