DEV Community

Cover image for jq in Shell Scripts: The Small Things That Trip You Up
Schiff Heimlich
Schiff Heimlich

Posted on

jq in Shell Scripts: The Small Things That Trip You Up

#devops #shell #bash #jq

The Setup

If you work with JSON APIs, configs, or log processing, you have probably used jq in a shell script. It is solid. But there are a few edge cases that trip people up regularly — things that work fine in tests but break in production.

Here is what I have run into.

The Problem: Null vs. Missing Keys

Say you have JSON like this:

{"name": "web01", "status": "running"}
Enter fullscreen mode Exit fullscreen mode

And you want to get the status field. Easy:

jq -r ".status" <<< "{\"name\": \"web01\", \"status\": \"running\"}"
# returns: running
Enter fullscreen mode Exit fullscreen mode

But what if the field is missing or null?

{"name": "web01"}
Enter fullscreen mode Exit fullscreen mode

jq will return null. Your script might handle that fine — or it might not. The problem comes when you try to do arithmetic or comparisons on null.

jq ".count + 1" <<< "{\"count\": null}"
# returns: null
Enter fullscreen mode Exit fullscreen mode

Not helpful. You need to handle this explicitly:

jq -r ".count // 0 + 1" <<< "{\"count\": null}"
# returns: 1
Enter fullscreen mode Exit fullscreen mode

The // operator provides a default value when the left side is null or missing.

The Problem: Piping Arrays Correctly

Another one. You have an array of objects and want to filter them:

[{"host": "web01", "cpu": 45}, {"host": "web02", "cpu": 12}, {"host": "web03", "cpu": 78}]
Enter fullscreen mode Exit fullscreen mode

You want hosts with cpu > 50.

jq ".[] | select(.cpu > 50)" <<< "[{\"host\": \"web01\", \"cpu\": 45}, {\"host\": \"web02\", \"cpu\": 12}, {\"host\": \"web03\", \"cpu\": 78}]"
Enter fullscreen mode Exit fullscreen mode

This outputs:

{"host": "web03", "cpu": 78}
Enter fullscreen mode Exit fullscreen mode

Fine, right? But now try to get just the hostnames as a list:

jq "[.[] | select(.cpu > 50) | .host]" <<< "[{\"host\": \"web01\", \"cpu\": 45}, {\"host\": \"web02\", \"cpu\": 12}, {\"host\": \"web03\", \"cpu\": 78}]"
Enter fullscreen mode Exit fullscreen mode

Returns:

["web03"]
Enter fullscreen mode Exit fullscreen mode

Works. But if nothing matches:

jq "[.[] | select(.cpu > 100) | .host]" <<< "[{\"host\": \"web01\", \"cpu\": 45}]"
Enter fullscreen mode Exit fullscreen mode

Returns:

[]
Enter fullscreen mode Exit fullscreen mode

Not null — an empty array. Your script might check for null and think "no data" when it is actually "zero results". Different things.

The Workaround

When I write jq in shell scripts that handle production data, I usually wrap things defensively:

# Get a value with a default, even if key is missing
cpu=$(echo "$json" | jq -r ".cpu // \"unknown\"")

# Check if result is valid before proceeding
if [ "$cpu" = "null" ] || [ "$cpu" = "unknown" ]; then
    echo "No CPU data" >&2
    exit 1
fi
Enter fullscreen mode Exit fullscreen mode

Or I use jq -e to check for null/empty results:

jq -e ".cpu" <<< "{\"cpu\": null}" > /dev/null
if [ $? -eq 5 ]; then
    echo "Key not found"
fi
Enter fullscreen mode Exit fullscreen mode

Error code 5 means the output was null.

The Point

jq is reliable. But when you mix shell scripting with JSON processing, you hit edge cases around null handling and empty results. A few defensive patterns save you from late-night debugging sessions.

These are not jq bugs — they are just things to know when you are writing scripts that process real data.

Schiff Heimlich | Sysadmin who learned this the hard way

Top comments (0)