A Practical Guide to jq in Shell Scripts

A Practical Guide to jq in Shell Scripts

While building sykd (a system health monitor daemon for linux servers) i needed to collect system data as json so i could process easily in python on doing research if found out about the jq shell program for creating and parsing json in bash, but learning the commands was quite tedious and the man page is large and very difficult to read so i came out with this guide to ease and summarize the major commands you might need.

jq is a lightweight and flexible command-line JSON processor. It is one of the most useful tools for working with JSON data directly inside shell scripts and pipelines.

Installation

• Debian / Ubuntu: sudo apt install jq
• macOS: brew install jq
• Other systems: Use your package manager or download binaries from the official site.

Basic Usage

jq [options] '<filter>' [file...]

Input can come from a file, a pipe, or standard input. The filter expression selects or transforms the data.

Quick note: jq reads JSON from stdin by default. If you want to generate JSON inside jq without reading input, use -n.

Useful Options

Option	Purpose
-r	Raw output (no JSON quotes around strings)
-c	Compact (single-line) output
-n	Start with null input (useful for generating JSON)
-s	Slurp entire input into an array
--arg	Pass a string variable
--argjson	Pass a JSON value (preferred for arrays/objects)

Other helpful runtime options:

Option	Purpose
--rawfile name filename	Read an entire file as a raw string into variable $name
--slurpfile name filename	Read file and slurp into array $name (like -s but into variable)

Example: jq --slurpfile all files.json '.[0] + $all[0]'

Essential Filters

Goal	Filter Example
Pretty-print	.
Extract a field	.name or .["name"]
Nested field	.user.address.city
Array element	.[0] or .[-1]
Iterate array	.[]
Select matching objects	`.[] \
Build a new object	{% raw %}{name, age}
Transform every element	map(.name)
Length	length
List keys	keys
Conditional	if .active then "yes" else "no" end

You can define reusable functions inside jq using def. Example:

def greet: "Hello, " + .name + "!";
. | greet

This will output "Hello, <name>!" for an object with a name key.

Practical Examples

Pretty-print a file

jq . data.json

Extract a single value

jq -r '.version' package.json

Filter an array of objects

jq '.[] | select(.status == "active") | .name' users.json

Create a new structure

jq '{names: map(.name), total: length}' people.json

Use in a pipeline

curl -s https://api.example.com/data | jq '.results[] | .id'

Generate JSON from scratch

jq -n '{name: "example", version: 1}'

Create multiple objects programmatically

jq -n '[range(3) | {index: ., squared: (. * .)}]'

This produces:

[
{"index":0,"squared":0},
{"index":1,"squared":1},
{"index":2,"squared":4}
]

Passing a Bash Array to jq

Bash arrays cannot be passed directly. Convert the array to a JSON array first, then supply it with --argjson.

Recommended Method

my_array=("apple" "banana" "cherry")

# Convert bash array → JSON array
json_array=$(printf '%s\n' "${my_array[@]}" | jq -R . | jq -s .)

# Use it inside jq
jq --argjson arr "$json_array" '$arr'

One-liner version:

jq --argjson arr "$(printf '%s\n' "${my_array[@]}" | jq -R . | jq -s .)" \
   '$arr[]'

Numeric Arrays

numbers=(10 20 30)
json_numbers=$(printf '%s\n' "${numbers[@]}" | jq -s .)

jq --argjson nums "$json_numbers" '$nums | map(. * 2)'

Feed as Input Instead

printf '%s\n' "${my_array[@]}" | jq -R . | jq -s .

Tips for Reliable Shell Scripts

• Always quote the filter when it contains spaces or special characters.
• Prefer --argjson for arrays and objects; use --arg only for plain strings.
• Test filters interactively before embedding them in scripts.

• For complex logic, store the filter in a separate .jq file and call it with -f.

• When embedding multi-line filters in shell variables, use a heredoc to keep quoting simple:
  

filter=$(cat <<'EOF'
.[] | select(.status=="active") | {id, name}
EOF
)

curl -s api | jq -f <(echo "$filter")

• Prefer --argjson when you need to pass non-string structured data; --arg always passes a string.

• When values may contain newlines or special characters, convert them to JSON with jq -R . before passing.

• For newline-delimited JSON (NDJSON/JSONL), use jq -c . to ensure compact output and --slurp if you need to combine records into an array.

• Use try ... catch for safe extraction and defaults:
  

try .foo.bar catch "default"

• Use inputs to read multiple JSON documents from stdin (useful with -n):

# echo multiple JSON objects and process them with inputs
printf '%s
' '{"x":1}' '{"x":2}' | jq -n '[inputs | .x]'

Additional Examples and Patterns

Selecting and Transforming

Extract and rename fields:

jq '.[] | {identifier: .id, user: .name, active: .status == "active"}' users.json

Build an index (map id → object):

jq 'map({ (.id|tostring): . }) | add' users.json

Group by a key and count per group:

jq 'group_by(.category) | map({category: .[0].category, count: length})' items.json

Working with Strings

Trim and lowercase a field:

jq '.name | ascii_downcase | gsub("^ +| +$"; "")' person.json

Join array of strings:

jq -r '.tags | join(",")' article.json

Numeric operations

Sum field values:

jq '[.[] | .size] | add' files.json

Find maximum:

jq 'max_by(.score) | .score' scores.json

Advanced: reduce, recurse, and more

Use reduce to accumulate values:

jq 'reduce .[] as $item (0; . + $item.value)' metrics.json

Recursively find all keys named id in nested structure:

jq '.. | objects | .id? // empty' nested.json

Streaming large JSON (jq --stream)

For very large JSON documents use --stream to get path/value pairs; processing is more manual but uses much less memory:

jq --stream 'select(.[0][-1]=="targetKey") | .[1]' large.json

Working with NDJSON / JSONL

To parse newline-delimited JSON and combine into an array:

jq -s '.' file.jsonl

Process each line independently (streaming):

jq -c '. | {id: .id, ok: (.value > 0)}' file.jsonl

Filters in separate .jq files

Save a complex filter to filters.jq:

def summarize:
   {total: (map(.value) | add), count: length};

.items | map({k: .key, v: .value}) | summarize

Call it from shell:

jq -f filters.jq data.json

Passing environment variables into jq

Pass a shell variable as JSON (string):

name="O'Reilly"
jq --arg name "$name" '. + {author: $name}' book.json

Pass an environment variable that is already JSON:

json_obj='{"a":1}'
jq --argjson o "$json_obj" '. + $o' base.json

Error handling and safe accesses

Use ? to avoid errors when a key may not exist:

jq '.maybe?.nested // "default"' data.json

Use try to catch runtime errors:

jq 'try .items[] catch empty' data.json

Common pitfalls

• Forgetting to quote filters when they contain shell-sensitive characters — always wrap your filter in single quotes in shell.
• Passing arrays with --arg (creates a JSON string, not an array) — use --argjson or jq -s conversion.
• Assuming jq modifies files in-place — it writes to stdout; redirect output to a file to save changes.

Performance tips

• Use --stream for very large JSON that can't fit in memory.
• Avoid unnecessary map + add for simple reductions; reduce can be more efficient.
• Use compiled filters in -f files for complex repeated transformations.

Examples: Real shell script snippets

Filter API result and extract IDs into a bash array:

readarray -t ids < <(curl -s "https://api.example.com/items" | jq -r '.items[].id')

for id in "${ids[@]}"; do
   echo "Got id: $id"
done

Pass a bash array of words safely into jq and filter by membership:

words=(alpha beta "complex value")
json_words=$(printf '%s\n' "${words[@]}" | jq -R . | jq -s .)

curl -s dataset.json | jq --argjson words "$json_words" '[.[] | select(.name as $n | $words | index($n))]'

Create a small portable CLI using jq and bash:

#!/usr/bin/env bash
data_file=${1:-data.json}
case $2 in
   list)
      jq -r '.items[] | "\(.id) - \(.name)"' "$data_file" ;;
   stats)
      jq '{total: (.items | length), sum: (.items | map(.value) | add)}' "$data_file" ;;
   *) echo "Usage: $0 <file> {list|stats}"; exit 2 ;;
esac

Conclusion

With a few core filters and the correct technique for passing arrays, jq becomes a powerful and reliable tool inside any shell workflow. The patterns shown above cover the majority of day-to-day JSON processing needs.

Happy scripting!