JSON Manipulation

There are many ways to manipulate JSON. I reviewed a few rapid ways today, which are using a command line tool called jq and libraries that support JSONPath query language.

jq

Awk is a powerful domain-specific language for text processing, but it lacks built-in support for manipulating hierarchical data structures like trees. jq fills this gap by providing a tool for transforming JSON data. However, one potential drawback is that jq requires a separate installation, which may add complexity to my workflow.

So I write a shell script using jq to extract user's names and user's urls from a cURL response, and then output it in TSV format.

curl -s 'https://mstdn.in.th/api/v1/timelines/public?limit=10' | jq '.[].account | [.username, .url] | @tsv' -r

The result looks like this:

kuketzblog      https://social.tchncs.de/@kuketzblog
cats    https://social.goose.rodeo/@cats
AlJazeera       https://flipboard.com/@AlJazeera
TheHindu        https://flipboard.com/@TheHindu
GossiTheDog     https://cyberplace.social/@GossiTheDog
kuketzblog      https://social.tchncs.de/@kuketzblog
weeklyOSM       https://en.osm.town/@weeklyOSM
juanbellas      https://masto.es/@juanbellas
noborusudou     https://misskey.io/@noborusudou
jerryd  https://mastodon.social/@jerryd

With a TSV file, we can use Awk, sed, etc. to manipulate them as usual.

JSONPath

JSONPath, which was explained in RFC 9535, is supported by many libraries and applications, e.g. PostgreSQL. Still, I try to it in Python by the jsonpath_nq library.

from jsonpath_ng import parse
import requests


res = requests.get("https://mstdn.in.th/api/v1/timelines/public?limit=10")

compiled_path = parse("$[*].account")


for matched_node in compiled_path.find(res.json()):
    print(matched_node.value["username"] + "\t" + matched_node.value["url"])

It gave the same result to the shell script above. The code is a bit longer than the shell script. However, it can be integrated with many Python libraries.