oq - A portable/performant jq wrapper Part 2

OQ Revisited

Over a year ago I published a blog post introducing my jq wrapper oq. Given it has been quite a while since then I decided to rerun the benchmarks I did in the original post with the goal of seeing if things changed, both in regards to oq itself and the greater ecosystem.

Methodology

I tested the latest versions of the packages in the first post. I also added the Go version of yq.

The tested packages, and versions, for the benchmarks include:

• jq (1.6) - Installed via pacman -S jq
• yq (go) (4.6.1) - Downloaded latest binary from the latest GH Release
• yq (python) (2.12.0) - Installed via pip3 install yq
• oq (1.2.0) - Built locally with Crystal 0.36.1 via shards build --production --release

Each package was tested against the same 3 benchmarks in my original post, plus an additional benchmark for XML => JSON, mainly since yq (python) has a dedicated command for handling XML input/output that I didn't account for in the first post. The data is gathered via the GNU Time command. E.g. /usr/bin/time -v.

Setup

OS: 5.10.15-1-MANJARO x86_64
CPU: Intel i7-7700k
Memory: 32GB @ 3,000 MHz
SSD: Samsung 970 EVO Plus - 1TB

Results

This section summarizes the results, with key metrics in table form. See the appendix for the raw benchmark data.

Simple

Simply doing a pass through JSON => JSON then counting the lines on a pretty small input file:

{
  "guests": [
    {
      "name": "Jim",
      "age": 17,
      "numbers": [
        1,
        2,
        3
      ]
    },
    {
      "name": "Bob",
      "age": 51,
      "numbers": [
        4,
        5,
        6
      ]
    },
    {
      "name": "Susan",
      "age": 85,
      "numbers": [
        7,
        8,
        9
      ]
    }
  ]
}

Name	Max Memory (MB)	Wall Clock Time (seconds)
jq	3.04	0.02
yq (go)	6.02	0.01
yq (python)	14.728	0.07
oq	6.536	0.07

All packages performed relatively equally, nothing noteworthy.

Jeopardy.json

Getting the length of a 53MB JSON file.

The file used: jeopardy.json.

Name	Max Memory (MB)	Wall Clock Time (seconds)
jq	230.32	0.66
yq (go)	705.292	2.54
yq (python)	2,922.996	108.89
oq	230.304	0.74

This test resulted in some more useful data, causing the true performance of each package to be more obvious. jq and oq fared the best, being pretty comparable in both memory usage and execution time. yq (go) was the next best, using 3x the memory and took ~3.5x longer than oq. yq (python) was by far the worse; using 12.7x the memory and taking 147x longer than oq.

YAML => XML/JSON

Consuming a ~57MB YAML file, and outputting XML and JSON.

The file used: sde/bsd/invItems.yaml from the EVE Online SDE Export.

Name	Max Memory (MB)	Wall Clock Time (seconds)
jq	N/A	N/A
yq (go) - JSON	5,202.032	32.77
yq (python) - JSON	5,742.736	206.2
yq (python) - XML	5,742.704	217.17
oq - JSON	575.6	14.01
oq - JSON - SimpleYAML	334.632	14.16
oq - XML	649.984	15.77
oq - XML - SimpleYAML	334.504	15.83

The final test showed similar metrics as the previous one. jq doesn't support non JSON formats so it was excluded. oq did the best with the execution time being fairly stable, while the memory usage was halved when using the new SimpleYAML format for the input. Interestingly yq (go) fared better execution wise in this test, being only 2.34x slower, but using ~15.5x more memory. yq (python) once again was the worse, with memory usage slightly worse than yq (go), but with an execution time of 15.5x slower than oq.

XML => JSON

Consume the output of the last benchmark, and convert it back to JSON. This test is mainly to not be biased as yq (python) includes a dedicated sub-package xq for handling XML input/output.

Name	Max Memory (MB)	Wall Clock Time (seconds)
jq	N/A	N/A
yq (go)	N/A	N/A
yq (python)	795.06	15.59
oq	2600.964	20.28

Neither jq or yq (go) support XML input, so they were excluded. The dedicated sub-package paid off for yq (python), beating oq by a fair margin in both memory consumption and execution time. It looks like it's able to stream the XML input, while XML input for oq is the only input format that cannot be streamed (yet).

Conclusion

It seems the performance of yq (python) has gotten a bit better since last time. Shaving off over 2GB of memory and 2 minutes of execution time in the jeopardy.json benchmark. However, it still is by far the slowest, and most memory hungry package (when not using XML as the input format). yq (go) is in a better spot, but still has pretty high memory usage depending on the exact use case. oq seems to still be the most efficient overall, with both memory and execution time being on par with jq. Thanks to its focus on streaming of data when possible (all but XML input at the moment); oq is able to use much less memory comparatively.

Appendix

Simple

jq

Command being timed: "jq . data.json | wc -l"
User time (seconds): 0.02
System time (seconds): 0.00
Percent of CPU this job got: 100%
Elapsed (wall clock) time (h:mm:ss or m:ss): 0:00.02
Average shared text size (kbytes): 0
Average unshared data size (kbytes): 0
Average stack size (kbytes): 0
Average total size (kbytes): 0
Maximum resident set size (kbytes): 3040
Average resident set size (kbytes): 0
Major (requiring I/O) page faults: 0
Minor (reclaiming a frame) page faults: 288
Voluntary context switches: 1
Involuntary context switches: 0
Swaps: 0
File system inputs: 0
File system outputs: 0
Socket messages sent: 0
Socket messages received: 0
Signals delivered: 0
Page size (bytes): 4096
Exit status: 0
31

yq (go)

Command being timed: "./go-yq -Pj e . data.json | wc -l"
User time (seconds): 0.01
System time (seconds): 0.00
Percent of CPU this job got: 105%
Elapsed (wall clock) time (h:mm:ss or m:ss): 0:00.01
Average shared text size (kbytes): 0
Average unshared data size (kbytes): 0
Average stack size (kbytes): 0
Average total size (kbytes): 0
Maximum resident set size (kbytes): 6020
Average resident set size (kbytes): 0
Major (requiring I/O) page faults: 0
Minor (reclaiming a frame) page faults: 617
Voluntary context switches: 76
Involuntary context switches: 2
Swaps: 0
File system inputs: 0
File system outputs: 0
Socket messages sent: 0
Socket messages received: 0
Signals delivered: 0
Page size (bytes): 4096
Exit status: 0
31

Using the -j and -P options to replicate the default behavior of the other binaries.

yq (python)

Command being timed: "yq . data.json | wc -l"
User time (seconds): 0.06
System time (seconds): 0.00
Percent of CPU this job got: 87%
Elapsed (wall clock) time (h:mm:ss or m:ss): 0:00.07
Average shared text size (kbytes): 0
Average unshared data size (kbytes): 0
Average stack size (kbytes): 0
Average total size (kbytes): 0
Maximum resident set size (kbytes): 14728
Average resident set size (kbytes): 0
Major (requiring I/O) page faults: 4
Minor (reclaiming a frame) page faults: 3637
Voluntary context switches: 43
Involuntary context switches: 1
Swaps: 0
File system inputs: 1328
File system outputs: 0
Socket messages sent: 0
Socket messages received: 0
Signals delivered: 0
Page size (bytes): 4096
Exit status: 0
31

oq

Command being timed: "oq . data.json | wc -l"
User time (seconds): 0.06
System time (seconds): 0.01
Percent of CPU this job got: 104%
Elapsed (wall clock) time (h:mm:ss or m:ss): 0:00.07
Average shared text size (kbytes): 0
Average unshared data size (kbytes): 0
Average stack size (kbytes): 0
Average total size (kbytes): 0
Maximum resident set size (kbytes): 6536
Average resident set size (kbytes): 0
Major (requiring I/O) page faults: 0
Minor (reclaiming a frame) page faults: 808
Voluntary context switches: 58
Involuntary context switches: 2
Swaps: 0
File system inputs: 0
File system outputs: 0
Socket messages sent: 0
Socket messages received: 0
Signals delivered: 0
Page size (bytes): 4096
Exit status: 0
31

Jeopardy.json

jq

216930
Command being timed: "jq length jeopardy.json"
User time (seconds): 0.59
System time (seconds): 0.06
Percent of CPU this job got: 99%
Elapsed (wall clock) time (h:mm:ss or m:ss): 0:00.66
Average shared text size (kbytes): 0
Average unshared data size (kbytes): 0
Average stack size (kbytes): 0
Average total size (kbytes): 0
Maximum resident set size (kbytes): 230320
Average resident set size (kbytes): 0
Major (requiring I/O) page faults: 0
Minor (reclaiming a frame) page faults: 59454
Voluntary context switches: 1
Involuntary context switches: 26
Swaps: 0
File system inputs: 0
File system outputs: 0
Socket messages sent: 0
Socket messages received: 0
Signals delivered: 0
Page size (bytes): 4096
Exit status: 0

yq (go)

216930
Command being timed: "./go-yq e length jeopardy.json"
User time (seconds): 3.41
System time (seconds): 0.22
Percent of CPU this job got: 143%
Elapsed (wall clock) time (h:mm:ss or m:ss): 0:02.54
Average shared text size (kbytes): 0
Average unshared data size (kbytes): 0
Average stack size (kbytes): 0
Average total size (kbytes): 0
Maximum resident set size (kbytes): 705292
Average resident set size (kbytes): 0
Major (requiring I/O) page faults: 0
Minor (reclaiming a frame) page faults: 176999
Voluntary context switches: 787
Involuntary context switches: 163
Swaps: 0
File system inputs: 0
File system outputs: 0
Socket messages sent: 0
Socket messages received: 0
Signals delivered: 0
Page size (bytes): 4096
Exit status: 0

yq (python)

216930
Command being timed: "yq length jeopardy.json"
User time (seconds): 108.62
System time (seconds): 0.86
Percent of CPU this job got: 100%
Elapsed (wall clock) time (h:mm:ss or m:ss): 1:48.89
Average shared text size (kbytes): 0
Average unshared data size (kbytes): 0
Average stack size (kbytes): 0
Average total size (kbytes): 0
Maximum resident set size (kbytes): 2922996
Average resident set size (kbytes): 0
Major (requiring I/O) page faults: 0
Minor (reclaiming a frame) page faults: 816903
Voluntary context switches: 13512
Involuntary context switches: 586
Swaps: 0
File system inputs: 0
File system outputs: 0
Socket messages sent: 0
Socket messages received: 0
Signals delivered: 0
Page size (bytes): 4096
Exit status: 0

oq

216930
Command being timed: "oq length jeopardy.json"
User time (seconds): 0.70
System time (seconds): 0.12
Percent of CPU this job got: 110%
Elapsed (wall clock) time (h:mm:ss or m:ss): 0:00.74
Average shared text size (kbytes): 0
Average unshared data size (kbytes): 0
Average stack size (kbytes): 0
Average total size (kbytes): 0
Maximum resident set size (kbytes): 230304
Average resident set size (kbytes): 0
Major (requiring I/O) page faults: 0
Minor (reclaiming a frame) page faults: 59966
Voluntary context switches: 13569
Involuntary context switches: 7
Swaps: 0
File system inputs: 0
File system outputs: 0
Socket messages sent: 0
Socket messages received: 0
Signals delivered: 0
Page size (bytes): 4096
Exit status: 0

YAML => XML/JSON

jq

N/A

yq (go)

JSON

Command being timed: "./go-yq -Pj e . invItems.yaml > invItems.go-yq.json"
User time (seconds): 47.66
System time (seconds): 1.46
Percent of CPU this job got: 149%
Elapsed (wall clock) time (h:mm:ss or m:ss): 0:32.77
Average shared text size (kbytes): 0
Average unshared data size (kbytes): 0
Average stack size (kbytes): 0
Average total size (kbytes): 0
Maximum resident set size (kbytes): 5202032
Average resident set size (kbytes): 0
Major (requiring I/O) page faults: 0
Minor (reclaiming a frame) page faults: 1395040
Voluntary context switches: 37244
Involuntary context switches: 1922
Swaps: 0
File system inputs: 0
File system outputs: 138984
Socket messages sent: 0
Socket messages received: 0
Signals delivered: 0
Page size (bytes): 4096
Exit status: 0

yq (python)

JSON

Command being timed: "yq . invItems.yaml > invItems.python-yq.json"
User time (seconds): 205.35
System time (seconds): 1.82
Percent of CPU this job got: 100%
Elapsed (wall clock) time (h:mm:ss or m:ss): 3:26.20
Average shared text size (kbytes): 0
Average unshared data size (kbytes): 0
Average stack size (kbytes): 0
Average total size (kbytes): 0
Maximum resident set size (kbytes): 5742736
Average resident set size (kbytes): 0
Major (requiring I/O) page faults: 0
Minor (reclaiming a frame) page faults: 1587436
Voluntary context switches: 13384
Involuntary context switches: 1836
Swaps: 0
File system inputs: 0
File system outputs: 138984
Socket messages sent: 0
Socket messages received: 0
Signals delivered: 0
Page size (bytes): 4096
Exit status: 0

XML

Command being timed: "yq -s -x --xml-root items --xml-dtd {"item": .[] | .} invItems.yaml > invItems.python-yq.xml"
User time (seconds): 215.17
System time (seconds): 2.03
Percent of CPU this job got: 100%
Elapsed (wall clock) time (h:mm:ss or m:ss): 3:37.17
Average shared text size (kbytes): 0
Average unshared data size (kbytes): 0
Average stack size (kbytes): 0
Average total size (kbytes): 0
Maximum resident set size (kbytes): 5742704
Average resident set size (kbytes): 0
Major (requiring I/O) page faults: 0
Minor (reclaiming a frame) page faults: 1683522
Voluntary context switches: 32842
Involuntary context switches: 1031
Swaps: 0
File system inputs: 0
File system outputs: 195072
Socket messages sent: 0
Socket messages received: 0
Signals delivered: 0
Page size (bytes): 4096
Exit status: 0

oq

JSON

Command being timed: "./oq -i yaml . invItems.yaml > invItems.oq.json"
User time (seconds): 12.90
System time (seconds): 12.24
Percent of CPU this job got: 179%
Elapsed (wall clock) time (h:mm:ss or m:ss): 0:14.01
Average shared text size (kbytes): 0
Average unshared data size (kbytes): 0
Average stack size (kbytes): 0
Average total size (kbytes): 0
Maximum resident set size (kbytes): 575600
Average resident set size (kbytes): 0
Major (requiring I/O) page faults: 0
Minor (reclaiming a frame) page faults: 231391
Voluntary context switches: 289799
Involuntary context switches: 166
Swaps: 0
File system inputs: 0
File system outputs: 138984
Socket messages sent: 0
Socket messages received: 0
Signals delivered: 0
Page size (bytes): 4096
Exit status: 0

Command being timed: "./oq -i simpleyaml . invItems.yaml > invItems.oq.simple.json"
User time (seconds): 12.27
System time (seconds): 12.18
Percent of CPU this job got: 172%
Elapsed (wall clock) time (h:mm:ss or m:ss): 0:14.16
Average shared text size (kbytes): 0
Average unshared data size (kbytes): 0
Average stack size (kbytes): 0
Average total size (kbytes): 0
Maximum resident set size (kbytes): 334632
Average resident set size (kbytes): 0
Major (requiring I/O) page faults: 0
Minor (reclaiming a frame) page faults: 89229
Voluntary context switches: 2981745
Involuntary context switches: 246
Swaps: 0
File system inputs: 0
File system outputs: 138984
Socket messages sent: 0
Socket messages received: 0
Signals delivered: 0
Page size (bytes): 4096
Exit status: 0

XML

Command being timed: "./oq -i yaml -o xml --xml-root items . invItems.yaml"
User time (seconds): 16.00
System time (seconds): 12.04
Percent of CPU this job got: 177%
Elapsed (wall clock) time (h:mm:ss or m:ss): 0:15.77
Average shared text size (kbytes): 0
Average unshared data size (kbytes): 0
Average stack size (kbytes): 0
Average total size (kbytes): 0
Maximum resident set size (kbytes): 649984
Average resident set size (kbytes): 0
Major (requiring I/O) page faults: 0
Minor (reclaiming a frame) page faults: 249933
Voluntary context switches: 381174
Involuntary context switches: 266
Swaps: 0
File system inputs: 0
File system outputs: 195072
Socket messages sent: 0
Socket messages received: 0
Signals delivered: 0
Page size (bytes): 4096
Exit status: 0

Example output:

<?xml version="1.0" encoding="utf-8"?>
<items>
  <item>
    <flagID>0</flagID>
    <itemID>0</itemID>
    <locationID>0</locationID>
    <ownerID>0</ownerID>
    <quantity>-1</quantity>
    <typeID>0</typeID>
  </item>
  <item>
    <flagID>0</flagID>
    <itemID>1</itemID>
    <locationID>0</locationID>
    <ownerID>0</ownerID>
    <quantity>-1</quantity>
    <typeID>0</typeID>
  </item>
  ...
</items>

Command being timed: "oq -i simpleyaml -o xml --xml-root items . invItems.yaml"
User time (seconds): 15.27
System time (seconds): 11.99
Percent of CPU this job got: 172%
Elapsed (wall clock) time (h:mm:ss or m:ss): 0:15.83
Average shared text size (kbytes): 0
Average unshared data size (kbytes): 0
Average stack size (kbytes): 0
Average total size (kbytes): 0
Maximum resident set size (kbytes): 334504
Average resident set size (kbytes): 0
Major (requiring I/O) page faults: 0
Minor (reclaiming a frame) page faults: 89233
Voluntary context switches: 3029485
Involuntary context switches: 298
Swaps: 0
File system inputs: 0
File system outputs: 195072
Socket messages sent: 0
Socket messages received: 0
Signals delivered: 0
Page size (bytes): 4096
Exit status: 0

XML => JSON

jq

N/A

yq (go)

N/A

yq (python)

Command being timed: "xq .items invItems.python-yq.xml > invItems.python-yq.xml.json"
User time (seconds): 16.37
System time (seconds): 0.39
Percent of CPU this job got: 107%
Elapsed (wall clock) time (h:mm:ss or m:ss): 0:15.59
Average shared text size (kbytes): 0
Average unshared data size (kbytes): 0
Average stack size (kbytes): 0
Average total size (kbytes): 0
Maximum resident set size (kbytes): 795060
Average resident set size (kbytes): 0
Major (requiring I/O) page faults: 0
Minor (reclaiming a frame) page faults: 341577
Voluntary context switches: 14927
Involuntary context switches: 87
Swaps: 0
File system inputs: 0
File system outputs: 168064
Socket messages sent: 0
Socket messages received: 0
Signals delivered: 0
Page size (bytes): 4096
Exit status: 0

oq

Command being timed: "./oq -i xml .items invItems.oq.xml > invItems.oq.xml.json"
User time (seconds): 20.25
System time (seconds): 16.87
Percent of CPU this job got: 183%
Elapsed (wall clock) time (h:mm:ss or m:ss): 0:20.28
Average shared text size (kbytes): 0
Average unshared data size (kbytes): 0
Average stack size (kbytes): 0
Average total size (kbytes): 0
Maximum resident set size (kbytes): 2600964
Average resident set size (kbytes): 0
Major (requiring I/O) page faults: 0
Minor (reclaiming a frame) page faults: 766666
Voluntary context switches: 2117666
Involuntary context switches: 1249
Swaps: 0
File system inputs: 0
File system outputs: 168064
Socket messages sent: 0
Socket messages received: 0
Signals delivered: 0
Page size (bytes): 4096
Exit status: 0