OQ Revisited
Over a year ago I published a blog post introducing my jq wrapper oq. Given it has been quite a while since then I decided to rerun the benchmarks I did in the original post with the goal of seeing if things changed, both in regards to oq itself and the greater ecosystem.
Methodology
I tested the latest versions of the packages in the first post. I also added the Go version of yq.
The tested packages, and versions, for the benchmarks include:
-
jq (
1.6) - Installed viapacman -S jq -
yq (go) (
4.6.1) - Downloaded latest binary from the latest GH Release -
yq (python) (
2.12.0) - Installed viapip3 install yq -
oq (
1.2.0) - Built locally with Crystal0.36.1viashards build --production --release
Each package was tested against the same 3 benchmarks in my original post, plus an additional benchmark for XML => JSON, mainly since yq (python) has a dedicated command for handling XML input/output that I didn't account for in the first post. The data is gathered via the GNU Time command. E.g. /usr/bin/time -v.
Setup
OS: 5.10.15-1-MANJARO x86_64
CPU: Intel i7-7700k
Memory: 32GB @ 3,000 MHz
SSD: Samsung 970 EVO Plus - 1TB
Results
This section summarizes the results, with key metrics in table form. See the appendix for the raw benchmark data.
Simple
Simply doing a pass through JSON => JSON then counting the lines on a pretty small input file:
{
"guests": [
{
"name": "Jim",
"age": 17,
"numbers": [
1,
2,
3
]
},
{
"name": "Bob",
"age": 51,
"numbers": [
4,
5,
6
]
},
{
"name": "Susan",
"age": 85,
"numbers": [
7,
8,
9
]
}
]
}
| Name | Max Memory (MB) | Wall Clock Time (seconds) |
|---|---|---|
| jq | 3.04 | 0.02 |
| yq (go) | 6.02 | 0.01 |
| yq (python) | 14.728 | 0.07 |
| oq | 6.536 | 0.07 |
All packages performed relatively equally, nothing noteworthy.
Jeopardy.json
Getting the length of a 53MB JSON file.
The file used: jeopardy.json.
| Name | Max Memory (MB) | Wall Clock Time (seconds) |
|---|---|---|
| jq | 230.32 | 0.66 |
| yq (go) | 705.292 | 2.54 |
| yq (python) | 2,922.996 | 108.89 |
| oq | 230.304 | 0.74 |
This test resulted in some more useful data, causing the true performance of each package to be more obvious. jq and oq fared the best, being pretty comparable in both memory usage and execution time. yq (go) was the next best, using 3x the memory and took ~3.5x longer than oq. yq (python) was by far the worse; using 12.7x the memory and taking 147x longer than oq.
YAML => XML/JSON
Consuming a ~57MB YAML file, and outputting XML and JSON.
The file used: sde/bsd/invItems.yaml from the EVE Online SDE Export.
| Name | Max Memory (MB) | Wall Clock Time (seconds) |
|---|---|---|
| jq | N/A | N/A |
| yq (go) - JSON | 5,202.032 | 32.77 |
| yq (python) - JSON | 5,742.736 | 206.2 |
| yq (python) - XML | 5,742.704 | 217.17 |
| oq - JSON | 575.6 | 14.01 |
| oq - JSON - SimpleYAML | 334.632 | 14.16 |
| oq - XML | 649.984 | 15.77 |
| oq - XML - SimpleYAML | 334.504 | 15.83 |
The final test showed similar metrics as the previous one. jq doesn't support non JSON formats so it was excluded. oq did the best with the execution time being fairly stable, while the memory usage was halved when using the new SimpleYAML format for the input. Interestingly yq (go) fared better execution wise in this test, being only 2.34x slower, but using ~15.5x more memory. yq (python) once again was the worse, with memory usage slightly worse than yq (go), but with an execution time of 15.5x slower than oq.
XML => JSON
Consume the output of the last benchmark, and convert it back to JSON. This test is mainly to not be biased as yq (python) includes a dedicated sub-package xq for handling XML input/output.
| Name | Max Memory (MB) | Wall Clock Time (seconds) |
|---|---|---|
| jq | N/A | N/A |
| yq (go) | N/A | N/A |
| yq (python) | 795.06 | 15.59 |
| oq | 2600.964 | 20.28 |
Neither jq or yq (go) support XML input, so they were excluded. The dedicated sub-package paid off for yq (python), beating oq by a fair margin in both memory consumption and execution time. It looks like it's able to stream the XML input, while XML input for oq is the only input format that cannot be streamed (yet).
Conclusion
It seems the performance of yq (python) has gotten a bit better since last time. Shaving off over 2GB of memory and 2 minutes of execution time in the jeopardy.json benchmark. However, it still is by far the slowest, and most memory hungry package (when not using XML as the input format). yq (go) is in a better spot, but still has pretty high memory usage depending on the exact use case. oq seems to still be the most efficient overall, with both memory and execution time being on par with jq. Thanks to its focus on streaming of data when possible (all but XML input at the moment); oq is able to use much less memory comparatively.
Appendix
Simple
jq
Command being timed: "jq . data.json | wc -l"
User time (seconds): 0.02
System time (seconds): 0.00
Percent of CPU this job got: 100%
Elapsed (wall clock) time (h:mm:ss or m:ss): 0:00.02
Average shared text size (kbytes): 0
Average unshared data size (kbytes): 0
Average stack size (kbytes): 0
Average total size (kbytes): 0
Maximum resident set size (kbytes): 3040
Average resident set size (kbytes): 0
Major (requiring I/O) page faults: 0
Minor (reclaiming a frame) page faults: 288
Voluntary context switches: 1
Involuntary context switches: 0
Swaps: 0
File system inputs: 0
File system outputs: 0
Socket messages sent: 0
Socket messages received: 0
Signals delivered: 0
Page size (bytes): 4096
Exit status: 0
31
yq (go)
Command being timed: "./go-yq -Pj e . data.json | wc -l"
User time (seconds): 0.01
System time (seconds): 0.00
Percent of CPU this job got: 105%
Elapsed (wall clock) time (h:mm:ss or m:ss): 0:00.01
Average shared text size (kbytes): 0
Average unshared data size (kbytes): 0
Average stack size (kbytes): 0
Average total size (kbytes): 0
Maximum resident set size (kbytes): 6020
Average resident set size (kbytes): 0
Major (requiring I/O) page faults: 0
Minor (reclaiming a frame) page faults: 617
Voluntary context switches: 76
Involuntary context switches: 2
Swaps: 0
File system inputs: 0
File system outputs: 0
Socket messages sent: 0
Socket messages received: 0
Signals delivered: 0
Page size (bytes): 4096
Exit status: 0
31
Using the -j and -P options to replicate the default behavior of the other binaries.
yq (python)
Command being timed: "yq . data.json | wc -l"
User time (seconds): 0.06
System time (seconds): 0.00
Percent of CPU this job got: 87%
Elapsed (wall clock) time (h:mm:ss or m:ss): 0:00.07
Average shared text size (kbytes): 0
Average unshared data size (kbytes): 0
Average stack size (kbytes): 0
Average total size (kbytes): 0
Maximum resident set size (kbytes): 14728
Average resident set size (kbytes): 0
Major (requiring I/O) page faults: 4
Minor (reclaiming a frame) page faults: 3637
Voluntary context switches: 43
Involuntary context switches: 1
Swaps: 0
File system inputs: 1328
File system outputs: 0
Socket messages sent: 0
Socket messages received: 0
Signals delivered: 0
Page size (bytes): 4096
Exit status: 0
31
oq
Command being timed: "oq . data.json | wc -l"
User time (seconds): 0.06
System time (seconds): 0.01
Percent of CPU this job got: 104%
Elapsed (wall clock) time (h:mm:ss or m:ss): 0:00.07
Average shared text size (kbytes): 0
Average unshared data size (kbytes): 0
Average stack size (kbytes): 0
Average total size (kbytes): 0
Maximum resident set size (kbytes): 6536
Average resident set size (kbytes): 0
Major (requiring I/O) page faults: 0
Minor (reclaiming a frame) page faults: 808
Voluntary context switches: 58
Involuntary context switches: 2
Swaps: 0
File system inputs: 0
File system outputs: 0
Socket messages sent: 0
Socket messages received: 0
Signals delivered: 0
Page size (bytes): 4096
Exit status: 0
31
Jeopardy.json
jq
216930
Command being timed: "jq length jeopardy.json"
User time (seconds): 0.59
System time (seconds): 0.06
Percent of CPU this job got: 99%
Elapsed (wall clock) time (h:mm:ss or m:ss): 0:00.66
Average shared text size (kbytes): 0
Average unshared data size (kbytes): 0
Average stack size (kbytes): 0
Average total size (kbytes): 0
Maximum resident set size (kbytes): 230320
Average resident set size (kbytes): 0
Major (requiring I/O) page faults: 0
Minor (reclaiming a frame) page faults: 59454
Voluntary context switches: 1
Involuntary context switches: 26
Swaps: 0
File system inputs: 0
File system outputs: 0
Socket messages sent: 0
Socket messages received: 0
Signals delivered: 0
Page size (bytes): 4096
Exit status: 0
yq (go)
216930
Command being timed: "./go-yq e length jeopardy.json"
User time (seconds): 3.41
System time (seconds): 0.22
Percent of CPU this job got: 143%
Elapsed (wall clock) time (h:mm:ss or m:ss): 0:02.54
Average shared text size (kbytes): 0
Average unshared data size (kbytes): 0
Average stack size (kbytes): 0
Average total size (kbytes): 0
Maximum resident set size (kbytes): 705292
Average resident set size (kbytes): 0
Major (requiring I/O) page faults: 0
Minor (reclaiming a frame) page faults: 176999
Voluntary context switches: 787
Involuntary context switches: 163
Swaps: 0
File system inputs: 0
File system outputs: 0
Socket messages sent: 0
Socket messages received: 0
Signals delivered: 0
Page size (bytes): 4096
Exit status: 0
yq (python)
216930
Command being timed: "yq length jeopardy.json"
User time (seconds): 108.62
System time (seconds): 0.86
Percent of CPU this job got: 100%
Elapsed (wall clock) time (h:mm:ss or m:ss): 1:48.89
Average shared text size (kbytes): 0
Average unshared data size (kbytes): 0
Average stack size (kbytes): 0
Average total size (kbytes): 0
Maximum resident set size (kbytes): 2922996
Average resident set size (kbytes): 0
Major (requiring I/O) page faults: 0
Minor (reclaiming a frame) page faults: 816903
Voluntary context switches: 13512
Involuntary context switches: 586
Swaps: 0
File system inputs: 0
File system outputs: 0
Socket messages sent: 0
Socket messages received: 0
Signals delivered: 0
Page size (bytes): 4096
Exit status: 0
oq
216930
Command being timed: "oq length jeopardy.json"
User time (seconds): 0.70
System time (seconds): 0.12
Percent of CPU this job got: 110%
Elapsed (wall clock) time (h:mm:ss or m:ss): 0:00.74
Average shared text size (kbytes): 0
Average unshared data size (kbytes): 0
Average stack size (kbytes): 0
Average total size (kbytes): 0
Maximum resident set size (kbytes): 230304
Average resident set size (kbytes): 0
Major (requiring I/O) page faults: 0
Minor (reclaiming a frame) page faults: 59966
Voluntary context switches: 13569
Involuntary context switches: 7
Swaps: 0
File system inputs: 0
File system outputs: 0
Socket messages sent: 0
Socket messages received: 0
Signals delivered: 0
Page size (bytes): 4096
Exit status: 0
YAML => XML/JSON
jq
N/A
yq (go)
JSON
Command being timed: "./go-yq -Pj e . invItems.yaml > invItems.go-yq.json"
User time (seconds): 47.66
System time (seconds): 1.46
Percent of CPU this job got: 149%
Elapsed (wall clock) time (h:mm:ss or m:ss): 0:32.77
Average shared text size (kbytes): 0
Average unshared data size (kbytes): 0
Average stack size (kbytes): 0
Average total size (kbytes): 0
Maximum resident set size (kbytes): 5202032
Average resident set size (kbytes): 0
Major (requiring I/O) page faults: 0
Minor (reclaiming a frame) page faults: 1395040
Voluntary context switches: 37244
Involuntary context switches: 1922
Swaps: 0
File system inputs: 0
File system outputs: 138984
Socket messages sent: 0
Socket messages received: 0
Signals delivered: 0
Page size (bytes): 4096
Exit status: 0
yq (python)
JSON
Command being timed: "yq . invItems.yaml > invItems.python-yq.json"
User time (seconds): 205.35
System time (seconds): 1.82
Percent of CPU this job got: 100%
Elapsed (wall clock) time (h:mm:ss or m:ss): 3:26.20
Average shared text size (kbytes): 0
Average unshared data size (kbytes): 0
Average stack size (kbytes): 0
Average total size (kbytes): 0
Maximum resident set size (kbytes): 5742736
Average resident set size (kbytes): 0
Major (requiring I/O) page faults: 0
Minor (reclaiming a frame) page faults: 1587436
Voluntary context switches: 13384
Involuntary context switches: 1836
Swaps: 0
File system inputs: 0
File system outputs: 138984
Socket messages sent: 0
Socket messages received: 0
Signals delivered: 0
Page size (bytes): 4096
Exit status: 0
XML
Command being timed: "yq -s -x --xml-root items --xml-dtd {"item": .[] | .} invItems.yaml > invItems.python-yq.xml"
User time (seconds): 215.17
System time (seconds): 2.03
Percent of CPU this job got: 100%
Elapsed (wall clock) time (h:mm:ss or m:ss): 3:37.17
Average shared text size (kbytes): 0
Average unshared data size (kbytes): 0
Average stack size (kbytes): 0
Average total size (kbytes): 0
Maximum resident set size (kbytes): 5742704
Average resident set size (kbytes): 0
Major (requiring I/O) page faults: 0
Minor (reclaiming a frame) page faults: 1683522
Voluntary context switches: 32842
Involuntary context switches: 1031
Swaps: 0
File system inputs: 0
File system outputs: 195072
Socket messages sent: 0
Socket messages received: 0
Signals delivered: 0
Page size (bytes): 4096
Exit status: 0
oq
JSON
Command being timed: "./oq -i yaml . invItems.yaml > invItems.oq.json"
User time (seconds): 12.90
System time (seconds): 12.24
Percent of CPU this job got: 179%
Elapsed (wall clock) time (h:mm:ss or m:ss): 0:14.01
Average shared text size (kbytes): 0
Average unshared data size (kbytes): 0
Average stack size (kbytes): 0
Average total size (kbytes): 0
Maximum resident set size (kbytes): 575600
Average resident set size (kbytes): 0
Major (requiring I/O) page faults: 0
Minor (reclaiming a frame) page faults: 231391
Voluntary context switches: 289799
Involuntary context switches: 166
Swaps: 0
File system inputs: 0
File system outputs: 138984
Socket messages sent: 0
Socket messages received: 0
Signals delivered: 0
Page size (bytes): 4096
Exit status: 0
Command being timed: "./oq -i simpleyaml . invItems.yaml > invItems.oq.simple.json"
User time (seconds): 12.27
System time (seconds): 12.18
Percent of CPU this job got: 172%
Elapsed (wall clock) time (h:mm:ss or m:ss): 0:14.16
Average shared text size (kbytes): 0
Average unshared data size (kbytes): 0
Average stack size (kbytes): 0
Average total size (kbytes): 0
Maximum resident set size (kbytes): 334632
Average resident set size (kbytes): 0
Major (requiring I/O) page faults: 0
Minor (reclaiming a frame) page faults: 89229
Voluntary context switches: 2981745
Involuntary context switches: 246
Swaps: 0
File system inputs: 0
File system outputs: 138984
Socket messages sent: 0
Socket messages received: 0
Signals delivered: 0
Page size (bytes): 4096
Exit status: 0
XML
Command being timed: "./oq -i yaml -o xml --xml-root items . invItems.yaml"
User time (seconds): 16.00
System time (seconds): 12.04
Percent of CPU this job got: 177%
Elapsed (wall clock) time (h:mm:ss or m:ss): 0:15.77
Average shared text size (kbytes): 0
Average unshared data size (kbytes): 0
Average stack size (kbytes): 0
Average total size (kbytes): 0
Maximum resident set size (kbytes): 649984
Average resident set size (kbytes): 0
Major (requiring I/O) page faults: 0
Minor (reclaiming a frame) page faults: 249933
Voluntary context switches: 381174
Involuntary context switches: 266
Swaps: 0
File system inputs: 0
File system outputs: 195072
Socket messages sent: 0
Socket messages received: 0
Signals delivered: 0
Page size (bytes): 4096
Exit status: 0
Example output:
<?xml version="1.0" encoding="utf-8"?>
<items>
<item>
<flagID>0</flagID>
<itemID>0</itemID>
<locationID>0</locationID>
<ownerID>0</ownerID>
<quantity>-1</quantity>
<typeID>0</typeID>
</item>
<item>
<flagID>0</flagID>
<itemID>1</itemID>
<locationID>0</locationID>
<ownerID>0</ownerID>
<quantity>-1</quantity>
<typeID>0</typeID>
</item>
...
</items>
Command being timed: "oq -i simpleyaml -o xml --xml-root items . invItems.yaml"
User time (seconds): 15.27
System time (seconds): 11.99
Percent of CPU this job got: 172%
Elapsed (wall clock) time (h:mm:ss or m:ss): 0:15.83
Average shared text size (kbytes): 0
Average unshared data size (kbytes): 0
Average stack size (kbytes): 0
Average total size (kbytes): 0
Maximum resident set size (kbytes): 334504
Average resident set size (kbytes): 0
Major (requiring I/O) page faults: 0
Minor (reclaiming a frame) page faults: 89233
Voluntary context switches: 3029485
Involuntary context switches: 298
Swaps: 0
File system inputs: 0
File system outputs: 195072
Socket messages sent: 0
Socket messages received: 0
Signals delivered: 0
Page size (bytes): 4096
Exit status: 0
XML => JSON
jq
N/A
yq (go)
N/A
yq (python)
Command being timed: "xq .items invItems.python-yq.xml > invItems.python-yq.xml.json"
User time (seconds): 16.37
System time (seconds): 0.39
Percent of CPU this job got: 107%
Elapsed (wall clock) time (h:mm:ss or m:ss): 0:15.59
Average shared text size (kbytes): 0
Average unshared data size (kbytes): 0
Average stack size (kbytes): 0
Average total size (kbytes): 0
Maximum resident set size (kbytes): 795060
Average resident set size (kbytes): 0
Major (requiring I/O) page faults: 0
Minor (reclaiming a frame) page faults: 341577
Voluntary context switches: 14927
Involuntary context switches: 87
Swaps: 0
File system inputs: 0
File system outputs: 168064
Socket messages sent: 0
Socket messages received: 0
Signals delivered: 0
Page size (bytes): 4096
Exit status: 0
oq
Command being timed: "./oq -i xml .items invItems.oq.xml > invItems.oq.xml.json"
User time (seconds): 20.25
System time (seconds): 16.87
Percent of CPU this job got: 183%
Elapsed (wall clock) time (h:mm:ss or m:ss): 0:20.28
Average shared text size (kbytes): 0
Average unshared data size (kbytes): 0
Average stack size (kbytes): 0
Average total size (kbytes): 0
Maximum resident set size (kbytes): 2600964
Average resident set size (kbytes): 0
Major (requiring I/O) page faults: 0
Minor (reclaiming a frame) page faults: 766666
Voluntary context switches: 2117666
Involuntary context switches: 1249
Swaps: 0
File system inputs: 0
File system outputs: 168064
Socket messages sent: 0
Socket messages received: 0
Signals delivered: 0
Page size (bytes): 4096
Exit status: 0
Top comments (0)