DEV Community

C.C.
C.C.

Posted on

I kept hitting jq: parse error on AI streams, so I built sjq

#ai #cli #opensource #rust

If you work with LLMs, you've done this: curl -N <streaming endpoint> | jq. And you've seen this:

jq: parse error: Invalid numeric literal at line 1, column 9
Enter fullscreen mode Exit fullscreen mode

Column 9. That's where data: ends and the JSON begins. An SSE stream isn't JSON — it's a framing protocol: data: {...} lines separated by blank lines, [DONE] at the end. jq never stood a chance.

So I built sjq — jq, but for AI streaming. A small Rust CLI that parses OpenAI-compatible server-sent event streams (/v1/chat/completions and /v1/responses) and prints one short line per event, showing only what each event added.

What it looks like 

sjq curl -s $BASE/v1/chat/completions \
  -H "Authorization: Bearer $KEY" -H 'Content-Type: application/json' \
  -d '{"model":"gemini-4-argon","stream":true,"messages":[{"role":"user","content":"hi"}]}'
Enter fullscreen mode Exit fullscreen mode 
model: gemini-4-argon
role: assistant
data: 'Hi'
data: ' there'
data: ' 👋'
…
time: ttft=0.42s total=3.1s tok/s=258.2
Enter fullscreen mode Exit fullscreen mode

Instead of 99% envelope JSON, you watch the actual content arrive — plus a timing line: TTFT, total, tokens/sec. Run through sjq curl and it also breaks out connect, TLS handshake, and TTFB, which is genuinely useful when you're debugging a slow provider.

It understands tool calls (name + arguments as they stream in), reasoning_content, refusals, logprobs, usage blocks, and multiple choices — each rendered as its own labeled line.

Install 

brew tap vivgrid/tap
brew trust vivgrid/tap  # Homebrew 6+ requires trusting third-party taps
brew install sjq
Enter fullscreen mode Exit fullscreen mode

Or from source: cargo install --git https://github.com/vivgrid/sjq (one dependency: serde_json).

Why not just strip the framing with sed?

You can — sed 's/^data: //' | jq — and plenty of people do. But then you're staring at full envelope objects per chunk: ids, timestamps, object: "chat.completion.chunk", finish_reason: null fifty times over. The signal (the tokens) drowns in the envelope. sjq inverts that: the delta is the output; the envelope gets summarized once.

MIT licensed, single binary. The repo ships 14 test fixtures covering the gnarly edges: multi-line data payloads, mid-stream errors, parallel tool calls, logprobs, CJK/emoji text.

Repo: https://github.com/vivgrid/sjq — feedback and weird streams welcome.

Top comments (0)