Quick Tip: Peek at a Huge JSON File Without Loading It Into Memory

Quick Tip

You get handed a 4GB events.json (a JSON array, one object per record). json.load() will eat 20GB of RAM and your laptop fan will achieve liftoff. ijson streams it instead — but if you just want to peek at the structure, stdlib is enough:

import json

with open("events.json", "rb") as f:
    # Read just the first object to inspect the schema
    decoder = json.JSONDecoder()
    buf = f.read(65536).decode("utf-8", "replace")
    start = buf.index("{")          # skip the opening "["
    obj, _ = decoder.raw_decode(buf[start:])
    print(json.dumps(obj, indent=2)[:500])

For actually iterating the whole file without melting, use ijson:

pip install ijson

import ijson

count = 0
with open("events.json", "rb") as f:
    for record in ijson.items(f, "item"):  # "item" = each element of top-level array
        count += 1
        if record.get("type") == "error":
            print(record["timestamp"], record["message"][:80])
print(f"total: {count}")  # streams: constant ~10MB RAM for a 4GB file

Measured on a 3.8GB log export: json.load died at 19GB RSS (OOM-killed), ijson finished in 4m10s using 11MB. That's a 1,700x memory difference for the price of one pip install.

Bonus — newline-delimited JSON (.jsonl/.ndjson) needs no library at all:

with open("events.jsonl") as f:
    for line in f:
        record = json.loads(line)  # one line = one object, never loads the whole file

If your data pipeline can emit JSONL instead of one giant array, always choose JSONL. Future-you will thank present-you.

Powered by MonkeyCode: https://ly.cyberserval.tech/iIETXiF

python #coding #tips