DEV Community: Ryan B

From JSONB to Bitmasks: Faster Flags & Cleaner RLS in Postgres with Go/TS

Ryan B — Fri, 26 Sep 2025 15:03:55 +0000

When you're juggling dozens of user preferences and permission levels, it's easy to reach for JSONB columns or sprawling tables of boolean fields. But there's a leaner way: bitmasks. With a single BIGINT column you can encode up to 64 flags, check them in constant time, and keep your schema compact. Pair this with generated columns in Postgres, and you get the best of both worlds - compact storage and human-readable queries.

Note: This comes with some caveats. Bitmasks are fast and small, but they can add complexity - sometimes unnecessarily. Generally I would only implement this approach on a very specific class of project and where the flags are immutable or fixed - migrations can be a pain and over large, long-lived projects it's probably not the best choice.

In the real world, I use this approach when I have a LOT of feature flags that are predictable and can be classified.

What is bitmasking?

Bitmasking is a way of packing many true/false values into a single integer by treating each bit as a flag. For example:

Decimal 1 in binary is 0001 - means flag 0 is set.
Decimal 2 in binary is 0010 - means flag 1 is set.
Decimal 4 in binary is 0100 - means flag 2 is set.
Decimal 8 in binary is 1000 - means flag 3 is set.

You can combine them:

1 | 2 = 3 (binary 0011) - flags 0 and 1 are set.
To check if a flag is present: (value & mask) != 0.
To remove a flag: value & ~mask.

In Postgres, BIGINT gives you 64 bits to work with. In Go, uint64 maps directly to the same concept. In Javascript... well, you get about 10 less just because how numbers work. More on that later.

Why bitmasks?

Compact: one integer column can track 64 permissions/flags/settings.
Fast: bitwise checks are CPU-cheap.
Simple writes: toggle flags with |, &, and ^.

Example schema:

CREATE TABLE users (
  id     BIGSERIAL PRIMARY KEY,
  username TEXT NOT NULL,
  flags  BIGINT  NOT NULL DEFAULT 0, -- permissions
  prefs  BIGINT  NOT NULL DEFAULT 0 -- user preferences
);

Generated columns for readability

Bitmasks are efficient but seven hells... they can be cryptic. You have two options that let you expose named booleans that are indexable and easy to query.

Option A - VIRTUAL (computed on read, zero storage)

With Postgres 18 released, you can use virtual generated columns (read at runtime, not stored). These can't be indexed however, so you have to use an expression to index them.

ALTER TABLE users
ADD COLUMN can_read boolean GENERATED ALWAYS AS 
  ((flags & (1::bigint << 0)) <> 0) VIRTUAL,
ADD COLUMN can_write boolean GENERATED ALWAYS AS
  ((flags & (1::bigint << 1)) <> 0) VIRTUAL,
ADD COLUMN is_admin  boolean GENERATED ALWAYS
  AS ((flags & (1::bigint << 2)) <> 0) VIRTUAL;

Indexing: For VIRTUAL columns, create expression/partial indexes on the underlying expression:

CREATE INDEX users_is_admin_expr_idx
ON users (((flags & (1::bigint << 2)) <> 0));

CREATE INDEX users_can_read_partial_idx
  ON users (id) WHERE (flags & (1::bigint << 0)) <> 0;

Query exactly with those predicates so the planner uses the indexes:

SELECT * FROM users WHERE (flags & (1::bigint << 2)) <> 0;

Option B - STORED (computed on write, indexable by name)

ALTER TABLE users
ADD COLUMN can_read boolean GENERATED ALWAYS AS
  ((flags & (1::bigint << 0)) <> 0) STORED,
ADD COLUMN can_write boolean GENERATED ALWAYS AS
  ((flags & (1::bigint << 1)) <> 0) STORED,
ADD COLUMN is_admin  boolean GENERATED ALWAYS AS
  ((flags & (1::bigint << 2)) <> 0) STORED;

CREATE INDEX users_can_read_idx ON users (can_read);
CREATE INDEX users_is_admin_idx ON users (is_admin);

Whichever option you choose, now your queries look like:

SELECT * FROM users WHERE is_admin;

Instead of juggling (flags & (1<<2)) <> 0 everywhere, you can reference the index. This can keep your queries cleaner.

Common operations

-- grant multiple permissions
UPDATE users SET flags = flags | :mask WHERE id = :id;

-- revoke multiple permissions
UPDATE users SET flags = flags & ~:mask WHERE id = :id;

-- has all bits
SELECT * FROM users WHERE (flags & :mask) = :mask;

-- has any bits
SELECT * FROM users WHERE (flags & :mask) <> 0;

RLS that reads like English

Row-level security policies become cleaner when you can reference virtual/generated booleans:

CREATE POLICY doc_read ON documents
USING (
  owner_id = current_setting('app.user_id')::bigint
  OR EXISTS (
    SELECT 1 FROM users u
    WHERE u.id = current_setting('app.user_id')::bigint
      AND (u.can_read OR u.is_admin)
  )
);

Go integration

On the application side, bitmasks map naturally to Go's uint64 with bitwise helpers:

package perms

const (
  PermRead uint64 = 1 << iota
  PermWrite
  PermAdmin
)

func HasAll(f, m uint64) bool { return f&m == m }
func HasAny(f, m uint64) bool { return f&m != 0 }
func Grant(f, m uint64) uint64 { return f | m }
func Revoke(f, m uint64) uint64 { return f &^ m } // AND-NOT

You can grant or check flags in Go, and Postgres queries can use the same bit positions. Just be careful with the signed BIGINT - practically speaking, avoid using the 63rd bit so values fit safely in int64.

TypeScript / Node.js integration

If you're writing your backend in Node & TypeScript - you can use number for up to 53 bits safely. Remember, JavaScript numbers are IEEE 754 doubles. For a full 64-bit mask you'd want bigint.

Here's a sample with bigint:

export const PermRead = 1n << 0n
export const PermWrite = 1n << 1n
export const PermAdmin = 1n << 2n

export function hasAll(flags: bigint, mask: bigint): boolean {
  return (flags & mask) === mask
}

export function hasAny(flags: bigint, mask: bigint): boolean {
  return (flags & mask) !== 0n
}

export function grant(flags: bigint, mask: bigint): bigint {
  return flags | mask
}

export function revoke(flags: bigint, mask: bigint): bigint {
  return flags & ~mask
}

// usage
let flags = 0n
flags = grant(flags, PermRead | PermWrite)
console.log(hasAll(flags, PermRead | PermWrite)) // true

Indexing strategies

VIRTUAL path: use expression or partial indexes on the bit-flag checks you actually query. Keep the conditions text identical in your queries for index use.
STORED path: create plain btree indexes on the generated boolean columns (or on grouped smallint windows) and query with the column names.
Grouped bits: expose an 8‑bit window as a smallint with another generated column if you filter by ranges.

How to pick (decision box)

Pick VIRTUAL when:

You want zero storage overhead and always‑fresh values.
You're comfortable creating expression/partial indexes on a handful of hot conditions.
You mostly read; write amplification from recompute is a non‑issue (there is none on write).

Pick STORED when:

You want simple indexes by column name and ORM friendliness.
You prefer materialized computed values for consistent ETL.
You expect the same generated columns to be used across many queries and dashboards.

Tip: You can mix them. Make 2-3 high‑traffic flags STORED (index by name) and keep the rest VIRTUAL with expression/partial indexes as needed.

Real-world tradeoffs

Phasing out old flags: You can keep a central registry of bits (position, name, rollout date). When a flag is deprecated, leave the bit unused instead of reassigning it immediately. This keeps historical data sane and avoids subtle bugs when replaying old records.
Mixing with JSONB: Bitmasks shine for hot-path checks (permissions, feature toggles), but you don't want to burn bits for long-tail preferences. Choose your hot & cold split: bitmask for the top 20-30 booleans, JSONB for rare or experimental prefs. Defaults live in code, so missing JSON keys don't break anything.
Migration ceiling: If you run out of bits (63 practical flags), you can add a second BIGINT column (flags2) or move a category of flags to a BIT(n) column. Plan bit ranges by domain (0-15 auth, 16-31 billing, et cetera) so it's easier to split later.
Auditing: Since bit flips can be security-sensitive, consider an audit table (i.e. users_flag_events table) to record who granted/revoked which bits and when.

When not to use bitmasks

If you need per-resource ACLs, use join tables.
If you need user-defined, dynamic permissions, use a roles/permissions graph table.
If you rely on ad-hoc queries across many flags, JSONB or a normalized schema might be better.

Bitmasks in Postgres give you compact storage and fast checks, while generated columns make your SQL ergonomic and indexable. Together, they're a practical middle ground between raw integers and sprawling permission tables - and with a little planning, they can scale cleanly as your flags evolve.

Building SW: Lessons from the Backend

Ryan B — Thu, 18 Sep 2025 13:49:56 +0000

Small warning: This post is about swearing in code.

For the last few months, I've been building SW - a backend service that ingests GitHub events, extracts human-written text, and detects tongue-in-cheek profanity. The idea is lighthearted analytics - not moderation - with consent pipelines baked in at the core. Along the way, I've fought with storage engines, schema design, NLP quirks, and the usual Go pitfalls. This post is about the journey - the problems, dead ends, and the fixes that got me here.

Why?

I was on github a few months ago and I saw a commit that caught my attention:

"Fuck you, dependabot."

It got me to thinking: How often do developers experience this kind of tooling rage? I dug into GH Archive a bit and the answer was pretty clear: It happens... A lot. And not just tooling rage; code breaks all the time, language nuance, self-owns... And while there's been research papers investigating toxicity in programming, no one has dug into what I affectionately call Expletive driven development.

I had questions:

Are there trends in how often people curse in commits?
Any kind of rhythm to it?
Is tooling_rage more pervasive than lang_rage or framework_rage?
Are some languages more likely to cause developers to drop F-bombs than others?

With the zeitgeist of the day being AI & LLMs, I wanted to do it my way: No fancy algos, no machine learning; as close to stdlib as I could get (with some exceptions).

1. Data Volume and Storage

I started with Postgres because it was familiar, battle-tested, and offered strong guarantees. For a while it was fine. But as I scaled, a few issues became unavoidable:

Maintaining views: Materialized views were brittle and expensive to refresh. I was constantly docker compose down and docker compose up to rebuild the db, and views kept slowing me down.
Ingestion time: High-volume inserts triggered deadlocks and lock contention.
Nuances: VACUUM tuning, WAL bloat, index churn - the usual Postgres headaches at scale. Just because we're used to it doesn't mean we can't try different approaches.

Migration: I adopted a split model:

Postgres (v18 beta) -> The Control-plane. Perfect for ingest accounting, consent, and rulepack metadata. Postgres 18 features (like improved parallelism, UUIDv7) gave me better ergonomics without giving up safety.
ClickHouse -> High-volume facts. Utterances and hits live here, with pruning-friendly tables and cheap rollups. In practice, hits are the real workhorse - powering stats and samples - while utterances (events) serve as the canonical backing.

Example (ClickHouse hits schema, simplified):

CREATE TABLE hits (
  bucket_hour   DateTime,
  detver        UInt16,
  lang_code     LowCardinality(Nullable(String)),
  term_id       UInt64,
)
ENGINE = SummingMergeTree
PARTITION BY toYYYYMM(bucket_hour)
ORDER BY (bucket_hour, detver, lang_code, term_id);

This balance - Postgres as the control plane, ClickHouse as the fact store - is where I finally stopped fighting the store.

2. Text Normalization & Detection Drift

Profanity isn't written straight. Developers get creative with leetspeak, homoglyphs, zero-width spaces. And if you over-normalize, you hit Scunthorpe-class false positives.

I built a normalizer pipeline:

func Normalize(s string) string {
    s = utf8Repair(s)
    s = nfkcFold(s)
    s = caseFold(s)
    s = stripZeroWidth(s)
    s = leetMap(s)
    return collapseSpaces(s)
}

Then came detection drift. Early on, I had a flat rules.json. Over time, I introduced the rulepacker JSON format, with templates, lemmas, categories, and versioning baked in. Every hit carries a detector_version - so backfills can compare old vs new results without rewriting history.

{
  "version": 2,
  "templates": [
    {"pattern": "fuck you, {TARGET_BOT}", "category": "bot_rage", "severity": "strong"}
  ],
  "lemmas": ["fuck", "shit"]
}

This let me evolve detection rules without reprocessing 14 years of commits every time. Implementing tried & true algorithms like Aho-Corasick gave me a path I needed to follow, I just needed to wire it up.

3. Language Handling

Language detection is fucking hard. And it sucks that GitHub doesn't provide language metadata on commit messages or issues.

I went through several stages:

Custom hinter: A script-based detector (langhint) that counted Unicode ranges. Rough, but decisive for Japanese/Korean/Arabic.
pg_cld2_detect_language: Hooked into Postgres triggers for per-utterance detection.
ClickHouse beta NLP: Functions like detectLanguageMixed(text_raw) gave me speed and scale.

Snippet from our first hinter:

// DetectScriptAndLang returns a coarse script and a best-effort lang code.
func DetectScriptAndLang(s string) (script string, lang string) {
    // Count letters by Unicode block, map to probable languages
}

Final setup: ClickHouse. While using experimental features isn't something I would recommend, I was already set on using the latest tech anyway (image: postgres:18beta3, GOEXPERIMENT=jsonv2) so at this point it's really just a "throw the baby out with the bathwater" kind of thing.

4. Consent, Anonymity & Guardrails

Analytics without doxxing. That was the bar.

While I initially was storing repo names & github usernames as well as github event ids, I dropped them completely. I turned github user ids and repo ids into HIDs, forced any identity related information into a postgres schema... with strict rules to prevent accidental API leaks. When repo/user information was fetched, the consent boundary would either write full information, or anonymous. If an opt-out exists in the consent receipts, we just don't write anything into CH.

CREATE DOMAIN hid_bytes AS bytea CHECK (octet_length(VALUE) = 32);

-- Minimal FK targets
CREATE TABLE principals_repos  (repo_hid  hid_bytes PRIMARY KEY);
CREATE TABLE principals_actors (actor_hid hid_bytes PRIMARY KEY);

CREATE TABLE repositories (
  repo_hid        hid_bytes  PRIMARY KEY REFERENCES principals_repos(repo_hid) ON DELETE CASCADE,
  consent_id      uuid UNIQUE REFERENCES consent_receipts(consent_id) ON DELETE CASCADE,
  full_name       text,
  default_branch  text,
  primary_lang    text,
  languages       jsonb,
  stars           int,
  forks           int,
  subscribers     int,
  open_issues     int,
  license_key     text,
  -- ...
  gone_at         timestamptz,
  gone_code       int2,
  gone_reason     text
);
CREATE TABLE actors (
  actor_hid        hid_bytes  PRIMARY KEY REFERENCES principals_actors(actor_hid) ON DELETE CASCADE,
  consent_id       uuid UNIQUE REFERENCES consent_receipts(consent_id) ON DELETE CASCADE,
  login            text,
  name             text,
  type             text,
  company          text,
  location         text,
  bio              text,
  blog             text,
  twitter_username text,
  followers        int,
  following        int,
  public_repos     int,
  public_gists     int,
  -- ...
  gone_at          timestamptz,
  gone_code        int2,
  gone_reason      text
);

CREATE SCHEMA ident;

CREATE TABLE ident.gh_repo_map (
  repo_hid   hid_bytes PRIMARY KEY,
  gh_repo_id bigint      NOT NULL,
  seen_at    timestamptz NOT NULL DEFAULT now()
);

CREATE OR REPLACE FUNCTION can_expose_repo(hid public.hid_bytes)
RETURNS boolean;
CREATE OR REPLACE FUNCTION can_expose_actor(hid public.hid_bytes)
RETURNS boolean;

CREATE OR REPLACE FUNCTION ident.resolve_repo
(p_hid hid_bytes, p_who text, p_purpose text)
RETURNS bigint
SECURITY DEFINER;

CREATE OR REPLACE FUNCTION ident.resolve_actor
(p_hid hid_bytes, p_who text, p_purpose text)
RETURNS bigint
SECURITY DEFINER;

I had to respect opt-outs and allow opt-ins with verifiable proof - without OAuth, tokens, or brand-specific files.

Guardrails v2:

Opt-out: Commit a zero-byte dotfile (.<hash>.txt) at repo root.
Opt-in: Create a public gist with the challenge hash.
Receipts: Postgres stores durable receipts.
Revocation: Delete the file or gist, state flips to revoked.

Enforcement:

Opt-outs override everything.
Opt-ins allow scoped de-masking (repo name or actor handle).
By default, everything stays anonymized, and slurs get masked.

This made consent auditable, cheap to verify, and ethically defensible. Repo/actor names would be hashed to allow API consumers to graph a particular idenity over time, but if they say "Hey, I don't mind everyone knowing that I swear like a sailor" (like me), we'd expose their information.

5. Backfill & Replays

Processing a decade and a half of GH Archive means retries are inevitable. Without careful design, retries lead to duplication or drift.

Our fixes:

Deterministic IDs: UUIDv7 + SHA-256. Replays are idempotent.
Leasing: Postgres coordinates hourly leases. A single worker claims an hour, runs it, marks it complete. No collisions, no double work. Multiple workers at the same time is our default, and everything is configurable.
Telemetry: Every lease logs ingest stats (fetch_ms, read_ms, db_ms).

sha := sha256.Sum256(rawJSON)
id := uuidv7FromHash(sha[:])

Every developer should understand this: idempotency is survival.

6. API & Service Design

I standardized on a ports-first architecture, but I also made deliberate choices that Go style purists would call violations:

I use http in package names instead of aliasing everywhere. I believe aliasing should be the responsibility of the importer, and I will die on that hill.
I use explicit service, repo, and module directories for every domain.

Ports strategy:

type StatsPort interface {
    Timeseries(ctx context.Context, in TimeseriesInput) (TimeseriesResp, error)
}

Modules only export ports. This killed cyclic deps, made testing easier, and kept module surfaces boring and predictable.

Wiring is done via Modkit - a package built specificly to alias my internal platform libraries and dependencies. It could be argued that this makes it harder to debug issues, and I will agree with that. I do however counter with "that's what tests are for"; Developer experience can be improved with syntactic sugar - all I cared about was cleaner imports & predictability.

I also had a bit of fun with the service names. Bouncer is the service that handles consent API calls. Nightshift is my rollup & aggregate service. Let's be honest... It's a project about cussing. Who cares?!

7. Observability & Ops

Long backfills + heavy inserts = ops pain if you're blind. I wired observability in from the start:

Structured logs with per-op timings.
ingest_hours table in Postgres tracks hourly telemetry and drops.
ClickHouse queries for compression vs uncompressed sizes.
Nightshift jobs record metrics as they prune, archive, or reverify policy.

Example log:

2025-09-14T15:52:04Z INF ch op args="table=hits rows=1" elapsed_ms=2.663 sql="INSERT BULK"

Ops stopped being about babysitting jobs - I trust the metrics, and when something spikes, it's immediately visible.

8. Data retention policies

The Nightshift service was born out of necessity.

One of the biggest challenges with GH Archive is sheer volume. Public events have been flowing since 2011, adding up to billions of records - on the order of 2 to 4 billion utterances by 2025. In our ClickHouse schema that translates into anywhere from ~160 GB (best case) to over a terabyte (worst case) just to keep the raw utterances and detector hits online. Add a few more years of growth and the storage bill balloons by tens to hundreds of gigabytes per year. Even at cloud-cheap disk prices, that's still way more money just for cold facts we rarely query.

That's why Nightshift exists. Instead of hoarding every commit message and issue comment forever, we roll them up into the a summary - hourly and daily counts, categories, top terms, and language breakdowns. Once that's written, we can safely nuke the raw utterances and hits for that window. The savings are enormous: dropping from hundreds of gigabytes per billion utterances down to a thin slice of summarized facts measured in a few gigabytes. We keep all the trend data we need for charts and exploration, while avoiding the operational nightmare of running a profanity time machine at terabyte scale.

The other reason I did this configurable retention policy is because the project is (will be) open sourced. It should be explorable on a home laptop, and a few gigs is a lot easier to digest.

Takeaways

Postgres -> ClickHouse split was the win. Control-plane vs fact-store is the right balance.
Normalization + rulepacker solved detection drift without reprocessing history.
Language detection is hard, but ClickHouse works. Fallback to lang_code un when it's not clear.
Consent flows built on GitHub primitives (dotfiles, gists) kept things simple and ethical.
Determinism + leasing made replays safe and repeatable.
Ports-first design killed cyclic deps and gave me clean seams, even if it breaks Go orthodoxy.
Observability baked-in made ops boring (and boring is good).
Data growth configuraton kept disk usage low (which is also good).

I'm not ready to share the repo for SW yet, but the backend development is solid: hybrid Postgres/ClickHouse, versioned detector, consent guardrails, replay-safe backfills, and metrics you can trust.

Of course... Now this means I have to write frontend code too... Bloody hell.

Docker, Windows & chasing missing disk space

Ryan B — Tue, 16 Sep 2025 16:29:10 +0000

Honestly, I love Docker. Long gone are the days of XAMPP and installing dependencies on my host machine, and it works so much better day to day than Vagrant ever did.

But even though I'm a Linux fan through and through, I still use Windows in my day to day development. So Docker Desktop on my host machine, WSL 2, with Ubuntu 24 running all my instances. Honestly, it works pretty well and I've been doing it this way since I left my old city (and work computer) behind.

But... WSL and Docker Desktop don't store files directly in NTFS - they live inside giant .vhdx virtual disks. What's more, when you delete stuff inside Ubuntu/Docker the space is freed inside ext4 but the .vhdx file doesn't shrink; it only ever grows.

Over time, that leads to those hundreds of gigs just locked up in AppData. The fix is to compact the VHDX files (with Optimize-VHD or diskpart) after shutting down WSL/Docker, which reclaims the unused blocks and shrinks the files back down.

If you forget to do this occassionally like I do, here's a quick one liner that you can use in Powershell to inspect how much space is used.

@(
  Get-ChildItem "$env:LOCALAPPDATA\Docker\wsl" -Filter *.vhdx -Recurse -ErrorAction SilentlyContinue
  Get-ChildItem "$env:LOCALAPPDATA\Packages\*\LocalState\ext4.vhdx" -File -ErrorAction SilentlyContinue
) | Select-Object FullName, @{n='SizeGB';e={[math]::Round($_.Length/1GB,2)}} |
    Sort-Object SizeGB -Descending

Note: You MAY need to adjust your paths.

Now obviously you may want to prune existing data - dangling docker volumes, images that are just sitting there collecting dust, but I needed space pronto - without affecting any used containers. So I skipped that step.

Then you want to kick off a new Elevated powershell terminal, and run this:

wsl --shutdown; Stop-Process -Name "Docker Desktop" -ErrorAction SilentlyContinue; $vhdx = @(Get-ChildItem "$env:LOCALAPPDATA\Docker\wsl" -Filter *.vhdx -Recurse -ErrorAction SilentlyContinue; Get-ChildItem "$env:LOCALAPPDATA\Packages\*\LocalState\ext4.vhdx" -File -ErrorAction SilentlyContinue); Write-Host "`nBefore:"; $vhdx | Select-Object FullName,@{n='SizeGB';e={[math]::Round($_.Length/1GB,2)}} | Sort-Object SizeGB -Descending; try { Import-Module Hyper-V -ErrorAction Stop; $vhdx | ForEach-Object { Write-Host "Optimizing $_.FullName"; Optimize-VHD -Path $_.FullName -Mode Full } } catch { Write-Warning "Hyper-V PowerShell module not available; falling back to diskpart"; $tmp=[IO.Path]::GetTempFileName(); Set-Content -Path $tmp -Value (($vhdx|ForEach-Object{"select vdisk file=""$($_.FullName)""`ncompact vdisk"}) -join "`r`n"); diskpart /s $tmp; Remove-Item $tmp -Force }; Write-Host "`nAfter:"; $vhdx | Select-Object FullName,@{n='SizeGB';e={[math]::Round((Get-Item $_.FullName).Length/1GB,2)}} | Sort-Object SizeGB -Descending

Obviously, don't run commands copied from the internet on your computer unless you know what they're doing. This will try to:

Stop WSL
Stop Docker Desktop
Find VHDX files under common paths
Evaluate file size.
Use Hyper-V PowerShell to resize each virtual disk; if not available, use diskpart
Evaluate the files again to see how much space was cleared.

Then, go install the latest game you've been wanting to try.

Bloody hell I miss working on Linux. Powershell scripting is awful.

Go 1.25 JSON v2: Benchmarks, Raptor Escapes, and a 1.8 Speedup

Ryan B — Sun, 31 Aug 2025 20:54:54 +0000

For years, Go developers have been stalked by the velociraptor of JSON performance - always close enough to hear the claws, never quite fast enough to outrun it. With Go 1.25, we finally strapped a jetpack to our frameworks.

Near the end of June, I read a roundup on Go 1.25 that reminded me how much of my life I've spent dodging JSON raptors. Let's be honest: JSON is everywhere and kind of awful. As a salty old dinosaur, I still reach for it more than ProtoBuf or MessagePack. So if the language gives me sharper teeth and faster legs... I'm in.

TL;DR: Go 1.25 brings an opt-in overhaul for JSON (encoding/json/v2, plus a lower-level encoding/json/jsontext). Boot 'er up and you'll often see meaningful decode speedups with fewer allocations, coupled with semantics that finally match what most of us meant all along.

Then vs Now: outrunning the old fossils

Then (pre-1.25): using encoding/json often felt like chiseling fossils with a spoon; reflection heavy, allocation hungry & slower than popular third-party libs. Many teams - sometimes mine included - adopted alternatives just to shave off milliseconds.

Now (1.25 with v2): enable the experiment and you get a reworked engine:

faster, especially on Unmarshal paths
fewer surprise behaviours (case sensitivity, dup keys, nil collections)
a lower-level jsontext layer that's strict about what JSON is (UTF-8, uniqueness)

In other words, the spoon got upgraded to a jackhammer - oh, and someone finally put guardrails around the dig site. Cool.

Turn it on with the env var:
GOEXPERIMENT=jsonv2

Caution: json/v2 is opt-in and still evolving; test thoroughly before enabling in prod.

Show me the raptors: real-world benchmarks

You can micro-bench toy structs all day, but the jungle test is big, messy, heterogeneous JSON. We want measurable impact in real-world scenarios. I used GitHub Archive (hourly JSON lines of public events) because it's authless and large.

Dataset

Download any hour you like. I tested with data from this month:

mkdir -p data
wget -O data/2025-08-01-0.json.gz https://data.gharchive.org/2025-08-01-0.json.gz

Benchmark harness

The trick is to load the data once (outside the timed loop), then hammer the decoders. Here's a minimal _test.go that samples and decodes into both a struct and map[string]any:

package ghbench

import (
  "bufio"
  "bytes"
  "compress/gzip"
  "encoding/json"
  jsonv2 "encoding/json/v2" // requires GOEXPERIMENT=jsonv2
  "io"
  "os"
  "testing"
)

const (
  pathGz      = "data/2025-08-01-0.json.gz"
  sampleLines = 5000
)

var samples [][]byte

type EventLoose struct {
  ID        string                 `json:"id"`
  Type      string                 `json:"type"`
  Actor     map[string]any         `json:"actor"`
  Repo      map[string]any         `json:"repo"`
  Payload   map[string]any         `json:"payload"`
  CreatedAt string                 `json:"created_at"`
  Public    *bool                  `json:"public"`
}

func TestMain(m *testing.M) {
  f, err := os.Open(pathGz)
  if err != nil { panic(err) }
  defer f.Close()

  var r io.Reader = f
  if len(pathGz) > 3 && pathGz[len(pathGz)-3:] == ".gz" {
    gz, err := gzip.NewReader(f)
    if err != nil { panic(err) }
    defer gz.Close()
    r = gz
  }

  sc := bufio.NewScanner(r)
  const maxLine = 20 << 20 // 20MB
  sc.Buffer(make([]byte, 0, 1<<20), maxLine)

  for sc.Scan() {
    if len(samples) >= sampleLines { break }
    line := append([]byte(nil), sc.Bytes()...)
    if len(bytes.TrimSpace(line)) > 0 {
      samples = append(samples, line)
    }
  }
  if err := sc.Err(); err != nil { panic(err) }
  if len(samples) == 0 { panic("no samples read") }

  os.Exit(m.Run())
}

func BenchmarkJSON_V1_Loose(b *testing.B) {
  b.ReportAllocs()
  var e EventLoose
  b.ResetTimer()
  for i := 0; i < b.N; i++ {
    for _, line := range samples {
      e = EventLoose{}
      if err := json.Unmarshal(line, &e); err != nil {
        b.Fatalf("v1: %v", err)
      }
    }
  }
}

func BenchmarkJSON_V2_Loose(b *testing.B) {
  b.ReportAllocs()
  var e EventLoose
  b.ResetTimer()
  for i := 0; i < b.N; i++ {
    for _, line := range samples {
      e = EventLoose{}
      if err := jsonv2.Unmarshal(line, &e); err != nil {
        b.Fatalf("v2: %v", err)
      }
    }
  }
}

func BenchmarkJSON_V1_Map(b *testing.B) {
  b.ReportAllocs()
  var m map[string]any
  b.ResetTimer()
  for i := 0; i < b.N; i++ {
    for _, line := range samples {
      m = nil
      if err := json.Unmarshal(line, &m); err != nil {
        b.Fatalf("v1 map: %v", err)
      }
    }
  }
}

func BenchmarkJSON_V2_Map(b *testing.B) {
  b.ReportAllocs()
  var m map[string]any
  b.ResetTimer()
  for i := 0; i < b.N; i++ {
    for _, line := range samples {
      m = nil
      if err := jsonv2.Unmarshal(line, &m); err != nil {
        b.Fatalf("v2 map: %v", err)
      }
    }
  }
}

Our test harness shape scales cost by N * sampleLines. That's okay if runs are short but if samples get large, set a fixed benchtime.

Run it

# Classic
go test -bench=. -benchmem -count=10 > old.txt

# v2 engine + APIs (Go 1.25 only)
GOEXPERIMENT=jsonv2 go test -bench=. -benchmem -count=10 > new.txt

# Compare
go install golang.org/x/perf/cmd/benchstat@latest

benchstat old.txt new.txt

Note: Some of you may need to echo 'export PATH=$PATH:$(go env GOPATH)/bin' >> ~/.bashrc && source ~/.bashrc if benchstat isn't found.
This is where I insert a disclaimer advising against pasting random internet commands into your terminal without knowing what they do, but I don't need to do that... right?

Remember, one of the rules with benchmarks are to run your own! But for me, here were my results.

Results

I ran these on 13th Gen Intel(R) Core(TM) i9-13900KF, Linux amd64.

$ benchstat old.txt new.txt

name             old time/op    new time/op    delta
JSON_V1_Loose-32   116.7ms ±1%    65.7ms ±0%   -44%
JSON_V1_Map-32     122.5ms ±0%    64.5ms ±1%   -47%

name             old alloc/op   new alloc/op   delta
JSON_V1_Loose-32   61.4MiB ±0%   57.1MiB ±0%   -7%
JSON_V1_Map-32     64.6MiB ±0%   59.8MiB ±0%   -8%

name             old allocs/op  new allocs/op  delta
JSON_V1_Loose-32   1.19M ±0%     885k ±0%      -25%
JSON_V1_Map-32     1.29M ±0%     975k ±0%      -24%

Benchmark	V1 (old)	V2 (new)	Improvement
Loose sec/op	116.7 ms	65.7 ms	🚀 ~44% faster
Map sec/op	122.5 ms	64.5 ms	🚀 ~47% faster
Loose B/op	61.4 MiB	57.1 MiB	💾 ~7% less mem
Map B/op	64.6 MiB	59.8 MiB	💾 ~8% less mem
Loose allocs/op	1.19 M	885 k	✂️ ~25% fewer
Map allocs/op	1.29 M	975 k	✂️ ~24% fewer
Overall	—	—	⚡ ≈1.8× faster overall

I saw a consistent ~1.8× speedup with lower memory use and fewer allocations.

You should see lower ns/op and allocs/op for v2 on most heterogeneous payloads. If your raptor doesn't slow down by much, try a bigger sample size, add a typed struct path, or throw in another "real" dataset.

Sneak in your own test for json-iterator/go and bytedance/sonic to the same harness for a 4-way boss fight. I didn't do this because I wasn't interested in third party libs; we want to know only about Go, and Go's chain.

Field guide: v2 behavior changes

These are the semantic guardrails that separate Jurassic quirks from modern predators. Most are wins; a few may require tweaks. These examples are all on the Go Playground, but because I can't set build args on the playground I import github.com/go-json-experiment/json.

1) Naming & tags

A) Case sensitivity: field matching is case-sensitive by default. Fewer "mystery matches" means safer refactors.
B) omitempty: omits based on JSON emptiness (null, [], {}, "") rather than Go's concept of zero values.
C) string tag option: narrower, predictable behaviour (no odd deep conversions).

A) Case-sensitive field matching**

https://go.dev/play/p/MB5BTCPB6yB

summary

package main

import (
    "encoding/json"
    "fmt"

    jsonv2 "github.com/go-json-experiment/json"
)

type User struct{ FullName string } // no tag on purpose

func main() {
    b := []byte(`{"fullname":"Casey"}`) // lower-case key
    var u1, u2 User
    _ = json.Unmarshal(b, &u1)               // v1: case-insensitive match
    _ = jsonv2.Unmarshal(b, &u2)             // v2: case-sensitive (no match)
    fmt.Println("v1 FullName:", u1.FullName) // "Casey"
    fmt.Println("v2 FullName:", u2.FullName) // ""
}

B) omitempty is about JSON emptiness (not Go zero)

https://go.dev/play/p/8_hjrF6ECSC

summary

package main

import (
    "encoding/json"
    "fmt"

    jsonv2 "github.com/go-json-experiment/json"
)

type Box struct{} // encodes to {}

type Demo struct {
    B Box `json:",omitempty"`
}

func main() {
    in := Demo{} // B is a non-pointer struct with zero value

    v1, _ := json.Marshal(in)
    v2, _ := jsonv2.Marshal(in)

    fmt.Println("v1:", string(v1)) // {"B":{}}  (struct is never “empty” to v1's omitempty)
    fmt.Println("v2:", string(v2)) // {}  ({} is JSON-empty, so v2 omits it)
}

C) `,string` tag

https://go.dev/play/p/hvdl5UIOIbD

summary

package main

import (
    "encoding/json"
    "fmt"

    jsonv2 "github.com/go-json-experiment/json"
)

type Demo struct {
    Nums []int `json:",string"`
}

func main() {
    // Elements are strings, not numbers
    input := []byte(`{"Nums":["1","2","3"]}`)

    var a, b Demo
    err1 := json.Unmarshal(input, &a)   // v1: invalid use of ,string on []int
    err2 := jsonv2.Unmarshal(input, &b) // v2: OK - applies to each element

    fmt.Printf("v1 Nums=%v err=%v\n", a.Nums, err1)
    fmt.Printf("v2 Nums=%v err=%v\n", b.Nums, err2)
}

The interesting differences with ,string in v2 show up in edge cases. Your test suite can cover stricter semantics in real code.

2) Collections & binaries

A) Nil slices/maps: marshal as [] / {} (not null). Client code rejoices.
B) Arrays: fixed lengths are enforced on unmarshal - no silent trim/extend.
C) Byte arrays: encoded as base64 strings, not integer lists.

A) Nil slices/maps: []/{} (not null)

https://go.dev/play/p/tJqxHht8lMA

summary

package main

import (
    "encoding/json"
    "fmt"

    jsonv2 "github.com/go-json-experiment/json"
)

func main() {
    var s []int          // nil slice
    var m map[string]int // nil map
    b1s, _ := json.Marshal(s)
    b1m, _ := json.Marshal(m)
    b2s, _ := jsonv2.Marshal(s)
    b2m, _ := jsonv2.Marshal(m)
    fmt.Println("v1 slice:", string(b1s)) // null
    fmt.Println("v1 map: ", string(b1m))  // null
    fmt.Println("v2 slice:", string(b2s)) // []
    fmt.Println("v2 map: ", string(b2m))  // {}
}

If your clients expect null, verify and adapt.

B) Arrays: fixed length enforced

https://go.dev/play/p/G9EBWREyLub

summary

package main

import (
    "encoding/json"
    "fmt"

    jsonv2 "github.com/go-json-experiment/json"
)

type Trio [3]int

func main() {
    good := []byte(`[1,2,3]`)
    short := []byte(`[1,2]`)
    var a, b Trio
    _ = json.Unmarshal(good, &a) // ok in v1
    fmt.Println("v1 Trio:", a)
    // v1 on short: silently fills first elements, leaves the rest zero
    _ = json.Unmarshal(short, &a)
    fmt.Println("v1 short Trio:", a)
    // v2: strict about array length
    fmt.Println("v2 good err:", jsonv2.Unmarshal(good, &b))
    fmt.Println("v2 short err:", jsonv2.Unmarshal(short, &b)) // expect error
}

C) Byte arrays: base64 string (not number lists)

https://go.dev/play/p/1xLblPna3gi

summary

package main

import (
    "encoding/json"
    "fmt"

    jsonv2 "github.com/go-json-experiment/json"
)

func main() {
    s := []byte{1, 2, 3}
    a := [3]byte{1, 2, 3}
    b1s, _ := json.Marshal(s)   // v1: "AQID"
    b1a, _ := json.Marshal(a)   // v1: [1,2,3]
    b2s, _ := jsonv2.Marshal(s) // v2: "AQID"
    b2a, _ := jsonv2.Marshal(a) // v2: "AQID"   // (byte *array* also base64)
    fmt.Println("v1:", string(b1s))
    fmt.Println("v1:", string(b1a))
    fmt.Println("v2:", string(b2s))
    fmt.Println("v2:", string(b2a))
}

3) Consistency knobs

A) Pointer receiver methods: MarshalJSON / UnmarshalJSON with pointer receivers are handled consistently.
B) Merging vs replacing: null clears; objects merge; scalars replace. Goodbye half-merges.

A) Pointer receiver methods are honored consistently

https://go.dev/play/p/4NdtlaH_9eK

summary

package main

import (
    "encoding/json"
    "fmt"

    jsonv2 "github.com/go-json-experiment/json"
)

type T struct{ X int }

func (t *T) MarshalJSON() ([]byte, error) { return []byte(`{"x":` + fmt.Sprint(t.X) + `}`), nil }

func main() {
    v := T{X: 7}
    b1, _ := json.Marshal(v)   // v1 behavior can be surprising for non-pointer receivers
    b2, _ := jsonv2.Marshal(v) // v2 is more consistent about honoring pointer-receiver methods; if you relied on v1's value/pointer quirks, add tests
    fmt.Println("v1:", string(b1))
    fmt.Println("v2:", string(b2))
}

B) Merge vs replace on updates

https://go.dev/play/p/nPkIiUNBQ4z

summary

package main

import (
    "encoding/json"
    "fmt"

    jsonv2 "github.com/go-json-experiment/json"
)

type Profile struct {
    Name string
    Meta map[string]any
}

func main() {
    dst1 := Profile{Name: "Raptor", Meta: map[string]any{"a": 1, "b": 2}}
    patch := []byte(`{"Meta":{"b":99},"Name":null}`)
    _ = json.Unmarshal(patch, &dst1) // v1: merging behavior historically inconsistent
    fmt.Println("v1:", dst1)

    dst2 := Profile{Name: "Raptor", Meta: map[string]any{"a": 1, "b": 2}}
    _ = jsonv2.Unmarshal(patch, &dst2) // v2: objects merge; null clears; scalars replace
    fmt.Println("v2:", dst2)
}

Expect Name cleared in v2, Meta.b updated to 99 (with a preserved). Exact v1 behavior may vary by shape.

4) Safety & correctness

A) Duplicate object names: now an error on both read and write.
B) HTML: HTML isn't escaped by default.
C) UTF-8: stricter UTF-8 (invalid is error).
D) Map key order: no guaranteed sorting (faster). If you need determinism for tests, sort before marshal.

A) Duplicate object names: v2 errors

https://go.dev/play/p/x60uPnwX8-U

summary

package main

import (
    "encoding/json"
    "fmt"

    jsonv2 "github.com/go-json-experiment/json"
)

func main() {
    dup := []byte(`{"a":1,"a":2}`)
    var m1, m2 map[string]int
    _ = json.Unmarshal(dup, &m1)       // v1: last one wins
    err := jsonv2.Unmarshal(dup, &m2)  // v2: error on duplicates
    fmt.Println("v1 m:", m1)           // map[a:2]
    fmt.Println("v2 err:", err != nil) // true
}

B) HTML escaping: v1 escapes by default; v2 doesn't

https://go.dev/play/p/CU8rV0R-GmI

summary

package main

import (
    "encoding/json"
    "fmt"

    jsonv2 "github.com/go-json-experiment/json"
)

func main() {
    s := map[string]string{"h": "<tag>&"}
    b1, _ := json.Marshal(s)   // v1: {"h":"\u003ctag\u003e\u0026"}
    b2, _ := jsonv2.Marshal(s) // v2: {"h":"<tag>&"}
    fmt.Println("v1:", string(b1))
    fmt.Println("v2:", string(b2))
}

C) UTF-8 validation: v2 is strict

https://go.dev/play/p/8npfRTcQafv

summary

package main

import (
    "encoding/json"
    "fmt"

    jsonv2 "github.com/go-json-experiment/json"
)

func main() {
    // {"x":"a\xffb"} – 0xff is invalid UTF-8 inside a JSON string
    bad := []byte{'{', '"', 'x', '"', ':', '"', 'a', 0xff, 'b', '"', '}'}

    var v1, v2 map[string]string
    err1 := json.Unmarshal(bad, &v1)   // v1 may accept & coerce to U+FFFD
    err2 := jsonv2.Unmarshal(bad, &v2) // v2: error on invalid UTF-8

    fmt.Printf("v1 err: %v\n", err1)
    fmt.Printf("v1 x: %q\n", v1["x"]) // usually "a�b"
    fmt.Printf("v2 err: %v\n", err2)  // non-nil
}

D) Map key order: no guaranteed sorting

https://go.dev/play/p/1Vpr6LLKDOe

summary

package main

import (
    "encoding/json"
    "fmt"

    jsonv2 "github.com/go-json-experiment/json"
)

func main() {
    m := map[string]int{"b": 2, "a": 1}
    b1, _ := json.Marshal(m)   // v1: sorted keys -> {"a":1,"b":2}
    b2, _ := jsonv2.Marshal(m) // v2: order not guaranteed
    fmt.Println("v1:", string(b1))
    fmt.Println("v2:", string(b2))
}

Final Result

End result? Fewer sharp edges, fewer "it works on my computer" bugs in production.

I should point out that we'll pay a small tax for checks like duplicate keys, but the crash-proofing is worth it.

Migration playbook (a.k.a. "don't get eaten")

1) Flip it in CI

Run your suite with and without:

GOEXPERIMENT=jsonv2

This lets you discover behavior diffs before you submit your pull request.

2) Audit your hot spots

Endpoints relying on case-insensitive struct binding.
Tests that assert sorted JSON byte-for-byte.
Places that tolerated duplicate keys or invalid UTF-8.
Frontends that expect null instead of []/{}.

3) Make two decisions

time.Duration representation (string vs number).
omitempty expectations for your public API (document it).

4) Be explicit

If you want deterministic JSON for diffs, manually sort keys before marshal.
If dup keys may arrive, pre-clean inputs (v2 will reject them).

5) Celebrate the wins

Share your benchstat table.
Include a flamegraph before/after if you want to flex. I won't mind, promise.

Closing

While the zeitgeist is AI, Rust, and whatever's going on in TypeScript land, Go 1.25 quietly sharpened the tools most of us use every day. It didn't just add speed - it drained the swamp and posted signs so we stop stepping on the same branches twice.

Of course it's never recommended to turn on experimental flags in production code, we all want fewer dependencies, fewer moving parts, fewer places things go wrong. With modern large codebases serializing and deserializing millions of JSON payloads a day, a 50% speed boost across the top is nothing to sneer at. I've played with json/v2 quite a lot, and it seems stable.

The JSON raptors still lurk, but with json/v2 we've got a flare gun, night-vision goggles, and a pack of loyal compys.

Lace up.
Benchmark.
Ship.

Dear Recruiters...

Ryan B — Fri, 28 Jul 2023 17:37:21 +0000

Dear recruiters,

Thank you for letting me know about the exciting / fantastic / other overused expletive you use to describe your opportunity. I'm repeatedly thrilled to learn that you don't understand my resume, job experience or the industry you're recruiting for at all, and dodge inquiries to test your understanding of the job by asking me counter questions.

I am happy that you decide to reach out to me with a brief description to get me interested, but fail to tell me anything about the company or the work I'd supposedly be doing. No, really, please keep sending me spam about your exciting opportunities. I love the fact that you're a middleman when I could be better spend my time wowing a potential employer rather than jumping through your hoops. I feel great knowing you get a placement bonus if I am offered and accept a job, when secretly you regard me as mere meat for the grinder.

I'm delighted you have chosen to add my profile to your internal database of candidates without my consent, and that the security of your system is worse than the PERL code I wrote in my teens. No, it's fine, please keep a database of my prior addresses, financial information and phone numbers.

Moreover, I'm still happy to work with you despite spending twenty years in this industry and not a single blasted time being placed by you or anyone of your ilk. Trying to get me to accept a lower salary than I am worth puts a smile on my face, and I would love it if you continued.

Oh, and if I say I'm not interested in an opportunity - however exciting - I would be more than happy to send you my Rolodex of developers I've worked with; I'm certain that they'll enjoy being headhunted, contacted out of the blue, and otherwise spammed regardless if they're happy with their current employment or not.

From the bottom of my heart, thank you. You bottom-feeding parasites are a much cherished benefit to my industry, and I look forward to doing the same song and dance with you in the future.

Charmingly Yours,

-Ryan

Authors Note: This post is designed for humour only, and not intended to devalue technical recruiters, be inflammatory or cause deep discussions about recruiters in your province/state/country/city.

DEV Community: Ryan B

From JSONB to Bitmasks: Faster Flags & Cleaner RLS in Postgres with Go/TS

What is bitmasking?

Why bitmasks?

Generated columns for readability

Option A - VIRTUAL (computed on read, zero storage)

Option B - STORED (computed on write, indexable by name)

Common operations

RLS that reads like English

Go integration

TypeScript / Node.js integration

Indexing strategies

How to pick (decision box)

Real-world tradeoffs

When not to use bitmasks

Building SW: Lessons from the Backend

Why?

1. Data Volume and Storage

2. Text Normalization & Detection Drift

3. Language Handling

4. Consent, Anonymity & Guardrails

5. Backfill & Replays

6. API & Service Design

7. Observability & Ops

8. Data retention policies

Takeaways

Docker, Windows & chasing missing disk space

Go 1.25 JSON v2: Benchmarks, Raptor Escapes, and a 1.8 Speedup

Then vs Now: outrunning the old fossils

Show me the raptors: real-world benchmarks

Dataset

Benchmark harness

Run it

Results

Field guide: v2 behavior changes

1) Naming & tags

A) Case-sensitive field matching**

B) omitempty is about JSON emptiness (not Go zero)

C) ,string tag

2) Collections & binaries

A) Nil slices/maps: []/{} (not null)

B) Arrays: fixed length enforced

C) Byte arrays: base64 string (not number lists)

3) Consistency knobs

A) Pointer receiver methods are honored consistently

B) Merge vs replace on updates

4) Safety & correctness

A) Duplicate object names: v2 errors

B) HTML escaping: v1 escapes by default; v2 doesn't

C) UTF-8 validation: v2 is strict

D) Map key order: no guaranteed sorting

Final Result

Migration playbook (a.k.a. "don't get eaten")

1) Flip it in CI

2) Audit your hot spots

3) Make two decisions

4) Be explicit

5) Celebrate the wins

Closing

Dear Recruiters...

C) `,string` tag