DEV Community: Ville Vesilehto

Go disassembly: branches on arm64 vs amd64

Ville Vesilehto — Mon, 17 Nov 2025 18:18:55 +0000

TL;DR

On arm64 the negative-branch check typically compiles to a single-bit test + branch (e.g., TBNZ on the sign bit). On amd64 it's a flags-setting ALU instruction (TESTQ) followed by a conditional jump and an unconditional jump.
We'll compile and disassemble a small toy program written in Go.
A simple micro-benchmark suggests that in this toy program the conditional itself costs very little compared to everything around it. In real code predictability usually matters more than the exact encoding of the branch.

The toy program

We'll keep the program trivial on purpose: call getNumber(), branch on x < 0, print, done. Since we're targeting 64-bit GOARCH values (arm64 and amd64), int is 64-bit and the sign bit we care about is bit 63.

Install the latest Go version. I've used v1.25.4 here. Then save this main program into a directory.

main.go:

package main

import (
    "fmt"
    "math/rand"
)

func getNumber() int { return int(int64(rand.Uint64())) }

func main() {
    x := getNumber()
    if x < 0 {
        fmt.Println("branch")
        return
    }
    fmt.Println("main")
}

Build a version with and without inlining/optimisations for both architectures. The unoptimised one uses gcflags to:

Disable inlining with -l
Disable most optimisations with -N

These flags are passed to go tool compile at build time. See compile cmd docs for full list of flags and background info.

# arm64
GOARCH=arm64 go build -o bin/main_arm64_optimised .
GOARCH=arm64 go build -gcflags='all=-N -l' -o bin/main_arm64_unoptimised .

# amd64
GOARCH=amd64 go build -o bin/main_amd64_optimised .
GOARCH=amd64 go build -gcflags='all=-N -l' -o bin/main_amd64_unoptimised .

Side note, read this post by Dave Cheney if you're interested how the mid-stack inlining works in Go!

Disassemble the things

Recall that our main point of interest is in line 12:

if x < 0 { ... }

ARM64 disassembly

Let's first disassemble the unoptimised arm64 binary, by outputting the main symbols only.

go tool objdump -s main.main bin/main_arm64_unoptimised

This yields the following:

main.go:12    0x1000bc540    b7f80040    TBNZ $63, R0, 2(PC)
main.go:12    0x1000bc544    14000015    JMP 21(PC)

First three columns:

Source line main.go:12 to which the instruction maps to.
Instruction addresses 0x1000bc540 and 0x1000bc544.
Encoded machine bytes b7f80040 and 14000015.

TBNZ is an AArch64-specific instruction which literally means "test bit if non-zero".

In this case it tests if bit 63 (the sign bit) of R0 (referring to variable x) is non-zero.
If it is, then offset by 2 instructions and jump forward, past the next instruction, and enter the negative branch body. The 2(PC) means "skip the next instruction", because each AArch64 instruction is 4 bytes.
If it isn't, then the JMP instruction is executed and we go back to the join/else path in the main function.

Let's compare it to the optimised one:

go tool objdump -s main.main bin/main_arm64_optimised

This yields:

main.go:12    0x10009e180    b6f80220    TBZ $63, R0, 17(PC)

TBZ is another AArch64-specific instruction which means "test bit and branch if zero". But why is it TBZ here (and not TBNZ or CMP)?

Our condition is x < 0, i.e. "sign bit set". After rand.Uint64() the value is cast to int64 without changing bits, so "negative" is exactly "bit 63 is 1". The compiler recognizes this as a sign-bit check and lowers it to a single bit test plus branch.

In this build, the compiler lays out the "x < 0" path as the fallthrough and the "x ≥ 0" path later, so it can encode the condition as a single TBZ that branches over the fallthrough when the sign bit is zero. If the layout flipped, you'd see TBNZ or even a compare-and-branch sequence instead.

Conceptually the surrounding objdump looks roughly like this:

main.go:10   prologue / stack check
main.go:8    CALL math/rand.Uint64
main.go:12   TBZ $63, R0, 17(PC)   # if sign bit == 0 → jump forward (x >= 0)
main.go:13   …                     # fallthrough: x < 0 path
                                   # → fmt.Fprintln("branch"), return
…                return
main.go:16        …                # target of TBZ: x >= 0 path¨
                                   # → fmt.Fprintln("main"), return

So:

If x < 0 (sign bit = 1): TBZ not taken -> execute "branch" path and return.
If x ≥ 0 (sign bit = 0): TBZ taken -> skip over the branch path into the "main" path.

You may still see a CMP/BPL style sequence in other shapes of code (e.g., range checks like x < 100) or with different optimization decisions. A pure sign test is where TBZ/TBNZ shines.

All in all a pretty nice optimisation. Exact block layout and call sequences can change between Go versions and build flags, so don't rely on this exact listing, just the general shape.

AMD64 disassembly

Let's then disassemble the AMD64 unoptimised binary:

go tool objdump -s main.main bin/main_amd64_unoptimised

From line 12 we get this:

main.go:12    0x10d0f00    4885c0    TESTQ AX, AX
main.go:12    x10d0f03       7c02    JL 0x10d0f07
main.go:12    x10d0f05       eb54    JMP 0x10d0f5b

Before moving forwards, it's worth mentioning that on AMD64 (x86-64) the processor status register RFLAGS has these flags which are set by ALU operations:

ZF (bit 6): zero flag set to 1 if the result of the operation is 0, otherwise 0.
SF (bit 7): sign flag, copies the most significant bit of the result, and for signed values 1 means negative.
OF (bit 11): overflow flag, set to 1 if a signed overflow occurred.

On AArch64 you also have NZCV condition flags and generic conditional branches (B.EQ, B.LT, etc.), but TBZ/TBNZ bypass these condition flags and test the selected bit directly.

Breaking it down to a mouthful:

TESTQ AX, AX is a bitwise test of AX against itself. It leaves AX unchanged and only sets flags (ZF set if x==0, SF reflects the sign, OF is cleared).
JL 0x10d0f07 is "jump if less" for signed values, which means "branch if SF != OF". After TESTQ we know OF=0, so JL is effectively "jump if SF==1", i.e., if the sign bit is set.
JMP 0x10d0f5b is an unconditional jump that skips over the negative-branch block to the "x >= 0" (or "main") path.

Micro-benchmark: does any of this matter?

Let's do a micro-benchmark that's mostly about "shape". We don't measure branch mispredict cost. Just that the single "x < 0" check does not dominate the function.

Save this as bench_test.go next to main.go:

package main

import "testing"

var sink int

func hotpathPredictable(x int) {
    if x < 0 {
        sink++
    }
}

func hotpathRandom(x int) {
    if x < 0 {
        sink++
    }
}

func BenchmarkBranchPredictable(b *testing.B) {
    x := -1
    for i := 0; i < b.N; i++ {
        hotpathPredictable(x)
    }
}

func BenchmarkBranchRandom(b *testing.B) {
    var x int
    r := uint64(1)
    for i := 0; i < b.N; i++ {
        // cheap LCG to flip sign bit “randomly”
        r = r*1103515245 + 12345
        x = int(int64(r)) // sign depends on high bit of r
        hotpathRandom(x)
    }
}

Run it with default settings, letting full inline and optimisation do its thing:

go test -count=10 -bench=BenchmarkBranchPredictable | tee predictable_optimised.out

Then run it by disabling inlining and optimisations:

go test -count=10 -bench=BenchmarkBranchPredictable -gcflags=all='-N -l' | tee predictable_unoptimised.out

And the same for the random version:

go test -count=10 -bench=BenchmarkBranchRandom | tee random_optimised.out
go test -count=10 -bench=BenchmarkBranchRandom -gcflags=all='-N -l' | tee random_unoptimised.out

On my M1 Pro (arm64), the predictable branch averages about 2.2 ns/op. Making the condition pseudo-random via a tiny integer LCG bumps that to about 3.6 ns/op in an optimised build (~1.6× slower), and to about 5.7 ns/op without optimisations.

The predictable case barely changes between optimised and unoptimised builds, while the random case is hit much harder. That matches the intuitive picture: the branch itself is cheap when the predictor can learn it, but once you add noise and extra work around it, the cost of missed predictions and extra instructions starts to dominate.

Closing

This is a tiny example, but it's a nice lens into how Go maps simple conditions onto each instruction set architecture (or ISA).

Finally, I do want to mention that the micro-benchmark here is intentionally simple and not a rigorous study of branch prediction. Its main purpose is to show that the conditional itself compiles to a handful of instructions and doesn't dominate runtime in this toy.

Brand Tagging with VLMs

Ville Vesilehto — Sat, 15 Nov 2025 20:58:39 +0000

TL;DR

Build a two-stage logo pipeline:

Retrieval - generate image embeddings for small crops and match against a logo dictionary with FAISS cosine search. Use SigLIP-2 (NaFlex) so logos are not distorted and small marks still pop.
Verification - for top matches, ask LLaVA-OneVision-1.5 a strict JSON question ("Is this the X logo?") and accept only high-confidence "yes". It's a good model sir.

A bit longer post this time around.

Intro

Brand tagging in real-world video is hard: logos are tiny, partly occluded, moving, and often appear on textured backgrounds. A practical approach is a two-stage pipeline:

First retrieve likely logo crops with a fast contrastive image encoder.
Then verify each candidate with a vision-language model (VLM) that can read text and reason about shapes and context. Each candidate is a (frame, bbox, brand, retrieval_score) record that we pass to the VLM.

This tutorial combines a modern image-text encoder (SigLIP-2, NaFlex variant) for high-recall retrieval with a VLM from the LLaVA family for precise, structured yes/no verification.

You could train a YOLO-style detector for specific brands. This post, however, focuses on a more flexible "embedding + VLM" approach that adapts quickly to new logos without retraining. Some might find this fun.

Research

Contrastive encoders such as CLIP/SigLIP produce embeddings where similar visuals are close. FAISS makes nearest-neighbor search over many references instantaneous. You can treat logo search as nearest-neighbor lookup instead of training a custom detector.

Verification reduces false positives. A VLM can explicitly answer "Is this the Red Bull logo?" and justify the decision, improving precision on lookalikes, partial views, and blur.

Pointers to the underlying research for the initiated:

Setup

You’ll need to be comfortable with basic Python and PyTorch. No prior experience with FAISS or VLMs required.

Hardware/software requirements:

At least one NVIDIA GPU (at least 32 GB VRAM recommended)
CUDA 12.x, Python 3.10–3.12
Disk: 5–10 GB (models + caches)
Tools: ffmpeg for frames

This tutorial was built and tested on a single NVIDIA H200.

Prepare the Python environment as follows:

python3 -m venv .venv && source .venv/bin/activate
pip install -U torch torchvision torchaudio \
  "transformers>=4.45" accelerate pillow opencv-python faiss-cpu \
  numpy pydantic polars

For simplicity we'll use ƒaiss-cpu as it's fine at this scale. It's a single logo after all. GPU for the models.

Example video (Creative Commons):

Title: Red Bull Racing Pit Stop Practice (2015), 44 s, 1920x1080.
License: CC BY-SA 4.0 - attribute ProtoplasmaKid / Wikimedia Commons.
It has tiny, moving logos, occlusion, uniforms, car bodywork, pit rig and acts as a great stress-test.

Architecture

frames (2–4 FPS) ──▶ crops (multi-scale grid)
         │
         ▼
 [Stage 1] Retrieval (SigLIP-2 image features + FAISS cosine)
         │      └─ top-K per frame/brand with heuristics
         ▼
 [Stage 2] VLM verification (LLaVA-OneVision-1.5 JSON verdict)
         │
         ▼
    JSONL evidence

Models used in this post:

SigLIP-2 reports better zero-shot and retrieval performance than prior SigLIP/CLIP models on public benchmarks. NaFlex means the encoder resizes each crop to a grid of flexible patches instead of forcing a fixed square, so long thin logos don't get squashed.

OneVision-1.5 is an open VLM family. Its card reports strong benchmark leads vs other open models.

Both are Apache-2.0 licensed.

So let's do it!

Quickstart

1) Grab the video:

wget https://upload.wikimedia.org/wikipedia/commons/b/ba/Red_Bull_Racing_Pit_Stop_Practice.webm

Then extract frames. Two FPS, scaled down to 1280x720:

mkdir -p frames
ffmpeg -i "Red_Bull_Racing_Pit_Stop_Practice.webm" -vf "fps=2,scale=1280:-1:flags=lanczos" -q:v 3 frames/f_%06d.jpg

This yields 88 JPG files.

2) Prepare a logo dictionary

In this example we're only interested in the Red Bull logo. Create a logos directory you want to use. Grab the logo:

mkdir -p logos
wget -P logos https://upload.wikimedia.org/wikipedia/fi/a/a5/Red_Bull_logo.png

Disclaimer: This tutorial is for educational purposes only and is not affiliated with or endorsed by Red Bull. Red Bull is a registered trademark of Red Bull GmbH. It's a decent energy drink though.

3) Build the logo index (FAISS, SigLIP-2)

Save to build_logo_index.py:

import json, faiss, torch
from pathlib import Path
from PIL import Image
import numpy as np
from transformers import AutoModel, AutoProcessor

MODEL_ID = "google/siglip2-base-patch16-naflex"  # NaFlex = native aspect ratio
OUT_DIR = Path("artifacts"); OUT_DIR.mkdir(exist_ok=True, parents=True)

def embed_images(paths, model, proc, batch=16):
    imgs = [Image.open(p).convert("RGB") for p in paths]
    feats = []
    for i in range(0, len(imgs), batch):
        chunk = imgs[i:i+batch]
        inputs = proc(images=chunk, return_tensors="pt")
        inputs = {k: v.to(model.device) for k, v in inputs.items()}
        with torch.no_grad():
            f = model.get_image_features(**inputs)  # (B, d)
            f = torch.nn.functional.normalize(f, dim=-1)
        feats.append(f.cpu())
    return torch.cat(feats, dim=0).numpy()

def main():
    model = AutoModel.from_pretrained(MODEL_ID, dtype=torch.float32,
                                      device_map="auto")
    proc  = AutoProcessor.from_pretrained(MODEL_ID)

    logo_paths = sorted(list(Path("logos").glob("*.*")))
    if not logo_paths:
        raise SystemExit("No logo files found in 'logos' directory.")
    brands = [p.stem for p in logo_paths]
    vecs = embed_images(logo_paths, model, proc)
    d = vecs.shape[1]
    index = faiss.IndexFlatIP(d); index.add(vecs)

    faiss.write_index(index, str(OUT_DIR / "logos.faiss"))
    (OUT_DIR / "logos_meta.json").write_text(
        json.dumps({"brands": brands, "files": [str(p) for p in logo_paths]},
                   indent=2)
    )
    print(f"Indexed {len(brands)} brands into {OUT_DIR/'logos.faiss'}")

if __name__ == "__main__":
    main()

SigLIP-2 provides get_image_features and NaFlex dynamic resizing to minimize distortion on non-square inputs — useful for narrow/wide logos.

Run the tool:

$ python build_logo_index.py
...
Indexed 1 brands into artifacts/logos.faiss

4) Generate crops + Stage-1 retrieval

Save the following to retrieve.py:

import json, math, os, faiss, torch
from pathlib import Path
from PIL import Image, ImageDraw
import numpy as np
from transformers import AutoModel, AutoProcessor

EMB_ID = "google/siglip2-base-patch16-naflex"
ART = Path("artifacts"); ART.mkdir(exist_ok=True)

SIZES = [192, 256, 320]     # square windows
STRIDE = 0.5                # 50% overlap
TOPK = int(os.environ.get("TOPK", "3"))
COSINE_TH = float(os.environ.get("COSINE_TH", "0.7"))  # keep candidates above this

def _metric_type(index):
    try:
        return index.metric_type
    except Exception:
        return None

def _l2_to_cosine(d: np.ndarray) -> np.ndarray:
    # If vectors are L2-normalized, cos = 1 - 0.5 * ||a - b||^2
    return 1.0 - 0.5 * d

def _as_float32(x: np.ndarray) -> np.ndarray:
    if x.dtype != np.float32:
        x = x.astype(np.float32, copy=False)
    return np.ascontiguousarray(x)

def grid_crops(im: Image.Image):
    W, H = im.size
    for s in SIZES:
        step = max(1, int(s * STRIDE))
        for y in range(0, max(1, H - s + 1), step):
            for x in range(0, max(1, W - s + 1), step):
                yield (x, y, s, s)

def embed(model, proc, pil_list, bs=24):
    out = []
    for i in range(0, len(pil_list), bs):
        chunk = pil_list[i:i+bs]
        inp = proc(images=chunk, return_tensors="pt")
        inp = {k: v.to(model.device) for k, v in inp.items()}
        with torch.no_grad():
            f = model.get_image_features(**inp)
            f = torch.nn.functional.normalize(f, dim=-1)
        out.append(f.detach().to(torch.float32).cpu())
    vecs = torch.cat(out, dim=0).numpy()
    return _as_float32(vecs)

def main():
    # load logo index
    index = faiss.read_index(str(ART / "logos.faiss"))
    meta = json.loads((ART / "logos_meta.json").read_text())
    brands = meta["brands"]
    try:
        print(f"[retrieve] index ntotal={index.ntotal}, brands={len(brands)}")
    except Exception:
        pass

    # load embedder
    model = AutoModel.from_pretrained(EMB_ID, dtype=torch.float32, device_map="auto")
    proc  = AutoProcessor.from_pretrained(EMB_ID)

    frames = sorted(Path("frames").glob("f_*.jpg"))
    print(f"[retrieve] frames found={len(frames)}")
    out = []
    dump_all = bool(int(os.environ.get("DEBUG_DUMP_ALL", "0")))
    dbg_all = [] if dump_all else None
    debug_draw = dump_all and bool(int(os.environ.get("DEBUG_DRAW", "0")))
    debug_draw_dir = ART / "debug_vis"
    if debug_draw:
        debug_draw_dir.mkdir(parents=True, exist_ok=True)
    debug_draw_th = float(os.environ.get("DEBUG_DRAW_TH", "-1.0"))
    debug_draw_max = int(os.environ.get("DEBUG_DRAW_MAX", "0"))  # 0 = unlimited
    best = {"score": -1.0, "frame": None, "brand": None, "bbox": None}

    for fpath in frames:
        im = Image.open(fpath).convert("RGB")
        boxes, crops = [], []
        for (x, y, w, h) in grid_crops(im):
            boxes.append((x, y, w, h))
            crops.append(im.crop((x, y, x+w, y+h)))

        if not crops: continue
        vecs = embed(model, proc, crops)
        D, I = index.search(_as_float32(vecs), TOPK)

        mt = _metric_type(index)
        if mt == getattr(faiss, "METRIC_L2", 1):
            scores_mat = _l2_to_cosine(D)
        else:
            scores_mat = D

        # collect frame-local debug matches as well
        frame_dbg = [] if dump_all else None
        for i, (scores, ids) in enumerate(zip(scores_mat, I)):
            if dump_all:
                for r, (score, idx) in enumerate(zip(scores.tolist(), ids.tolist())):
                    rec = {
                        "frame": fpath.name,
                        "bbox": boxes[i],
                        "rank": int(r),
                        "score": float(score),
                        "brand": brands[idx]
                    }
                    dbg_all.append(rec)
                    if frame_dbg is not None:
                        frame_dbg.append(rec)
            for score, idx in zip(scores, ids):
                if score > best["score"]:
                    best.update({
                        "score": float(score),
                        "frame": fpath.name,
                        "brand": brands[idx],
                        "bbox": boxes[i]
                    })
                if score < COSINE_TH:
                    continue
                bx = boxes[i]
                out.append({
                    "frame": fpath.name,
                    "bbox": bx,
                    "score_retr": float(score),
                    "brand": brands[idx]
                })

        # draw annotations for this frame if requested
        if debug_draw and frame_dbg:
            # sort by score desc
            frame_dbg.sort(key=lambda r: r["score"], reverse=True)
            if debug_draw_max > 0:
                frame_dbg = frame_dbg[:debug_draw_max]
            canvas = im.copy()
            draw = ImageDraw.Draw(canvas)
            for rec in frame_dbg:
                if rec["score"] < debug_draw_th:
                    continue
                x, y, w, h = rec["bbox"]
                x2, y2 = x + w, y + h
                color = (255, 0, 0)
                draw.rectangle([x, y, x2, y2], outline=color, width=2)
                label = f"{rec['brand']} {rec['score']:.3f}#{rec['rank']}"
                # simple text; if background needed, draw a small filled box then text
                draw.text((x + 3, y + 3), label, fill=color)
            out_path = debug_draw_dir / f"{fpath.stem}_debug.jpg"
            canvas.save(out_path, quality=90)
            print(f"[retrieve] wrote debug visualization → {out_path}")

    Path("candidates.jsonl").write_text("\n".join(json.dumps(x) for x in out))
    print(f"wrote {len(out)} retrieval candidates → candidates.jsonl")
    if not out and best["frame"] is not None:
        print(f"[retrieve] no candidates above threshold {COSINE_TH}. "
              f"Best observed: score={best['score']:.3f}, frame={best['frame']}, "
              f"brand={best['brand']}, bbox={best['bbox']} — consider lowering COSINE_TH.")
    if dump_all and dbg_all is not None:
        Path("debug_matches.jsonl").write_text("\n".join(json.dumps(x) for x in dbg_all))
        print(f"[retrieve] wrote {len(dbg_all)} raw matches → debug_matches.jsonl (DEBUG_DUMP_ALL=1)")

if __name__ == "__main__":
    main()

This is quite a few things so breaking it down:

Crops: multi-scale grid over each frame using SIZES and STRIDE, yielding square patches and their bbox tuples.
Embeddings: SigLIP‑2 get_image_features on each crop batch, then L2‑normalize features.
Search: FAISS over the logo index; uses inner‑product on normalized vectors (cosine). If the index is L2, we convert to cosine (1 − 0.5·L2²).
Types: FAISS expects contiguous float32; embeddings are cast/contiguous before index.search.
Thresholds: keep top‑TOPK per crop, then filter by COSINE_TH. You can override at runtime:
- TOPK=10 COSINE_TH=0.65 python retrieve.py
Debugging:
- DEBUG_DUMP_ALL=1 writes every raw match to debug_matches.jsonl (ranked with scores).
- DEBUG_DRAW=1 also saves artifacts/debug_vis/*_debug.jpg with [brand score#rank] boxes.
- Optional: DEBUG_DRAW_TH=0.3 (only draw ≥ threshold), DEBUG_DRAW_MAX=200 (cap boxes).
Outputs: filtered candidates land in candidates.jsonl and are fed to the verifier stage.
Knobs to tune recall/precision: crop sizes, stride, TOPK, COSINE_TH. For higher recall, increase sizes or TOPK; for precision, raise the threshold and later add a margin or temporal smoothing.

Output:

$ python retrieve.py
[retrieve] index ntotal=1, brands=1
[retrieve] frames found=88
wrote 25 retrieval candidates → candidates.jsonl

Depending on the source material this probably needs tuning for recall/precision. The debugging knobs are quite nice. Here's an example frame from the video showing how it looks like on frame #7:

Frame from Red Bull Racing Pit Stop Practice (2015), ProtoplasmaKid / Wikimedia Commons / CC BY-SA 4.0.

5) VLM verification

Save the following to verify.py:

import json, torch, os
from pathlib import Path
from PIL import Image
from transformers import AutoProcessor, AutoModelForCausalLM

VLM_ID = "lmms-lab/LLaVA-OneVision-1.5-8B-Instruct"

SYSTEM = (
  "You are a logo verification API. "
  "Given an image crop and a target brand, answer in strict JSON with no extra text."
)

def build_user_prompt(brand: str):
    return (
      "Task: Verify whether the crop contains the specified brand's logo.\n"
      f"Brand: {brand}\n\n"
      "Return JSON only:\n"
      "{\n"
      '  "verdict": "yes" | "no",\n'
      '  "confidence": 0.0-1.0,\n'
      '  "visual_cues": "short, literal cues proving the verdict (colors/shapes/text)"\n'
      "}\n"
      "Rules: output only the JSON object; no prose before or after. "
      "Be literal; do not speculate; base confidence on how clearly the logo is visible."
    )

def _normalize_quotes(s: str) -> str:
    # Replace smart quotes with ASCII equivalents
    return (
        s.replace("\u201c", '"').replace("\u201d", '"')
         .replace("\u2018", "'").replace("\u2019", "'")
    )

def _extract_json_object(text: str):
    text = _normalize_quotes(text)
    # Find first balanced {...} block; handle braces within strings
    in_str = False
    escape = False
    depth = 0
    start = None
    for i, ch in enumerate(text):
        if ch == "\\" and not escape:
            escape = True
            continue
        if ch == '"' and not escape:
            in_str = not in_str
        escape = False
        if in_str:
            continue
        if ch == "{":
            if depth == 0:
                start = i
            depth += 1
        elif ch == "}":
            if depth > 0:
                depth -= 1
                if depth == 0 and start is not None:
                    candidate = text[start:i+1]
                    try:
                        j = json.loads(candidate)
                        if isinstance(j, dict) and "verdict" in j and "confidence" in j:
                            return j
                    except Exception:
                        pass
    return None

def run_once(proc, model, crop: Image.Image, brand: str, max_new=128):
    msgs = [
        {"role":"system", "content": SYSTEM},
        {"role":"user", "content":[{"type":"image"}, {"type":"text", "text": build_user_prompt(brand)}]}
    ]
    text = proc.apply_chat_template(msgs, tokenize=False, add_generation_prompt=True)
    inputs = proc(text=[text], images=[crop], padding=True, return_tensors="pt")
    inputs = {k: (v.to(model.device) if hasattr(v,"to") else v) for k,v in inputs.items()}
    with torch.inference_mode():
        ids = model.generate(**inputs, max_new_tokens=max_new, temperature=0.0, do_sample=False)
    out = proc.batch_decode(ids, skip_special_tokens=True)[0]
    j = _extract_json_object(out)
    if j is None:
        if os.environ.get("DEBUG_VLM", "0") == "1":
            Path("vlm_raw.txt").write_text(out)
        return {"verdict":"no", "confidence":0.0, "visual_cues":"parse_error"}
    # Coerce fields
    verdict = str(j.get("verdict", "no")).strip().lower()
    if verdict not in ("yes", "no"):
        verdict = "no"
    try:
        conf = float(j.get("confidence", 0.0))
    except Exception:
        conf = 0.0
    cues = str(j.get("visual_cues", ""))
    return {"verdict": verdict, "confidence": conf, "visual_cues": cues}

def main():
    proc = AutoProcessor.from_pretrained(VLM_ID, trust_remote_code=True)
    model = AutoModelForCausalLM.from_pretrained(
        VLM_ID, dtype=torch.float32, device_map="auto", trust_remote_code=True
    )
    results = []
    for line in Path("candidates.jsonl").read_text().splitlines():
        c = json.loads(line)
        im = Image.open(Path("frames")/c["frame"]).convert("RGB")
        x,y,w,h = c["bbox"]
        crop = im.crop((x,y,x+w,y+h))
        j = run_once(proc, model, crop, c["brand"])
        results.append({
          **c,
          "verdict": j.get("verdict","no"),
          "confidence_vlm": float(j.get("confidence",0.0)),
          "rationale": j.get("visual_cues","")
        })
    Path("detections.jsonl").write_text("\n".join(json.dumps(r, ensure_ascii=False) for r in results))
    print("wrote detections.jsonl")

if __name__ == "__main__":
    main()

Again, breaking it down:

Inputs: candidates.jsonl entries with frame, bbox, score_retr, brand.
Crop: open the frame and im.crop(bbox) per candidate.
Prompting: one system + one user message; user contains the target brand and an explicit "JSON only" schema.
Generation: temperature=0.0, do_sample=False, max_new_tokens=128 for deterministic outputs.
Parsing: balanced-brace JSON extraction with smart-quote normalization; on failure returns parse_error. Set DEBUG_VLM=1 to dump raw text to vlm_raw.txt.
Output: writes detections.jsonl with verdict, confidence_vlm, rationale merged onto each candidate.
Cost control: keep retrieval strict (lower TOPK, higher COSINE_TH) to limit VLM calls as that's the main runtime driver.

Output:

$ python verify.py
wrote detections.jsonl

And what does detections.jsonl look like? It has items like this (one per line):

{
  "frame": "f_000007.jpg",
  "bbox": [
    384,
    288,
    192,
    192
  ],
  "score_retr": 0.7020304203033447,
  "brand": "Red_Bull_logo",
  "verdict": "yes",
  "confidence_vlm": 0.95,
  "rationale": "red bull charging bull silhouette, red and yellow colors"
}

Here 0.702 is the cosine similarity between the crop and the logo embedding.

Conclusion

I hope this showcases how a simple two-stage recipe - SigLIP-2 retrieval + VLM verification - can turn a semi-noisy video into reviewable brand evidence. I do want to mention that a pipeline like this is meant as a powerful filter, not an oracle. Human in the loop needed.

On this specific 44-second clip, with the thresholds above, I get N true positives, M false positives, and 0 missed clear logos (subjective visual check).

That being said, this is probably not a production-ready detector. We trade some accuracy and runtime for simplicity and transparency. Some improvement ideas I had in mind:

float32 dtype could be replaced with lower precision, like bfloat16.
Add a margin filter (top1 − top2 ≥ 0.15) and simple temporal smoothing. Confirm across 2–3 adjacent frames.
Add a lightweight OCR gate for texty marks to backstop retrieval.
Calibrate thresholds per brand, as some logos need higher COSINE_TH or TH_VLM.
Expand the logo dictionary and try multi-resolution templates (flat vs curved surfaces).
Speed/scale: use FAISS IVF/PQ for larger dictionaries.
Quantize the VLM or batch crops.
Maybe consider a second encoder for consensus (SigLIP‑2 + CLIP) to reduce lookalikes.
For 4K frames, consider either down-scaling first or increasing STRIDE (e.g. 0.75) to avoid generating tens of thousands of crops per frame.

As always, credits: thanks to the SigLIP-2 authors & maintainers and the LLaVA-OneVision team for open releases. And attribution for the example video: ProtoplasmaKid / Wikimedia Commons / CC BY-SA 4.0.

ClipTagger-12B VLM: Frame Captioning Tutorial

Ville Vesilehto — Sun, 02 Nov 2025 15:54:55 +0000

TL;DR

The inference-net/ClipTagger-12b is a Gemma-3-12B based VLM with an Apache-2.0 license. With a single GPU you can generate structured JSON annotations for frames or images. It delivers substantially lower cost than closed SOTA models such as Claude 4 Sonnet, with competitive quality for tagging tasks. Details and benchmarks in inference.net blog post.

Requirements

GPU
CUDA 12.x runtime
Python 3.10–3.12
Disk: ~20–30 GB free (model + deps + cache)

Optional but recommended: ffmpeg for extracting video frames.

GPU note. Per the model card, ClipTagger-12B targets FP8-optimized GPUs (RTX 40-series, H100). This tutorial shows BF16 inference in PyTorch for portability. It was tested on a single NVIDIA H200 (SXM5). For smaller VRAM budgets, use quantization or for e.g. TensorRT-LLM/FP8.

Quickstart (PyTorch, BF16, single GPU)

1) Create environment and install CUDA-enabled PyTorch + deps

python3 -m venv .venv
source .venv/bin/activate
pip install torch torchvision torchaudio "transformers>=4.44" accelerate pillow safetensors compressed-tensors

2) Prepare test image

Use a small JPEG/PNG (≤1 MB, as per the model card). This example uses a CC0 licensed picture from magdeleine.co from this post:

curl -L -o frame.jpg \
  https://magdeleine.co/wp-content/uploads/2022/11/43865945842_6f89d901fc_o-1.jpg

If your frames are large, you can downscale/compress to keep under ~1 MB, for example:

ffmpeg -y -i input.jpg -vf "scale=1280:-1,setsar=1" -pix_fmt yuv420p -q:v 3 -frames:v 1 -update 1 frame.jpg

3) Run inference (BF16):

Create run_cliptagger.py:

import json
from pathlib import Path
from PIL import Image
import torch
from transformers import AutoProcessor, AutoModelForCausalLM

MODEL_ID = "inference-net/ClipTagger-12b"

# Prompts copied from the model card (citation link in the post body).
SYSTEM_PROMPT = (
    "You are an image annotation API trained to analyze YouTube video keyframes. "
    "You will be given instructions on the output format, what to caption, and how to perform your job. "
    "Follow those instructions. For descriptions and summaries, provide them directly and do not lead them with "
    "'This image shows' or 'This keyframe displays...', just get right into the details."
)

USER_PROMPT = (
    "You are an image annotation API trained to analyze YouTube video keyframes. "
    "You must respond with a valid JSON object matching the exact structure below.\n\n"
    "Your job is to extract detailed factual elements directly visible in the image. Do not speculate or interpret "
    "artistic intent, camera focus, or composition. Do not include phrases like 'this appears to be', 'this looks like', "
    "or anything about the image itself. Describe what is physically present in the frame, and nothing more.\n\n"
    "Return JSON in this structure:\n\n"
    "{\n"
    "    \"description\": \"A detailed, factual account of what is visibly happening (4 sentences max). Only mention concrete elements or actions that are clearly shown. Do not include anything about how the image is styled, shot, or composed. Do not lead the description with something like 'This image shows' or 'this keyframe is...', just get right into the details.\",\n"
    "    \"objects\": [\"object1 with relevant visual details\", \"object2 with relevant visual details\", ...],\n"
    "    \"actions\": [\"action1 with participants and context\", \"action2 with participants and context\", ...],\n"
    "    \"environment\": \"Detailed factual description of the setting and atmosphere based on visible cues (e.g., interior of a classroom with fluorescent lighting, or outdoor forest path with snow-covered trees).\",\n"
    "    \"content_type\": \"The type of content it is, e.g. 'real-world footage', 'video game', 'animation', 'cartoon', 'CGI', 'VTuber', etc.\",\n"
    "    \"specific_style\": \"Specific genre, aesthetic, or platform style (e.g., anime, 3D animation, mobile gameplay, vlog, tutorial, news broadcast, etc.)\",\n"
    "    \"production_quality\": \"Visible production level: e.g., 'professional studio', 'amateur handheld', 'webcam recording', 'TV broadcast', etc.\",\n"
    "    \"summary\": \"One clear, comprehensive sentence summarizing the visual content of the frame. Like the description, get right to the point.\",\n"
    "    \"logos\": [\"logo1 with visual description\", \"logo2 with visual description\", ...]\n"
    "}\n\n"
    "Rules:\n"
    "- Be specific and literal. Focus on what is explicitly visible.\n"
    "- Do NOT include interpretations of emotion, mood, or narrative unless it's visually explicit.\n"
    "- No artistic or cinematic analysis.\n"
    "- Always include the language of any text in the image if present as an object, e.g. \"English text\", \"Japanese text\", \"Russian text\", etc.\n"
    "- Maximum 10 objects and 5 actions.\n"
    "- Return an empty array for 'logos' if none are present.\n"
    "- Always output strictly valid JSON with proper escaping.\n"
    "- Output only the JSON, no extra text or explanation."
)

def load_model(device: str = "cuda"):
    processor = AutoProcessor.from_pretrained(MODEL_ID, trust_remote_code=True)
    model = AutoModelForCausalLM.from_pretrained(
        MODEL_ID,
        dtype=torch.bfloat16,
        device_map="auto",
        trust_remote_code=True,
    )
    return processor, model

def build_prompt(processor):
    # Include an image placeholder so the template has 1 image token.
    user_content = [
        {"type": "image"},
        {"type": "text", "text": USER_PROMPT},
    ]
    messages = [
        {"role": "system", "content": SYSTEM_PROMPT},
        {"role": "user", "content": user_content},
    ]
    if hasattr(processor, "apply_chat_template"):
        return processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
    # Fallback: include a literal <image> token expected by many VLM chat templates.
    return f"<system>\n{SYSTEM_PROMPT}\n</system>\n<user>\n<image>\n{USER_PROMPT}\n</user>\n"

def annotate_image(image_path: str, max_new_tokens: int = 1000, temperature: float = 0.1):
    device = "cuda" if torch.cuda.is_available() else "cpu"
    processor, model = load_model(device)
    prompt = build_prompt(processor)
    image = Image.open(image_path).convert("RGB")

    inputs = processor(images=image, text=prompt, return_tensors="pt")
    inputs = {k: (v.to(model.device) if hasattr(v, "to") else v) for k, v in inputs.items()}

    with torch.inference_mode():
        generated = model.generate(
            **inputs,
            max_new_tokens=max_new_tokens,
            do_sample=(temperature > 0),
            temperature=temperature,
        )
    text = processor.batch_decode(generated, skip_special_tokens=True)[0]

    # Try to parse JSON; if the model outputs surrounding text, trim to JSON braces.
    try:
        start = text.index("{")
        end = text.rindex("}") + 1
        parsed = json.loads(text[start:end])
        print(json.dumps(parsed, ensure_ascii=False, indent=2))
    except Exception:
        # Fallback: print raw output to help debugging
        print(text)

if __name__ == "__main__":
    import sys
    img = sys.argv[1] if len(sys.argv) > 1 else "frame.jpg"
    annotate_image(img)

The prompts have been taken from the model card, see required prompts for full reference.

Then run it:

python3 run_cliptagger.py frame.jpg

You should see the following kind of output:

{
  "actions": [],
  "content_type": "real-world footage",
  "description": "A small village with white buildings is situated in a valley between two large, green mountains. A paved road winds through the landscape, and a small river flows through the bottom of the valley. The peaks of the mountains are obscured by thick, white fog. A thin waterfall is visible on the side of the mountain on the left.",
  "environment": "A remote, rural mountain valley with lush green vegetation. The atmosphere is cool and damp, indicated by the thick fog covering the mountain peaks and the overcast sky.",
  "logos": [],
  "objects": [
    "Green mountains",
    "White buildings",
    "Paved road",
    "River",
    "Fog",
    "Waterfall"
  ],
  "production_quality": "professional studio",
  "specific_style": "Landscape photography",
  "summary": "A small village sits in a foggy, green mountain valley with a winding road and a river."
}

Ideas for production use

For batching, extract frames at a steady cadence and pre-resize to keep images under about 1 MB, then run annotations in parallel.

For scale, add an HTTP/gRPC front end, queue requests, and use GPU workers that micro-batch to maximize utilization. Track latency percentiles, tokens/sec, batch sizes, and GPU utilization.

Engines: start with PyTorch for simplicity. Switch to TensorRT-LLM for FP8 throughput where supported. If you prefer not to operate GPUs, use a managed API (e.g., inference.net).

Conclusion

That's it - run the quickstart on your frames and iterate on the JSON schema for your task. Feedback welcome!

And finally, thanks to the model authors.

Modernize Go with golangci-lint v2.6.0

Ville Vesilehto — Fri, 31 Oct 2025 19:42:18 +0000

TL;DR

golangci-lint v2.6.0 adds the new modernize analyzer. It surfaces suggestions to adopt modern Go features from the standard library (e.g., strings.CutPrefix, slices.Contains, maps.Clone) and language (any). Previously the tool was available as a separate binary, but it'll receive much more exposure now being included in golangci-lint.

What is modernize?

The modernize analyzer suggests clearer, more idiomatic code by using newer Go features. Each diagnostic includes a suggested fix designed to be behavior-preserving. Many improvements rely on Go ≥1.21 features. If your module uses an older go version, you'll see fewer suggestions and some fixes won't apply cleanly.

A few examples

Replace manual prefix handling with strings.CutPrefix

Before:

if strings.HasPrefix(s, "llama") {
    s = s[len("llama"):]
}

After:

if after, ok := strings.CutPrefix(s, "llama"); ok {
    s = after
}

Replace hand-rolled membership checks with slices.Contains

Before:

found := false
for _, v := range xs {
    if v == needle {
        found = true
        break
    }
}

After:

found := slices.Contains(xs, needle)

Replace manual map clone loops with maps.Clone

Before:

dst := make(map[string]int, len(src))
for k, v := range src {
    dst[k] = v
}

After:

dst := maps.Clone(src)

Prefer any over interface{}

Before:

var x interface{}

After:

var x any

How to enable it

With golangci-lint (recommended)

In .golangci.yml:

linters:
  enable:
    - modernize

Or via CLI:

golangci-lint run --enable modernize

Optionally apply fixes automatically (where supported):

golangci-lint run --enable modernize --fix

Real world example

I originally enabled modernize for CoreDNS in coredns/coredns#7536. I went through the analyser findings and applied fixes. To keep future code changes from diverging from modernize, I added a separate CI step that failed if modernize produced a non-empty diff. This worked fine.

Now with golangci-lint v2.6.0 out, I refactored this in coredns/coredns#7645. The reflecttypefor had a number of changes that will need to be addressed separately. Nice and clean now.

The CPU Cost of Signing NXDOMAINs

Ville Vesilehto — Thu, 30 Oct 2025 19:12:11 +0000

TL;DR

The CoreDNS dnssec plugin guidance to prefer ECDSA is there for a reason.

In this post we see that with the same NXDOMAIN-signing workload, RSA (RSASHA256/3072) used 30x the amount of CPU compute compared to ECDSA (P‑256).

Introduction

When you enable DNSSEC in CoreDNS, negative answers must be provably negative. CoreDNS implements authenticated denial with NSEC "black lies". It forges per-query NSEC owner names to prevent zone walking, which means responses can’t be broadly reused by resolvers and each miss requires fresh signing. That pushes cryptographic signing onto the hot path, so the key algorithm directly determines CPU and packet size.

Dry information for those who are initiated:

Note that ECDSA P-256 RRSIGs are 64 bytes, while RSASHA256-3072 RRSIGs are 256 bytes (size equals modulus). ECDSA signatures also save you bandwidth.

Test design

I ran two otherwise identical configs that differed only in the DNSSEC key algorithm: RSA (RSASHA256, 3072; algorithm 8) and ECDSA (P‑256; algorithm 13). As mentioned in RFC 6605:

"Current estimates are that ECDSA with curve P-256 has an approximate equivalent strength to RSA with 3072-bit keys"

To force signing work, I generated many unique, random non‑existent names, set the DO bit and EDNS to trigger signing. I measured throughput and latency with dnsperf, and captured CPU profiles with pprof to summarize and visualize.

Environment

CoreDNS was built as v1.13.1-dirty from commit coredns/coredns@60e2d455f9f2fd0f12b1ce3dcb64125913a26743 with Go 1.25.3 on macOS.

Supporting tools included dnsperf, bind or ldns key tools, and optionally graphviz for SVG outputs.

Configuration

Zone file

Save as db.example.test:

$ORIGIN example.test.
@ 3600 IN SOA ns1.example.test. hostmaster.example.test. (1 7200 3600 1209600 3600)
  3600 IN NS ns1.example.test.
ns1 3600 IN A 127.0.0.1

Keys

Generate one key for each algorithm:

ECDSA P-256:

mkdir -p keys/ecdsa
( cd keys/ecdsa && dnssec-keygen -a ECDSAP256SHA256 -n ZONE example.test )

RSA 3072:

mkdir -p keys/rsa
( cd keys/rsa && dnssec-keygen -a RSASHA256 -b 3072 -n ZONE example.test )

Note the basenames printed (e.g., Kexample.test.+013+15419 or Kexample.test.+008+09030). Use the exact basename in the Corefiles below.

Corefiles

Corefile.ecdsa:

.:1053 {
    pprof 127.0.0.1:6060
    file db.example.test example.test
    dnssec example.test {
        key file keys/ecdsa/Kexample.test.+013+15419
    }
    bufsize 1232
}

Corefile.rsa:

.:1053 {
    pprof 127.0.0.1:6060
    file db.example.test example.test
    dnssec example.test {
        key file keys/rsa/Kexample.test.+008+09030
    }
    bufsize 1232
}

Buffer size is set to 1232 to prevent IP fragmentation with EDNS0. See bufsize plugin docs for more information if curious.

log and errors are omitted due to profiling. They add syscall noise.

Load generation

Create a large set of unique NXDOMAIN queries:

jot -w "nonexist-%08d.example.test A" 200000 1 > queries.txt

Run CoreDNS for the chosen Corefile:

./coredns -conf Corefile.ecdsa
# or
./coredns -conf Corefile.rsa

Drive the load (DO bit + EDNS, match bufsize):

dnsperf -s 127.0.0.1 -p 1053 -d queries.txt -l 60 -Q 2000 -c 50 -e -D -b 1232

Profiling

Capture a 30‑second CPU profile while the load is running:

go tool pprof -proto http://127.0.0.1:6060/debug/pprof/profile?seconds=30 > rsa.cpu.pb.gz
# or
go tool pprof -proto http://127.0.0.1:6060/debug/pprof/profile?seconds=30 > ecdsa.cpu.pb.gz

Generate clean visuals from call graphs in SVG format:

go tool pprof -svg -hide='^(syscall|runtime\.)' -focus='^crypto|dnssec' rsa.cpu.pb.gz > rsa.user.svg
# or
go tool pprof -svg -hide='^(syscall|runtime\.)' -focus='^crypto|dnssec' ecdsa.cpu.pb.gz > ecdsa.user.svg

Interactive exploration:

go tool pprof ./coredns rsa.cpu.pb.gz
# inside pprof:
hide=^(syscall|runtime\.)
top
top -cum
list sign

Results

From pprof we can see that RSA used about 176 seconds of samples over 30 seconds:

(pprof) top
Showing nodes accounting for 168.37s, 95.49% of 176.32s total

ECDSA used about 5.6 seconds of samples over 30 seconds:

(pprof) top
Showing nodes accounting for 5.33s, 95.35% of 5.59s total

For this workload, RSA required approximately 30× more CPU than ECDSA.

Latency

See table below.

Algorithm	QPS	Avg Lat (ms)	Min (ms)	Max (ms)	StdDev (ms)	Avg resp bytes
ECDSA P-256	1999.65	0.298	0.083	40.003	1.333	495
RSA-3072	1342.79	73.412	4.608	1055.376	69.898	1135

Detailed pprof output from RSA

RSA top

RSA call graph

RSA flame graph

Detailed pprof output from ECDSA

ECDSA top

ECDSA call graph

ECDSA flame graph

Conclusion

Prefer ECDSA (P‑256) keys for online DNSSEC signing in CoreDNS, unless you have a compelling reason not to. Monitor coredns_dnssec_cache_hits_total and coredns_dnssec_cache_misses_total metrics series, and naturally overall CPU saturation from the process or container runtime.

If you end up debugging this further, use the pprof plugin in CoreDNS.

Comments and feedback welcome - and thanks for reading!