DEV Community: Nicolas Micaud

We're taking pre-orders for an on-prem inference box built on four Intel Arc Pro B60 — here's exactly what it does

Nicolas Micaud — Sat, 12 Sep 2026 07:45:27 +0000

Since 10 September, four Intel Arc Pro B60 cards in a tower in Meyrin, Geneva, have been running three language models at once — for our own products first. This week we turned that configuration into something you can put in your own building, and opened pre-orders. This post is the spec sheet with the numbers we actually measured, the part that doesn't scale, and what we are not promising.

The problem it solves

We kept hearing the same sentence from law firms, clinics and fiduciaries in Switzerland: "the cloud is not allowed here." Not "expensive", not "we prefer local" — not allowed. For them the choice is not ChatGPT versus something cheaper; it is a machine inside the walls, or nothing.

So SOKKAN Anchor is a box. It runs the models, the retrieval and the gateway on premises. A key can be declared sovereign-only, and the server — not a client-side setting — guarantees that no request under that key will ever reach a cloud provider, not even as a fallback. If the local model is down, you get an error, not a detour.

What is in it

The entry configuration, Anchor E, carries four Intel Arc Pro B60 (24 GB each, 96 GB total) and runs three resident models:

Card	Model	Format / runtime	Weights	Role
0	gpt-oss-20b	MXFP4 · vLLM (Intel XPU build)	13 GB	overnight automation, batch, embeddings
1	Qwen3-Coder-30B-A3B	GGUF Q4_K_XL · llama.cpp SYCL	17.7 GB	developers
2 + 3	Qwen3-Next-80B-A3B	GGUF Q3_K_XL · llama.cpp SYCL	35.6 GB	finance, HR, management assistants

All three are MoE models with about 3B active parameters. That is what makes the cohabitation possible: 130B of cumulated weights at the speed of a small model. A single dense 70B would have eaten two cards to be four times slower.

The production platform is an AMD EPYC 9115 (16 cores, SP5) on a Supermicro board with IPMI, 128 GB of DDR5 ECC, two 2 TB NVMe drives in a ZFS mirror, a 1.6 kW ATX 3.1 supply, in a silent office tower or a 4U rack case. Why EPYC rather than a desktop chip: 128 PCIe 5.0 lanes (four cards at full width, no bifurcation games), ECC, and remote management. IPMI is not a luxury for a box we support remotely — it is how we reboot a machine in a law office at 7 am without anyone touching it. ZFS mirror because the box holds the customer's document index, and a dead NVMe must not mean a re-ingestion.

What it does — measured

Every number below was measured on our prototype: a 2018 X299 workstation (i9-9980XE, 64 GB DDR4) that also carries our production job platform — Postgres, twelve Celery workers, two web front ends. It is a pessimistic rig. The EPYC build will not be slower.

Each model alone, warm:

Model	Decode	Prefill (~4k tokens)
Qwen3-Coder-30B	82.4 tok/s	4,098 tok in 4.2 s
gpt-oss-20b	41.2 tok/s	3,307 tok in 2.4 s
Qwen3-Next-80B	36.2 tok/s	4,098 tok in 10.3 s

All three under load at once — 12 concurrent requests, 4 per model, 300 tokens each: 3,600 tokens in 30.3 s, 119 tok/s aggregate, no errors, no measurable interference between cards. The 20B alone batches to 134 tok/s at 4 requests and 241 tok/s at 8.

Power: 163 W idle with the three models loaded, 211 W peak under the 12-request load, for the four cards. The whole machine is around 350 W. At Swiss electricity prices, running around the clock, that is roughly CHF 750 a year.

Host RAM stayed above 41 GB free at the worst moment — with a full production stack running next to it. That is why the spec sheet does not need 256 GB.

Since 10 September our job platform sends about 107 million tokens a week of batch work — job-ad classification, extraction, enrichment — to this box instead of a cloud provider. That is not a benchmark; it is the workload that paid for the cards.

The part that doesn't scale

The 80B is the model people want: it is the better writer and analyst. It is also the one that does not scale under concurrency, and the reason is the runtime, not the model.

Concurrent requests	gpt-oss-20b (vLLM)	Qwen3-Next-80B (llama.cpp)
1	35.1 tok/s	32.0 tok/s
2	35.0 tok/s each	14.5 tok/s each
4	33.6 tok/s each	7.0 tok/s each
aggregate at 4	133.9 tok/s	20.7 tok/s (less than solo)

vLLM does continuous batching; throughput grows almost linearly with load. llama.cpp does not; its aggregate throughput regresses as concurrency rises. Four users on the 80B get less total work done than one.

Why is it on llama.cpp then? Because vLLM on Intel XPU currently serves only quantisations with a native XPU kernel. MXFP4 gpt-oss has one. Every other quantised MoE we tried routes its expert layers to Marlin — an NVIDIA kernel — and dies with device_capability = -1, whatever you pass to --quantization. Dense AWQ works (int4_gemm_w4a16, and you must set --dtype float16); AWQ MoE does not. So everything that is not gpt-oss runs through llama.cpp SYCL, which is honestly good — Qwen3-Next's hybrid Gated DeltaNet attention runs on it — but serves one user well and four badly.

This goes on the spec sheet as a design rule, not a footnote: the 80B serves two simultaneous users, not a department. Anything real-time — voice, interactive completion — must hit a model served by vLLM. Voice needs roughly 15 tok/s per speaker to stay fluid; the 80B holds two conversations, the 20B at least four.

Two more lessons that cost us real time: models must be loaded sequentially (the RAM peak is at load, not at serve — three simultaneous docker start at boot reproduced a crash), and the first request after loading is 3–4× slower than steady state, so the box warms itself up before a customer touches it.

What we are not promising

It does not replace ChatGPT or Claude for general use. A developer who compares side by side will prefer the frontier model. Anchor is for the work you are not allowed to send there.
We have not measured Anchor S (two Arc Pro B70, 64 GB). Its numbers will be published when the cards arrive; until then the page says "estimate".
vLLM tensor-parallel across the four cards wedged our X299 prototype three times in a day (a load-time DMA/PCIe storm, not RAM — it failed with 59 GB free and production stopped). We will retry on EPYC. It is not needed for the three-model configuration above.
The larger M (128 GB) and L (192 GB) configurations are described, not open.

What is on top of the hardware

SOKKAN Inference, the gateway from our open-source cockpit (github.com/ninabot-ch/sokkan), runs on the box: one LAN endpoint, OpenAI- and Anthropic-Messages-compatible, so Claude Code, any OpenAI SDK and any agent framework point at it unchanged. Per-key metering, per-key sovereign-only policy, Prometheus/Grafana, WireGuard for support, Ansible for fleet updates. Plus Nina, a document assistant with retrieval over the customer's own files — the same chain as this demo we built for a truck garage's technical manuals.

Price, and how the pre-order works

Anchor E — 4× B60, 96 GB: CHF 10,450 excl. VAT.
Anchor S — 2× B70, 64 GB: CHF 11,100 excl. VAT.
Built to order, delivered and installed in about six weeks, 24-month warranty on new parts. Managed operation (model updates, monitoring, remote support) from CHF 300 a month, optional.

We are a small company and we do not stock EPYC boards and GPUs on the shelf: parts are ordered against a pre-order. The deposit is 20 % (CHF 2,100 for E), fully refundable until the machine ships, deducted from the final invoice. The first five units are sold at this price whatever the GPU market does — B70s went up 26–48 % in August alone — and their serial number is engraved on the front plate.

Page, specs and the full measurement report: ninabot.ch/en/anchor.

If you run Arc Pro cards for inference and hit the same walls — or different ones — I would genuinely like to compare notes. I answer the comments myself.

SOKKAN Anchor is a product of Ninabot Sàrl, Geneva. The Geneva-hosted "Swiss" tier of SOKKAN Inference runs on the same prototype.

I stopped asking my AI agents to read the project memory. Now the server does it for them.

Nicolas Micaud — Sat, 12 Sep 2026 06:21:17 +0000

Follow-up to I run 9 parallel Claude Code sessions — the bottleneck wasn't
the model, it was memory.
That post described the problem. This one is about the fix I shipped, and the
uncomfortable thing it taught me about "instructing" a model.

The setup

I build and run our products — a Swiss job platform, a handful of sites, the
infrastructure under them — with a cockpit that keeps ~9 Claude Code sessions
alive in parallel. Each session is a task: a kanban card you drag, and it
becomes a working agent.

The recurring failure was never intelligence. It was amnesia. Session #7 would
happily reimplement an error format we'd standardized weeks ago, because nothing
told it that decision existed. The decision was written down — in a markdown
note, in the project memory — but the session never looked.

v1: I asked nicely

My first fix was the obvious one. When a session spawns, append an instruction
to its first prompt:

{the task}
Start by calling the memory_search tool to load any relevant project context,
then propose a plan — don't execute anything without my go-ahead.

This is what most "give your agent memory" setups do: a system prompt, a tool,
and trust that the model uses them. And it mostly works. Mostly.

Here's the thing about "mostly" when you have nine of them running. One session
in nine ignoring the instruction isn't a 1-in-9 annoyance — it's a silent,
guaranteed defect generator. The model doesn't announce "I skipped the memory
this time." It just dives into the code, does something locally reasonable, and
violates a convention you'd forgotten you even needed to defend. You find out at
review, or worse, after merge.

I spent a while trying to make the instruction stickier — stronger wording,
putting it last, tool descriptions that begged. That's when it clicked: I was
trying to make a non-deterministic thing reliable by asking harder. Wrong
layer.

v2: I stopped asking

The recall doesn't belong in the prompt. It belongs in the plumbing.

Now, when a session spawns, the server runs the semantic search — before the
model gets a single turn — and injects the results directly into the session's
first message:

def _memory_preseed(query: str, top_k: int = 5) -> str:
    hits = memory_search(query, top_k=top_k)   # server-side, deterministic
    if not hits:
        return ""                               # empty memory → fall back to the ritual
    lines = ["=== Project memory (auto-recalled) ==="]
    for h in hits:
        star = "★ " if h.get("priority") else ""
        lines.append(f"- {star}[{h['note_name']}] {h['description']} — {h['snippet']}")
    return "\n".join(lines)

The task the model receives now looks like:

Fix the healthcheck flapping on staging.

=== Project memory (auto-recalled) ===
- ★ [staging-stack] Staging API is on port 6443, deploys land on host "callisto"
- [deploy-ritual] Deploys go exclusively through `make ship-v2`, never raw rsync

The notes above were auto-recalled from project memory for this task. Call
memory_get on any note you need in full… then propose a plan and wait for my go.

The model can't not have the context now. It's not a tool it might call. It's
in the first tokens it reads.

Crucially, the old ritual is still there — as a fallback. If the memory is
empty (fresh project) or the embedding backend is down, the injected block is
empty and the prompt reverts to "please go search." Determinism where it
matters, graceful degradation where it doesn't.

Does it actually work? I planted a trap.

Claims about memory systems are cheap. So the release has an end-to-end test
that seeds a throwaway instance with two notes containing facts that exist
nowhere in the code: the staging port is 6443, deploys go to a host called
callisto, via make ship-v2. The workspace README is deliberately wrong
about all three.

Then it spawns a session asking: "what port does staging run on, which host do
we deploy to, and with what command?"

A session that reads the code (or the lying README) fails. A session with
working recall answers 6443 / callisto / make ship-v2 — and, in the transcript,
literally says it answered "without needing to search the codebase." That's
the whole product in one assertion: the fact reached the model because the
server put it there, not because the model went looking.

The boring parts (on purpose)

Notes are just markdown with YAML frontmatter — the same files Claude Code already writes. One fact per file, a description: that doubles as the embedding text, optional priority: high.
The index is incremental sqlite. Vectors live in the same store. No pgvector, no ANN library — at a few hundred notes, dot-product over a warm in-process model is instant, and I'd rather ship boring than clever.
Embeddings run locally by default (fastembed / ONNX, a ~120MB multilingual MiniLM). No external embedding service unless you point one.
[[wikilinks]] are indexed, so a session can walk the graph with a memory_links tool, and the cockpit draws it with type filters and connected clusters.
priority: high is a bounded, multiplicative nudge — I started with a flat additive boost and it let a weak match jump above genuinely relevant notes. Multiply, don't add.

What I deliberately didn't build

This isn't an autopilot orchestrator. There's no "let the swarm run overnight."
Every irreversible action waits for a human click — the helm, not the
autopilot. The point of the memory work isn't to remove me from the loop; it's
to make sure that when I'm steering nine sessions, none of them is flying blind.

The lesson I keep relearning

When you need a model to do something every single time, don't put it in the
prompt and hope. Move it to a layer that doesn't have opinions. Prompts are for
judgment. Plumbing is for guarantees. I keep having to relearn which is which.

We serve a sovereign inference tier from four Intel Arc Pro B60 — here's what it actually does (and doesn't)

Nicolas Micaud — Fri, 11 Sep 2026 13:40:02 +0000

Since this week, SOKKAN Inference has a tier called "Swiss". It is served from a machine in Meyrin, Geneva, on hardware we own, with no NVIDIA card inside. This post is the honest version of what that machine does, with the numbers we measured — including the ones that don't scale.

Why Intel

We are a small Swiss company. Our inference product already had two cloud tiers, EU-sovereign, served from French datacenters. What we didn't have was an answer for the customer who says "the data cannot leave Switzerland, full stop" — and, longer term, for the customer who says "the data cannot leave the building".

That meant buying GPUs, in 2026, in the middle of a memory shortage. We chased second-hand RTX 3090s for a month and lost every auction above market price. Meanwhile the Intel Arc Pro B60 — 24 GB of VRAM, a 2-slot blower, natively x8 — was still selling near MSRP, at CHF 614 a card. Four of them came to about CHF 2,450 for 96 GB of VRAM. That was the whole argument: VRAM per franc, available now.

The bet was on the software stack, not the silicon. The risk with Intel in 2026 is not the hardware; it is whether vLLM, llama.cpp and the XPU runtime hold up under a service you sell.

What we run

The chassis is a 2018 X299 workstation (i9-9980XE, 64 GB DDR4) that also carries our production job platform — Postgres, Celery workers, a few web front ends. Not a clean benchmark rig. Every number below is therefore pessimistic; the production appliance is an EPYC board dedicated to inference.

Four B60s, each on its own x8 Gen3 root port, run three resident models:

Card	Model	Format / runtime	Weights
0	gpt-oss-20b	MXFP4 · vLLM (Intel XPU build 0.21)	13 GB
1	Qwen3-Coder-30B-A3B	GGUF Q4_K_XL · llama.cpp SYCL	17.7 GB
2 + 3	Qwen3-Next-80B-A3B	GGUF Q3_K_XL · llama.cpp SYCL	35.6 GB

All three are MoE models with about 3B active parameters, which is what makes the cohabitation possible: 130B of cumulated weights at the speed of a small model.

The Swiss tier is card 0.

What it does — measured

gpt-oss-20b under vLLM, warm, on one B60:

Concurrent requests	Per request	Aggregate
1	35.1 tok/s	35.1 tok/s
2	35.0 tok/s	70 tok/s
4	33.6 tok/s	133.9 tok/s

Prefill is around 5,000 tok/s on a 4.3k-token prompt. In an earlier run at 8 concurrent requests the same card reached 241 tok/s aggregate. Continuous batching does exactly what it says on a model this size: throughput grows almost linearly with load, and per-request speed barely moves.

Power for the four cards: 163 W idle with all three models loaded, 211 W peak under 12 concurrent requests across the three models. Temperatures 52–62 °C. On the UPS, the whole machine — including the production it also carries — moves from 13% to 19% load. At Swiss electricity prices that is roughly CHF 750 a year for the full box, running around the clock.

Tool calling works on all three models. For gpt-oss you need --enable-auto-tool-choice --tool-call-parser openai; without the harmony parser you get zero tool calls and an empty content field, and you will spend an evening wondering why.

What does not scale

The 80B model is the one people want — it is the better writer, the better analyst. It is also the one that does not scale, and the reason is the runtime, not the model.

llama.cpp has no continuous batching. Under concurrency, the same 80B goes from 32.0 tok/s for one request, to 14.5 tok/s each for two, to 7.0 tok/s each for four. Worse: the aggregate throughput regresses — 32 → 28.8 → 20.7 tok/s. Four users on that model get less total work done than one.

So why is it on llama.cpp? Because vLLM on Intel XPU currently only serves quantisations that have a native XPU kernel. MXFP4 gpt-oss does. Every other quantised MoE we tried routes its expert layers to Marlin, an NVIDIA kernel, and dies with device_capability = -1 regardless of the --quantization flag. Dense AWQ works (int4_gemm_w4a16, and you must pass --dtype float16); AWQ MoE does not. Everything that is not gpt-oss goes through llama.cpp SYCL, and llama.cpp SYCL is honestly good — Qwen3-Next's hybrid attention runs on it — but it serves one user well and four users badly.

We also could not get vLLM tensor-parallel across the four cards to stay up on this platform. Loading four workers in parallel wedged the machine three times in one day, once with 59 GB of RAM free and production stopped, so it is not memory; it looks like a DMA/PCIe storm at load on an old chipset. llama.cpp, which loads sequentially, spreads gpt-oss-120b across the four cards without incident — 18.8 tok/s decode, ~250 tok/s prefill, stable — but that prefill number is why you won't see the 120b behind a public tier yet. We will retry TP on the EPYC board.

Two smaller lessons that cost us real time: models must be loaded sequentially (the memory peak is at load, not at serve — three simultaneous docker start at boot reproduce a crash), and the first request after loading is 3–4× slower than steady state, so warm the service before a customer touches it.

The Swiss tier, as sold

Given all that, the Swiss tier is deliberately small and deliberately honest:

A 20B-class model (gpt-oss-20b), best for chat, RAG and agents. Not a frontier coding model. If you need that, our other tiers are EU-sovereign and better at it.
Served in Geneva on hardware we own. Data never leaves Switzerland.
Never rerouted to an EU provider. If the primary model is down, a second model on a neighbouring card takes over, still in Geneva; if that is down too, you get a clear error, not a silent detour.
Capacity: 4 concurrent requests. Above that, a 429. We would rather tell you than queue you.
CHF 0.60 / 2.40 per million tokens (in / out), prepaid, metered to the token.
Anthropic Messages-compatible, so Claude Code and any Anthropic client work unchanged:

  export ANTHROPIC_BASE_URL=https://infer.sokkan.ch
  export ANTHROPIC_AUTH_TOKEN=sik_your_token
  export ANTHROPIC_MODEL=sokkan-swiss

Reselling raw tokens from one card will never be a business — at these prices a saturated B60 earns about what it burns in electricity. The point is different: the same machine hosts services on top of the model (a website assistant for SMEs, a finance assistant for our ERP), and the Swiss tier is the door for developers who want to build the same kind of thing with the same residency guarantee.

What comes next

The same four-card configuration becomes SOKKAN Anchor, an on-prem appliance for organisations that have banned the cloud altogether. The dimensioning rule we learned the hard way goes into the sales sheet: anything real-time — voice, interactive completion — must hit a model served by vLLM; the big llama.cpp model serves two simultaneous users, not a department.

If you are running Arc Pro cards for inference and hit the same walls (or different ones), I would genuinely like to compare notes.

The Swiss tier is live at sokkan.ch/en/inference. SOKKAN is a product of Ninabot Sàrl, Geneva.

I run 9 parallel Claude Code sessions. The bottleneck wasn't the model — it was memory.

Nicolas Micaud — Wed, 22 Jul 2026 13:08:08 +0000

For the past few months I’ve been building my products — a job platform, a 3D jewelry shop, an internal ERP, a handful of sites — almost entirely through parallel Claude Code sessions. Usually around nine at a time, in tmux, each one owning a domain: backend, frontend, infra, marketing.

Throughput roughly doubled. And one problem never went away.

Every new session starts amnesiac.

I caught myself typing the same sentence dozens of times a day: “check your memory, we were working on X.” Every morning, every context switch, every fresh session: reload the context, re-explain the project, start over.

The failure analysis that changed my mind
At some point I started root-causing my agent failures seriously — the sessions that went off the rails, rebuilt something that existed, or contradicted a decision made two days earlier.

Almost none of them were model intelligence failures.

They were memory architecture failures. My setup at the time was the obvious one: a big CLAUDE.md-style context file, ~100KB of accumulated project knowledge. Three problems with that, and they compound:

Truncation. A 100KB file doesn’t fit the injection budget. It gets cut, silently. The session doesn’t know what it doesn’t know — and neither do you, until it bites.
No recall. The file is injected linearly. The one paragraph that matters for this task is buried between four unrelated ones. Relevant knowledge exists but doesn’t surface.
Concurrent writers. Nine sessions appending to one file is a recipe for overwrites and merge noise. The memory gets worse as you use it more.
The fix isn’t a bigger context window. It’s the same thing it’s always been in software: don’t load everything — index, and retrieve what’s relevant.

Spawning a session IS the memory ritual
So I built the tool I wanted, used it internally for months, and open-sourced it three weeks ago as SOKKAN (Apache-2.0). The core idea fits in one sentence:

Spawning a session is the “check your memory” ritual.

Concretely: memory is one markdown file per fact, with a one-line description: in the frontmatter. Claude Code already writes files like this natively — SOKKAN just indexes them:

name: api-port-convention

description: "The api container listens on 8097 internally; health endpoint is /healthz (not /health)"

Decided 2026-05-12 after the nginx 502 incident. All new services follow
this: internal port from the 809x range, /healthz returns {"ok": true}.

When you spawn a session, the task description seeds a semantic search over those notes, and the top hits are injected as the session’s starting context. The session begins already knowing the port convention, the incident that motivated it, and what the last session shipped — facts that exist nowhere in the code.

The stack is deliberately boring
Here’s the part that gets raised eyebrows: there is no vector database.

markdown notes → local ONNX embeddings (multilingual MiniLM) → SQLite → dot product

That’s it. Brute-force cosine over a few hundred chunks. A project memory is hundreds of facts, not millions of documents — at that scale, pgvector or a hosted vector DB is pure operational overhead. The whole index rebuilds in seconds, runs on CPU, and nothing ever leaves your machine (the only egress is your prompts to Anthropic, same as any Claude Code use).

A side effect I didn’t plan: the embeddings are multilingual, so recall is cross-lingual. Half my notes are in French. Sessions ask in English and recall them anyway.

The cockpit around it
Memory was the reason; the rest grew from daily use:

A kanban as the front door. A card describes a task; ▶ spawn turns it into a session pre-seeded with the memory context. The agent proposes a plan and waits for your go.
Human-in-the-loop gates, enforced. The chat is built on the Claude Agent SDK — permission prompts and questions render as buttons. Nothing irreversible happens without a click. With nine sessions running, this stops being a UX nicety and becomes the safety model.
Per-session cost tracking, aggregated from the transcripts. When sessions are cheap to spawn, you spawn a lot of them; you want to know which ones eat your budget.
An audit journal — who spawned, moved, deleted what. The basis for reverting.
Sessions can talk back through bundled MCP servers: search the memory, create board cards, push a preview.
The name says the philosophy: SOKKAN (سکان) is Persian for helm. The AI rows, the human steers. It’s explicitly not an autopilot — I tried the autopilot-orchestrator route first, and I dropped it: token-hungry, slow, and the memory/context quality was worse than doing it by hand.

What running this for months taught me
Memory quality beats model quality for multi-session work. A mid-size model with the right three facts beats a frontier model guessing.
One fact per file is the discipline that makes everything work. Big notes rot; atomic notes compose. The description: line doubles as the retrieval hook — write it like you’d write a commit subject.
Cost visibility changes behavior. Once each session shows its burn, you stop leaving zombie sessions around.
The permission gate catches real incidents. Not hypothetically — it has stopped a destructive migration and one wrong-directory rm at the click prompt.
Try it
Docker + an Anthropic API key (or a Claude Pro/Max setup-token):

Fenêtre de terminal
curl -fsSL https://sokkan.ch/install.sh | sh

Repo: https://github.com/ninabot-ch/sokkan — Apache-2.0, self-hosted, BYOK.

Honest caveats: it’s v0.2. Claude-only today (the session protocol is provider-neutral; adapters are on the roadmap), single-user by default (OIDC pluggable), and some UI chrome is still in French — it was our internal tool until three weeks ago. Full disclosure: there’s a paid managed version hosted in Switzerland; that’s the business model. Self-hosted is identical and free forever.

I’m genuinely curious how others handle this: if you run multiple agent sessions in parallel, what does your memory setup look like — one big context file, git worktrees, something custom?