<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/">
  <channel>
    <title>DEV Community: Nicolas Micaud</title>
    <description>The latest articles on DEV Community by Nicolas Micaud (@nicolas_micaud_20671fb4f2).</description>
    <link>https://dev.to/nicolas_micaud_20671fb4f2</link>
    <image>
      <url>https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4042051%2Fe2430749-7609-4884-a05b-ed992092e7a5.png</url>
      <title>DEV Community: Nicolas Micaud</title>
      <link>https://dev.to/nicolas_micaud_20671fb4f2</link>
    </image>
    <atom:link rel="self" type="application/rss+xml" href="https://dev.to/feed/nicolas_micaud_20671fb4f2"/>
    <language>en</language>
    <item>
      <title>Your agent's memory fails silently. CortHeXis 2.0 is the part that notices.</title>
      <dc:creator>Nicolas Micaud</dc:creator>
      <pubDate>Mon, 05 Oct 2026 10:24:54 +0000</pubDate>
      <link>https://dev.to/nicolas_micaud_20671fb4f2/your-agents-memory-fails-silently-corthexis-20-is-the-part-that-notices-581f</link>
      <guid>https://dev.to/nicolas_micaud_20671fb4f2/your-agents-memory-fails-silently-corthexis-20-is-the-part-that-notices-581f</guid>
      <description>&lt;p&gt;Your coding agent's memory does not fail loudly. It fails the way a slow leak does: the agent just knows a little less every week, and you blame the model.&lt;/p&gt;

&lt;p&gt;I've been running my company's products through Claude Code sessions for five months, with a memory of short markdown notes behind them — 417 notes today, about 440,000 words, every one written while actually shipping something. Over that time I kept a list of every way that memory broke. Not one of them raised an error.&lt;/p&gt;

&lt;p&gt;Today we're releasing &lt;strong&gt;CortHeXis 2.0&lt;/strong&gt;, the engine that came out of that list. It's open source (Apache-2.0), it runs alone next to any agent that speaks MCP, and it's the same code that runs our own memory. This post is about the failures first, because they are the product.&lt;/p&gt;

&lt;h2&gt;
  
  
  Four ways a memory dies quietly
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;1. The memory server vanished for nine days.&lt;/strong&gt; A tool rewrote our project's &lt;code&gt;.mcp.json&lt;/code&gt; to add a browser server, and dropped the memory server in the process. Every session kept working. None of them could search the memory. Agents started answering from an index file that had been truncated to fit the context window. It took nine days to notice — because nothing was &lt;em&gt;wrong&lt;/em&gt;, things were just &lt;em&gt;worse&lt;/em&gt;.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;2. One colon broke recall for a note.&lt;/strong&gt; A note whose &lt;code&gt;description:&lt;/code&gt; contains an unquoted &lt;code&gt;:&lt;/code&gt; is invalid YAML. The indexer skipped its header, the note lost its description, and with it the few words the search was matching on. It still existed. It was just never found again.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;3. A ghost note.&lt;/strong&gt; A session appended a fact with &lt;code&gt;cat &amp;gt;&amp;gt; pricing_rules.md&lt;/code&gt; while the note was called &lt;code&gt;pricing-rules.md&lt;/code&gt;. A new file appeared, with no header, carrying a corrected fact — while the old note kept the wrong one. Two truths, and the search happily returned the stale one.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;4. A better model made recall worse.&lt;/strong&gt; We switched embeddings from MiniLM to EmbeddingGemma and recall jumped on our bench. But the hook that injects notes at every message kept the score threshold I had tuned "by eye" for the old model. Measured afterwards: it was injecting the right note in &lt;strong&gt;22 %&lt;/strong&gt; of cases. Re-tuned on the bench, not on examples: &lt;strong&gt;87 %&lt;/strong&gt;. Scores don't mean the same thing from one model to the next.&lt;/p&gt;

&lt;p&gt;None of these is exotic. Each was invisible from inside a session.&lt;/p&gt;

&lt;h2&gt;
  
  
  What CortHeXis does about it
&lt;/h2&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fw5mxiz887obnyokulkcs.jpg" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fw5mxiz887obnyokulkcs.jpg" alt="The CortHeXis dashboard: graph of the notes, recall, health" width="800" height="413"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Recall at every turn — including sub-agents.&lt;/strong&gt; A Claude Code hook (&lt;code&gt;UserPromptSubmit&lt;/code&gt;) puts the relevant notes in front of the agent before it answers, not only when the session starts. A second hook (&lt;code&gt;PreToolUse&lt;/code&gt; on &lt;code&gt;Task|Agent&lt;/code&gt;) does the same for every sub-agent it launches. Sub-agents start with an empty context, and that's where most of our memory used to get lost. The injected block is framed as data, not instructions: notes are written by agents, and a note can carry an order.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;A review that re-reads everything, every hour.&lt;/strong&gt; Twenty-five checks, each born from a failure like the ones above: the MCP server not declared or not answering, the embedding server down, the index lagging behind the files, broken links, notes with no description, a description that contradicts the body, near-duplicates, projects still "in progress" two months later, secrets pasted into a note, and a mass rewrite that made every note look new. You get a score, findings with their remedy, a history, and a daily digest to your webhook. On the public demo, it looks like this:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;score 92
warn  broken_links  1   [[modele-restitution-2025]] cited by methode-restitution-en-une-page
                        → point the link to the note that replaced the old one
info  orphans       3   notes with no link in or out: found by search only
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;That broken link is real, not staged: the note was renamed and one reference was forgotten. A demo that shows 100/100 proves nothing.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Repairs you approve.&lt;/strong&gt; Naming is fixed losslessly at indexing time (and links written with the old name are reattached). Anything that needs judgement — relinking, merging two notes, closing a stale project — comes as a proposal with a diff, applied all-or-nothing once you approve it, with a backup.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Dates you can trust.&lt;/strong&gt; Every recalled note carries its age &lt;em&gt;and where that date comes from&lt;/em&gt;: written in the note, reconstructed by the indexer, or unknown. A script that touches every file no longer makes the whole memory look fresh.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Measured, not promised.&lt;/strong&gt; &lt;code&gt;corthexis eval&lt;/code&gt; builds a recall bench from your own sessions: a prompt followed by &lt;code&gt;memory_get(note)&lt;/code&gt; is a question with a known answer, no annotation needed. A model or profile change is judged on the same questions, and a drop in recall blocks the switch.&lt;/p&gt;

&lt;h2&gt;
  
  
  The numbers, and what they don't say
&lt;/h2&gt;

&lt;p&gt;On our private memory, 300 questions (65 % French, 25 % English, 10 % German), hybrid search:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;&lt;/th&gt;
&lt;th&gt;hit@1&lt;/th&gt;
&lt;th&gt;MRR&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;1.x engine (MiniLM)&lt;/td&gt;
&lt;td&gt;42 %&lt;/td&gt;
&lt;td&gt;0.55&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;2.0, CPU, 4 cores, no GPU&lt;/td&gt;
&lt;td&gt;73 %&lt;/td&gt;
&lt;td&gt;0.82&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;2.0 with a reranker on a GPU&lt;/td&gt;
&lt;td&gt;82 %&lt;/td&gt;
&lt;td&gt;0.88&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;The biggest gap is on questions whose answer lives only in a note's &lt;em&gt;body&lt;/em&gt;: MiniLM reads the first 128 tokens of a passage and barely sees them.&lt;/p&gt;

&lt;p&gt;You can't rerun that bench — the corpus is full of infrastructure and clients, and it stays private. So 2.0 ships a &lt;strong&gt;public bench&lt;/strong&gt; you can: 60 questions in French, English and German over the fictional demo corpus, rerunnable in one command against your install. Honest caveat, written in the repo too: 38 tidy notes are an easy corpus. Every model lands at MRR 0.89–0.97 there; the bench shows the ordering (MiniLM drops to 0.73 on German questions over French notes) and lets you check your install gives the same figures. It doesn't replace the bench you build on your own notes.&lt;/p&gt;

&lt;h2&gt;
  
  
  Try it
&lt;/h2&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;git clone https://github.com/ninabot-ch/corthexis &lt;span class="o"&gt;&amp;amp;&amp;amp;&lt;/span&gt; &lt;span class="nb"&gt;cd &lt;/span&gt;corthexis
./setup.sh ~/.claude/projects/&amp;lt;your-project&amp;gt;/memory   &lt;span class="c"&gt;# or any folder of markdown notes&lt;/span&gt;
docker compose up &lt;span class="nt"&gt;-d&lt;/span&gt;                                   &lt;span class="c"&gt;# dashboard on http://localhost:8420&lt;/span&gt;

claude mcp add &lt;span class="nt"&gt;-s&lt;/span&gt; user &lt;span class="nt"&gt;--transport&lt;/span&gt; http corthexis http://localhost:8420/mcp &lt;span class="se"&gt;\&lt;/span&gt;
  &lt;span class="nt"&gt;--header&lt;/span&gt; &lt;span class="s2"&gt;"Authorization: Bearer &lt;/span&gt;&lt;span class="si"&gt;$(&lt;/span&gt;&lt;span class="nb"&gt;sed&lt;/span&gt; &lt;span class="nt"&gt;-n&lt;/span&gt; &lt;span class="s1"&gt;'s/^CORTHEXIS_TOKEN=//p'&lt;/span&gt; .env&lt;span class="si"&gt;)&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt;
python3 corthexis/hook.py &lt;span class="nb"&gt;install&lt;/span&gt;                      &lt;span class="c"&gt;# recall at every message + sub-agents&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Docker, about 2 GB of RAM, four cores, no GPU. Postgres with pgvector for the index; the notes stay plain markdown files you own, and the index can be rebuilt from them at any time. The default embedding model is downloaded on first run after you accept its licence, with an MIT-licensed fallback if you don't. After that, nothing leaves your machine to index or recall.&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Repo: &lt;a href="https://github.com/ninabot-ch/corthexis" rel="noopener noreferrer"&gt;github.com/ninabot-ch/corthexis&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;Live demo, fictional corpus, no account: &lt;a href="https://demo.corthexis.com" rel="noopener noreferrer"&gt;demo.corthexis.com&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;Site: &lt;a href="https://corthexis.com" rel="noopener noreferrer"&gt;corthexis.com&lt;/a&gt;
&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;If you already use Mem0, Zep, Letta or your framework's RAG: CortHeXis isn't trying to retrieve better than they do. It starts from the next question — how do you know that what your agent retrieves is still there, still true, and still found? I'd love to hear how your own agent memory has failed. I suspect most of those failures never threw an error either.&lt;/p&gt;

</description>
      <category>ai</category>
      <category>opensource</category>
      <category>mcp</category>
      <category>productivity</category>
    </item>
    <item>
      <title>We're taking pre-orders for an on-prem inference box built on four Intel Arc Pro B60 — here's exactly what it does</title>
      <dc:creator>Nicolas Micaud</dc:creator>
      <pubDate>Sat, 12 Sep 2026 07:45:27 +0000</pubDate>
      <link>https://dev.to/nicolas_micaud_20671fb4f2/were-taking-pre-orders-for-an-on-prem-inference-box-built-on-four-intel-arc-pro-b60-heres-2fjd</link>
      <guid>https://dev.to/nicolas_micaud_20671fb4f2/were-taking-pre-orders-for-an-on-prem-inference-box-built-on-four-intel-arc-pro-b60-heres-2fjd</guid>
      <description>&lt;p&gt;Since 10 September, four Intel Arc Pro B60 cards in a tower in Meyrin, Geneva, have been running three language models at once — for our own products first. This week we turned that configuration into something you can put in your own building, and opened pre-orders. This post is the spec sheet with the numbers we actually measured, the part that doesn't scale, and what we are not promising.&lt;/p&gt;

&lt;h2&gt;
  
  
  The problem it solves
&lt;/h2&gt;

&lt;p&gt;We kept hearing the same sentence from law firms, clinics and fiduciaries in Switzerland: &lt;em&gt;"the cloud is not allowed here."&lt;/em&gt; Not "expensive", not "we prefer local" — not allowed. For them the choice is not ChatGPT versus something cheaper; it is a machine inside the walls, or nothing.&lt;/p&gt;

&lt;p&gt;So SOKKAN Anchor is a box. It runs the models, the retrieval and the gateway on premises. A key can be declared &lt;code&gt;sovereign-only&lt;/code&gt;, and the server — not a client-side setting — guarantees that no request under that key will ever reach a cloud provider, not even as a fallback. If the local model is down, you get an error, not a detour.&lt;/p&gt;

&lt;h2&gt;
  
  
  What is in it
&lt;/h2&gt;

&lt;p&gt;The entry configuration, &lt;strong&gt;Anchor E&lt;/strong&gt;, carries four Intel Arc Pro B60 (24 GB each, 96 GB total) and runs three resident models:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Card&lt;/th&gt;
&lt;th&gt;Model&lt;/th&gt;
&lt;th&gt;Format / runtime&lt;/th&gt;
&lt;th&gt;Weights&lt;/th&gt;
&lt;th&gt;Role&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;0&lt;/td&gt;
&lt;td&gt;gpt-oss-20b&lt;/td&gt;
&lt;td&gt;MXFP4 · vLLM (Intel XPU build)&lt;/td&gt;
&lt;td&gt;13 GB&lt;/td&gt;
&lt;td&gt;overnight automation, batch, embeddings&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;1&lt;/td&gt;
&lt;td&gt;Qwen3-Coder-30B-A3B&lt;/td&gt;
&lt;td&gt;GGUF Q4_K_XL · llama.cpp SYCL&lt;/td&gt;
&lt;td&gt;17.7 GB&lt;/td&gt;
&lt;td&gt;developers&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;2 + 3&lt;/td&gt;
&lt;td&gt;Qwen3-Next-80B-A3B&lt;/td&gt;
&lt;td&gt;GGUF Q3_K_XL · llama.cpp SYCL&lt;/td&gt;
&lt;td&gt;35.6 GB&lt;/td&gt;
&lt;td&gt;finance, HR, management assistants&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;All three are MoE models with about 3B active parameters. That is what makes the cohabitation possible: 130B of cumulated weights at the speed of a small model. A single dense 70B would have eaten two cards to be four times slower.&lt;/p&gt;

&lt;p&gt;The production platform is an AMD EPYC 9115 (16 cores, SP5) on a Supermicro board with IPMI, 128 GB of DDR5 ECC, two 2 TB NVMe drives in a ZFS mirror, a 1.6 kW ATX 3.1 supply, in a silent office tower or a 4U rack case. Why EPYC rather than a desktop chip: 128 PCIe 5.0 lanes (four cards at full width, no bifurcation games), ECC, and remote management. IPMI is not a luxury for a box we support remotely — it is how we reboot a machine in a law office at 7 am without anyone touching it. ZFS mirror because the box holds the customer's document index, and a dead NVMe must not mean a re-ingestion.&lt;/p&gt;

&lt;h2&gt;
  
  
  What it does — measured
&lt;/h2&gt;

&lt;p&gt;Every number below was measured on our prototype: a 2018 X299 workstation (i9-9980XE, 64 GB DDR4) that &lt;em&gt;also&lt;/em&gt; carries our production job platform — Postgres, twelve Celery workers, two web front ends. It is a pessimistic rig. The EPYC build will not be slower.&lt;/p&gt;

&lt;p&gt;Each model alone, warm:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Model&lt;/th&gt;
&lt;th&gt;Decode&lt;/th&gt;
&lt;th&gt;Prefill (~4k tokens)&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Qwen3-Coder-30B&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;82.4 tok/s&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;4,098 tok in 4.2 s&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;gpt-oss-20b&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;41.2 tok/s&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;3,307 tok in 2.4 s&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Qwen3-Next-80B&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;36.2 tok/s&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;4,098 tok in 10.3 s&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;All three under load at once — 12 concurrent requests, 4 per model, 300 tokens each: &lt;strong&gt;3,600 tokens in 30.3 s, 119 tok/s aggregate&lt;/strong&gt;, no errors, no measurable interference between cards. The 20B alone batches to 134 tok/s at 4 requests and 241 tok/s at 8.&lt;/p&gt;

&lt;p&gt;Power: 163 W idle with the three models loaded, &lt;strong&gt;211 W peak&lt;/strong&gt; under the 12-request load, for the four cards. The whole machine is around 350 W. At Swiss electricity prices, running around the clock, that is roughly CHF 750 a year.&lt;/p&gt;

&lt;p&gt;Host RAM stayed above 41 GB free at the worst moment — with a full production stack running next to it. That is why the spec sheet does not need 256 GB.&lt;/p&gt;

&lt;p&gt;Since 10 September our job platform sends about &lt;strong&gt;107 million tokens a week&lt;/strong&gt; of batch work — job-ad classification, extraction, enrichment — to this box instead of a cloud provider. That is not a benchmark; it is the workload that paid for the cards.&lt;/p&gt;

&lt;h2&gt;
  
  
  The part that doesn't scale
&lt;/h2&gt;

&lt;p&gt;The 80B is the model people want: it is the better writer and analyst. It is also the one that does not scale under concurrency, and the reason is the runtime, not the model.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Concurrent requests&lt;/th&gt;
&lt;th&gt;gpt-oss-20b (vLLM)&lt;/th&gt;
&lt;th&gt;Qwen3-Next-80B (llama.cpp)&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;1&lt;/td&gt;
&lt;td&gt;35.1 tok/s&lt;/td&gt;
&lt;td&gt;32.0 tok/s&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;2&lt;/td&gt;
&lt;td&gt;35.0 tok/s each&lt;/td&gt;
&lt;td&gt;14.5 tok/s each&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;4&lt;/td&gt;
&lt;td&gt;33.6 tok/s each&lt;/td&gt;
&lt;td&gt;7.0 tok/s each&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;aggregate at 4&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;133.9 tok/s&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;
&lt;strong&gt;20.7 tok/s&lt;/strong&gt; (less than solo)&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;vLLM does continuous batching; throughput grows almost linearly with load. llama.cpp does not; its aggregate throughput &lt;em&gt;regresses&lt;/em&gt; as concurrency rises. Four users on the 80B get less total work done than one.&lt;/p&gt;

&lt;p&gt;Why is it on llama.cpp then? Because vLLM on Intel XPU currently serves only quantisations with a native XPU kernel. MXFP4 gpt-oss has one. Every other quantised MoE we tried routes its expert layers to Marlin — an NVIDIA kernel — and dies with &lt;code&gt;device_capability = -1&lt;/code&gt;, whatever you pass to &lt;code&gt;--quantization&lt;/code&gt;. Dense AWQ works (&lt;code&gt;int4_gemm_w4a16&lt;/code&gt;, and you must set &lt;code&gt;--dtype float16&lt;/code&gt;); AWQ MoE does not. So everything that is not gpt-oss runs through llama.cpp SYCL, which is honestly good — Qwen3-Next's hybrid Gated DeltaNet attention runs on it — but serves one user well and four badly.&lt;/p&gt;

&lt;p&gt;This goes on the spec sheet as a design rule, not a footnote: &lt;strong&gt;the 80B serves two simultaneous users, not a department.&lt;/strong&gt; Anything real-time — voice, interactive completion — must hit a model served by vLLM. Voice needs roughly 15 tok/s per speaker to stay fluid; the 80B holds two conversations, the 20B at least four.&lt;/p&gt;

&lt;p&gt;Two more lessons that cost us real time: models must be loaded &lt;strong&gt;sequentially&lt;/strong&gt; (the RAM peak is at load, not at serve — three simultaneous &lt;code&gt;docker start&lt;/code&gt; at boot reproduced a crash), and the first request after loading is 3–4× slower than steady state, so the box warms itself up before a customer touches it.&lt;/p&gt;

&lt;h2&gt;
  
  
  What we are not promising
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;It does not replace ChatGPT or Claude for general use. A developer who compares side by side will prefer the frontier model. Anchor is for the work you are &lt;em&gt;not allowed&lt;/em&gt; to send there.&lt;/li&gt;
&lt;li&gt;We have not measured &lt;strong&gt;Anchor S&lt;/strong&gt; (two Arc Pro B70, 64 GB). Its numbers will be published when the cards arrive; until then the page says "estimate".&lt;/li&gt;
&lt;li&gt;vLLM tensor-parallel across the four cards wedged our X299 prototype three times in a day (a load-time DMA/PCIe storm, not RAM — it failed with 59 GB free and production stopped). We will retry on EPYC. It is not needed for the three-model configuration above.&lt;/li&gt;
&lt;li&gt;The larger M (128 GB) and L (192 GB) configurations are described, not open.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  What is on top of the hardware
&lt;/h2&gt;

&lt;p&gt;SOKKAN Inference, the gateway from our open-source cockpit (&lt;a href="https://github.com/ninabot-ch/sokkan?utm_source=devto" rel="noopener noreferrer"&gt;github.com/ninabot-ch/sokkan&lt;/a&gt;), runs on the box: one LAN endpoint, OpenAI- and Anthropic-Messages-compatible, so Claude Code, any OpenAI SDK and any agent framework point at it unchanged. Per-key metering, per-key &lt;code&gt;sovereign-only&lt;/code&gt; policy, Prometheus/Grafana, WireGuard for support, Ansible for fleet updates. Plus Nina, a document assistant with retrieval over the customer's own files — the same chain as &lt;a href="https://demo.ninabot.ch/app/?mode=onprem&amp;amp;utm_source=devto" rel="noopener noreferrer"&gt;this demo&lt;/a&gt; we built for a truck garage's technical manuals.&lt;/p&gt;

&lt;h2&gt;
  
  
  Price, and how the pre-order works
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Anchor E&lt;/strong&gt; — 4× B60, 96 GB: &lt;strong&gt;CHF 10,450&lt;/strong&gt; excl. VAT.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Anchor S&lt;/strong&gt; — 2× B70, 64 GB: &lt;strong&gt;CHF 11,100&lt;/strong&gt; excl. VAT.&lt;/li&gt;
&lt;li&gt;Built to order, delivered and installed in about six weeks, 24-month warranty on new parts. Managed operation (model updates, monitoring, remote support) from CHF 300 a month, optional.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;We are a small company and we do not stock EPYC boards and GPUs on the shelf: parts are ordered against a pre-order. The deposit is 20 % (CHF 2,100 for E), &lt;strong&gt;fully refundable until the machine ships&lt;/strong&gt;, deducted from the final invoice. The first five units are sold at this price whatever the GPU market does — B70s went up 26–48 % in August alone — and their serial number is engraved on the front plate.&lt;/p&gt;

&lt;p&gt;Page, specs and the full measurement report: &lt;a href="https://ninabot.ch/en/anchor?utm_source=devto" rel="noopener noreferrer"&gt;ninabot.ch/en/anchor&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;If you run Arc Pro cards for inference and hit the same walls — or different ones — I would genuinely like to compare notes. I answer the comments myself.&lt;/p&gt;




&lt;p&gt;&lt;em&gt;SOKKAN Anchor is a product of Ninabot Sàrl, Geneva. The Geneva-hosted "Swiss" tier of &lt;a href="https://sokkan.ch/en/inference/?utm_source=devto" rel="noopener noreferrer"&gt;SOKKAN Inference&lt;/a&gt; runs on the same prototype.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>ai</category>
      <category>hardware</category>
      <category>opensource</category>
      <category>selfhosted</category>
    </item>
    <item>
      <title>I stopped asking my AI agents to read the project memory. Now the server does it for them.</title>
      <dc:creator>Nicolas Micaud</dc:creator>
      <pubDate>Sat, 12 Sep 2026 06:21:17 +0000</pubDate>
      <link>https://dev.to/nicolas_micaud_20671fb4f2/i-stopped-asking-my-ai-agents-to-read-the-project-memory-now-the-server-does-it-for-them-238n</link>
      <guid>https://dev.to/nicolas_micaud_20671fb4f2/i-stopped-asking-my-ai-agents-to-read-the-project-memory-now-the-server-does-it-for-them-238n</guid>
      <description>&lt;blockquote&gt;
&lt;p&gt;Follow-up to &lt;a href="https://dev.to/nicolas_micaud_20671fb4f2/i-run-9-parallel-claude-code-sessions-the-bottleneck-wasnt-the-model-it-was-memory-1n7c"&gt;&lt;em&gt;I run 9 parallel Claude Code sessions — the bottleneck wasn't&lt;br&gt;
the model, it was memory&lt;/em&gt;&lt;/a&gt;.&lt;br&gt;
That post described the problem. This one is about the fix I shipped, and the&lt;br&gt;
uncomfortable thing it taught me about "instructing" a model.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;h2&gt;
  
  
  The setup
&lt;/h2&gt;

&lt;p&gt;I build and run our products — a Swiss job platform, a handful of sites, the&lt;br&gt;
infrastructure under them — with a cockpit that keeps ~9 Claude Code sessions&lt;br&gt;
alive in parallel. Each session is a task: a kanban card you drag, and it&lt;br&gt;
becomes a working agent.&lt;/p&gt;

&lt;p&gt;The recurring failure was never intelligence. It was amnesia. Session #7 would&lt;br&gt;
happily reimplement an error format we'd standardized weeks ago, because nothing&lt;br&gt;
told it that decision existed. The decision &lt;em&gt;was&lt;/em&gt; written down — in a markdown&lt;br&gt;
note, in the project memory — but the session never looked.&lt;/p&gt;
&lt;h2&gt;
  
  
  v1: I asked nicely
&lt;/h2&gt;

&lt;p&gt;My first fix was the obvious one. When a session spawns, append an instruction&lt;br&gt;
to its first prompt:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;{the task}
Start by calling the memory_search tool to load any relevant project context,
then propose a plan — don't execute anything without my go-ahead.
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;This is what most "give your agent memory" setups do: a system prompt, a tool,&lt;br&gt;
and trust that the model uses them. And it mostly works. &lt;em&gt;Mostly.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;Here's the thing about "mostly" when you have nine of them running. One session&lt;br&gt;
in nine ignoring the instruction isn't a 1-in-9 annoyance — it's a silent,&lt;br&gt;
guaranteed defect generator. The model doesn't announce "I skipped the memory&lt;br&gt;
this time." It just dives into the code, does something locally reasonable, and&lt;br&gt;
violates a convention you'd forgotten you even needed to defend. You find out at&lt;br&gt;
review, or worse, after merge.&lt;/p&gt;

&lt;p&gt;I spent a while trying to make the instruction &lt;em&gt;stickier&lt;/em&gt; — stronger wording,&lt;br&gt;
putting it last, tool descriptions that begged. That's when it clicked: &lt;strong&gt;I was&lt;br&gt;
trying to make a non-deterministic thing reliable by asking harder.&lt;/strong&gt; Wrong&lt;br&gt;
layer.&lt;/p&gt;
&lt;h2&gt;
  
  
  v2: I stopped asking
&lt;/h2&gt;

&lt;p&gt;The recall doesn't belong in the prompt. It belongs in the plumbing.&lt;/p&gt;

&lt;p&gt;Now, when a session spawns, the &lt;strong&gt;server&lt;/strong&gt; runs the semantic search — before the&lt;br&gt;
model gets a single turn — and injects the results directly into the session's&lt;br&gt;
first message:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;_memory_preseed&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;query&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;str&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;top_k&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;int&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;5&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;-&amp;gt;&lt;/span&gt; &lt;span class="nb"&gt;str&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="n"&gt;hits&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;memory_search&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;query&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;top_k&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;top_k&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;   &lt;span class="c1"&gt;# server-side, deterministic
&lt;/span&gt;    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="ow"&gt;not&lt;/span&gt; &lt;span class="n"&gt;hits&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="sh"&gt;""&lt;/span&gt;                               &lt;span class="c1"&gt;# empty memory → fall back to the ritual
&lt;/span&gt;    &lt;span class="n"&gt;lines&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;=== Project memory (auto-recalled) ===&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
    &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;h&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;hits&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="n"&gt;star&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;★ &lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;h&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;priority&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;else&lt;/span&gt; &lt;span class="sh"&gt;""&lt;/span&gt;
        &lt;span class="n"&gt;lines&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;append&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;- &lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;star&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s"&gt;[&lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;h&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="s"&gt;note_name&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s"&gt;] &lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;h&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="s"&gt;description&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s"&gt; — &lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;h&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="s"&gt;snippet&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="se"&gt;\n&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;join&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;lines&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The task the model receives now looks like:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Fix the healthcheck flapping on staging.

=== Project memory (auto-recalled) ===
- ★ [staging-stack] Staging API is on port 6443, deploys land on host "callisto"
- [deploy-ritual] Deploys go exclusively through `make ship-v2`, never raw rsync

The notes above were auto-recalled from project memory for this task. Call
memory_get on any note you need in full… then propose a plan and wait for my go.
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The model can't &lt;em&gt;not&lt;/em&gt; have the context now. It's not a tool it might call. It's&lt;br&gt;
in the first tokens it reads.&lt;/p&gt;

&lt;p&gt;Crucially, the old ritual is still there — as a &lt;strong&gt;fallback&lt;/strong&gt;. If the memory is&lt;br&gt;
empty (fresh project) or the embedding backend is down, the injected block is&lt;br&gt;
empty and the prompt reverts to "please go search." Determinism where it&lt;br&gt;
matters, graceful degradation where it doesn't.&lt;/p&gt;

&lt;h2&gt;
  
  
  Does it actually work? I planted a trap.
&lt;/h2&gt;

&lt;p&gt;Claims about memory systems are cheap. So the release has an end-to-end test&lt;br&gt;
that seeds a throwaway instance with two notes containing facts that exist&lt;br&gt;
&lt;strong&gt;nowhere in the code&lt;/strong&gt;: the staging port is &lt;code&gt;6443&lt;/code&gt;, deploys go to a host called&lt;br&gt;
&lt;code&gt;callisto&lt;/code&gt;, via &lt;code&gt;make ship-v2&lt;/code&gt;. The workspace README is deliberately &lt;em&gt;wrong&lt;/em&gt;&lt;br&gt;
about all three.&lt;/p&gt;

&lt;p&gt;Then it spawns a session asking: &lt;em&gt;"what port does staging run on, which host do&lt;br&gt;
we deploy to, and with what command?"&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;A session that reads the code (or the lying README) fails. A session with&lt;br&gt;
working recall answers &lt;code&gt;6443 / callisto / make ship-v2&lt;/code&gt; — and, in the transcript,&lt;br&gt;
literally says it answered &lt;em&gt;"without needing to search the codebase."&lt;/em&gt; That's&lt;br&gt;
the whole product in one assertion: the fact reached the model because the&lt;br&gt;
server put it there, not because the model went looking.&lt;/p&gt;

&lt;h2&gt;
  
  
  The boring parts (on purpose)
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Notes are just markdown&lt;/strong&gt; with YAML frontmatter — the same files Claude Code
already writes. One fact per file, a &lt;code&gt;description:&lt;/code&gt; that doubles as the
embedding text, optional &lt;code&gt;priority: high&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;The index is incremental sqlite.&lt;/strong&gt; Vectors live in the same store. No
pgvector, no ANN library — at a few hundred notes, dot-product over a warm
in-process model is instant, and I'd rather ship boring than clever.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Embeddings run locally by default&lt;/strong&gt; (fastembed / ONNX, a ~120MB multilingual
MiniLM). No external embedding service unless you point one.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;&lt;code&gt;[[wikilinks]]&lt;/code&gt; are indexed&lt;/strong&gt;, so a session can walk the graph with a
&lt;code&gt;memory_links&lt;/code&gt; tool, and the cockpit draws it with type filters and connected
clusters.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;&lt;code&gt;priority: high&lt;/code&gt; is a bounded, multiplicative nudge&lt;/strong&gt; — I started with a flat
additive boost and it let a weak match jump above genuinely relevant notes.
Multiply, don't add.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  What I deliberately didn't build
&lt;/h2&gt;

&lt;p&gt;This isn't an autopilot orchestrator. There's no "let the swarm run overnight."&lt;br&gt;
Every irreversible action waits for a human click — &lt;em&gt;the helm, not the&lt;br&gt;
autopilot&lt;/em&gt;. The point of the memory work isn't to remove me from the loop; it's&lt;br&gt;
to make sure that when I'm steering nine sessions, none of them is flying blind.&lt;/p&gt;

&lt;h2&gt;
  
  
  The lesson I keep relearning
&lt;/h2&gt;

&lt;p&gt;When you need a model to do something &lt;em&gt;every single time&lt;/em&gt;, don't put it in the&lt;br&gt;
prompt and hope. Move it to a layer that doesn't have opinions. Prompts are for&lt;br&gt;
judgment. Plumbing is for guarantees. I keep having to relearn which is which.&lt;/p&gt;

</description>
      <category>ai</category>
      <category>llm</category>
      <category>productivity</category>
      <category>opensource</category>
    </item>
    <item>
      <title>We serve a sovereign inference tier from four Intel Arc Pro B60 — here's what it actually does (and doesn't)</title>
      <dc:creator>Nicolas Micaud</dc:creator>
      <pubDate>Fri, 11 Sep 2026 13:40:02 +0000</pubDate>
      <link>https://dev.to/nicolas_micaud_20671fb4f2/we-serve-a-sovereign-inference-tier-from-four-intel-arc-pro-b60-heres-what-it-actually-does-and-49e2</link>
      <guid>https://dev.to/nicolas_micaud_20671fb4f2/we-serve-a-sovereign-inference-tier-from-four-intel-arc-pro-b60-heres-what-it-actually-does-and-49e2</guid>
      <description>&lt;p&gt;Since this week, SOKKAN Inference has a tier called "Swiss". It is served from a machine in Meyrin, Geneva, on hardware we own, with no NVIDIA card inside. This post is the honest version of what that machine does, with the numbers we measured — including the ones that don't scale.&lt;/p&gt;

&lt;h2&gt;
  
  
  Why Intel
&lt;/h2&gt;

&lt;p&gt;We are a small Swiss company. Our inference product already had two cloud tiers, EU-sovereign, served from French datacenters. What we didn't have was an answer for the customer who says "the data cannot leave Switzerland, full stop" — and, longer term, for the customer who says "the data cannot leave the building".&lt;/p&gt;

&lt;p&gt;That meant buying GPUs, in 2026, in the middle of a memory shortage. We chased second-hand RTX 3090s for a month and lost every auction above market price. Meanwhile the Intel Arc Pro B60 — 24 GB of VRAM, a 2-slot blower, natively x8 — was still selling near MSRP, at CHF 614 a card. Four of them came to about CHF 2,450 for 96 GB of VRAM. That was the whole argument: VRAM per franc, available now.&lt;/p&gt;

&lt;p&gt;The bet was on the software stack, not the silicon. The risk with Intel in 2026 is not the hardware; it is whether vLLM, llama.cpp and the XPU runtime hold up under a service you sell.&lt;/p&gt;

&lt;h2&gt;
  
  
  What we run
&lt;/h2&gt;

&lt;p&gt;The chassis is a 2018 X299 workstation (i9-9980XE, 64 GB DDR4) that also carries our production job platform — Postgres, Celery workers, a few web front ends. Not a clean benchmark rig. Every number below is therefore pessimistic; the production appliance is an EPYC board dedicated to inference.&lt;/p&gt;

&lt;p&gt;Four B60s, each on its own x8 Gen3 root port, run three resident models:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Card&lt;/th&gt;
&lt;th&gt;Model&lt;/th&gt;
&lt;th&gt;Format / runtime&lt;/th&gt;
&lt;th&gt;Weights&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;0&lt;/td&gt;
&lt;td&gt;gpt-oss-20b&lt;/td&gt;
&lt;td&gt;MXFP4 · vLLM (Intel XPU build 0.21)&lt;/td&gt;
&lt;td&gt;13 GB&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;1&lt;/td&gt;
&lt;td&gt;Qwen3-Coder-30B-A3B&lt;/td&gt;
&lt;td&gt;GGUF Q4_K_XL · llama.cpp SYCL&lt;/td&gt;
&lt;td&gt;17.7 GB&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;2 + 3&lt;/td&gt;
&lt;td&gt;Qwen3-Next-80B-A3B&lt;/td&gt;
&lt;td&gt;GGUF Q3_K_XL · llama.cpp SYCL&lt;/td&gt;
&lt;td&gt;35.6 GB&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;All three are MoE models with about 3B active parameters, which is what makes the cohabitation possible: 130B of cumulated weights at the speed of a small model.&lt;/p&gt;

&lt;p&gt;The Swiss tier is card 0.&lt;/p&gt;

&lt;h2&gt;
  
  
  What it does — measured
&lt;/h2&gt;

&lt;p&gt;gpt-oss-20b under vLLM, warm, on one B60:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Concurrent requests&lt;/th&gt;
&lt;th&gt;Per request&lt;/th&gt;
&lt;th&gt;Aggregate&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;1&lt;/td&gt;
&lt;td&gt;35.1 tok/s&lt;/td&gt;
&lt;td&gt;35.1 tok/s&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;2&lt;/td&gt;
&lt;td&gt;35.0 tok/s&lt;/td&gt;
&lt;td&gt;70 tok/s&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;4&lt;/td&gt;
&lt;td&gt;33.6 tok/s&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;133.9 tok/s&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Prefill is around 5,000 tok/s on a 4.3k-token prompt. In an earlier run at 8 concurrent requests the same card reached 241 tok/s aggregate. Continuous batching does exactly what it says on a model this size: throughput grows almost linearly with load, and per-request speed barely moves.&lt;/p&gt;

&lt;p&gt;Power for the four cards: 163 W idle with all three models loaded, &lt;strong&gt;211 W peak&lt;/strong&gt; under 12 concurrent requests across the three models. Temperatures 52–62 °C. On the UPS, the whole machine — including the production it also carries — moves from 13% to 19% load. At Swiss electricity prices that is roughly CHF 750 a year for the full box, running around the clock.&lt;/p&gt;

&lt;p&gt;Tool calling works on all three models. For gpt-oss you need &lt;code&gt;--enable-auto-tool-choice --tool-call-parser openai&lt;/code&gt;; without the harmony parser you get zero tool calls and an empty content field, and you will spend an evening wondering why.&lt;/p&gt;

&lt;h2&gt;
  
  
  What does not scale
&lt;/h2&gt;

&lt;p&gt;The 80B model is the one people want — it is the better writer, the better analyst. It is also the one that does not scale, and the reason is the runtime, not the model.&lt;/p&gt;

&lt;p&gt;llama.cpp has no continuous batching. Under concurrency, the same 80B goes from 32.0 tok/s for one request, to 14.5 tok/s each for two, to 7.0 tok/s each for four. Worse: the &lt;em&gt;aggregate&lt;/em&gt; throughput regresses — 32 → 28.8 → 20.7 tok/s. Four users on that model get less total work done than one.&lt;/p&gt;

&lt;p&gt;So why is it on llama.cpp? Because vLLM on Intel XPU currently only serves quantisations that have a native XPU kernel. MXFP4 gpt-oss does. Every other quantised MoE we tried routes its expert layers to Marlin, an NVIDIA kernel, and dies with &lt;code&gt;device_capability = -1&lt;/code&gt; regardless of the &lt;code&gt;--quantization&lt;/code&gt; flag. Dense AWQ works (&lt;code&gt;int4_gemm_w4a16&lt;/code&gt;, and you must pass &lt;code&gt;--dtype float16&lt;/code&gt;); AWQ MoE does not. Everything that is not gpt-oss goes through llama.cpp SYCL, and llama.cpp SYCL is honestly good — Qwen3-Next's hybrid attention runs on it — but it serves one user well and four users badly.&lt;/p&gt;

&lt;p&gt;We also could not get vLLM tensor-parallel across the four cards to stay up on this platform. Loading four workers in parallel wedged the machine three times in one day, once with 59 GB of RAM free and production stopped, so it is not memory; it looks like a DMA/PCIe storm at load on an old chipset. llama.cpp, which loads sequentially, spreads gpt-oss-120b across the four cards without incident — 18.8 tok/s decode, ~250 tok/s prefill, stable — but that prefill number is why you won't see the 120b behind a public tier yet. We will retry TP on the EPYC board.&lt;/p&gt;

&lt;p&gt;Two smaller lessons that cost us real time: models must be loaded &lt;strong&gt;sequentially&lt;/strong&gt; (the memory peak is at load, not at serve — three simultaneous &lt;code&gt;docker start&lt;/code&gt; at boot reproduce a crash), and the first request after loading is 3–4× slower than steady state, so warm the service before a customer touches it.&lt;/p&gt;

&lt;h2&gt;
  
  
  The Swiss tier, as sold
&lt;/h2&gt;

&lt;p&gt;Given all that, the Swiss tier is deliberately small and deliberately honest:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;A 20B-class model (gpt-oss-20b), best for chat, RAG and agents. &lt;strong&gt;Not a frontier coding model.&lt;/strong&gt; If you need that, our other tiers are EU-sovereign and better at it.&lt;/li&gt;
&lt;li&gt;Served in Geneva on hardware we own. Data never leaves Switzerland.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Never rerouted&lt;/strong&gt; to an EU provider. If the primary model is down, a second model on a neighbouring card takes over, still in Geneva; if that is down too, you get a clear error, not a silent detour.&lt;/li&gt;
&lt;li&gt;Capacity: 4 concurrent requests. Above that, a 429. We would rather tell you than queue you.&lt;/li&gt;
&lt;li&gt;CHF 0.60 / 2.40 per million tokens (in / out), prepaid, metered to the token.&lt;/li&gt;
&lt;li&gt;Anthropic Messages-compatible, so Claude Code and any Anthropic client work unchanged:
&lt;/li&gt;
&lt;/ul&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;  &lt;span class="nb"&gt;export &lt;/span&gt;&lt;span class="nv"&gt;ANTHROPIC_BASE_URL&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;https://infer.sokkan.ch
  &lt;span class="nb"&gt;export &lt;/span&gt;&lt;span class="nv"&gt;ANTHROPIC_AUTH_TOKEN&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;sik_your_token
  &lt;span class="nb"&gt;export &lt;/span&gt;&lt;span class="nv"&gt;ANTHROPIC_MODEL&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;sokkan-swiss
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Reselling raw tokens from one card will never be a business — at these prices a saturated B60 earns about what it burns in electricity. The point is different: the same machine hosts services on top of the model (a website assistant for SMEs, a finance assistant for our ERP), and the Swiss tier is the door for developers who want to build the same kind of thing with the same residency guarantee.&lt;/p&gt;

&lt;h2&gt;
  
  
  What comes next
&lt;/h2&gt;

&lt;p&gt;The same four-card configuration becomes SOKKAN Anchor, an on-prem appliance for organisations that have banned the cloud altogether. The dimensioning rule we learned the hard way goes into the sales sheet: anything real-time — voice, interactive completion — must hit a model served by vLLM; the big llama.cpp model serves two simultaneous users, not a department.&lt;/p&gt;

&lt;p&gt;If you are running Arc Pro cards for inference and hit the same walls (or different ones), I would genuinely like to compare notes.&lt;/p&gt;




&lt;p&gt;&lt;em&gt;The Swiss tier is live at &lt;a href="https://sokkan.ch/en/inference/?utm_source=devto&amp;amp;utm_medium=article&amp;amp;utm_campaign=swiss_tier" rel="noopener noreferrer"&gt;sokkan.ch/en/inference&lt;/a&gt;. SOKKAN is a product of Ninabot Sàrl, Geneva.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>ai</category>
      <category>llm</category>
      <category>intel</category>
      <category>selfhosted</category>
    </item>
    <item>
      <title>I run 9 parallel Claude Code sessions. The bottleneck wasn't the model — it was memory.</title>
      <dc:creator>Nicolas Micaud</dc:creator>
      <pubDate>Wed, 22 Jul 2026 13:08:08 +0000</pubDate>
      <link>https://dev.to/nicolas_micaud_20671fb4f2/i-run-9-parallel-claude-code-sessions-the-bottleneck-wasnt-the-model-it-was-memory-1n7c</link>
      <guid>https://dev.to/nicolas_micaud_20671fb4f2/i-run-9-parallel-claude-code-sessions-the-bottleneck-wasnt-the-model-it-was-memory-1n7c</guid>
      <description>&lt;p&gt;For the past few months I’ve been building my products — a job platform, a 3D jewelry shop, an internal ERP, a handful of sites — almost entirely through parallel Claude Code sessions. Usually around nine at a time, in tmux, each one owning a domain: backend, frontend, infra, marketing.&lt;/p&gt;

&lt;p&gt;Throughput roughly doubled. And one problem never went away.&lt;/p&gt;

&lt;p&gt;Every new session starts amnesiac.&lt;/p&gt;

&lt;p&gt;I caught myself typing the same sentence dozens of times a day: “check your memory, we were working on X.” Every morning, every context switch, every fresh session: reload the context, re-explain the project, start over.&lt;/p&gt;

&lt;p&gt;The failure analysis that changed my mind&lt;br&gt;
At some point I started root-causing my agent failures seriously — the sessions that went off the rails, rebuilt something that existed, or contradicted a decision made two days earlier.&lt;/p&gt;

&lt;p&gt;Almost none of them were model intelligence failures.&lt;/p&gt;

&lt;p&gt;They were memory architecture failures. My setup at the time was the obvious one: a big CLAUDE.md-style context file, ~100KB of accumulated project knowledge. Three problems with that, and they compound:&lt;/p&gt;

&lt;p&gt;Truncation. A 100KB file doesn’t fit the injection budget. It gets cut, silently. The session doesn’t know what it doesn’t know — and neither do you, until it bites.&lt;br&gt;
No recall. The file is injected linearly. The one paragraph that matters for this task is buried between four unrelated ones. Relevant knowledge exists but doesn’t surface.&lt;br&gt;
Concurrent writers. Nine sessions appending to one file is a recipe for overwrites and merge noise. The memory gets worse as you use it more.&lt;br&gt;
The fix isn’t a bigger context window. It’s the same thing it’s always been in software: don’t load everything — index, and retrieve what’s relevant.&lt;/p&gt;

&lt;p&gt;Spawning a session IS the memory ritual&lt;br&gt;
So I built the tool I wanted, used it internally for months, and open-sourced it three weeks ago as SOKKAN (Apache-2.0). The core idea fits in one sentence:&lt;/p&gt;

&lt;p&gt;Spawning a session is the “check your memory” ritual.&lt;/p&gt;

&lt;p&gt;Concretely: memory is one markdown file per fact, with a one-line description: in the frontmatter. Claude Code already writes files like this natively — SOKKAN just indexes them:&lt;/p&gt;




&lt;p&gt;name: api-port-convention&lt;/p&gt;

&lt;h2&gt;
  
  
  description: "The api container listens on 8097 internally; health endpoint is /healthz (not /health)"
&lt;/h2&gt;

&lt;p&gt;Decided 2026-05-12 after the nginx 502 incident. All new services follow&lt;br&gt;
this: internal port from the 809x range, /healthz returns {"ok": true}.&lt;br&gt;
￼&lt;br&gt;
When you spawn a session, the task description seeds a semantic search over those notes, and the top hits are injected as the session’s starting context. The session begins already knowing the port convention, the incident that motivated it, and what the last session shipped — facts that exist nowhere in the code.&lt;/p&gt;

&lt;p&gt;The stack is deliberately boring&lt;br&gt;
Here’s the part that gets raised eyebrows: there is no vector database.&lt;/p&gt;

&lt;p&gt;markdown notes → local ONNX embeddings (multilingual MiniLM) → SQLite → dot product&lt;br&gt;
￼&lt;br&gt;
That’s it. Brute-force cosine over a few hundred chunks. A project memory is hundreds of facts, not millions of documents — at that scale, pgvector or a hosted vector DB is pure operational overhead. The whole index rebuilds in seconds, runs on CPU, and nothing ever leaves your machine (the only egress is your prompts to Anthropic, same as any Claude Code use).&lt;/p&gt;

&lt;p&gt;A side effect I didn’t plan: the embeddings are multilingual, so recall is cross-lingual. Half my notes are in French. Sessions ask in English and recall them anyway.&lt;/p&gt;

&lt;p&gt;The cockpit around it&lt;br&gt;
Memory was the reason; the rest grew from daily use:&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F4bnee83b41v8pzakperi.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F4bnee83b41v8pzakperi.png" alt=" " width="800" height="480"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;A kanban as the front door. A card describes a task; ▶ spawn turns it into a session pre-seeded with the memory context. The agent proposes a plan and waits for your go.&lt;br&gt;
Human-in-the-loop gates, enforced. The chat is built on the Claude Agent SDK — permission prompts and questions render as buttons. Nothing irreversible happens without a click. With nine sessions running, this stops being a UX nicety and becomes the safety model.&lt;br&gt;
Per-session cost tracking, aggregated from the transcripts. When sessions are cheap to spawn, you spawn a lot of them; you want to know which ones eat your budget.&lt;br&gt;
An audit journal — who spawned, moved, deleted what. The basis for reverting.&lt;br&gt;
Sessions can talk back through bundled MCP servers: search the memory, create board cards, push a preview.&lt;br&gt;
The name says the philosophy: SOKKAN (سکان) is Persian for helm. The AI rows, the human steers. It’s explicitly not an autopilot — I tried the autopilot-orchestrator route first, and I dropped it: token-hungry, slow, and the memory/context quality was worse than doing it by hand.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F4t5lgaotefcidkmlw201.gif" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F4t5lgaotefcidkmlw201.gif" alt=" " width="600" height="375"&gt;&lt;/a&gt;&lt;br&gt;
What running this for months taught me&lt;br&gt;
Memory quality beats model quality for multi-session work. A mid-size model with the right three facts beats a frontier model guessing.&lt;br&gt;
One fact per file is the discipline that makes everything work. Big notes rot; atomic notes compose. The description: line doubles as the retrieval hook — write it like you’d write a commit subject.&lt;br&gt;
Cost visibility changes behavior. Once each session shows its burn, you stop leaving zombie sessions around.&lt;br&gt;
The permission gate catches real incidents. Not hypothetically — it has stopped a destructive migration and one wrong-directory rm at the click prompt.&lt;br&gt;
Try it&lt;br&gt;
Docker + an Anthropic API key (or a Claude Pro/Max setup-token):&lt;/p&gt;

&lt;p&gt;Fenêtre de terminal&lt;br&gt;
curl -fsSL &lt;a href="https://sokkan.ch/install.sh" rel="noopener noreferrer"&gt;https://sokkan.ch/install.sh&lt;/a&gt; | sh&lt;br&gt;
￼&lt;br&gt;
Repo: &lt;a href="https://github.com/ninabot-ch/sokkan" rel="noopener noreferrer"&gt;https://github.com/ninabot-ch/sokkan&lt;/a&gt; — Apache-2.0, self-hosted, BYOK.&lt;/p&gt;

&lt;p&gt;Honest caveats: it’s v0.2. Claude-only today (the session protocol is provider-neutral; adapters are on the roadmap), single-user by default (OIDC pluggable), and some UI chrome is still in French — it was our internal tool until three weeks ago. Full disclosure: there’s a paid managed version hosted in Switzerland; that’s the business model. Self-hosted is identical and free forever.&lt;/p&gt;

&lt;p&gt;I’m genuinely curious how others handle this: if you run multiple agent sessions in parallel, what does your memory setup look like — one big context file, git worktrees, something custom?&lt;/p&gt;

</description>
    </item>
  </channel>
</rss>
