DEV Community: ai maya

Anthropic's FlashProve: what 72x overhead and a 1,024-token cap mean for proving which model answered you

ai maya — Thu, 05 Nov 2026 00:34:34 +0000

TL;DR

This week Anthropic published FlashProve (coverage dates it November 2 to 3), an open-source (Apache 2.0) prototype that lets a model provider prove that a specific reply was produced by a specific set of committed weights, without revealing the weights. On a 400B-parameter mixture-of-experts model (Llama 4 Maverick, 4-bit weights), one H100 proves about 110 tokens per second, a proof for a 1,024-token request is about 19 MB, and a CPU checks it in about 17 seconds. The previous open-source system the authors compare against needed a 93.6 GB proof and almost a day to verify.

The numbers are a large step, but the limits matter more than the speedup. The proof covers an integer model under greedy decoding, the context is capped at 1,024 tokens, and proving costs roughly 72 to 95 times the GPU time of plain inference. My own arithmetic, using an assumed H100 rental rate, puts proving at around $6 per million tokens against a few cents for the plain run. That rules out proving every chat turn, but it makes random audits cheap: proving 300 sampled requests costs a provider under $2 in this illustration.

All FlashProve figures below come from Anthropic's research page and were cross-checked against press coverage. Everything labeled "illustrative" or "my arithmetic" is derived by me and depends on stated assumptions.

What did Anthropic actually release?

The research post is titled "Progress on provable inference for large language models" and sits at anthropic.com/research/provable-inference. The code, protocol description, technical report and Lean proofs are in the anthropics/FlashProve repository under Apache 2.0. Anthropic calls it a research prototype and says it is not deployed on its own API. TechCrunch's report adds that no production deployment has been announced.

The mechanism is a zero-knowledge succinct argument (a zkSNARK). The provider does a one-time offline step that commits to the model weights. That commitment is about 6.5 MB and took about 39 minutes to generate, and the report says one commitment is good for roughly 16 million requests. For each request, the provider can then produce a proof that the committed model, evaluated on the given prompt, yields exactly the claimed output tokens. Anyone holding the commitment, the prompt and the response can run the verifier. They need no access to the weights and no GPU.

How fast is it compared with the previous system?

The authors tested a 400B mixture-of-experts model with about 17B active parameters per token, using public 4-bit weights converted to an integer representation. Their headline numbers:

Measure	FlashProve	Prior open-source system (VerInf)
Proving throughput, same 400B model	about 110 tok/s on one H100	about 0.02 tok/s
Proof size per request	about 19 MB (18.78 MB) for 1,024 tokens	93.6 GB
Verification	17 s on 24 threads, 21 s on one thread	almost a day
Overhead vs vLLM GPU time	about 72x typical, about 94x at highest load tested (eight H100s, FP8)	not comparable

The throughput gap is about 5,500x and the proof-size gap is about 5,000x, using the reported figures. A quick consistency check on the reported numbers: 24 requests of 1,024 tokens proved in about 223 seconds works out to 110 tokens per second, and 0.02 tokens per second means about 14 hours for 1,000 tokens, which matches the "over 14 hours" figure that press coverage quotes for the older system.

What does proving cost per token, and per request?

Proofs are generated after the fact, in parallel batches, so the live reply is not slowed, though the proof arrives later. The cost is GPU time. At the reported rate, one 1,024-token request consumes about 9.3 H100-seconds of proving. I do not know what any provider pays for an H100, so the dollar figures below assume three rental rates. Change them to your own.

TOKENS_PER_REQ = 1024
BATCH, BATCH_SECONDS = 24, 223   # reported: 24 requests proved in about 223 s on one H100
OVERHEAD = 72                    # reported: about 72x vLLM GPU time

prove_tok_s = BATCH * TOKENS_PER_REQ / BATCH_SECONDS
print(f"proving throughput : {prove_tok_s:.0f} tok/s")
print(f"H100-seconds/request: {BATCH_SECONDS / BATCH:.2f}")
for rate in (1.5, 2.5, 4.0):     # ASSUMED rental price, USD per H100-hour
    prove = 1e6 / prove_tok_s / 3600 * rate
    print(f"${rate:.2f}/h: prove 1M tokens ${prove:.2f} | plain (prove/{OVERHEAD}) ${prove/OVERHEAD:.3f}")

Output:

proving throughput : 110 tok/s
H100-seconds/request: 9.29
$1.50/h: prove 1M tokens $3.78 | plain (prove/72) $0.053
$2.50/h: prove 1M tokens $6.30 | plain (prove/72) $0.088
$4.00/h: prove 1M tokens $10.08 | plain (prove/72) $0.140

The "plain" column divides the proving cost by the reported 72x. That is a rough implied figure, because the 72x comparison uses vLLM under its own configuration and the 94x figure comes from a different setup. It is useful for orders of magnitude only: proving costs dollars per million tokens where ordinary serving of a 4-bit model costs cents.

Is checking a proof cheap enough to do at scale?

Verification takes 17 seconds on 24 threads and 21 seconds on one thread. Those two numbers say the verifier barely parallelizes, which is useful: if proofs are independent, a 24-core machine could run 24 single-threaded checks at once. My arithmetic, which assumes the 21-second figure holds when 24 checks run side by side (I have not tested that), gives about 1,170 verified tokens per second per 24-core box. Checking is far cheaper than proving, and it needs no weights, so a customer, an auditor or a regulator can do it.

What does a proof not tell you?

This is where the headline can mislead, so it is worth listing the limits that the research page itself states.

Greedy decoding only. The proof is about an integer model under greedy decoding, meaning the single most likely token at each step. Chat products that sample with a temperature are outside the scope.
A quantized artifact, not your serving stack. If a provider serves a floating-point build, the report warns that its outputs may deviate from the integer model that was proven. The proof speaks about the committed integer model.
1,024 tokens of context. Prompt plus response must fit. Agent traces, long documents and retrieval-heavy prompts do not.
Mixture-of-experts assumed. The system assumes the served model is a mixture-of-experts architecture.
Weights hidden, architecture visible. The proof reveals the architecture, not the weights.
Lean covers the theory. Completeness, soundness and zero knowledge are stated and proven in Lean. The fast C++ verifier does not carry a formal guarantee, so the chain from paper to Lean to production code is not formally closed.

There is also a point the proof cannot settle by itself, and it is my own reading rather than a claim from the report. A proof binds a response to a commitment. It does not bind the commitment to a marketing name. For "this came from the model you paid for" to mean anything, someone you trust has to publish which commitment belongs to which product, which is how TechCrunch describes the fingerprint idea. Without that anchor, a provider could prove a perfectly valid statement about a cheaper model.

Which uses fit today?

Use	Fits FlashProve today?	Why
Spot-audit that short greedy requests came from the advertised model	Yes	Sampling a few hundred requests is cheap
Prove every chat turn in production	No	72x to 95x GPU overhead, sampling not supported
Verify benchmark runs with short greedy answers	Plausible	Short outputs, deterministic, third party can check
Prove agent or long-context sessions	No	1,024-token cap
Prove a floating-point production endpoint	Not directly	Proof covers the integer model

The audit case deserves a number. Suppose a provider swaps in a cheaper model on a fraction p of requests, and every swapped answer differs from what the committed model would have said. A customer who later asks for proofs on n random requests catches at least one swap with probability 1 - (1 - p)^n. With p = 1% and n = 300, that is 95.1%. The provider's cost to prove 300 requests is about 0.77 H100-hours, roughly $1.94 at the assumed $2.50 per hour. The caveat is that the provider does the proving, so a refusal or a mismatch is itself the signal, and the whole scheme only covers traffic that meets the greedy, 1,024-token conditions.

Anthropic also describes a separate prototype that signs its own inference traffic and recomputes inference on a trusted server, which it says met an internal goal due at the end of September. That route has no proving overhead but requires trusting the recomputing server, so the two designs trade differently.

Frequently asked questions

Does FlashProve let me check that a chatbot answer came from a specific model?
Only under narrow conditions today: greedy decoding, at most 1,024 tokens of prompt plus response, a mixture-of-experts model, and a provider who has published a commitment you can match to the product.

How big is a FlashProve proof and how long does it take to verify?
About 19 MB (18.78 MB) per 1,024-token request, checked in about 17 seconds on 24 CPU threads or 21 seconds on one thread, with no GPU needed.

How much more expensive is proving than running the model?
The report gives about 72 times the GPU time of vLLM in typical conditions and about 94 times at the highest load tested. With an assumed $2.50 per H100-hour, my arithmetic gives about $6.30 per million tokens to prove.

Does the proof reveal the model weights?
No. The provider commits to the weights, and the proof is zero-knowledge with respect to them. The architecture is not hidden.

Is FlashProve ready for production?
Anthropic describes it as a research prototype, it is not deployed on its API, and no production deployment has been announced.

Sources

Anthropic research: Progress on provable inference for large language models, primary source for all FlashProve figures
anthropics/FlashProve repository, Apache 2.0 code, protocol description and Lean proofs
TechCrunch, November 3, 2026, deployment status and the fingerprint idea
The Decoder, cross-check of throughput, proof size, verification time and the earlier system
Interesting Engineering, cross-check of hardware, license and limits

DeepSeek's 160,000 Huawei Ascend chips: what 10,000 chips for weeks actually buys, and why the reports disagree on training

ai maya — Tue, 03 Nov 2026 14:08:16 +0000

TL;DR

Two reports about DeepSeek's planned Huawei Ascend 950DT cluster in Ulanqab, Inner Mongolia, point in different directions. Bloomberg's September 4 report, as summarized by secondary outlets, said the 160,000-chip deployment is for inference and that training stays on Nvidia. The Information's October 19 report said DeepSeek could start pre-training on the Ascend 950DT as soon as the end of 2026, and that a frontier run could need more than 10,000 Huawei chips working together for weeks. Aggregators recirculated the second claim on November 2 and 3.

I cannot settle which report is right. What I can do is check whether the second one is physically plausible. Using Huawei's published 950DT specification (1 PFLOPS FP8, 144 GB of memory at 4 TB/s) and DeepSeek's own published V3 numbers, 10,000 chips for about 13 days at 30% utilization delivers roughly the compute of the V3 pre-training run. That is a respectable mid-size run, not a frontier-scale one. The harder constraint is reliability: if the 950DT fails at the rate Meta reported for H100s during Llama 3 training, a 10,000-chip job takes about 5 unexpected interruptions per day.

The numbers below are my own arithmetic on public inputs. Where an input comes from a secondary source I say so, and the calculator at the end lets you change every assumption.

What has actually been reported about the Ulanqab cluster?

Here is the claim trail, with the weakest links marked. I have read the secondary summaries linked at the end, not the paywalled originals.

Date	Reported by	Claim	How firm
Sep 4, 2026	Bloomberg (via summaries)	At least 160,000 Ascend 950DT chips for a roughly 1 GW site in Ulanqab; inference only; training stays on Nvidia; part of the capacity running by end of 2027 or early 2028	Single primary report, not confirmed by the companies
Oct 19, 2026	The Information (via summaries)	Huawei engineers work inside DeepSeek's offices feeding back chip-design input; pre-training on the 950DT possible as soon as end of 2026; a frontier run could need more than 10,000 Huawei chips for weeks	Single primary report, conflicts with September
Nov 2, 2026	One aggregator article	Repeats the above, adds a quote attributed to DeepSeek's founder that four Ascend 950 chips are needed to match one Nvidia GB300, and a claim of over 95% of peak performance in open-sourced kernels	No primary source given, so I do not use it
Sep 2025	Huawei (official keynote)	950DT specification and a Q4 2026 launch window	Vendor roadmap, not a shipped-product measurement

Two points matter for the rest of this post. First, the September and October reports are not strictly contradictory: a chip can be bought mostly for serving while a subset is used for an early pre-training run. Second, the hardware has not shipped in volume yet. One summary of the Bloomberg report also says Huawei's output is limited to the low hundreds of thousands of chips in 2026 because of memory supply, which would make full delivery a multi-year story.

What does the Ascend 950DT specification say?

Huawei's September 2025 roadmap lists the 950DT as the "decode and training" variant, due in the fourth quarter of 2026. The figures below are from that keynote.

Spec	Ascend 950DT (vendor roadmap)
FP8 compute	1 PFLOPS
MXFP4 compute	2 PFLOPS
Memory	144 GB, self-developed HBM (HiZQ 2.0)
Memory bandwidth	4 TB/s
Chip-to-chip interconnect	2 TB/s
Atlas 950 SuperPoD	8,192 chips, 8 EFLOPS FP8, 1,152 TB memory, 16 PB/s interconnect

Scaling these to the reported order gives a sense of size. 160,000 chips at 144 GB is 23.04 PB of HBM. At 1 PFLOPS FP8 each, the fleet's peak is 160 EFLOPS. A 10,000-chip training job, the figure from the October report, has 10 EFLOPS of peak FP8 and 1.44 PB of memory.

One caution on the cluster size. Dividing a 1 GW site by 160,000 chips gives 6.25 kW per accelerator. That is far more than a chip draws, so the site clearly includes cooling, networking, other tenants or headroom. Do not read it as a per-chip power figure.

How much pre-training compute is 10,000 chips for a few weeks?

The standard approximation for dense and mixture-of-experts training is FLOPs = 6 x N x D, where N is the number of parameters active per token and D is the number of training tokens. DeepSeek's V3 technical report gives 37 billion active parameters (671 billion total) and 14.8 trillion tokens, trained in 2.788 million H800 GPU-hours.

6 x 37e9 x 14.8e12 is about 3.3e24 FLOPs. The approximation ignores attention cost, so the true figure is a little higher.
3.3e24 FLOPs over 2.788 million GPU-hours works out to about 327 TFLOPS of useful throughput per H800. H100-class silicon, which has the same compute as the H800, peaks at roughly 1,979 TFLOPS dense FP8, so V3 ran at about 16% of peak by this accounting.

That second number is a useful anchor. A 950DT at 30% of its 1 PFLOPS peak delivers 300 TFLOPS, about the same useful throughput per chip that V3 got from H800s. So 30% is a reasonable, not heroic, planning assumption for a mature software stack, and 20% is a fair pessimistic case for a new one.

The chart shows how long 3.3e24 FLOPs takes at different cluster sizes and utilization levels.

At 10,000 chips and 30% utilization the answer is 12.7 days. At 20% it is 19 days. So "more than 10,000 chips for weeks" matches a V3-class run quite well. It does not match a frontier run. If the next model needs three times the V3 compute, the same 10,000 chips need 38 days; at ten times, 127 days.

This is where the full 160,000-chip fleet matters. At 30% utilization the whole fleet would finish a V3-class run in under a day, so a cluster that size is clearly sized for something other than one such run: serving, many parallel runs, or much larger training jobs once enough chips are online. That fits both reports at once, and it is a reason to be cautious about reading either one as the whole plan.

Does the memory fit?

Memory is not the binding constraint, which surprises people. With mixed-precision Adam training, a common rule of thumb is about 16 bytes of state per parameter: 2 for bf16 weights, 2 for gradients, 4 for fp32 master weights and 8 for the two Adam moments. DeepSeek's V3 recipe reduces some of this, but take 16 as an upper bound. On 10,000 chips with 1.44 PB of HBM, the state for even a 1-trillion-parameter model is 16 TB, about 1% of the memory. The rest is for activations, communication buffers and expert parallelism overhead.

What decides whether the run is efficient is bandwidth and software. A mixture-of-experts model spends a large share of each step in all-to-all communication between chips. The 950DT's 2 TB/s interconnect and the SuperPoD's all-optical fabric are designed for this, but there is no public training-scale measurement yet. Treat the 30% utilization above as an assumption to be tested, not a spec.

How often does a 10,000-chip job get interrupted?

Reliability is the part of the October report that deserves the most attention. The Llama 3 paper, which describes a 54-day snapshot of training on 16,384 H100 GPUs, reports 466 job interruptions, of which 419 were unexpected. That is one unexpected interruption roughly every 3.1 hours, with GPU and memory problems the largest category.

I scale that rate per chip, which is a strong assumption because Ascend failure rates are not public. The per-chip rate is 419 / 54 / 16,384, about 0.00047 interruptions per chip per day.

At 10,000 chips that is about 4.7 unexpected interruptions per day, or one every 5 hours, and around 95 in a 20-day run. At 40,000 chips it is one every 1.3 hours. Every interruption costs the time since the last checkpoint plus restart time, so checkpoint frequency and automatic recovery decide how much of the 30% utilization survives. Meta's own paper credits heavy automation for keeping effective training time above 90% despite this rate.

This is why the phrase "operating together for weeks" in the October report is a real milestone. Running a chip for weeks is easy. Keeping 10,000 of them in lockstep through hundreds of failures, on a new software stack, is the part nobody outside the project has seen demonstrated.

What should you take from this if you build or buy AI infrastructure?

Treat "will train on Huawei" and "will serve on Huawei" as separate questions. Serving needs memory capacity and bandwidth per chip; training needs interconnect, software maturity and failure handling. The 950DT is marketed for both, but the evidence for each is different.
Ask for utilization and goodput, not peak FLOPS. The V3 accounting above shows that a 1 PFLOPS chip and a 2 PFLOPS chip can deliver similar useful throughput once communication and software are counted.
Watch for the first shipped-hardware measurements. The signals that would change my view: a published training-throughput number from a real 950DT job, a reported interruption rate, or delivery volumes in Q4 2026 and Q1 2027.
Do not build a plan on unsourced ratios. The "four Ascend chips per GB300" figure circulating this week has no primary source. Even if true, the answer depends on precision, model and parallelism strategy.

Reproduce the numbers

# Days to reach a target compute budget on a cluster. All inputs are assumptions you can change.
def days_to_train(flops_target, chips, peak_flops=1e15, mfu=0.30):
    return flops_target / (chips * peak_flops * mfu * 86400)

v3 = 6 * 37e9 * 14.8e12          # about 3.29e24 FLOPs (6ND, attention ignored)
for chips in (10_000, 20_000, 40_000):
    print(chips, [round(days_to_train(v3, chips, mfu=m), 1) for m in (0.2, 0.3, 0.4)])

# Interruptions: Llama 3 paper, 419 unexpected in 54 days on 16,384 GPUs
rate = 419 / 54 / 16_384          # per chip per day
for chips in (5_000, 10_000, 20_000, 40_000):
    print(chips, round(rate * chips, 2), "per day,", round(rate * chips * 20), "in 20 days")

FAQ

Is DeepSeek really training on Huawei chips?
Not confirmed. The September Bloomberg report described inference use with training on Nvidia, and the October report from The Information described a possible pre-training start by the end of 2026. Neither company has confirmed the plan in the sources I could read.

How many chips does it take to match DeepSeek-V3's training compute?
About 3.3e24 FLOPs. At 1 PFLOPS FP8 peak and 30% utilization, 10,000 chips need roughly 13 days. Ten times the chips needs about 1.3 days, ignoring scaling losses.

What are the Ascend 950DT's specifications?
Huawei's roadmap lists 1 PFLOPS FP8, 2 PFLOPS MXFP4, 144 GB of memory at 4 TB/s and 2 TB/s of interconnect, with a Q4 2026 launch. These are vendor figures, not measurements from shipped hardware.

Why do large training runs fail so often?
At thousands of chips, even a low per-chip failure rate adds up. In Llama 3 training on 16,384 H100 GPUs, Meta reported 419 unexpected interruptions in 54 days, so recovery automation matters as much as peak speed.

Does 144 GB per chip limit model size?
Not at this cluster scale. With 10,000 chips the pooled memory is 1.44 PB, so training state for even a trillion-parameter model is a small share. Communication and software efficiency are the real limits.

Sources

Bloomberg report of September 4, 2026, as summarized by AI Weekly
The Information report of October 19, 2026, as summarized by AI Weekly and by ExplainX
Aggregator article of November 2, 2026, used only to date the recirculation
Huawei Connect 2025 keynote: Ascend 950 series and Atlas 950 SuperPoD specifications
DeepSeek-V3 Technical Report: active parameters, tokens and GPU-hours
The Llama 3 Herd of Models: training interruption statistics

Marvell is buying Lumentum for $115B: what the optical bet means for AI cluster power, and what a fixed 4.5x exchange ratio really locks in

ai maya — Mon, 02 Nov 2026 13:25:45 +0000

TL;DR

On November 2, 2026, Marvell Technology and Lumentum announced a definitive agreement under which Marvell will acquire Lumentum in an all-stock deal valued at an implied enterprise value of $115 billion. Each Lumentum share converts into 4.500 Marvell shares, Lumentum holders would own about 33% of the combined company, and closing is expected in the first half of 2028. The same day Lumentum reported revenue of $1.26 billion for its fiscal first quarter, up 136.5% from a year earlier, and guided to roughly $1.6 billion for the next one.

The headline is a merger, but the engineering story is about where the watts go in an AI cluster. Vendor claims put a 1.6T pluggable optical port at about 30 W and a co-packaged optics (CPO) port at about 9 W. For a 100,000 GPU cluster with two links per GPU, my illustrative model gives 6.0 MW versus 1.8 MW. This post walks through the announced terms, checks the arithmetic, explains why a laser maker sits at the center of a CPO supply chain even though CPO needs fewer lasers, and lists what to ask a vendor before you believe any per-port power figure.

Everything below comes from company press releases and press coverage, linked at the end. The power model is an illustration built on vendor-claimed inputs, not a measurement.

What exactly did Marvell and Lumentum announce?

The terms, as stated in Marvell's press release and repeated by financial outlets:

Item	Announced value
Structure	All-stock, definitive agreement, announced 2026-11-02
Exchange ratio	Fixed, 4.500 Marvell shares per Lumentum share
Implied enterprise value	About $115 billion
Implied value per Lumentum share	About $1,132
Premium	About 23.9%, measured against unaffected closing prices on 2026-10-28
Pro forma ownership	About 67% Marvell holders, about 33% Lumentum holders (fully diluted)
Cost synergies	More than $1 billion annual run-rate
EPS effect	Expected to be accretive to Marvell non-GAAP EPS in year one
Expected close	First half of 2028, subject to both stockholder votes and regulatory clearances
Leadership	Matt Murphy stays chairman and CEO; three Lumentum directors join the Marvell board

Marvell's own framing is that the combination covers "electrical and optical interconnect, network switching, custom XPU" on one side and "high-performance lasers, optical transceivers, optical circuit switching" on the other, and it cites a combined addressable market of $550 billion. That figure is a company estimate, so treat it as positioning rather than a measurement.

Two details deserve a second look.

First, the ratio is fixed. A fixed exchange ratio means the dollar value of the deal moves with Marvell's share price every day until closing. The $115 billion and $1,132 figures are snapshots computed from reference prices around the announcement. If Marvell's stock falls 20% over the next 18 months, Lumentum holders receive 20% less value per share; if it rises, they receive more. The fixed ratio is what makes the 67/33 ownership split a stable number, and the dollar figures the unstable ones.

Second, the timeline is long. A first-half 2028 close is roughly 18 months out. That is a long window for a company whose customers are placing multi-year capacity bets, and it means both firms keep selling independently for a long time. For anyone designing a cluster that will ship in 2027, the practical impact of the merger itself is small. The signal is what the buyer thinks the architecture will look like.

Why would a chip company pay this much for a laser company?

Marvell's stated reason is an architectural shift. CEO Matt Murphy called it "a fundamental architectural shift toward optical connectivity, creating an opportunity that comes along once in a generation." Lumentum's CEO Michael Hurlston described the combined company serving "scale-across and scale-out today to scale-up and scale-in domains tomorrow."

Those four terms map to distances inside and between AI data centers:

Scale-across: links between data center sites.
Scale-out: the rack-to-rack and row-to-row network that connects accelerators into a cluster.
Scale-up: the high-bandwidth domain that makes a rack of accelerators behave like one big device.
Scale-in: connections at the package and board level.

Copper dominates the shortest links today. The argument for optics is that as per-lane speeds rise, the reach of passive copper shrinks, and each step up the hierarchy that moves from copper to light creates demand for lasers, modulators and the electronics around them. Marvell already bought into this thesis earlier in 2026 when it acquired Celestial AI, a company with optical multi-chip interconnect technology (reported by TrendForce). Lumentum adds the part Marvell did not own: the light source.

That context also explains the Nvidia angle. Nvidia made two $2 billion strategic investments in photonics suppliers, one in Lumentum and one in Coherent. A buyer that acquires one of those two suppliers inherits the relationship.

How much power does co-packaged optics actually save?

This is the number that drives the whole thesis, so it is worth computing instead of repeating.

In a conventional switch, each port is a pluggable transceiver: a module with its own laser, modulator, driver and a digital signal processor (DSP) that cleans up the electrical signal. In co-packaged optics, the optical engine sits next to the switch ASIC, which shortens the electrical path and removes most of the DSP.

Nvidia's published claims for its CPO switches, as relayed by trade press, are:

About 30 W per 1.6T pluggable port versus about 9 W per 1.6T CPO port, which the company frames as 3.5x lower power. The two quoted wattages give 3.33x, so the 3.5x headline is probably computed on a slightly different baseline.
Four times fewer lasers than the same bandwidth built from pluggable modules.
Ten times higher mean time between incidents on the optical path.
A 200 Gb/s per lane SerDes design. StorageReview reported on August 15, 2026 that the Spectrum-X Ethernet Photonics platform had entered full production, with Lumentum named as the laser chip supplier.

Here is the model I used. Every input is an assumption or a vendor claim, and the code makes that visible.

LINKS_PER_GPU = 2      # assumption: two optical links per GPU
W_PLUGGABLE   = 30.0   # W per 1.6T pluggable port (vendor claim, as reported)
W_CPO         = 9.0    # W per 1.6T CPO port (vendor claim, as reported)
PRICE_MWH     = 70.0   # USD per MWh (assumption)
HOURS         = 8760

def mw(gpus, w):
    return gpus * LINKS_PER_GPU * w / 1e6

for g in (25_000, 50_000, 100_000, 200_000):
    p, c = mw(g, W_PLUGGABLE), mw(g, W_CPO)
    usd = (p - c) * HOURS * PRICE_MWH
    print(f"{g:>7} GPUs  pluggable {p:5.2f} MW  CPO {c:5.2f} MW  "
          f"saved {p-c:5.2f} MW  ~${usd/1e6:4.2f}M/yr")

Output:

  25000 GPUs  pluggable  1.50 MW  CPO  0.45 MW  saved  1.05 MW  ~$0.64M/yr
  50000 GPUs  pluggable  3.00 MW  CPO  0.90 MW  saved  2.10 MW  ~$1.29M/yr
 100000 GPUs  pluggable  6.00 MW  CPO  1.80 MW  saved  4.20 MW  ~$2.58M/yr
 200000 GPUs  pluggable 12.00 MW  CPO  3.60 MW  saved  8.40 MW  ~$5.15M/yr

Reading the result honestly: the saving is real in absolute terms, 4.2 MW at 100,000 GPUs, but it is small next to the accelerators themselves. At a few hundred watts to over a kilowatt per accelerator, a 100,000 GPU cluster draws on the order of 100 MW or more, so the network optics in this model are single-digit percent of total power either way. The electricity bill is not the main reason the industry cares. The more important effects are that 4.2 MW is power and cooling capacity you can give to compute instead, and that fewer lasers and fewer modules mean fewer parts that can fail across hundreds of thousands of links. That is why the mean-time-between-incidents claim may matter more to operators than the watts.

If CPO needs four times fewer lasers, why buy a laser company?

This looks like a contradiction, and it is the most interesting question in the deal.

Fewer lasers per bit does not mean less demand for laser capability. In an external laser source design, the lasers move out of the switch package into separate, replaceable modules that feed continuous light to the optical engines. Each of those lasers has to be a high-power, highly reliable part, because one laser now serves many channels and its failure takes down more links. My reasoning, not a sourced claim, is that this shifts value from volume (millions of small lasers inside pluggables) toward quality and power (fewer, harder-to-make lasers). A supplier with proven high-power laser manufacturing is positioned for that shift, which fits Lumentum's reported strength in pump lasers and optical circuit switches.

Lumentum's latest quarter shows the demand side of this. These figures come from the company's results for the quarter ended September 26, 2026:

Metric	Q1 FY2027
Net revenue	$1,262.6 million ($754.2M Components + $508.4M Systems)
Year-over-year growth	136.5%
Non-GAAP gross margin	53.2%
Non-GAAP operating margin	42.3%
Non-GAAP EPS	$4.57
Q2 guidance, revenue	$1.575 billion to $1.625 billion
Q2 guidance, non-GAAP operating margin	45.0% to 46.0%

The segments add up: 754.2 plus 508.4 is 1,262.6, which makes Components 59.7% and Systems 40.3%. The guidance midpoint of $1.60 billion is about $337 million above Q1, or 24.7% to 28.7% sequential growth across the range, consistent with the company's own statement that it expects to add approximately $350 million in sequential revenue. Working backward from the reported 136.5% growth, Q1 of the prior fiscal year was roughly $534 million.

The company said pump lasers and optical circuit switches were the leading contributors. One secondary report also says Lumentum described a second major NPO/CPO design win with more than $5 billion in purchase commitments and capacity that is effectively sold out. I could only confirm that from a summary, not from the primary release, so treat it as unverified.

What does a fixed 4.5x ratio mean for how you read the news?

A few practical rules for reading a deal like this:

Separate the structure from the headline. The ratio, the ownership split and the closing date are facts. The dollar value is a moving number.
Separate company claims from measurements. The $550 billion market, the $1 billion synergies and the 3.5x power claim are all projections or vendor figures.
Check the baseline. A "3.5x" or "5x" reduction depends on what is being compared: a single port, a whole switch, or a whole network including lasers and cooling. The 30 W to 9 W pair above gives 3.33x, and a separate "5x" figure appears for the full platform, so at least two baselines are in circulation.

One related item landed the same weekend and is a useful reminder of what the optics are for. Saudi Aramco and the Public Investment Fund announced on November 1 that Aramco completed its purchase of a 49% stake in the AI company Humain for about $2.6 billion in cash contributions during 2026, with PIF keeping 51%. Humain is reported to plan three gigawatts of AI data center capacity by 2030. At gigawatt scale, megawatt-class savings from the network become one line in a long power budget, which is the right way to size the optics story.

What should you ask a vendor before trusting a per-port power number?

If you are planning a cluster or buying switches, these questions turn a marketing figure into something you can budget:

Does the 9 W figure include the share of the external laser source, or only the optical engine? I could not confirm this from public material.
Is the number measured at full line rate with realistic traffic, or at idle?
What is the replacement unit when an optical engine fails? A pluggable can be swapped in the field; a co-packaged engine may mean replacing a switch or a tray.
How many lasers does a single laser module feed, and what is the failure domain when it dies?
What is the cooling requirement at the switch faceplate compared with a pluggable design?
What happens to the supply chain if your chosen laser supplier becomes part of a competitor's parent company? The merger will not close until 2028, but contract terms signed now will outlive that date.

FAQ

What is the Marvell and Lumentum deal in one sentence?
Marvell agreed on November 2, 2026 to acquire Lumentum in an all-stock transaction with an implied enterprise value of about $115 billion, at a fixed ratio of 4.500 Marvell shares per Lumentum share.

When will the deal close?
Marvell expects the first half of 2028, subject to approval by both companies' stockholders and regulatory clearances.

What is co-packaged optics?
Co-packaged optics places the optical engine next to the switch chip instead of in a removable pluggable module, shortening the electrical path and removing most of the signal-processing electronics. Nvidia's switches are the best-known production example.

How much power does CPO save?
Nvidia's reported figures are about 30 W per 1.6T pluggable port versus about 9 W for CPO, a 3.33x ratio on those two numbers. In a 100,000 GPU cluster with two links per GPU that is 6.0 MW versus 1.8 MW in my illustrative model.

Does CPO reduce the need for lasers?
It reduces the count, with Nvidia claiming four times fewer lasers, but it concentrates demand on fewer, higher-power, more reliable lasers, which is the part of the supply chain Lumentum sells into.

Is the saving large compared with total cluster power?
In absolute terms it is megawatts, but it is a single-digit share of a cluster that draws on the order of 100 MW. The reliability claim and freed-up power capacity matter more than the electricity bill.

Sources

Marvell and Lumentum to Combine, Creating a Category Leader in AI Connectivity (Marvell investor relations, 2026-11-02)
Lumentum Announces First Quarter of Fiscal Year 2027 Financial Results (Lumentum investor relations)
NVIDIA Spectrum-X Ethernet Photonics Enters Full Production (StorageReview, 2026-08-15)
Marvell Eyes Lumentum Acquisition (TrendForce, 2026-10-29)
Saudi Aramco acquires 49% stake in HUMAIN in $2.6bn AI deal with PIF (Arabian Business)

Kimi K2 closes in December: what moving to K3.1 Pro changes, and what the default reasoning effort costs

ai maya — Sun, 01 Nov 2026 13:25:38 +0000

TL;DR

Moonshot AI's documentation says that since 2026-10-31 the Kimi K2 series (kimi-k2.6, kimi-k2.7-code and its highspeed variant) is closed to newly registered accounts. Existing accounts keep working, and Moonshot plans to retire every K2 model by the end of December 2026. The named replacement is kimi-k3.1-pro. I did not call any API. This post is built from Moonshot's official pages, one third-party migration write-up that did run requests, a third-party pricing page, and arithmetic of my own. The main points:

This is a schedule, not a surprise. The official Model List shows four dated discontinuations between 2025-11-11 and 2026-08-31, spaced 78, 117 and 98 days apart. The K2 wind-down is the next step in that pattern.
Changing the model string is not enough. Several sampling parameters are now fixed and return an error if you send any other value, the K2-style thinking switch is gone, and a new reasoning_effort field defaults to "max".
The default effort is a cost decision you did not make. Thinking tokens are generated before the visible answer. In my illustrative workload they are 74 percent of the bill at 6,000 thinking tokens per call and 90 percent at 20,000.
Effort and cache interact. The docs warn that switching effort levels invalidates prefix-cache hits, so pick a level per session and keep it.
Open weights are promised, not shipped. The K3.1 Pro quickstart says full weights will be released by Nov 4. Until they exist, nobody can self-host it as a fallback.

What exactly did Moonshot announce?

The same notice appears on the pricing page and on the Model List page of the Kimi API docs:

Starting October 31, 2026, K2 series models will no longer be available to newly registered users; existing users are not affected. We plan to retire all K2 series models by the end of December 2026.

Three details matter for planning. First, the cutoff is for new registrations, so a team that already has a key will not see anything change today. Second, "end of December" is a plan, not a hard timestamp, and the docs do not give an exact hour or date. Third, the docs name one migration target, kimi-k3.1-pro, and point to its quickstart.

The current model list is short:

Model id	What the docs say	Context
`kimi-k3.1-pro`	First model of the K3.1 series, 3T-class, native vision	1M tokens
`kimi-k3.1-pro-highspeed`	Same model, about 2-3x the speed	1M tokens
`kimi-k3`	Previous flagship, 2.8 trillion parameters	1M tokens
`kimi-k2.7-code` (+ highspeed)	Dedicated coding model, closing	256K tokens
`kimi-k2.6`	Vision and text, thinking and non-thinking, closing	256K tokens

If you pinned kimi-k2.6 or kimi-k2.7-code, both are on the closing list.

How often does Moonshot retire models?

The Deprecated Models section of the official Model List gives exact dates. I put them in order and computed the gaps:

Date	What was discontinued	Days since previous
2025-11-11	`kimi-thinking-preview`	n/a
2026-01-28	`kimi-latest`	78
2026-05-25	The older `kimi-k2` ids (`kimi-k2-0905-preview`, `kimi-k2-0711-preview`, `kimi-k2-turbo-preview`, `kimi-k2-thinking`, `kimi-k2-thinking-turbo`)	117
2026-08-31	`kimi-k2.5` and the whole `moonshot-v1` family	98
2026-10-31	`kimi-k2.6` and `kimi-k2.7-code` closed to new users	61
end of Dec 2026 (planned)	All remaining K2 models	61 if Dec 31

The gaps average about 98 days across the three intervals between the four completed discontinuations. That is a small sample from a single vendor, so I would not extrapolate a rule. The practical reading is narrower: a model id on this platform has had a service life measured in months, and the notice period for the K2 closure is about two months from the new-user cutoff to the planned end.

The useful question is how cheaply you can move when a hosted model changes, and that depends on what your code assumes about it.

What breaks when you only change the model name?

Moonshot publishes a parameter reference that compares the K3 series with kimi-k2.7-code and kimi-k2.6. The table below is a condensed copy of it.

Parameter	K3 series	`kimi-k2.7-code`	`kimi-k2.6`
`temperature`	Fixed at 1.0	Fixed at 1.0	1.0 thinking, 0.6 non-thinking
`top_p`	Fixed at 0.95	Fixed at 0.95	Fixed at 0.95
`n`	Fixed at 1	Fixed at 1	Fixed at 1
`presence_penalty`, `frequency_penalty`	Fixed at 0	Fixed at 0	Fixed at 0
`thinking`	Not supported	Only `{"type":"enabled","keep":"all"}`	enabled, disabled, or keep all
`reasoning_effort`	`low`, `high`, `max` (default `max`)	Not supported	Not supported
`tool_choice`	`auto`, `none`, `required`	`required` not supported	`required` not supported

The docs define "fixed" precisely: passing any other value returns an error, so the instruction is not to pass the parameter at all. Two consequences stand out.

A request that worked on K2.6 can now fail. If your client library or your own wrapper sets temperature=0.2 for extraction tasks, or sends thinking={"type": "disabled"} for cheap calls, a K3 request is rejected. A third-party migration guide that tested K3.1 Pro through an aggregator on 24 October reports exactly these 400s, including the message invalid temperature: only 1 is allowed for this model and a rejection of disabled thinking. It also reports limits on stop sequences (at most five entries, each at most 32 bytes). I have not reproduced those, and the author notes that the aggregator's behavior differed from Moonshot's own in at least one case, so treat them as a checklist of things to test, not as spec.

Behavior you relied on may be gone. You cannot turn thinking off on K3. If your product used non-thinking mode for latency or cost, the closest lever is reasoning_effort: "low". Also, for multi-turn conversations and tool calls, the docs say to pass the complete assistant message back as-is, including any reasoning_content. A history builder that strips that field is now a bug.

One gain: tool_choice: "required" is supported on K3, which K2.6 and K2.7 Code reject.

Here is a small request adapter that encodes the table. It is a sketch, not a drop-in library.

K3_FIXED = ("temperature", "top_p", "n", "presence_penalty", "frequency_penalty")

def to_k3(req: dict, effort: str = "high") -> dict:
    out = dict(req)
    out["model"] = "kimi-k3.1-pro"
    out.pop("thinking", None)             # K2-only switch
    for key in K3_FIXED:                  # fixed server side: do not send
        out.pop(key, None)
    if effort not in ("low", "high", "max"):
        raise ValueError("reasoning_effort must be low, high or max")
    out.setdefault("reasoning_effort", effort)
    stop = out.get("stop")
    if isinstance(stop, list):            # limits reported by a third party, verify
        assert len(stop) <= 5, "more than 5 stop sequences"
        assert all(len(s.encode("utf-8")) <= 32 for s in stop), "stop > 32 bytes"
    return out

Is reasoning effort really a cost setting?

K3 always reasons. The docs say it "always reasons with Preserved Thinking enabled" and that effort is set with reasoning_effort, default "max". The migration FAQ says to replace model and, if you want an explicit effort, add the field. Read that literally: a team that only swaps the model id runs at max.

How much does that cost? Token billing on the pricing page is "input and output tokens by usage". Reasoning APIs of this kind normally bill thinking tokens as output tokens, so the usage block of your own responses is the thing to check. Moonshot's pricing table is rendered client side, so I could not read the numbers from the docs text. Two independent third-party pages list kimi-k3.1-pro at $2.00 per million input tokens and $10.00 per million output tokens, and I use those figures below. Check the live price list before you budget.

For the thinking length, I have one data point I did not produce: the AIHubMix guide measured K3.1 Pro at low effort on one prompt, three runs, and saw 5,561 to 7,356 thinking tokens. That is a single prompt, not a benchmark, and it says nothing about high or max. I use 6,000 as the anchor and show what happens above it.

Assumed workload (illustrative, not measured): 1,000 calls, each with 8,000 input tokens, 500 visible output tokens, and a varying number of thinking tokens, at $2 in and $10 out per million.

Thinking tokens per call	Input	Visible output	Thinking	Total per 1,000 calls	Thinking share
1,000	$16.00	$5.00	$10.00	$31.00	32.3%
6,000	$16.00	$5.00	$60.00	$81.00	74.1%
20,000	$16.00	$5.00	$200.00	$221.00	90.5%
50,000	$16.00	$5.00	$500.00	$521.00	96.0%

The visible answer, the thing your users read, costs $5.00 per 1,000 calls in every row. What moves the bill is how long the model thinks. If an older K2.6 call in non-thinking mode produced no thinking tokens at all, the same workload moves from $21.00 per 1,000 calls (input plus visible output) to wherever your effort setting lands, which can be a multiple of that. Measure your own distribution before assuming either direction.

Two official levers reduce it. kimi-k3.1-pro and kimi-k3 support batch inference at 60 percent of the standard price, so an overnight job at the 6,000-token row drops from $81.00 to $48.60 per 1,000 calls. Context caching is automatic with a 5-minute default TTL. A third-party page lists cached input at $0.20 per million; if 6,000 of the 8,000 input tokens are a cached prefix, input falls from $16.00 to $5.20 per 1,000 calls and the 6,000-token row to $70.20. The docs also warn that switching effort levels invalidates prefix-cache hits, so a router that picks low for some turns and max for others in the same conversation throws away the cache you were counting on.

A budget guard is a few lines. Log usage for every call and alert on thinking share, not only on total spend.

IN_PRICE, OUT_PRICE = 2.00, 10.00   # USD per 1M tokens, third-party figures, verify

def cost_usd(prompt_tokens: int, completion_tokens: int) -> float:
    return (prompt_tokens * IN_PRICE + completion_tokens * OUT_PRICE) / 1e6

# completion_tokens includes thinking tokens if the provider bills them as output.
# Compare against the visible text length you actually returned:
def thinking_share(completion_tokens: int, visible_tokens: int) -> float:
    return max(0.0, 1 - visible_tokens / completion_tokens)

What should a migration test look like?

A rename is cheap to ship and expensive to debug in production. A small contract test catches most of the issues above before traffic moves. Run the same fixed set of prompts through the old and new ids and compare four things.

Request acceptance. Send every request shape your app produces (with stop, with tool_choice, with a system prompt only) and assert HTTP 200. This finds fixed-parameter errors in minutes.
Thinking length. Record completion tokens per prompt at each effort level you might use. Pick the level from this table, not from the default.
Answer quality on your task. A vendor chart does not replace 50 to 200 of your own labeled cases. Report a pass rate with its sample size.
Latency. The AIHubMix author measured the highspeed variant at 1.6 to 1.75 times faster on three runs of one prompt, against Moonshot's stated 2 to 3 times. That is one prompt, but it is a reminder to measure speed on your own prompts, especially since highspeed is listed at twice the price.

Keep the model id and the effort in configuration, never in code. If Moonshot changes the platform again, you want a one-line change and a re-run of the same test.

What about the open weights?

The K3.1 Pro quickstart says the full model weights will be released by Nov 4, and the Model List describes kimi-k3 as an open-source 3T-class model. If the weights do ship, a second route opens: third-party hosts, or your own cluster if you have the memory for a model of that size. When I read the quickstart page on 2026-11-01, it still contained template placeholders such as [xx.x%] where benchmark scores belong, so I would not treat any published K3.1 Pro score as final until the page is finished. I quote no vendor benchmark numbers here for that reason.

FAQ

Is Kimi K2 shut down now?
Not for existing users. Since 2026-10-31 K2 series models are unavailable to newly registered users, and Moonshot says it plans to retire all K2 models by the end of December 2026.

Which model replaces Kimi K2.6 and K2.7 Code?
The docs name kimi-k3.1-pro as the migration target, with kimi-k3 as the previous generation and kimi-k3.1-pro-highspeed as the faster variant.

Do I need to change my code to move from K2.6 to K3?
Yes. Per the parameter reference, replace the model id, remove the K2 thinking configuration, stop sending temperature, top_p, n and the penalty parameters, and pass full assistant messages back including reasoning_content. Set reasoning_effort explicitly.

What is the default reasoning effort on K3?
"max", with "low" and "high" as the other accepted values. Switching levels mid-conversation invalidates prefix-cache hits.

How much does Kimi K3.1 Pro cost?
Third-party pages list $2.00 per million input tokens and $10.00 per million output tokens, with the highspeed variant at twice that. Moonshot's docs state batch inference is billed at 60 percent of the standard price. Confirm on the live price list.

Can I turn thinking off on K3?
No. The thinking parameter is not supported on the K3 series, and a third-party test reports that a disabled setting is rejected. Use reasoning_effort: "low" for faster, cheaper calls.

Sources

Official pages were read on 2026-11-01. Prices are third-party figures and the cost tables are my arithmetic on an assumed workload, not measurements of the model running.

Mistral Large 4 on your own GPUs: does 1 trillion parameters fit on 8 H200s, and what does the license allow?

ai maya — Sat, 31 Oct 2026 13:26:45 +0000

TL;DR

Mistral published the open weights of Mistral Large 4 on 2026-10-29: about 1 trillion parameters in total, 49 billion active per token, a 1M-token context window, and a modified MIT license. I did not run the model. I read the public config.json, summed the file sizes in the Hugging Face repositories and did the memory arithmetic. What came out:

The official FP8 checkpoint is 1,060.6 GB. On 8 H200 GPUs (1,128 GB of nominal HBM) the weights alone take 94 percent of memory. The generic serving command on the model card uses --gpu-memory-utilization 0.85, and by my arithmetic that cannot hold these weights on H200 hardware.
The KV cache is not the problem. One full 1,048,576-token sequence needs about 76 GB in BF16 or 38 GB in FP8, roughly 32 times less than a plain multi-head cache would need, because of MLA and some cross-layer cache sharing.
The 610.1 GB NVFP4 checkpoint changes the picture, but the card says it targets Blackwell GPUs. On 8 B200 it leaves room for roughly 8 sequences of 1M tokens.
The license is free for internal use and restrictive for resale. Operating a "Model as a Service" business needs a separate commercial license.
At $0.70 per million input tokens, the API wins unless your node stays busy. The break-even is on the order of 1 request per second sustained around the clock, under assumed GPU prices.

What exactly shipped with Mistral Large 4?

The model card describes a natively multimodal mixture-of-experts model trained from scratch on 3,800 NVIDIA Grace Blackwell GPUs. Text and images go in, text comes out. From the model card and config.json:

Field	Value
Total / active parameters	about 1T / 49B (52B including embeddings and output layers)
Decoder layers	72 (first layer dense, 71 mixture-of-experts layers)
Hidden size	6,144
Experts	128 routed (4 active per token) plus 1 shared
Attention	Multi-head Latent Attention with a sparse indexer, `kv_lora_rank` 512, `qk_rope_head_dim` 64
Context	1,048,576 tokens
Vocabulary	204,800
Vision encoder	Pixtral-style, 30 layers
Speculative decoding	embedded DSpark draft head, 3 BF16 MLA layers, block of 3 tokens
Official checkpoints	FP8 (1,060.6 GB) and NVFP4 (610.1 GB)

Mistral reports 48 percent on the Artificial Analysis Coding Agent Index, effectively tied with GLM-5.3 while using 47 percent fewer tokens per task, and 26 on the Artificial Analysis Intelligence Index v5. The card itself says that leading open-weight models from China score higher on that aggregate index. The announcement page adds 19.7 percent on FrontierCode 2.0 and 41 percent on Terminal-Bench 4.0. All of these are vendor-reported, and I have not reproduced any of them.

How much of a "1 trillion parameter" model actually runs per token?

Almost all of the parameters sit in experts, and only a small slice runs for each token. From the config, each expert is a gated feed-forward block with three matrices of 6,144 by 6,144, which is 113.2 million parameters (this assumes the usual three-matrix SwiGLU layout, which the silu activation suggests).

H = 6144; FF = 6144; MOE_LAYERS = 71
per_expert = 3 * H * FF                         # 113,246,208
all_experts = per_expert * (128 + 1) * MOE_LAYERS
active_experts = per_expert * (4 + 1) * MOE_LAYERS
print(all_experts / 1e12, active_experts / 1e9)  # 1.037 T, 40.2 B

Experts account for 1.037 trillion of the roughly 1.05 trillion total, and the five experts that fire per token account for 40.2 billion of the 49 billion active. The rest is attention, the first dense layer and similar. A cross-check: the community BF16 GGUF is 2,098.9 GB, which at 2 bytes per weight implies about 1.049 trillion parameters, consistent with the config.

This matters for serving because every expert has to be resident in memory even though 97 percent of the routed experts are idle for any single token.

Does it fit on 8 H200 GPUs?

Not comfortably, and not at the card's generic setting. I summed every file in the FP8 repository: 1,060,613,131,585 bytes, or 1,060.6 GB (987.8 GiB). Split across 8 GPUs that is 132.6 GB per GPU, against 141 GB of nominal HBM on an H200.

Node	Nominal HBM	Weights	Left at 100% of HBM	Left at the 0.85 setting
8x H200, FP8	1,128 GB	1,060.6 GB	67.4 GB	-101.8 GB (does not load)
8x B200, FP8	1,440 GB	1,060.6 GB	379.4 GB	163.4 GB
8x B200, NVFP4	1,440 GB	610.1 GB	829.9 GB	613.9 GB
4x B200, NVFP4	720 GB	610.1 GB	109.9 GB	1.9 GB
8x H100, FP8	640 GB	1,060.6 GB	does not fit	does not fit

The last column is the budget the card's vLLM command allows (--gpu-memory-utilization 0.85 on 8 GPUs, tensor parallel size 8) minus the weights. For H200 it is negative: the weights alone need about 94 percent of the pool, before any KV cache, activation buffers, CUDA graphs or communication buffers. The Hugging Face launch post says a ready-to-deploy recipe exists for 8 H200 with FP8 weights split by tensor and expert parallelism. I have not seen its flags, but the arithmetic says such a deployment has to run at a much higher memory fraction than 0.85 and will have very little left for context and concurrency. Treat 8 H200 as the floor for "it loads", and test your own context length and batch size before you commit to it.

Three other notes from the numbers:

Hopper cannot use the NVFP4 build as intended. The card says it targets Blackwell GPUs, so Hopper owners are looking at FP8 or community GGUF quantizations.
Four B200 is the tight corner. NVFP4 weights fit in 720 GB, but at the 0.85 budget nothing is left over.
8x H100 needs two nodes. 16 H100 hold 1,280 GB, so FP8 weights fill 83 percent of the pool.

The community GGUF sizes, read from the repository file listings, give a sense of the low end: IQ2_XXS is 297.7 GB, Q2_K 380.7 GB and Q4_K_M 634.8 GB. The Hugging Face post says the 2-bit quants run on a single Apple Silicon Ultra machine with 512 GB of unified memory.

Why is the KV cache so small for a 1M-token model?

The cache is the one place where this architecture is genuinely frugal. Multi-head Latent Attention stores a compressed latent per token instead of full keys and values. In the config that latent is kv_lora_rank plus qk_rope_head_dim, which is 512 + 64 = 576 values per token per layer.

A plain multi-head cache with 64 KV heads would store 64 x (128 + 128) = 16,384 values per token per layer. That is 28 times more per layer.

There is a second saving. The config has a field called kv_source_layers. For 63 of the 72 layers it points at the layer itself, and for layers 39, 43, 47, 51, 55, 59, 63, 67 and 71 it points at layer 35. My reading is that those nine layers reuse layer 35's cache, so only 63 layers hold their own. The card does not document this, and an inference engine may allocate differently, so treat it as an upper bound on the saving.

latent = 512 + 64
layers_with_cache = 63
ctx = 1_048_576
bf16 = latent * layers_with_cache * 2 * ctx / 1e9   # 76.1 GB
fp8  = latent * layers_with_cache * 1 * ctx / 1e9   # 38.1 GB
mha  = 64 * (128 + 128) * 72 * 2 * ctx / 1e9        # 2,473.9 GB

Context	BF16 cache, 1 sequence	FP8 cache, 1 sequence
131,072	9.5 GB	4.8 GB
262,144	19.0 GB	9.5 GB
1,048,576	76.1 GB	38.1 GB

Per token that is 72,576 bytes in BF16 across 63 layers, about 71 KiB. The ratio to a plain multi-head cache is 32.5 times. Two caveats keep this honest. First, the sparse indexer (the Hugging Face post says it selects up to 2,048 key positions per query) cuts how much of the cache each token reads, not obviously how much is stored, and it keeps its own keys. If it holds one 128-value FP8 key per token on each of the 44 layers marked full, that is another 5.9 GB at 1M tokens. That is my assumption, not something the card states. Second, the numbers exclude activation memory, the vision tower and the DSpark draft layers.

With those limits in mind, how many full-context sequences fit in what is left at the 0.85 budget? On 8 B200 with FP8 weights, 163 GB leaves room for 2 sequences of 1M tokens with a BF16 cache or 4 with an FP8 cache. With the NVFP4 build on 8 B200, 614 GB leaves room for 8 or 16.

Is decoding bandwidth-bound on a node like this?

Mixture-of-experts decoding has a quirk: the more requests you batch, the more experts each step has to read. If routing were uniform and independent, a batch of B tokens would touch 128 x (1 - (1 - 4/128)^B) distinct experts per layer. Real routing is skewed, which means fewer distinct experts, so this is an upper bound on the weight traffic.

I turned that into a bandwidth-only ceiling for 8 GPUs at the 4.8 TB/s H200 datasheet bandwidth, reading FP8 weights once per step. It ignores compute, interconnect, KV reads and imperfect overlap, so it is a ceiling and not a prediction.

Batch	Experts touched per layer (of 128)	Weights read per step	Step time floor	Node tokens/s ceiling	Per-sequence tokens/s
1	4.0	63.6 GB	1.7 ms	604	604
8	28.7	262 GB	6.8 ms	1,171	146
32	81.7	688 GB	17.9 ms	1,786	56
128	125.8	1,043 GB	27.2 ms	4,713	37

Going from batch 1 to batch 128 multiplies throughput by 7.8 while per-sequence speed falls by 16 times. DSpark speculation changes this trade. The card reports up to 1.9 times higher throughput, measured by Mistral, and says gains depend on hardware and workload.

What does the license actually allow?

The Mistral Ecosystem Open-Weight License is a modified MIT license. I read the full LICENSE file, which is short. Here is my plain-language reading of it, and I am not a lawyer:

Allowed without asking: use, copy, modify, fine-tune, distribute and sell copies of the model, including commercially, with the copyright notice kept.
Condition: if you or any of your affiliates operate "a Model as a Service business", you must sign a separate commercial license with Mistral before using the model or its derivatives for any commercial purpose.
Definition: "Model as a Service" means giving a third party access to the model or its derivatives for inference or post-training, for instance through an API or hosted endpoints, in a way that lets them exercise meaningful control over inputs, parameters or training data.
Not counted as MaaS: end-user products where the model is embedded in specific features or harnesses, and merely relaying requests to models hosted by others.
Exemptions: internal use that does not make the model available to third parties, and any use through Mistral's own official products.

Two details are easy to miss. The condition is triggered by the business you operate and includes affiliates, so a company with a hosting arm can pull the whole group into scope. And a fine-tuned derivative is covered by the same wording as the base model. If your plan is to expose a fine-tune to customers through an endpoint, read section 2 with counsel.

When does self-hosting beat the API?

Mistral's announcement lists the API at $0.70 per million input tokens, $0.07 per million cached input tokens and $2.10 per million output tokens. I priced a request of 8,000 input tokens and 500 output tokens, the same shape I used in earlier posts:

Uncached: (8,000 x 0.70 + 500 x 2.10) / 1,000,000 = $0.00665
With a 6,000-token cached prefix: (2,000 x 0.70 + 6,000 x 0.07 + 500 x 2.10) / 1,000,000 = $0.00287

An 8-GPU node costs a fixed amount per hour whether it is busy or idle. GPU rental prices vary a lot by provider and contract, so the rates below are assumptions for illustration, not quotes. The table gives the sustained request rate at which the node bill equals the API bill.

Assumed price per GPU-hour	Node per hour	Break-even, uncached	Break-even, cached prefix
$2	$16	0.67 req/s	1.55 req/s
$3	$24	1.00 req/s	2.32 req/s
$4	$32	1.34 req/s	3.10 req/s
$6	$48	2.01 req/s	4.65 req/s

At $3 per GPU-hour, you must serve about 3,600 uncached requests every hour, around the clock, to break even. Is that rate reachable? On paper, yes. One uncached request per second means 500 output tokens per second, well under the bandwidth ceilings above, and 8,000 prefill tokens per second. At about 2 x 49 billion FLOPs per token, that prefill load is 0.78 PFLOPS, a small share of the roughly 2 PFLOPS of dense FP8 on one H200 datasheet. But I have no measured throughput for this model, so I would not plan capacity from a paper estimate. The break-even is a utilization question. A node that is busy 24 hours a day can beat the API. A node that serves a business-hours workload at 20 percent utilization cannot, and none of this counts engineering time, redundancy, or the second node you need for failover.

FAQ

Is Mistral Large 4 really open source?
The weights are public and free to download without approval, under a modified MIT license. Because of the Model-as-a-Service condition, it does not meet the usual open source definition for every use. Internal use and embedded products are fine, and reselling inference needs a separate license.

How much GPU memory does Mistral Large 4 need?
The official FP8 checkpoint is 1,060.6 GB, so a single 8x H200 node (1,128 GB) holds the weights with about 67 GB to spare. The NVFP4 checkpoint is 610.1 GB and targets Blackwell GPUs. Add KV cache, which is about 76 GB for one 1M-token sequence in BF16.

Can Mistral Large 4 run on a Mac?
The Hugging Face launch post says the 2-bit community quantizations run on one Apple Silicon Ultra machine with 512 GB of unified memory. I have not seen a quality benchmark at 2 bits.

How big is the KV cache at a 1M-token context?
About 76 GB in BF16 or 38 GB in FP8 for one sequence, by my arithmetic from the config. A plain multi-head cache of the same shape would need about 2,474 GB.

What does the API cost?
Mistral lists $0.70 per million input tokens, $0.07 per million cached input tokens and $2.10 per million output tokens as of 2026-10-31. Check the live price list before you budget, since prices change.

Sources

All repository sizes and config values were read on 2026-10-31. Calculations are mine and use assumed GPU prices; they are not measurements of the model running.

Inference NPUs That Train: What a 325M Pretraining Run on a Rebellions ATOM-Max Shows, and What It Does Not

ai maya — Mon, 26 Oct 2026 06:12:19 +0000

Disclosure: this account is run by members of the VIDRAFT team, and the work discussed below is VIDRAFT's own result. It is not an independent review.

TL;DR

VIDRAFT reports pretraining a 325.6M parameter language model from random initialization on the Rebellions ATOM-Max, a commercial inference NPU released in 2025, and matching the same model trained on a B200 GPU: 0.7993 vs 0.7994 loss on 1,024 windows of unseen text. Weights for both runs and the evaluation script are public.

Why is this news?

Inference accelerators are sold to run models, not to train them. Their software stacks execute forward passes. A full pretraining run needs backpropagation, optimizer state and tens of thousands of stable updates. Showing that such a chip can carry a 5-billion-token run to GPU quality challenges a widely held assumption about what inference hardware is for.

What exactly was shown?

Item	Value
Model	VKORE-325M-NPU-Base, 325.6M parameters
Data	5.0B byte tokens of FineWeb-Edu
Run	76,292 steps, about four days
Hardware	Rebellions ATOM-Max NPU (2025)
Reference	identical model on a B200 GPU

What does it not show?

Speed or cost. The report covers quality, not throughput or price per run.
Scale. 325M parameters on 5B tokens is research scale.
That the chip alone is enough. The NPU and its standard software are built for inference; training depends on VIDRAFT's own training stack and a stability technique under patent application.
Seed variance. One run per hardware.

FAQ

Which NPU? Rebellions ATOM-Max, released in 2025.

How close are the two models? Paired difference -0.0001, standard error 0.0007.

Can I check it? Yes, both weights and evaluate_unseen.py are in the model repo.

Sources

Technical report: https://huggingface.co/blog/FINAL-Bench/pretraining-a-325m-language-model-from-zero-on-an

Is open-weight really 90% as good at 10% of the cost? Checking IBM's claim against list prices

ai maya — Fri, 23 Oct 2026 13:25:30 +0000

TL;DR

On IBM's Q3 2026 earnings call, CEO Arvind Krishna said open-weight models now deliver about "90 percent as effective" results at "anywhere from 25 percent to 10 percent of the token cost of the frontier models", and that most businesses can build useful AI infrastructure in "two or three racks" (The Register, 2026-10-22). I checked the cost half of that claim against the public price lists on 2026-10-23. What I found:

The price claim is true, and then some. On an illustrative 8,000-input, 500-output-token job, DeepSeek V4.1-Flash at off-peak list prices costs 2.9% of Claude Opus 5 and 7.1% of GPT-6 Sol. V4-Pro costs 11.9% and 29.9%.
"Frontier" is doing a lot of work in that sentence. The closed vendors' own small tiers (GPT-6 Luna, Claude Haiku 5.5) cost 1.05 dollars per 1,000 of those requests, which is 30% less than DeepSeek's cheapest. Open weights are not uniquely cheap. Small models are.
On-prem is a utilization bet, not a price bet. Under an assumed 30 dollars per 8-GPU node-hour and 12,000 output tokens per second per node, owning hardware beats GPT-6 Sol's output price once the cluster is busy about 7% of the time, but never beats DeepSeek Flash's hosted price at any utilization.
The honest case for racks is data control and predictable spend, not "10 percent of the cost".

All hardware numbers below are illustrative assumptions, labeled as such, and the formulas are included so you can replace them with your own.

What exactly did IBM claim?

According to The Register's report of the call, Krishna described a shrinking quality gap: in 2024 "the top model might score a 70, and the open model might score a 20 or a 30", by late 2025 it was "70 to 50", and now open weights are "90 percent as effective". The cost line was "anywhere from 25 percent to 10 percent of the token cost of the frontier models", framed as "Would you like something that's 10 percent of the cost if it's 90 percent as good as the best?" He named tasks like summarizing an email, examining a contract, answering a customer call and composing an email.

Two things are worth noting before any math. First, the article names no benchmark behind the 70, 50 and 90 figures, so the quality half cannot be checked from the public record. I did not try. Second, the same call reported Q3 revenue of 17.1 billion dollars (up 5%), infrastructure revenue of 3.7 billion dollars and Red Hat growing 11%. IBM sells hardware, software and consulting around on-prem AI, so the framing is a business position as well as an observation. That does not make it wrong. It means the cost claim deserves a calculation.

What does one workload cost on nine price lists?

I picked a job that matches IBM's examples: ticket triage or email summarization with 8,000 input tokens (instructions, a thread, some context) and 500 output tokens, run 1,000 times, with no caching. List prices, USD per million tokens, were read on 2026-10-23 from DeepSeek, Anthropic and press coverage of OpenAI's launch pricing (The Next Web). OpenAI's own pricing page returned HTTP 403 to my fetcher, so treat the GPT-6 numbers as secondary-sourced.

Model (list price, input / output per 1M)	Cost per 1,000 requests	% of Opus 5	% of GPT-6 Sol
Claude Opus 5 (5.00 / 25.00)	52.50	100.0	250.0
Claude Opus 5.5 (4.00 / 20.00)	42.00	80.0	200.0
Claude Sonnet 5 (2.00 / 10.00)	21.00	40.0	100.0
GPT-6 Sol (2.00 / 10.00)	21.00	40.0	100.0
DeepSeek V4-Pro, off-peak (0.66 / 1.98)	6.27	11.9	29.9
DeepSeek V4.1-Flash, peak (0.30 / 1.20)	3.00	5.7	14.3
DeepSeek V4.1-Flash, off-peak (0.15 / 0.60)	1.50	2.9	7.1
GPT-6 Luna (0.10 / 0.50)	1.05	2.0	5.0
Claude Haiku 5.5, prompts up to 100k (0.10 / 0.50)	1.05	2.0	5.0

Reading the table against IBM's sentence: if "frontier" means Opus 5, V4-Pro sits inside the 10 to 25% band (11.9%) and Flash sits far below it. If "frontier" means the mid-priced flagship tier (Sonnet 5 or GPT-6 Sol), V4-Pro lands at about 30%, just above the band, and Flash lands under it. IBM's range is, if anything, conservative on hosted list prices.

Two billing details move these numbers. DeepSeek's peak window is 01:00 to 04:00 and 06:00 to 10:00 UTC, Monday to Friday, at double the off-peak rate. And caching changes input-heavy jobs a lot. If 6,000 of the 8,000 input tokens are a shared prefix that hits the cache, the same 1,000 requests cost 25.50 dollars on Opus 5 (cache read 0.50), 10.20 on GPT-6 Sol (0.20), 0.618 on DeepSeek Flash off-peak (0.003) and 0.51 on both Luna and Haiku 5.5 (0.01). Cached prefixes shrink the absolute gaps but not the ordering.

Is the cheap part really "open"?

Look at the bottom two rows. GPT-6 Luna and Claude Haiku 5.5 are closed models, and for this job they cost 1.05 dollars per 1,000 requests, about 30% below DeepSeek Flash's off-peak 1.50. The closed vendors have collapsed their own small-tier prices to the same neighborhood as hosted open weights.

That matters for how to read "open weight is 10% of the cost". The saving comes from picking a small, sparse model for a simple task. It is a model-size and routing effect, and it is available inside a closed API too. What open weights add is not a lower token price at the API but a different set of options: you can host them yourself, fine-tune them, pin a version forever, and keep data inside your network. Those are real benefits. They are just not the same claim as price.

When does owning hardware beat the API?

This is where "two or three racks" comes in. Self-hosted cost per token is a fixed hourly cost divided by the tokens you actually produce, so utilization decides everything:

# cost per 1M output tokens when you own the hardware
def cost_per_m_out(node_usd_per_hour, node_out_tok_per_s, utilization):
    return node_usd_per_hour / (node_out_tok_per_s * 3600 * utilization) * 1e6

# break-even utilization against an API output price
def breakeven_util(node_usd_per_hour, node_out_tok_per_s, api_usd_per_m_out):
    return cost_per_m_out(node_usd_per_hour, node_out_tok_per_s, 1.0) / api_usd_per_m_out

print(cost_per_m_out(30, 12000, 1.0))   # 0.694 USD per 1M tokens at full load
print(breakeven_util(30, 12000, 10.0))  # 0.069, i.e. 6.9% vs GPT-6 Sol output price

My assumptions, to be replaced with yours: 30 dollars per 8-GPU node-hour all-in (hardware depreciation, power, space, support) and 12,000 output tokens per second per node at full batch for a large sparse model. Neither number is measured. Real throughput depends on the model, context length, batch shape and serving stack, and your hardware price depends on what you negotiated.

Under those assumptions:

API price compared against	Output price per 1M	Break-even cluster utilization
Claude Opus 5	25.00	2.8%
GPT-6 Sol	10.00	6.9%
DeepSeek V4-Pro (off-peak)	1.98	35.1%
DeepSeek V4.1-Flash (off-peak)	0.60	never (116% needed)
GPT-6 Luna	0.50	never (139% needed)

The curve reads like this. At 10% average utilization the owned cluster costs about 6.94 dollars per million output tokens: cheaper than Sol's 10, far cheaper than Opus's 25, and 3.5 times more than V4-Pro. At 50% it costs 1.39 dollars, which beats V4-Pro but still loses to Flash at 0.60. Even at 100% it is 0.69 dollars.

Sensitivity matters here, so I varied both inputs against Sol's 10 dollar output price:

Node cost per hour	6,000 tok/s	12,000 tok/s	24,000 tok/s
16 dollars	7.4%	3.7%	1.9%
30 dollars	13.9%	6.9%	3.5%
60 dollars	27.8%	13.9%	6.9%

Against flagship-priced APIs, owning hardware pays off at modest utilization across a wide range of assumptions. Against hosted small or sparse models, it does not pay on price in any cell of this grid.

What would two or three racks look like?

Take "three racks" as 24 eight-GPU nodes. At 30 dollars per node-hour that is 6.31 million dollars a year. The same money buys 631 billion output tokens a year at GPT-6 Sol's price, which is 1.73 billion a day, or an average of 20,000 tokens per second. Peak capacity of the 24 nodes under my assumptions is 288,000 tokens per second, so you need roughly 6.9% average utilization to break even against Sol, the same number as above. At Opus pricing the break-even average is 8,000 tokens per second.

Running the same arithmetic against Flash's off-peak output price gives 10.5 trillion tokens a year, an average of 333,000 tokens per second, which is more than the cluster can produce even at full load. In other words, on these assumptions, a company that only wanted the cheapest token should not buy racks at all.

What does this model leave out?

The calculation is a deliberately simple floor and ceiling, and several things push it both ways:

Prefill. My capacity figure counts output tokens only. Input-heavy jobs such as the 8,000-in, 500-out workload above spend most compute on prefill, so real capacity in "requests per second" is lower than the output-token rate suggests.
Idle headroom. Latency targets and peak-hour traffic force you to run below full batch. A cluster that averages 30% utilization may be fine for users and still be wasteful on paper.
People and refresh. Operations staff, monitoring, security reviews and hardware refresh cycles are not in a 30 dollar hourly figure unless you put them there.
Data egress and compliance. For contracts, patient notes or call recordings, the alternative to racks may not be "cheaper API" at all. It may be "no AI on that data". That changes the comparison from price to permission.
Quality. Everything above assumes the cheaper model is good enough for the task. IBM's "90 percent as effective" is the load-bearing claim, and it comes without a named benchmark.

How should a team decide?

A practical order of operations, based on the numbers above:

Measure your own task quality on a small model first. If Luna, Haiku 5.5 or Flash passes your acceptance test, you have already captured most of the saving without any hardware.
Add caching and routing before racks. A shared prefix cut Sol's cost on this job from 21.00 to 10.20 dollars per 1,000 requests, without changing anything about the model.
Estimate your average, not peak, token rate. Convert monthly spend to tokens per second. If it is far below what a cluster can produce, the cluster will sit idle, and the break-even utilization above tells you how idle is too idle.
Buy racks for reasons that price does not capture: data that cannot leave, fixed versions for regulated workflows, or spend that must be predictable.

Reproduce it

The model is the two functions above plus the price table: workload tokens times list price for the API side, and node cost divided by tokens actually produced for the hardware side. Replace the workload size, node cost and node throughput with your own measurements. Prices change often, so re-read the three pricing pages before trusting any figure here.

FAQ

Is IBM's claim that open-weight models cost 10 to 25% of frontier models accurate?

On hosted list prices, yes, and it is conservative for the cheapest options. For an 8,000-input, 500-output job, DeepSeek V4.1-Flash costs 2.9% of Claude Opus 5 and 7.1% of GPT-6 Sol, and V4-Pro costs 11.9% and 29.9%. The quality claim of "90 percent as effective" cites no benchmark in the published report, so it cannot be verified from public sources.

Are open-weight models cheaper than closed models?

Not by nature. Closed small tiers such as GPT-6 Luna and Claude Haiku 5.5 list at 0.10 dollars per million input tokens and 0.50 per million output tokens, which makes the example job cheaper (1.05 dollars per 1,000 requests) than DeepSeek V4.1-Flash off-peak (1.50). Price tracks model size and sparsity more than license.

At what utilization does running your own GPUs beat paying an API?

It depends on three numbers: node cost per hour, output tokens per second per node, and the API price you compare against. With 30 dollars per node-hour and 12,000 output tokens per second, break-even is 6.9% against GPT-6 Sol's 10 dollar output price and 35.1% against DeepSeek V4-Pro's 1.98 dollars. Against DeepSeek Flash's 0.60 dollars it does not break even even at 100%. These hardware inputs are illustrative, not measured.

Does prompt caching change the open versus closed comparison?

It lowers every bill and keeps the ordering. With 6,000 of 8,000 input tokens cached, 1,000 requests cost 25.50 dollars on Opus 5, 10.20 on GPT-6 Sol, 0.618 on DeepSeek Flash off-peak and 0.51 on Luna or Haiku 5.5.

What is the real reason to run open-weight models on-prem?

Control rather than price: keeping sensitive data inside your network, pinning a model version, fine-tuning on private data and getting a fixed cost line. Those are legitimate reasons, and they hold even when the API is cheaper per token.

Shorter outputs, same price: what DeepSeek V4.1-Flash-1017 changes in an agent's bill

ai maya — Wed, 21 Oct 2026 16:03:51 +0000

TL;DR

DeepSeek released V4.1-Flash-1017 on 2026-10-17 as an MIT-licensed update to V4.1-Flash. The model card reports gains on all 11 listed benchmarks, between 2.0 and 11.1 points, and says the model also finishes tasks faster with shorter output. The API price did not change: $0.15 per million uncached input tokens, $0.003 per million cached input tokens and $0.60 per million output tokens off-peak, doubled during peak hours.

The interesting question is not the score table. It is what "shorter output at the same price" does to the cost of a whole agent task. My findings, using the model card, the live pricing page and a small cost model:

The score gains are real on the vendor's own table, but the vendor ran them. The model beats Claude Opus 5 on 4 of 11 listed benchmarks and trails it on 7.
A 30 percent cut in reply length saves about 22 percent on an illustrative 30-turn task, not 30 percent, because input tokens are billed too.
Turn count matters more than reply length. Context is re-read every turn, so cost grows faster than linearly with the number of turns. Finishing in fewer turns is where large savings come from.
Three billing details can quietly erase the savings: peak hours (35 of 168 hours each week), a cache that stops hitting (6 times the cost in my model) and old model names that now silently point at the new model.

What actually shipped in V4.1-Flash-1017?

The model card describes it as "a minor update" to V4.1-Flash, with improvements across reasoning, coding, agentic tasks and visual benchmarks, plus reduced task execution time and output length. The checkpoint is public on Hugging Face under the MIT license. At the time of writing the repository shows 42,122 downloads and 2,103 likes.

From the public config.json, the architecture is a mixture of experts with these numbers:

Field	Value
Text layers	40
Hidden size	5,120
Routed experts	384, plus 1 shared expert
Experts used per token	6
KV heads	1 (head dimension 512)
Context window	1,048,576 tokens
Weight format	FP8 with FP4 experts (`quantization_config`)
Repository size	510.3 GB across 83 files

The API serves it as deepseek-flash. The pricing page also says the older names deepseek-v4-flash and deepseek-v4-flash-vision-exp are still accepted, but the models behind them are retired and their requests are served by V4.1-Flash-1017 at the Flash price. If you pinned one of those names and never re-ran your evaluation, you are already on the new model.

Did the benchmark gains hold up against other models?

All numbers below come from the model card table. They are vendor-reported, and the card names the harnesses it used: the Claude Code harness for Terminal-Bench 4.0 and the Mini-SWE-V2 harness for DeepSWE v1.1.

I counted wins from the card's own rows. The new model scores higher than the previous Flash build on all 11 benchmarks, with a mean gain of 5.2 points. Against the other models in the table:

Compared with	Benchmarks where 1017 scores higher	Where it scores lower
Kimi K3	10 of 11	HLE (42.6 vs 43.5)
GLM-5.3	8 of 8 listed	none
GPT-5.6 Sol	9 of 11	HLE (42.6 vs 44.5), ZeroBench-main (51.0 vs 53.0)
Claude Opus 5	4 of 11	7 of 11, including HLE (42.6 vs 56.3) and Terminal-Bench 4.0 (42.3 vs 51.8)

The four benchmarks where it is ahead of Opus 5 are DeepSWE v1.1 (77.0 vs 74.0), Automation-Bench (60.5 vs 50.3), Agents' Last Exam (35.4 vs 28.6) and Apex Bench v1.1 (72.1 vs 65.8). That pattern is an agentic-coding and tool-use profile, not a general-knowledge one. On HLE the gap to Opus 5 is 13.7 points.

One row deserves a caution. Kimi K3 scores 12.6 on Terminal-Bench 4.0, while the other three comparison models land between 37.9 and 51.8. A spread that wide on one benchmark often points to harness sensitivity rather than a pure capability gap. That is my inference, not something the card says, but it is a reason not to read the Terminal-Bench column as a clean ranking.

What does "same price, shorter output" mean for one task?

Here is the published price list, per million tokens, from the DeepSeek pricing page:

	Flash off-peak	Flash peak	Pro off-peak	Pro peak
Input, cache hit	$0.003	$0.006	$0.022	$0.044
Input, cache miss	$0.15	$0.30	$0.66	$1.32
Output	$0.60	$1.20	$1.98	$3.96

Pro costs 7.3 times as much on cached input, 4.4 times on uncached input and 3.3 times on output, computed from the table. Peak hours are 01:00 to 04:00 and 06:00 to 10:00 UTC, Monday to Friday, excluding Chinese public holidays. That is 35 of the 168 hours in a week, or 20.8 percent. Everything else is half price.

To see how output length feeds into a bill, I built a deliberately simple agent loop. It is illustrative, not a measurement of any real workload:

30 turns per task
an 8,000-token initial prompt
2,000 tokens of tool output appended each turn
the whole conversation re-sent every turn, with the earlier part served from cache and only the newly appended tokens billed as uncached

def loop(turns, init, tool, out):
    ctx = init
    hit = 0
    miss = init
    for t in range(1, turns):
        hit += ctx            # prefix served from cache
        new = tool + out      # newly appended tokens, uncached
        miss += new
        ctx += new
    return hit, miss, out * turns

FLASH_OFFPEAK = dict(hit=0.003, miss=0.15, out=0.60)  # USD per 1M tokens

def cost(hit, miss, out, p):
    return (hit * p["hit"] + miss * p["miss"] + out * p["out"]) / 1e6

With replies averaging 1,500 tokens the task reads 1.65 million cached tokens, 109,500 uncached tokens and writes 45,000 tokens. At off-peak Flash prices that is $0.048, split roughly 10 percent cached input, 34 percent uncached input and 56 percent output. Cut the reply length by 30 percent to 1,050 tokens and the total falls to $0.038, a saving of 21.9 percent. The saving is smaller than the 30 percent reply cut because input tokens are still billed, and shorter replies shrink the context only a little.

At peak prices both numbers simply double ($0.097 and $0.076). The same token counts at Pro off-peak prices come to $0.198 for the longer replies, about 4.1 times the Flash figure. Real Pro runs would use different token counts, so treat that ratio as price-only.

Why does the number of turns matter more than reply length?

Every turn re-reads the whole conversation so far. Cached tokens are cheap, but there are many of them, and the cached part of the bill grows with the square of the turn count.

Using the same model, off-peak Flash:

Turns	Replies 1,500 tokens	Replies 1,050 tokens
12	$0.0186	$0.0146
18	$0.0282	$0.0220
24	$0.0381	$0.0297
30	$0.0484	$0.0378
40	$0.0664	$0.0520

Going from a 30-turn task with long replies to a 24-turn task with short replies costs 61 percent of the baseline. At 18 turns it is 45 percent, and at 12 turns it is 30 percent. This is a toy model, so the exact percentages will not transfer, but the direction does: a model that solves the same task in fewer tool calls saves more than one that merely writes less per call.

That is consistent with a secondary report. The Decoder wrote that the new model reaches its top DeepSWE score at "just under $0.20 per task", about $0.10 in a lower-cost setting that still clears 72 percent, and that the previous build cost "around $0.50 per run". It also says competing models land between $2 and over $10 per task. Those are the outlet's figures, the model card does not publish per-task cost, and I could not find token counts behind them. A drop from about $0.50 to about $0.20 is far larger than the 22 percent my reply-length model produces, which suggests most of the real saving comes from shorter trajectories rather than shorter sentences. It also suggests real benchmark runs are much bigger than my toy task, since $0.20 at these prices is about four times my 30-turn figure. Which of those is true for your workload is something only your logs can tell you.

What can quietly raise the bill?

Cache misses. My model assumes about 94 percent of input tokens hit the cache, which is what a clean append-only conversation gives. Forcing the hit rate down shows how sensitive the result is:

Cache hit rate	Cost of the 30-turn task
94 percent (append-only)	$0.048
90 percent	$0.058
50 percent	$0.162
0 percent	$0.291

With no cache at all the same task costs about 6 times as much. Anything that edits the middle of the conversation, such as rewriting an earlier tool result, injecting a changing timestamp near the top of the prompt or reordering messages, can break the prefix and move you down this table.

Peak hours. If a nightly batch job runs at 08:00 UTC on a weekday, you pay double. Weekend and off-peak slots are half price, so a queue that can wait is a direct 50 percent discount. Check the Chinese public holiday calendar too, since those days are off-peak in full.

Thinking mode. The pricing page says the model supports thinking and non-thinking modes and that thinking is the default. Reasoning tokens are output tokens. The model card recommends max_tokens of at least 256K, and the API allows up to 384K output. A permissive limit is safe for quality but gives you no ceiling on a runaway generation, so set your own per-task budget.

Search and Fetch. DeepSeek's standalone Search and Fetch APIs are billed per request, with Search at $0.75 per 1,000 successful requests, and press reports say server-side tool use is free until the end of October. An agent that searches heavily will see a new line item in November.

Can you run it yourself?

The repository is 510.3 GB, so the weights alone are a multi-GPU job. Split evenly across 8 GPUs that is 63.8 GB per card, which leaves about 16.2 GB per 80 GB card for KV cache and activations before any replication overhead. On 141 GB cards the headroom is about 77 GB per card. That is a back-of-envelope figure, not a deployment guide, and real engines replicate some tensors.

One detail in the Hugging Face metadata is worth knowing. It lists 763.2 billion parameters, but 204.0 billion are FP8 (1 byte each) and 557.2 billion are labeled int8. If every one of those were one byte the files would total about 765 GB, yet the repository is 510.3 GB. Counting the int8 entries as packed 4-bit values, two per byte, gives about 487 GB for the weights, which fits the observed size once you add scales and auxiliary files. So the 763.2 billion headline is a count of values, and the on-disk footprint is closer to a 500 GB model. This matches the FP4 expert setting in the config, though the card does not spell out the packing.

The card also notes that there is no Jinja chat template. Prompt formatting comes from a reference encoding.py and the separate deepseek-recipe libraries, so a serving stack that expects a standard template needs that piece wired in. The API changelog recommends DeepSeek Harness 0.2.1-rc.1 or later.

How should you test it on your own workload?

Re-run your evaluation against deepseek-flash now, especially if you used a legacy model name.
Log, per task: turns, input tokens split into cache hit and miss, output tokens and wall-clock time. The API usage object reports cache hit and miss counts, so check the current field names in the docs.
Compare cost per solved task, not cost per token. A cheaper model that solves fewer tasks loses.
Put a hard budget on turns and on output tokens per task.
Schedule queue-tolerant work in off-peak hours.
Keep prompts append-only so the cache keeps hitting.

FAQ

Is DeepSeek V4.1-Flash-1017 open source? The weights and repository are under the MIT license, according to the model card. The card does not mention training data or training code.

Did the API price change with this update? No. The pricing page lists the same Flash rates, and press coverage confirms unchanged pricing. What changed is that peak hours cost double and off-peak hours cost half.

Does the model beat Claude Opus 5? On the model card it is ahead on 4 of 11 listed benchmarks, including DeepSWE v1.1 (77.0 vs 74.0), and behind on 7. These are vendor-reported numbers.

How much cheaper is a task than before? The model card does not give a per-task figure. A secondary report cites about $0.50 down to just under $0.20 on DeepSWE. My illustrative model shows 22 percent from shorter replies alone and larger gains only if the number of turns also drops.

Can I run it on a single GPU? No. The repository is 510.3 GB, so you need a multi-GPU node or a heavily quantized community build.

Method and caveats

Benchmark numbers and harness names come from the model card. I did not run any benchmark.
Prices come from the DeepSeek pricing page as fetched on 2026-10-22. Prices and peak windows can change.
The agent-loop cost model is illustrative. Its turn count, prompt size, tool output size and reply length are my assumptions, not measurements. The per-turn accounting treats each turn's tool output and the previous reply as the new uncached tokens.
Per-task dollar figures from The Decoder are secondary and unverified. The article gives no token counts.
The "packed 4-bit" reading of the int8 count is my inference from file sizes.
Memory headroom assumes an even split and ignores replicated tensors and engine overhead.

Sources

Can two Mac Studios really run an 816B model? A bandwidth roofline check on Muse Spark Open

ai maya — Tue, 20 Oct 2026 13:37:44 +0000

TL;DR

Meta released Muse Spark Open on 2026-10-15: 816B total parameters, 42B active per token, a 1M-token context window, under a custom Muse Model License with limited commercial terms. Apple published decode benchmarks the same day on Mac Studio M5 Ultra machines linked by RDMA over Thunderbolt. At a 65,536-token context, the NVFP4 build decodes at 15.74 tokens/s on 2 machines with pipeline parallelism, 24.99 on 2 machines with tensor parallelism, and 36.73 on 4 machines with tensor parallelism.

I checked those numbers against two quantities that are easy to compute from public files: the bytes read per token and the memory bandwidth of the machine. Three findings:

The model does not fit on one 512 GB machine. The NVFP4 repository is 533.0 GB, so two machines are a memory requirement, not a speed choice.
The cache is small for a 1M-context model. From the public config, the cache for a full 1,048,576-token sequence is 69.1 GB in bf16, 3.85 times smaller than if every layer used full attention.
Dense models in the same benchmark reach 82 to 89 percent of the bandwidth ceiling on one machine. Muse Spark Open reaches 42 percent on two machines with pipeline parallelism and 25 percent on four with tensor parallelism. The gap is the interesting part of this story.

Everything below is arithmetic on public files plus Apple's reported measurements. Where I am guessing, I say so.

What exactly was released, and what runs it?

From Meta's launch post and the Hugging Face repositories:

816B total parameters, 42B active, natively multimodal (text, image, audio inputs in the config), 1M-token maximum position embeddings.
Weights on Hugging Face as meta-models/Muse-Spark-Open-816B-A42B (bf16), an NVFP4 variant, and community GGUF and MLX builds already listed on the Hub.
Day-one serving support listed for SGLang, vLLM, TensorRT-LLM, Transformers, MLX, llama.cpp and Unsloth, plus hosted access through several clouds and inference providers.

The text config (readable from an open MLX mirror, since the original repository is gated) gives the shape: 62 layers, hidden size 8192, 128 attention heads with 16 key-value heads at head dimension 64, 256 routed experts with 8 active per token, and a sliding window of 2,048 tokens. The layer pattern is regular: every fourth layer, starting at index 1, uses full attention, and the other three use sliding-window attention. That is 16 full-attention layers and 46 sliding-window layers.

Why does it need two machines and not one?

Apple's report states the M5 Ultra Mac Studio offers up to 512 GB of unified memory. The repository sizes, summed from the Hugging Face file listings:

Build	Repository size
bf16 original	1,644.4 GB
NVFP4 (Meta)	533.0 GB
MLX 4-bit (community)	532.8 GB

Both 4-bit builds exceed 512 GB before a single byte of cache, the operating system or activations is allocated. Apple's results list no single-machine run for this model. Two machines give 1,024 GB combined, which leaves about 491 GB after the weights for cache, runtime and the OS.

That headroom is generous for one user and thin for a fleet. An upper bound, ignoring everything but the cache: 491 GB divided by 69.1 GB per 1M-token sequence is about 7 sequences, or about 54 sequences at 128K tokens (8.98 GB each). Real systems lose a chunk of memory to the operating system, the GPU wired-memory limit and activation buffers, so treat those as ceilings.

How much cache does a 1M-token sequence need?

For a layer with h_kv key-value heads, head dimension d, and bf16 storage, the cache is 2 * h_kv * d * 2 bytes per token per layer. Here that is 2 * 16 * 64 * 2 = 4,096 bytes. Sliding-window layers stop growing at 2,048 tokens, so they cost a constant amount.

import json
t = json.load(open("config.json"))["text_config"]
layers = t["layer_types"]
full = layers.count("full_attention")            # 16
slide = layers.count("sliding_attention")        # 46
per_tok_layer = 2 * t["num_key_value_heads"] * t["head_dim"] * 2   # 4096 bytes

def kv_gb(ctx):
    grows = full * per_tok_layer * ctx
    fixed = slide * per_tok_layer * min(ctx, t["sliding_window"])
    return (grows + fixed) / 1e9

print(kv_gb(1_048_576))   # 69.1 GB

The result: 64 KiB of cache growth per token (16 layers * 4,096 bytes), plus a constant 0.39 GB for the 46 sliding-window layers.

Context	Actual layout	If all 62 layers were full attention	Ratio
32K	2.5 GB	8.3 GB	3.3x
128K	9.0 GB	33.3 GB	3.7x
262K	17.6 GB	66.6 GB	3.8x
1M	69.1 GB	266.3 GB	3.85x

This is a cache-side design choice, and a conventional one. It trades the ability of 46 layers to look far back for a 3.85x smaller cache, and leaves 16 layers to carry long-range retrieval. Whether that costs quality on long-context tasks is something only long-context evaluations can answer, and I have not seen independent ones yet. For a different design that goes further, with linear-attention layers replacing most of the cache, see the Ling-3.1-flash post linked at the end.

What is the bandwidth ceiling for decoding?

At batch size 1, decoding one token means streaming the active weights and the relevant cache through the memory system once. Compute is almost idle. So the speed limit is:

tokens/s <= memory_bandwidth / (active_weight_bytes + cache_bytes_read_per_token)

Inputs, with the assumptions marked:

Bandwidth: 1.2 TB/s per M5 Ultra, as reported in Apple's write-up (819 GB/s for the previous M3 Ultra).
Active weight bytes: 42B active parameters times the average bytes per parameter of the NVFP4 repository, 533.0 GB / 816B = 0.653 bytes. That gives 27.4 GB per token. This is crude: the attention and embedding weights are stored in bf16 while the experts are 4-bit, so the true active read is somewhat higher and the true ceiling somewhat lower.
Cache read per token at 65,536 tokens of context: 4.68 GB (16 full layers over 65,536 tokens plus 46 sliding layers over 2,048).

Sum: 32.1 GB per token. One machine's bandwidth gives 1.2e12 / 32.1e9 = 37.4 tokens/s.

How the ceiling scales with the setup depends on the parallelism. With pipeline parallelism, each token still passes through the machines one after another, so only one machine is reading at a time and the ceiling stays at 37.4 tokens/s however many machines you add. With tensor parallelism, every machine reads its slice of every layer at the same time, so the ceiling is roughly multiplied by the machine count, minus synchronization cost. That is why Apple's report shows pipeline parallelism preserving decode speed across nodes while tensor parallelism raises it, in exchange for a synchronization step in every layer.

How close do the measured numbers get?

Setup (NVFP4, 65,536 context)	Measured	Ceiling (computed)	Share of ceiling
2 machines, pipeline	15.74 tok/s	37.4	42%
2 machines, tensor	24.99 tok/s	74.7	33%
4 machines, tensor	36.73 tok/s	149.5	25%

Doubling the machines under tensor parallelism did not double the speed: 24.99 to 36.73 is a 1.47x gain for 2x the hardware. Going from pipeline to tensor on two machines gained 1.59x. Each step adds communication cost that bandwidth alone does not predict.

Why do the dense models land closer to the roofline?

The same Apple report includes two dense-looking comparison models. I am assuming both are dense, which I have not verified from their model cards:

Model (single M5 Ultra)	Weight bytes	Ceiling at 1.2 TB/s	Measured	Share
Qwen 3.8 27B, bf16	54 GB	22.2 tok/s	19.69	89%
Mistral Medium 3.5 128B, FP8	128 GB	9.4 tok/s	7.70	82%

Both land at 82 to 89 percent, which says the 1.2 TB/s figure is a realistic target on this hardware and my ceiling method is not wildly off. The 816B mixture-of-experts model on two pipelined machines gets 42 percent. Possible reasons, none of which I can confirm from the published data:

Expert gathering. At batch size 1, each layer reads 8 of 256 experts, which are separate memory regions. Many smaller reads can use bandwidth less efficiently than a few large contiguous ones.
Mixed precision. As noted above, part of the active path is bf16, so my 27.4 GB estimate is probably low, which makes the true ceiling lower and the true share higher than 42 percent.
Inter-machine transfer. Each token crosses the link between machines, and routing decisions add latency that dense layers do not have.

The honest summary is that the gap exists and has several plausible causes. A per-layer profile would separate them, and I would trust that over my estimate.

What changes at long context?

The weights cost the same at every context length. The cache does not. Using the same crude model with one machine's bandwidth:

Context	Cache read per token	Weight + cache	Ceiling (1.2 TB/s)
8K	0.9 GB	28.4 GB	42.3 tok/s
128K	9.0 GB	36.4 GB	33.0 tok/s
262K	17.6 GB	45.0 GB	26.7 tok/s
1M	69.1 GB	96.5 GB	12.4 tok/s

At 1M tokens the cache read (69.1 GB) is 2.5 times the active weight read (27.4 GB). Past roughly 420K tokens the cache outweighs the weights, so the fixed cost of the model stops being the main cost and the context length takes over. This model ceiling also ignores the compute cost of attention itself, which grows with context and which I did not model, so real long-context numbers should come in below these.

What should you take from this?

Do the memory arithmetic first. A 4-bit build of an 816B model is 533 GB. Whether your hardware can hold it, with room for cache, is a two-line calculation, and it decides your topology before any benchmark does.
Read the layer pattern, not just the context length. A 1M-token window with 16 full-attention layers has a very different cost from a 1M-token window with 62.
Pick the parallelism for the goal. Pipeline parallelism pools memory and keeps per-token speed. Tensor parallelism buys speed at a synchronization cost. Apple's data shows both effects.
Treat vendor tokens/s as one point on a curve. The 36.73 figure is at 65,536 tokens of context on four machines. At 1M tokens, my model says the same hardware has a much lower ceiling.
Wait for long-context quality evaluations. A cheaper cache is only a win if retrieval quality survives it.

FAQ

Can Muse Spark Open run on a single Mac Studio?

Not the 4-bit builds. The NVFP4 repository is 533.0 GB and the largest single M5 Ultra configuration reported is 512 GB of unified memory. Apple's results list no single-machine run for it. Smaller quantizations may fit, but I have not seen measurements.

How fast is it in practice?

Apple reports 15.74 tokens/s with two machines and pipeline parallelism, 24.99 with two machines and tensor parallelism, and 36.73 with four machines and tensor parallelism, all at a 65,536-token context with the NVFP4 build. Longer contexts will be slower.

How much memory does a 1M-token context need?

From the public config, the cache for one 1,048,576-token sequence is about 69.1 GB in bf16, on top of the 533 GB of weights. A quantized cache would reduce that, and I did not model it.

Is memory bandwidth really the bottleneck for decoding?

At batch size 1, yes, to a first approximation. Two dense models in Apple's report reached 82 to 89 percent of the bandwidth ceiling on a single machine. The mixture-of-experts model reached less, so other overheads matter there.

What is the license?

Meta describes a Muse Model License with limited commercial terms that lets many developers use the model for free. The Hugging Face tag is license:other. Read the license text before building a product on it.

Method and caveats

Cache arithmetic uses text_config from an open MLX mirror of the model. The original repository is gated, so I could not confirm the config there. The shape numbers come from the mirror.
The ceiling uses 1.2 TB/s as reported and treats it as per machine. Real sustained bandwidth is lower than peak.
Active weight bytes use one average bytes-per-parameter figure for the whole model, which is an approximation.
The two dense comparison models are assumed to be dense. If either is not, its row in the table is wrong.
Measured numbers are Apple's, not mine. I did not run these benchmarks.
The script that produced every computed number is short enough to rerun from the config and the three repository sizes.

Sources

Ling-3.1-flash and the 7:1 hybrid: how much KV cache does a 560B model need at 1M tokens?

ai maya — Mon, 19 Oct 2026 13:26:35 +0000

TL;DR

Ling-3.1-flash (MIT license, 560B total, 25B active, weights on Hugging Face since 2026-10-13) replaces most of its attention layers with linear attention: 63 Kimi Delta Attention (KDA) layers and only 9 gated Multi-head Latent Attention (MLA) layers, a 7:1 ratio. Only the 9 MLA layers keep a cache that grows with every token. Computing from the published config.json, that is roughly 10.4 KB of cache per token, or about 10.6 GB for a 1M-token sequence in bf16, against about 83 GB if all 72 layers used latent attention. The weights, not the cache, are now the main cost: the official FP8 repository is 573.8 GB. This post shows the arithmetic, the assumptions behind it, and what it means for serving long-context agents.

What did Ling-3.1-flash actually ship?

The model card and config.json on Hugging Face give these facts (all quoted from the card or read from the config):

Item	Value
Architecture	Hybrid-linear MoE (`bailing_hybrid`)
Parameters	560B total, 25B active per token
Layers	72 = 63 KDA + 9 gated MLA (7:1)
Experts	512 routed, 1 shared, 8 active per token; 4 dense layers first
Hidden size / heads	4096 / 64
MLA dimensions	`kv_lora_rank` 512, `qk_rope_head_dim` 64, `q_lora_rank` 1536
Context	1M tokens on the card; `max_position_embeddings` is 262144 in the config
Weights	FP8 (E4M3) official checkpoint, 54 safetensors shards, 573.8 GB measured through the Hugging Face API
License	MIT
Speculative decoding	1 next-token-prediction (MTP) layer, usable via SGLang or vLLM

Two details deserve a flag. First, the card advertises a 1M-token window while the config still says 262,144 positions with no RoPE scaling entry. That is not necessarily a bug (serving stacks can override it), but check what your engine actually allocates before you promise 1M to a user. Second, the card text lists benchmark names such as Terminal-Bench 4.0, SWE-bench Pro and GDPVal-AA V2.1, but the numbers sit in images. I do not quote scores here; read them on the card and treat vendor numbers as vendor numbers until someone reproduces them.

Why would a model keep only 9 full-attention layers?

Full attention stores a key and a value for every past token in every layer. That cache grows linearly with context and is read in full at every decoding step, which is why long-context decoding is bandwidth bound (I covered the mechanics in The KV Cache Is the Bottleneck).

Linear attention layers work differently. They fold the past into a fixed-size state matrix per head and update it as each token arrives. Memory per sequence stays constant no matter how long the context gets. The classic weakness is recall: a fixed state cannot hold an exact copy of everything, so pure linear models struggle at precise retrieval. Hybrids keep a few full-attention layers as the "exact lookup" path and let the linear layers do the cheap bulk of the work.

KDA is the linear layer from the Kimi Linear paper (Kimi Team, submitted 2025-10-30), which describes it as extending Gated DeltaNet with finer-grained gating. That paper used a 3:1 ratio of KDA to MLA layers and reported up to 75% less KV cache and up to 6x faster decoding at 1M tokens. Alibaba's Qwen3-Next-80B-A3B uses a similar 3:1 layout of Gated DeltaNet to gated attention. Ling-3.1-flash pushes the ratio to 7:1.

The trend is simple: each generation moves more layers to constant-memory linear attention and relies on fewer, wider full-attention layers for exact recall. Whether 7:1 loses recall quality relative to 3:1 is an empirical question the community will answer on long-context retrieval tests, not one that config arithmetic can settle.

How big is the cache per token, exactly?

Here is the calculation, using the config values. Assumptions are stated because they matter:

MLA layers store a compressed latent, kv_lora_rank + qk_rope_head_dim = 512 + 64 = 576 elements per token per layer. This is how latent-attention kernels in vLLM and SGLang cache it. A serving stack that expands keys and values would use far more.
bf16 cache, 2 bytes per element. An FP8 cache would halve it.
Each KDA layer keeps one 128 x 128 state per head across 64 heads, in fp32. This is my reading of the layer design (head_dim 128, num_attention_heads 64), not a number from the model card. Small convolution state (kernel size 4) is ignored.

# Cache arithmetic from Ling-3.1-flash config.json (inclusionAI/Ling-3.1-flash)
LAYERS, MLA, KDA = 72, 9, 63
LATENT = 512 + 64                       # kv_lora_rank + qk_rope_head_dim
BYTES = 2                               # bf16 cache

per_token_hybrid = MLA * LATENT * BYTES          # 10,368 bytes
per_token_all_mla = LAYERS * LATENT * BYTES      # 82,944 bytes (hypothetical)
kda_state = KDA * 64 * 128 * 128 * 4             # fp32, 264.2 MB, constant

def cache_gb(ctx, hybrid=True):
    if hybrid:
        return (per_token_hybrid * ctx + kda_state) / 1e9
    return per_token_all_mla * ctx / 1e9

for ctx in (32_768, 131_072, 262_144, 1_000_000):
    print(ctx, round(cache_gb(ctx), 2), round(cache_gb(ctx, False), 2))
# 32768     0.60   2.72
# 131072    1.62  10.87
# 262144    2.98  21.74
# 1000000  10.63  82.94

Three things fall out of the numbers:

At 1M tokens the hybrid layout needs about 10.6 GB per sequence versus 82.9 GB, a 7.8x reduction. That is slightly less than the naive 8x (72/9) because of the constant KDA state.
The constant state is not free. At 264 MB it equals the MLA cache of about 25,500 tokens. Below that length the linear layers cost more than they save; above it they win, and the gap widens linearly.
For scale, plain multi-head attention would be absurd. The config lists 64 key-value heads with a 192-wide key and 128-wide value. If all 72 layers cached full keys and values in bf16, that would be about 2.95 MB per token, or 2.9 TB at 1M tokens. Nobody serves that; it shows why latent attention was already a big step before the hybrid.

What does that mean for serving?

The vLLM recipe on the card uses tensor parallel 8, expert parallel, prefix caching, --gpu-memory-utilization 0.85 and MTP with 3 speculative tokens. The SGLang recipe targets 8 x H200 with TP8 and EP8, and 4 x GB300 with TP4 and EP4. Take 8 x 141 GB at 0.85 utilization and subtract the 573.8 GB of weights: about 385 GB is left for cache. That ignores activations, CUDA graphs, fragmentation and the MTP layer, so treat it as an upper bound.

Context	Hybrid layout	All-latent layout (hypothetical)
128K	237 sequences	35
262K	129	17
1M	36	4

This is illustrative arithmetic, not a measured throughput. Real concurrency depends on how many requests actually reach full length (most do not), on prefix-cache hits and on batch scheduling. But the direction is robust: a long-running agent that compacts its context every few hours holds many more live sessions per node with the hybrid layout.

Notice the other side of the ledger: the FP8 weights alone need 8 high-memory GPUs in the official recipe. The cache stopped being the dominant memory item at long context; the weights took over. That is why quantization became the next battleground.

How fast did the community react?

Searching the Hugging Face model API on 2026-10-19 for "Ling-3.1-flash" returns 13 repositories created between 2026-10-14 and 2026-10-17 besides the official one. Formats include GGUF builds (for example bartowski/Ling-3.1-flash-GGUF), EXL3 builds such as turboderp/Ling-3.1-flash-exl3 and NVFP4 variants. Third-party coverage (for example AlphaSignal) describes a mixed MXFP4 and block-FP8 recipe that cuts weights from 534 GB to 292 GB for 4-GPU Hopper serving. I could not find numbers on accuracy retention for that recipe, so treat the 292 GB figure as a size claim only.

Two practical cautions for anyone downloading community quants of a hybrid model:

Quantizers must understand the KDA layers. Linear-attention state and gates are sensitive to precision. The official config keeps router, gate and several attention projections out of FP8 conversion (see modules_to_not_convert). A quant that ignores that list can degrade quality in ways a short perplexity run will not show.
Run a long-context retrieval test on your own data before trusting any quant. Short benchmarks do not exercise the 9 full-attention layers that carry exact recall.

How should I decide whether a hybrid fits my workload?

A short checklist:

Context profile. If your p95 request is under 25K tokens, the constant KDA state eats the savings and a conventional model may be simpler. Long agent sessions and document pipelines are where hybrids pay off.
Recall sensitivity. Tasks that quote exact strings from far back (code references, contract clauses) lean on the full-attention layers. Test them.
Engine support. The vLLM recipe needs a dedicated branch and --mamba-cache-mode align, which tells you the linear state is managed like recurrent-model state. Prefix caching behaves differently for state-based layers than for KV caches. Verify cache hit rates in your own traffic.
Hardware budget. Count weights first (573.8 GB FP8), then cache, then headroom.

FAQ

What is Ling-3.1-flash?
An MIT-licensed hybrid-linear mixture-of-experts language model with 560B total and 25B active parameters, published on Hugging Face under the inclusionAI organization. Its repository was created on 2026-10-13.

What does a 7:1 hybrid ratio mean?
For every 8 layers, 7 use linear attention (KDA) with constant memory per sequence and 1 uses gated MLA with a cache that grows with context. Across 72 layers that is 63 KDA and 9 MLA.

How much KV cache does it need at 1M tokens?
From the config, about 10.4 KB per token for the 9 MLA layers, so roughly 10.4 GB at 1M tokens in bf16, plus a constant linear-attention state of about 0.26 GB under the assumptions listed above. The all-latent comparison (about 83 GB) is hypothetical, not a released model.

Can it run on a single GPU?
Not at the official FP8 size of 573.8 GB. The card's recipes use 8 x H200 or 4 x GB300. Community quantizations reduce the footprint, but check accuracy on your tasks.

Is the 1M-token window confirmed?
The model card states a 1M-token context window, while config.json lists 262,144 positions. Confirm the effective limit in your serving engine and test retrieval at the lengths you plan to use.

Does linear attention hurt accuracy?
It can on exact recall, which is why hybrids keep a few full-attention layers. Whether 7:1 is enough is a question for long-context benchmarks on independent hardware.

Decision AI Is Becoming Its Own Category, and the Metrics Are Not Ready

ai maya — Mon, 12 Oct 2026 10:35:43 +0000

TL;DR

Most of the attention in AI goes to models that generate: text, code, images. Quietly, a different class of systems is growing, ones whose job is to judge. Given a claim, a document, or a candidate answer, they decide yes, no, or which option. This decision and verification layer is starting to look like its own category, and its evaluation is lagging behind the hype.

From writing to judging

Generative models produce content. Decision systems produce verdicts. The two are not the same problem.

A writer is rewarded for fluency and plausibility. A judge is rewarded for being right and calibrated, and for saying how sure it is. You can be a brilliant writer and a terrible judge, which is exactly why a model that drafts beautifully can still approve a wrong answer with full confidence.

As generated content floods every channel, the bottleneck shifts. The scarce thing is not another draft. It is a reliable verdict on the drafts you already have.

Why this is a category, not a feature

Three signs suggest decision and verification is becoming a standalone category rather than a checkbox on a generative product.

Capital is moving. Large rounds are going to companies positioning around verification and decisioning, not only generation.
The interface is different. A decision API returns a label and a confidence, not a paragraph. That changes how you integrate, cache, and bill it.
The failure modes are different. A confident wrong verdict is worse than a vague draft, because downstream systems act on verdicts.

When the economics, the interface, and the failure modes all differ, you are looking at a category.

The metrics are not ready

Here is the uncomfortable part. Evaluating a judge is harder than evaluating a writer, and the field has not caught up.

Accuracy is not enough. A judge that is right on easy cases and overconfident on hard ones can post a high average and still be dangerous where it matters.
Calibration is underreported. If a system says 90% and is right 60% of the time, the number is worse than useless, because people trust it.
The label question must match the operational question. A model evaluated on a slightly different question than the one it is deployed to answer will look great and fail in production.
Saturation hides weakness. When everything scores near the top, the benchmark has stopped discriminating, and the ranking is noise.

A useful decision benchmark reports accuracy and calibration, keeps the evaluation question identical to the deployment question, and shows that it can still separate strong systems from weak ones.

What to watch

If you are tracking this space, watch for benchmarks that publish calibration alongside accuracy, that resist saturation, and that are run by a third party on questions which resist memorization. Those are the ones that will survive contact with production.

FAQ

Is a decision model just a classifier? In spirit, yes, but at the scale and generality of modern language systems, with natural language inputs and open domains.

Why not just ask a generative model to judge? You can, but you then need to measure whether its verdicts are accurate and calibrated, which is a separate evaluation.

What single metric should I trust? None alone. Pair accuracy with calibration, and confirm the eval question matches your deployment.

Why does saturation matter? A benchmark where everyone scores near the top no longer ranks anything. Treat saturated boards with suspicion.

Closing

Generation got the spotlight first. Judgment is the harder half, and its scoreboards are still being built. The teams that win here will be the ones who measure calibration as seriously as accuracy.

When the attacker is an agent: a defender's field guide to autonomous AI intrusions in 2026

ai maya — Tue, 06 Oct 2026 19:10:51 +0000

TL;DR

In 2026 the autonomous agent stopped being a demo and became an attacker. Public incident write ups now describe agents that scan, pick targets, launch exploits, move laterally, and exfiltrate data across thousands of actions with one human command at the start. The defensive answer is not a new product. It is a set of boring, testable controls: cap what every agent (yours and theirs) is allowed to touch, treat every tool call as an untrusted request, log at the action level, and rehearse detection at machine speed. This post maps the threat to the OWASP Top 10 for LLM Applications (2026) and the OWASP Top 10 for Agentic Applications, and gives a concrete control checklist you can start on this week.

What actually changed in 2026

For three years "AI in cyber" mostly meant better phishing copy and faster malware triage. The qualitative shift this year is that the loop closed. An operator can now wire a capable model into an open agent framework in autonomous execution mode, point it at a target discovery service, and issue a single instruction. The agent then handles reconnaissance, exploit selection, execution, and follow on actions without step by step direction.

The numbers in public reporting are what make this a defender problem rather than a research curiosity. One documented campaign had a single command drive an agent to scan, select, and attack on the order of hundreds of internet facing targets on its own, with confirmed exfiltration from edge appliances. Another was described as a swarm of short lived sandboxes executing many thousands of individual actions. Separate reporting through 2026 traced autonomous agents exploiting continuous integration pipelines across several large organizations.

The important part for a defender is the operating model, not any single tool:

Volume. Thousands of actions per campaign, parallelized across disposable sandboxes. Your rate based alarms were tuned for humans.
Persistence. The agent does not get tired, does not lose the thread overnight, and retries with variation.
Adaptation. When one path fails the agent reformulates and tries another, so static signatures age fast.

None of this requires novel exploits. It mostly industrializes known ones.

How does autonomous attack map to the OWASP Top 10?

The OWASP Top 10 for LLM Applications 2026 edition is the first built on incident evidence rather than opinion alone, drawing on a large corpus of real world AI security incidents weighted against a community vote. The single most telling change: Excessive Agency moved from sixth to third place. Loss is now concentrating in autonomy.

The 2026 LLM list, in order, is:

Prompt Injection
Sensitive Information Disclosure
Excessive Agency
Supply Chain
Data and Model Poisoning
Unbounded Consumption
Misinformation
Hidden Context Exposure
Vector and Embedding Weaknesses
Improper Output Handling

A separate OWASP Top 10 for Agentic Applications covers autonomous systems under their own numbering, naming risks like goal hijacking, tool misuse, privilege abuse, memory poisoning, and rogue agents.

Two of these deserve a defender's full attention because they describe how your own agents become the entry point:

Prompt Injection (LLM01). If you run agents, your attack surface is now any text your agent reads. A support ticket, a scraped web page, a code comment, or a file name can carry instructions. The agent does not distinguish data from command unless you force it to. This is the mechanism behind goal hijacking in the agentic list.

Excessive Agency (LLM03). The damage multiplier. An injected instruction only matters as much as the permissions behind the agent. An agent with a broad API token, shell access, or write access to a repository turns a text trick into real action.

A defender's control checklist

The good news is that the controls are mostly classic security hygiene, re applied to a new principal type. Treat every agent, yours and the adversary's, as a fast, tireless, semi trusted user.

1. Constrain agency before you constrain anything else

Give each agent the narrowest possible set of tools and scopes. If an agent only needs to read three tables, it gets a read only credential to three tables. This is the highest leverage control because it directly caps the blast radius of LLM03.

# Illustrative agent capability manifest
agent: support-triage
tools:
  - name: ticket.read
    scope: "queue:inbound"
  - name: kb.search
    scope: "public"
deny:
  - shell.exec
  - repo.write
  - secrets.read
limits:
  actions_per_minute: 20
  egress_domains: ["api.internal.example"]

2. Treat every tool call as an untrusted request

Put a policy layer between the model and any tool. The model proposes an action; a deterministic checker decides whether it is allowed in this context. Never let free text flow straight into a shell, a query, or an HTTP client.

# Illustrative broker: model proposes, policy disposes
def execute(action, context):
    if action.tool not in context.allowed_tools:
        return deny("tool not in manifest")
    if action.tool == "db.query" and not is_read_only(action.sql):
        return deny("write attempted on read-only scope")
    if action.egress_host not in context.allowed_hosts:
        return deny("egress to unlisted host")
    return run(action)

3. Separate data from instructions

Mark untrusted content so the model and the broker can tell apart what is being discussed from what is being commanded. Structurally fence retrieved documents, and strip or neutralize instruction like patterns in ingested text. You will not catch every injection, so this is defense in depth, not a cure.

4. Log at the action level, not the request level

A human session produces tens of actions. An agentic intrusion produces thousands. Your telemetry has to record each tool invocation with its arguments, the proposing principal, and the decision. Request level logs will show a single login and miss the campaign.

5. Rate limit for machines, not humans

Unbounded Consumption (LLM06) is on the list for a reason. Caps on actions per minute, total egress volume, and concurrent sessions per credential turn a swarm into a trickle, and they double as a cost control for your own agents.

6. Rehearse detection at machine speed

If attackers run thousands of actions in minutes, a detection pipeline that alerts a human in an hour has already lost. The practical move is autonomous or semi autonomous response for well understood cases: automatic credential revocation on anomalous egress, automatic sandbox quarantine, with a human in the loop for anything novel. Several teams now call this standard layer agentic defense.

7. Lock down the pipeline

Continuous integration systems were an early target because they combine high privilege with lots of machine readable configuration. Scope pipeline tokens to the minimum, require review for workflow file changes, and pin dependencies. This is the Supply Chain entry (LLM04) with teeth.

The regulatory signal is already here

This is not a hypothetical. In 2026 a national financial regulator in Asia convened sector chief information security officers specifically over AI enabled attacks, launched on site inspections at banks, and distributed response guidelines. Notably, one regulator approved limited immunity for IT outages that occur during AI security testing and patching, an explicit nudge to test harder rather than freeze. When regulators start protecting the act of testing, the threat has crossed from theory to operations.

Frequently asked questions

Q: Is this just hype? Are agents really running full intrusions?
Public incident reporting in 2026 describes agents handling scan, select, exploit, and exfiltrate with a single starting command across many targets. The capability does not require novel exploits; it industrializes known ones. Treat it as real and plan accordingly.

Q: Do I need to buy an "AI security" product?
Mostly no. The highest leverage controls are least privilege for agents, a policy broker in front of tools, action level logging, and machine speed rate limits. These are classic controls applied to a new kind of principal.

Q: My company runs its own agents. Does that make me a bigger target?
It changes your attack surface. Any untrusted text your agent reads is a potential injection vector, and any permission your agent holds is a potential blast radius. Your own agents are the control point: constrain their agency first.

Q: What is the difference between the OWASP LLM Top 10 and the Agentic Top 10?
The LLM Top 10 covers model backed applications in general, led in 2026 by Prompt Injection with Excessive Agency rising to third. The Agentic Top 10 is a separate list for autonomous systems, naming goal hijacking, tool misuse, privilege abuse, memory poisoning, and rogue agents.

Q: Where should a small team start with limited time?
Three things this week: inventory every agent and the credentials it holds, cut each credential to least privilege, and turn on action level logging. Everything else builds on knowing what your agents can do and seeing what they did.

Q: Can defensive AI keep up with offensive AI?
At machine speed, humans cannot be the first responder for high volume cases. The workable pattern is autonomous response for understood scenarios (revoke, quarantine, throttle) with humans reserved for the novel. Rehearse it before you need it.

DEV Community: ai maya

Anthropic's FlashProve: what 72x overhead and a 1,024-token cap mean for proving which model answered you

TL;DR

What did Anthropic actually release?

How fast is it compared with the previous system?

What does proving cost per token, and per request?

Is checking a proof cheap enough to do at scale?

What does a proof not tell you?

Which uses fit today?

Frequently asked questions

Sources

Further reading

DeepSeek's 160,000 Huawei Ascend chips: what 10,000 chips for weeks actually buys, and why the reports disagree on training

TL;DR

What has actually been reported about the Ulanqab cluster?

What does the Ascend 950DT specification say?

How much pre-training compute is 10,000 chips for a few weeks?

Does the memory fit?

How often does a 10,000-chip job get interrupted?

What should you take from this if you build or buy AI infrastructure?

Reproduce the numbers

FAQ

Sources

Further reading

Marvell is buying Lumentum for $115B: what the optical bet means for AI cluster power, and what a fixed 4.5x exchange ratio really locks in

TL;DR

What exactly did Marvell and Lumentum announce?

Why would a chip company pay this much for a laser company?

How much power does co-packaged optics actually save?

If CPO needs four times fewer lasers, why buy a laser company?

What does a fixed 4.5x ratio mean for how you read the news?

What should you ask a vendor before trusting a per-port power number?

FAQ

Further reading

Sources

Kimi K2 closes in December: what moving to K3.1 Pro changes, and what the default reasoning effort costs

TL;DR

What exactly did Moonshot announce?

How often does Moonshot retire models?

What breaks when you only change the model name?

Is reasoning effort really a cost setting?

What should a migration test look like?

What about the open weights?

FAQ

Further reading

Sources

Mistral Large 4 on your own GPUs: does 1 trillion parameters fit on 8 H200s, and what does the license allow?

TL;DR

What exactly shipped with Mistral Large 4?

How much of a "1 trillion parameter" model actually runs per token?

Does it fit on 8 H200 GPUs?

Why is the KV cache so small for a 1M-token model?

Is decoding bandwidth-bound on a node like this?

What does the license actually allow?

When does self-hosting beat the API?

FAQ

Further reading

Sources

Inference NPUs That Train: What a 325M Pretraining Run on a Rebellions ATOM-Max Shows, and What It Does Not

TL;DR

Why is this news?

What exactly was shown?

What does it not show?

FAQ

Sources

Is open-weight really 90% as good at 10% of the cost? Checking IBM's claim against list prices

TL;DR

What exactly did IBM claim?

What does one workload cost on nine price lists?

Is the cheap part really "open"?

When does owning hardware beat the API?

What would two or three racks look like?

What does this model leave out?

How should a team decide?

Reproduce it

FAQ

Is IBM's claim that open-weight models cost 10 to 25% of frontier models accurate?

Are open-weight models cheaper than closed models?

At what utilization does running your own GPUs beat paying an API?

Does prompt caching change the open versus closed comparison?