DEV Community: byeongsoo kang

Running 35B–400B LLMs on a GPU-less Cluster to Mine 10,000 Papers — and the 4 Bugs That Almost Ruined the Data

byeongsoo kang — Wed, 03 Jun 2026 05:55:34 +0000

A field report from building a CPU-only, distributed LLM pipeline for large-scale scientific literature extraction. No GPUs. A lot of quantization. And four silent data-quality bugs that taught me more than the happy path ever did.

The constraint that started it all

Our team runs an internal research cluster: a couple dozen older x86 servers, plenty of RAM, zero GPUs. The mandate was to extract structured data — effect sizes, the entity each one describes, and the direction of effect — from ~10,000 full-text research papers, so a downstream meta-analysis could pool them.

The obvious 2024-era answer is "send it to a hosted LLM API." That wasn't on the table for data-governance reasons: the corpus had to stay on-prem. So the real question became:

Can you do serious LLM extraction at the 10k-document scale with CPUs only?

Spoiler: yes — but the interesting part isn't the throughput. It's that correctness, not speed, turned out to be the hard problem. Let me walk through the architecture, then the four bugs that each silently corrupted the data in a different way.

The stack

Everything is open source and CPU-friendly:

llama.cpp serving quantized GGUF models over its OpenAI-ish HTTP endpoint. We ran a MoE model (~35B total / ~3B active, Q8) as the high-throughput workhorse on 8 nodes, and a ~400B model (Q3) on a dedicated node for the heaviest pass.
BGE-M3 (1024-dim) for embeddings, also on llama.cpp, across 8 nodes.
Qdrant as the vector DB.
Plain Python (requests + ThreadPoolExecutor) for orchestration. No Ray, no fancy scheduler — just a queue and one worker bound per node, because each llama.cpp server runs --parallel 1: on CPU, inference is memory-bandwidth bound, so one in-flight request already saturates the memory bus and batching buys little.

Each node is a dual-socket Xeon, ~36 cores total (AVX-512), no accelerator. The 35B MoE generated ~6 tokens/s per node; with 8 nodes load-balanced, a sentence took ~10s end to end and the full 14k-sentence extraction finished in a few hours.

MoE was the unlock for CPU: ~3B active parameters per token means it generates at a usable rate even without a GPU, while delivering quality far above what its ~3B active count alone would suggest.

The ~400B Q3 model was reserved for a separate, earlier abstract-level pass — a different job at a different scale, out of scope for this post — where its stronger one-shot reading paid off. On a single CPU node it ran at low single-digit tokens/s, so routing the sentence-level corpus through it was never viable; everything below is the 35B MoE.

RAG is not extraction (the distinction that bites everyone)

First, a clarification I had to make repeatedly, because it confuses people (it confused me):

RAG (embed → vector search) answers questions. You chunk text, embed it, and at query time retrieve the top-k most semantically similar passages to ground an LLM's answer. Great for "find me passages about X."
Extraction for meta-analysis needs numbers — every effect size from every paper, aggregated. That is exhaustive structured extraction, not retrieval.

A vector DB stores d = -0.45 as a text token inside an embedding. It will happily find that sentence by meaning, but it cannot compute over the number. If your goal is to pool effect sizes, embeddings are the wrong tool. You want extraction.

So the pipeline is a hybrid: a cheap mechanical pass to find candidate sentences, then an LLM to interpret them.

10k full-text papers
   │
   ├─ ① regex pre-filter  (mechanical, no understanding)
   │     keep sentences that have a number near a target-entity keyword
   │     → ~14k candidate sentences
   │
   └─ ② LLM mapping       (the judgment step)
         each sentence → {entity, metric, direction, value, measure_type}
         → structured JSON for the meta-analysis

Regex is the funnel; the LLM is the brain. Neither replaces the other.

Now the fun part.

Bug #1 — the chunker that silently deleted 79% of the data

The embedding side (the RAG corpus) had its own chunking pipeline. It looked fine. Counts looked fine. Then someone asked a simple question — "how many points are actually in the collection?" — and the numbers didn't add up: ~1M chunks generated, ~217k points in the DB.

A 78% gap. Where did 800k chunks go?

The culprit was the point ID. Each chunk got an ID derived from (paper_id, chunk_index). Reasonable — except chunk_index was reset to 0 at the start of every section:

for section in sections:
    for j, chunk in enumerate(chunk_text(section)):   # j resets per section!
        point_id = make_id(paper_id, j)               # collision: (abstract,0) == (methods,0)
        upsert(point_id, ...)

So a paper's abstract chunk-0 and its methods chunk-0 and its results chunk-0 all hashed to the same point ID. Qdrant upserts are idempotent by ID, so each new section silently overwrote the previous one. Every paper collapsed to roughly max(chunks in any single section) points.

I confirmed it by replaying the raw chunks: 27,222 chunks across a sample → only 5,672 unique (paper_id, chunk_index) pairs. 79.2% collision on the sample, closely matching the 78% gap across the full DB (the small delta is just sampling — one is a replayed subset, the other the whole collection).

The fix is a one-liner — make chunk_index a running counter across the whole paper (and derive the ID with a deterministic hash like hashlib/UUID, not Python's per-process hash(), so IDs stay stable across runs) — but the lesson isn't the fix. It's that a silent overwrite produces a database that looks completely healthy: green status, fast queries, plausible counts. Nothing errors. You only catch it if you reconcile "things I generated" against "things that landed."

Reconcile your pipeline's input count against its output count at every hop. Silent data loss doesn't throw.

Bug #2 — recursive chunking that duplicated 75% of the text

While fixing #1, I re-ran the chunker on a fresh corpus and a sample paper produced 7,588 chunks, of which only 1,897 were unique — 75% duplicates.

The XML parser walked sections like this:

for sec in body.findall(".//sec"):          # ALL <sec>, including nested ones
    paragraphs = sec.findall(".//p")          # ALL <p>, recursively

In journal XML, sections nest. A parent <sec> contains child <sec>s. .//p is recursive, so the parent emitted all of its children's paragraphs — and then each child <sec> was visited separately and emitted them again. Deeply nested papers (a conference-proceedings document with 600 sub-sections was the worst) exploded.

Fix: take direct-child paragraphs only (sec.findall("p")), plus a within-paper dedup as a safety net. Chunks dropped to the honest count, embedding time dropped with it.

.// in XPath is a footgun when your tree is recursive and you also iterate the tree.

Bug #3 — the reasoning model that never stops thinking

Onto the extraction LLM — the 35B MoE workhorse. It's a reasoning model that emits a <think>…</think> block before its answer. The first run capped generation at 512 tokens with stop=["\n\n"]. Result: 0% parse rate. The \n\n stop fired inside the thinking block, truncating mid-thought; no JSON ever appeared.

OK, remove the bad stop, give it room. Bump to 1024 tokens. Now ~42% parse — better, but a third of outputs were still <think> with no </think>: the model hit the token cap still reasoning.

So give it more room. 2048 tokens, 600-second timeout, quality-first. I ran a single hard sentence as a test. It generated 6,144 characters in 269 seconds and still hadn't closed the think block — it was literally mid-sentence, "Let's draft the JSON:", when it ran out of budget. At that rate, 14k sentences would take ~5 days and still fail on the hard ones.

The model wasn't slow. It was non-terminating: on ambiguous inputs it reasoned in circles and never committed to an answer. More tokens didn't help; it just thought more.

The fix is a known trick for reasoning models in raw-completion mode: pre-close the think block in the prompt so the model skips deliberation and answers directly:

prompt = f"...<|im_start|>assistant\n<think>\n\n</think>\n\n"
#                                  ^ empty, pre-closed → no open-ended reasoning

Latency dropped from "minutes, maybe never" to ~10 seconds, deterministically. The whole 14k run finished in hours, at 99.96% parse.

A reasoning model with no thinking budget is a liability for bulk structured output. If you don't need the chain-of-thought, close it.

Bug #4 — empty outputs, and a one-character fix

No-think mode had its own quirk: on ~14% of the harder sentences, the model returned completely empty output. Not bad JSON — nothing. Deterministic (temperature 0), so retries reproduced the emptiness exactly.

The model, forced to answer immediately, was "blanking" on sentences it found ambiguous. The fix was almost insultingly small: seed the assistant turn with an opening bracket so the model is already inside a JSON array and must continue it:

prompt = f"...<|im_start|>assistant\n<think>\n\n</think>\n\n["
#                                                          ^ forces JSON to start

(You then prepend the [ back when parsing, since the completion only returns what comes after the prompt.) This recovered 298 of 301 empties → 99.86% parse on the hard subset.

When a model can output "nothing," constrain the output space so "nothing" isn't reachable.

The bug that wasn't a bug: precision vs. recall

The last lesson is subtler. The first extraction run mapped a number whenever a sentence had a number near a target-entity keyword. The audit found ~50% of the mapped "effect sizes" weren't the target effect at all — they were regression-predictor t-values (age, sex, medication), correlations with secondary task scores, even positional coordinates (x = -28) the regex had grabbed as if they were measurements.

That noise produced a confident-but-spurious aggregate signal. Garbage in, significant garbage out.

The fix had two halves, and getting it wrong in an instructive way:

Filter at the source. Only feed the LLM sentences from papers that are actually about the topic, with a target entity and a real effect statistic. (First attempt: filter at the sentence level — too aggressive, it dropped real effects whose context lived in the neighboring sentence. Second attempt: filter at the paper level — recovered ~1,100 real effects the sentence filter had thrown away.)
Make the prompt relationship-aware. Tell the model what kind of number counts and what to reject (predictors, task-performance correlations, coordinates, cluster stats), with a worked rejection example.

But I over-corrected: my first sharpened prompt rejected so aggressively it returned [] for valid patient-vs-control effects too (1/15 on a sanity sample). The filter and the prompt were fighting — the filter guaranteed the paper was on-topic, but the prompt still demanded an explicit topic keyword in the sentence. Once I told the prompt "you can trust that this sentence is from an on-topic paper; extract the entity's effect and only reject these specific noise types," recall snapped back (9/15) with zero coordinate leakage.

Precision in the mapped set went from ~49% to ~66% at the sentence level; at the paper level — meaning every paper that contributes an effect is genuinely on-topic — it was 100%. Total entries dropped from ~4,900 to ~1,700, almost all of it noise. The residual ~34% sentence-level noise isn't pooled blind, but be precise about what catches it: the load-bearing filter downstream is entity normalization against a controlled vocabulary — off-target entities (age, sex, medication) get dropped there — backed by a validation gate. (Stratifying by measure type and dedup are cleanup, not misclassification removal: a predictor t-value mislabeled as a target effect sails right through those.) The mapping's job is to maximize signal and flag; the controlled-vocabulary step is where the final noise is supposed to die.

The most dangerous extraction failure isn't a crash or a low parse rate. It's clean-looking data that's confidently wrong. Audit what your pipeline includes, not just what it drops.

Takeaways

CPU + quantized MoE is genuinely viable for 10k-document LLM work: at ~6 tok/s/node across 8 nodes, the full 14k-sentence extraction finished in a few hours. The bottleneck was never compute — it was correctness.
Reconcile counts at every hop. Both data-loss bugs (#1, #2) were invisible from status dashboards; only input-vs-output reconciliation surfaced them.
Reasoning models need a thinking budget — or none at all — for bulk structured output. Pre-closing <think> and seeding the output bracket turned a 5-day non-terminating job into a few-hour deterministic one.
For extraction, precision is the silent killer. A permissive regex + a literal LLM will hand you a statistically significant result built on coordinates and covariates. Filter at the right granularity, and tell the model what to reject.

None of these are exotic. They're the unglamorous correctness work that sits between "the demo runs" and "the numbers are trustworthy" — which, for anything feeding a real analysis, is the whole job.

This pipeline powered the large-scale literature extraction behind our chronic-stress scoping-review preprint (Research Square).

Tools used: llama.cpp, BGE-M3, Qdrant, Python. All on-prem, all CPU.

A MOGONET-Style Multi-Omics Biomarker Pipeline: Why a Near-Random Graph Net Still Earns Its Place

byeongsoo kang — Wed, 03 Jun 2026 05:24:09 +0000

TL;DR (Quick Answer)

This is an honest engineering write-up of a MOGONET-style multi-omics consensus biomarker pipeline built as an internal R&D project at sysofti.

The headline — on a small synthetic cohort (n=30), the graph network alone scores near-random in leak-free 5-fold cross-validation (AUC 0.53 ± 0.16). Yet as one voter in a 5-evidence consensus, the top-10 ranking is 90% real markers (9 of 10 are known periodontitis genes).
The lesson — a single model that looks weak in honest evaluation can still be a useful voter. That contrast is the whole point of the consensus design, and we show it with data.
What it is — per-omics Graph Convolutional Networks (GCN) over a sample-similarity graph, attention-fused, contributing to a consensus score alongside differential-expression hubs, Random Forest, a DNN, and co-expression modules.
What it is *not* — the official MOGONET. We dropped the original's VCDN fusion for attention fusion. Call it "MOGONET-based." All numbers are from synthetic data with embedded ground-truth markers — code validation, not a clinical claim.

If you're implementing multi-omics integration, the parts you can't get from the paper are below: the real results, the leakage-aware evaluation, and the bugs we hit.

What MOGONET Is (the One-Line Mental Model)

MOGONET (Multi-Omics Graph cOnvolutional NETwork) learns a separate GCN per omics view on a sample-similarity graph (patients as nodes, edges by feature similarity), then fuses the per-view embeddings for classification and biomarker discovery. Reference: Wang et al. 2021, Nature Communications 12:3445; the GCN itself is Kipf & Welling 2017.

Mental model: "build one graph net per omics layer, let each form an opinion, then combine those opinions."

What We Simplified — and Why

The original MOGONET fuses views with a View Correlation Discovery Network (VCDN). We replaced it with attention-weighted fusion:

Why — with tiny cohorts (tens of samples), VCDN's extra parameters were a liability; attention fusion gave a simpler intermediate-fusion scheme that still up-weights the more informative omics per sample.
The tradeoff — we lose the explicit cross-view correlation modeling that is part of MOGONET's original contribution. So this is honestly MOGONET-based, not a reimplementation. The source docstring says as much: "Simplified implementation of MOGONET."

Architecture

Input: X_views = [omics1 (n×p1), omics2 (n×p2), ...]   (n = common samples)
  └─ per-view StandardScaler
  └─ per-view k-NN (cosine) adjacency  (n×n)
ViewEncoder (per omics):  GraphConv(p→128) → BN → ReLU → GraphConv(128→64)
  → view embedding (n×64)
Attention fusion:  softmax(Linear(64→1)) over views → weighted sum (n×64)
Classifier:  Linear(64→32) → ReLU → Linear(32→n_classes)

class GraphConvLayer(nn.Module):
    def __init__(self, in_features, out_features):
        super().__init__()
        self.linear = nn.Linear(in_features, out_features)
    def forward(self, x, adj):
        return torch.mm(adj, self.linear(x))   # propagate over the sample graph

class MOGONET(nn.Module):
    def __init__(self, input_dims, hidden_dim=128, latent_dim=64, n_classes=2):
        super().__init__()
        self.encoders = nn.ModuleList([ViewEncoder(d, hidden_dim, latent_dim) for d in input_dims])
        self.attention = nn.Linear(latent_dim, 1)
        self.classifier = nn.Sequential(nn.Linear(latent_dim, 32), nn.ReLU(), nn.Linear(32, n_classes))
    def forward(self, views, adjs):
        embeddings = [enc(x, adj) for enc, x, adj in zip(self.encoders, views, adjs)]
        stacked = torch.stack(embeddings, dim=0)                       # n_views × n × latent
        attn = F.softmax(self.attention(stacked).squeeze(-1), dim=0)   # per-view, per-sample
        fused = (stacked * attn.unsqueeze(-1)).sum(dim=0)              # n × latent
        return self.classifier(fused)

Sample-similarity graph — k-NN (cosine), no self-loops on purpose (see below):

def build_adjacency(X, k=5):
    sim = cosine_similarity(X)
    adj = np.zeros_like(sim)
    for i in range(len(sim)):
        top_k = np.argsort(sim[i])[-k-1:-1]      # top-k neighbours, excluding self
        adj[i, top_k] = sim[i, top_k]
        adj[top_k, i] = sim[top_k, i]            # symmetrize
    row_sum = adj.sum(axis=1, keepdims=True); row_sum[row_sum == 0] = 1   # guard zero-sum rows
    return adj / row_sum

The Engineering Decisions That Mattered

Sample-node graph, not feature graph. Nodes are patients; edges are patient-patient similarity. Same-group patients cluster, so the GCN smooths group signal.
No self-loops — on purpose. Standard GCN uses Ahat = A + I so a node keeps its own features. We deliberately omit the self-loop so each node's representation is built purely from its sample-neighborhood, pushing the model toward group structure rather than individual raw features. It is a tradeoff (you give up the node's own signal each layer), and we flag it as a choice, not an accident.
Per-view scaling + common-sample intersection. Each omics standardized independently; only samples present in all views are used.
Consensus over a single model. MOGONET is one of five evidence sources by design — Hub (DE+PPI), ML (Random Forest), DL (DNN), WGCNA co-expression, and MOGONET — with a multi-evidence bonus:

avg_score = sum(scores.values()) / max(len(scores), 1)
composite = avg_score * (1 + 0.3 * (n_sources - 1))   # reward agreement across sources

As the results show, this design choice is what makes the pipeline useful despite any single model being weak.

Results (Synthetic Data, with Ground Truth)

We validate on a synthetic periodontitis case-control set (3 omics — transcriptomics 500, proteomics 200, metabolomics 100 features × 30 samples, 15 disease / 15 control, seed-fixed) with known biomarkers deliberately embedded: up-regulated inflammatory genes (MMP8, MMP9, IL1B, IL6, TNF, RANKL, CTSK, TLR4 …) and down-regulated bone-formation genes (COL1A1, RUNX2, SP7, BGLAP, OPG …). Embedding known markers gives ground truth — you can check whether the pipeline recovers them, which is impossible on a real cohort.

Note on sources: the pipeline defines five evidence sources, but in this run WGCNA returned no co-expression hubs, so four sources actually contributed (Hub, ML, DL, MOGONET).

The consensus ranking surfaces real markers

Of 793 candidate features, the top-30 consensus included 13 of the 25 embedded markers. The ranking is strikingly clean at the top:

Rank	Gene	Composite	Sources	Known marker
1	MMP8	1.888	4	★
2	COL1A1	1.212	3	★
3	MMP9	1.020	4	★
4	IL6	1.000	1	★
5	IL1B	0.900	4	★
6	METAB_0031	0.866	1	—
7	TLR4	0.856	3	★
8	RANKL	0.838	3	★
9	CTSK	0.803	3	★
10	SP7	0.678	3	★
11	MYD88	0.672	3	★

Precision@10 = 0.90 — 9 of the top 10 are known markers (only METAB_0031 is not).
Recall@10 = 0.36, Recall@20 = 0.52 (9 then 13 of 25 known markers); it plateaus by 20 because a few embedded markers were given weak synthetic signal (e.g. TNF, fold-change ≈ 1.1).

More evidence = more trustworthy

Breaking the top-30 down by which sources agreed makes the consensus logic concrete:

4 sources → 3 genes, all 3 known (100%): MMP8, MMP9, IL1B.
3 sources → 17 genes, 9 known.
2 sources (DL + MOGONET) → 8 genes, 0 known — pure noise.
1 source → 2 genes, 1 known.

The signal lives where independent methods agree. A gene flagged by four sources was always real here; genes flagged by only two were not.

The honest part: the graph net alone is near-random

We cross-validated MOGONET as a standalone classifier, rebuilding the sample graph from training folds only to avoid leakage:

MOGONET 5-fold CV AUC = 0.53 ± 0.16 (folds: 0.44, 0.44, 0.78, 0.33, 0.67)

That is barely above chance. With n=30 (six test samples per fold) and a transductive sample-graph model, a single GCN simply cannot generalize here — and its training AUC near 1.0 is mostly the leakage and the injected signal talking. This is exactly why MOGONET is wired in as one voter, not the decision-maker. The consensus result above is strong because it doesn't trust any single model, including this one.

Honest Limitations

Simplified model. No VCDN fusion — attention instead. "MOGONET-based," not a reimplementation.
MOGONET is a weak standalone classifier here (CV AUC 0.53). Useful only in aggregate. It also scores all 793 features, so its solo discriminative power is low.
Synthetic, small (n=30). Results validate the code's ability to recover injected signal — not clinical performance. External cohorts are required for any real claim.
Single run (seed 42). Known markers are stable at the top; the unnamed GENE_xxxx candidates shuffle on re-runs.
Self-loop omission is a design choice with a cost — worth A/B testing against the standard A + I formulation.
Feature importance is an approximation (first-layer weight magnitude), not a gradient-based attribution.

What Broke Along the Way (Real Notes)

Zero-sum adjacency → NaN. If a sample's k-NN cosine similarities summed to zero, row-normalization divided by zero and propagated NaNs. Fixed with a row_sum[row_sum == 0] = 1 guard.
Attribute-name mismatches (fixed twice). Pulling feature importance broke on AttributeError when the sklearn-wrapper conventions clashed with the nn.Module attribute names (view_encoders → encoders, model → model_).
Common-sample collapse. When omics measured different sample sets, the intersection shrank fast. Added a "≥6 common samples" guard that skips gracefully instead of crashing.
MOGONET scores everything. It assigns weight to all 793 features, so it appeared in all top-30 entries — the multi-evidence bonus is what keeps it honest.

What We'd Improve Next

Report consensus performance under the same leak-free CV, not just MOGONET's.
A/B test self-loops (Ahat = A + I).
Gradient-based attribution (Integrated Gradients) instead of first-layer weights.
Add VCDN fusion and compare head-to-head with attention fusion.
External multi-omics cohort for real-world validation.

FAQ

Q: Is this the official MOGONET implementation?

No — a simplified, MOGONET-based design: per-omics GCN with attention fusion, without the original's VCDN view-correlation network.

Q: If MOGONET's CV AUC is only 0.53, why keep it?

Because it is one voter in a five-source consensus, not the classifier. Single models overfit small cohorts; consensus rewards agreement across independent methods, and that ranking recovered known markers at 90% precision in the top 10. A weak voter still adds signal when combined.

Q: Why validate on synthetic data?

Embedded known markers give ground truth, so you can measure recovery (recall/precision) — impossible on a real cohort where the answer is unknown. It validates the code, not clinical utility.

Q: Why omit GCN self-loops?

Intentional: without a self-loop, each node's representation comes purely from its sample-neighborhood, pushing the model toward group structure rather than individual features. It is a tradeoff worth A/B testing, not a universal recommendation.

Q: Can I use this on my own multi-omics data?

Yes — the classifier is sklearn-compatible (fit/predict/predict_proba). Build the sample graph from training data only to avoid leakage, and don't over-read AUC on small cohorts.

Resources

Reference implementation (clean, standalone, MIT): github.com/shoo99/mogonet_lite
Original paper: Wang T. et al. (2021), MOGONET integrates multi-omics data via graph convolutional networks for biomarker discovery, Nat Commun 12:3445.

Running a 35B MoE (Qwen3.6-35B-A3B) on 2x GTX 1080 Ti in 2026 — Real Benchmarks, and Does the Second GPU Actually Help?

byeongsoo kang — Wed, 03 Jun 2026 05:18:42 +0000

TL;DR (Quick Answer)

I actually ran Qwen3.6-35B-A3B — a 35B-parameter mixture-of-experts model (only 3B active per token) — on a pair of 8-year-old GTX 1080 Ti cards (22 GB combined). Real, measured numbers:

Generation speed: ~20 tokens/sec on 2× 1080 Ti (IQ4_XS quant), stable across runs (19.4 / 21.4 / 20.0).
Single GPU: ~16.8 tok/s. CPU-only (i9-14900K): ~17.1 tok/s. The second 1080 Ti buys only ~20% over one card — and, the kicker, the GPUs barely beat a modern CPU here (~+18%), because the MoE experts stay mmap'd in CPU RAM regardless. See the honest update below.
It only "fits" because of the MoE + CPU-mmap trick. ~13 GB of the model sits on the two GPUs; ~18 GB of expert weights are mmap'd from CPU RAM, and only the active 3B runs each token.
Quant matters for 22 GB: the default qwen3.6:35b-a3b tag is 23.9 GB and spills to CPU. You want ≤ IQ4_XS (~17.7 GB) to keep it (mostly) on the GPUs.

Bottom line: a 35B MoE is genuinely usable on this box in 2026 — but the honest workhorse turned out to be the i9-14900K CPU; the used 1080 Ti cards add only ~18%. Pick a sparse MoE and a quant that mostly fits — and know that for an offload-heavy MoE, a fast CPU + RAM bandwidth matters as much as the GPUs.

The setup (and one gotcha)

GPUs: 2× NVIDIA GeForce GTX 1080 Ti (11 GB each, 22 GB total), Pascal, compute capability 6.1.
Driver: 581.57 (Windows host, used via WSL2 passthrough). This matters — recent Ollama bundles CUDA 13, which refuses drivers older than 570. On the older 560 driver it silently fell back to CPU (total_vram=0). Updating to 581 fixed it.
Ollama: v0.30.2. Interesting detail: its cuda_v13 build skips Pascal ("compute capability not in compiled architectures", cc 6.1), so it auto-falls back to the bundled cuda_v12 build to use the 1080 Ti. Good to know if you're on old hardware.

Why a "35B" model runs on old cards at all

Qwen3.6-35B-A3B is a mixture-of-experts (MoE): 35B total parameters, but only ~3B are active for any given token. So the compute per token is small (3B-class), even though all the experts must be available in memory.

That's the whole reason this works on Pascal: the GTX 1080 Ti has no tensor cores and modest FP16, so a dense 35B would crawl. A sparse 3B-active MoE keeps the per-token math light, and the bottleneck shifts to where the weights live — which is exactly what the dual-GPU question is about.

Quant fit on 22 GB

You can't just ollama pull qwen3.6:35b-a3b — that default is 23.9 GB and won't sit on 22 GB of VRAM. Measured GGUF sizes:

Quant	Size	Fits 22 GB?
Q3_K_M	~16.6–17.1 GB	✅ comfortable
IQ4_XS	~17.7 GB	✅ best quality that fits
Q4_K_S	~21 GB	⚠️ too tight (spills with KV cache)
Q4_K_M / default	23.9 GB+	❌ offloads to CPU

I used IQ4_XS.

Results: single vs dual 1080 Ti

Same model (IQ4_XS), same prompt, num_predict=256, measured via Ollama's /api/generate:

Config	Generation	Prefill	Model on GPU	Model on CPU (mmap)
CPU only (i9-14900K)	~17.1 tok/s	—	0 GB	whole model in RAM
1× GTX 1080 Ti	~16.8 tok/s	~50 tok/s	~3 GB	~18 GB+
2× GTX 1080 Ti	~20.3 tok/s	~50 tok/s	~13 GB (4 + 9.3)	~18 GB

Under load, the busier card drew up to ~101 W, GPU utilization sat around 26–33% — telling: the cards are waiting a lot, because the CPU-mmap'd experts are the bottleneck, not raw GPU FLOPs.

Update (2026-06-03) — the honest punchline, after an r/ollama reader pushed back ("those numbers are slow for A3B"). I measured CPU-only on the same box — an Intel i9-14900K (32 threads, DDR5): ~17.1 tok/s. That's basically tied with a single 1080 Ti, and only ~18% behind both GPUs combined. So for this offload-heavy MoE, the old Pascal cards barely beat a modern CPU — the 14900K does most of the work and the GPUs mostly shave overhead. The honest framing isn't "a 35B runs on 2× 1080 Ti" so much as "a 35B MoE runs on a fast desktop CPU, and old GPUs add ~18%." When the experts have to live in CPU RAM, your CPU + memory bandwidth — not the GPU — set the ceiling. (On hardware where the whole MoE is VRAM-resident, the GPU story would look very different.)

So, does the second 1080 Ti help? A little — ~+20% over one card, ~+18% over CPU-only — by keeping ~9 GB more of the model in VRAM. But not 2×, and not the win you'd hope: an MoE that overflows your combined VRAM is gated by the CPU-side experts in every config here.

Reproduce it

# (driver must be 570+ for current Ollama; check with: nvidia-smi)
ollama pull hf.co/bartowski/Qwen_Qwen3.6-35B-A3B-GGUF:IQ4_XS

# generate + read the eval rate
curl -s http://127.0.0.1:11434/api/generate -d '{
  "model": "hf.co/bartowski/Qwen_Qwen3.6-35B-A3B-GGUF:IQ4_XS",
  "prompt": "Explain mixture-of-experts in 150 words.",
  "stream": false,
  "options": {"num_predict": 256}
}'
# tokens/sec = eval_count / (eval_duration / 1e9)

To force a single GPU for comparison, start the server with CUDA_VISIBLE_DEVICES=0 ollama serve.

Honest Limitations

One quant, one model, one box. IQ4_XS on 2× 1080 Ti; your tokens/sec will shift with quant, context length, CPU, and RAM speed.
Prefill measured on a short prompt (~55 tokens) — treat ~50 tok/s as a ballpark; long-context prefill on Pascal will be slower.
IQ4_XS is a ~4-bit quant — fine for chat/drafting, but it's not full-precision quality.
MoE-specific. These conclusions (the modest dual-GPU gain, the CPU-mmap behavior) are about this sparse MoE. A dense model that fully fits VRAM would scale differently across two cards.
A few runs, not a statistical study — numbers are representative, not p-valued.

FAQ

Q: Can a GTX 1080 Ti really run a 35B model in 2026?

A sparse MoE one, yes — Qwen3.6-35B-A3B at IQ4_XS ran ~20 tok/s on two of them. A dense 35B would not be usable. The 3B-active design is what makes it work.

Q: Will a second 1080 Ti double my speed?

No. Here it added ~20%. The MoE experts stay memory-mapped in CPU RAM in both single- and dual-GPU setups, so the second card helps but doesn't scale linearly.

Q: Why did Ollama ignore my GPU until I updated the driver?

Recent Ollama bundles CUDA 13, which requires NVIDIA driver ≥ 570. On an older driver it falls back to CPU silently. Update the driver; Ollama then uses its cuda_v12 build for Pascal cards.

Q: Which quant should I use on 22 GB?

IQ4_XS (~17.7 GB) for the best quality that stays (mostly) on the GPUs; Q3_K_M if you want more headroom for context. Avoid the 23.9 GB default — it spills to CPU.

Resources

Model: Qwen3.6-35B-A3B GGUF (bartowski)
Ollama · benchmark via /api/generate (eval_count / eval_duration).