You have search over your own documents. Next comes a report with a reference after every claim, and the Allen Institute for AI (Ai2) released AstaBrief-8B for that writing step. The weights (the trained model file you download) are Apache 2.0, and the base is the Qwen team's Qwen3-8B. We ran it on one NVIDIA RTX A5000, a 24 GB card, and counted every reference it wrote.
Serving it is easy. One check in the code around it is not optional.
Decision one: the prompt is not yours to write
Ai2 trained the model on one fixed instruction file. The model card warns that swapping it may give "degraded or inconsistent behavior". So your code fills two blanks, the question and the excerpts, and keeps even the line "Note that the current year is 2025". No report stated the wrong year; whether it ranks 2025 papers as the latest work, we left untested.
Each excerpt carries a label: an ID, the first author, the year and a citation count (how many other papers cite it). The model copies it into square brackets after each claim, and should write "(LLM Memory)" when it relies on its own recollection.
Check the licence first. Apache 2.0 covers the weights only; the instruction file is CC-BY-NC-4.0, non-commercial. A paid product must clear that licence or write its own prompt in the same format, with the risk the card describes. Our SDK example ships no copy: it fetches the file from Ai2 at run time, under Ai2's terms.
Decision two: size the context, then let it batch
vLLM, the open-source program that loads a model onto the GPU and answers requests, treats AstaBrief as an ordinary Qwen3 model, with no custom code. Text is counted in tokens, about three quarters of a word each. The weights fill 16.4 GB of the 24 GB. We capped each request at 16,384 tokens (question, excerpts and report), the length of Ai2's final training round. The leftover memory holds two requests at that cap side by side. Our real requests were nearer 7,000 tokens, so about four fit.
Alone on the card, one report took 41.5 to 64.7 s, at roughly 39 tokens a second. Sending four is more telling.
Sent together, four reports took 85 s, against 209 s one after another. The first try missed that, through a bug of ours. Concurrency is how many requests a model may work on at once, and pinning a model to one GPU machine through our SDK set it to one. The four waited in line: 198 s, and one report vanished. One settings call raises the limit today. The vanished report revealed a second, unfixed bug. If the first token (the opening piece of the answer) takes over 3 minutes, the request closes empty with no error. The card still finishes a report nobody receives. The AstaBrief-8B test notes on inferencekey.com cover both, plus the workaround.
On your stack, test the concurrency limit at each layer, first-token time under load, and whether a timeout fails loudly.
Decision three: the label check is part of the pipeline
The 11 reports held 380 references, and 363 (95.5%) match an excerpt we sent. Good, and also the trap: the other 17 look just like the genuine ones.
Eight of the 11 reports cited 1 to 3 labels the prompt never contained. Part of them were our own excerpts with the label garbled. We sent [W4411306417 | Thapliyal et al. | 2025 | Citations: 45] and got back [4411306415 | Thapliyal et al. | 2025 | Citations: 111]. The rest were recalled papers, dressed in the same format and missing the tag ("(LLM Memory)" showed up once across 11 reports). Plain numeric IDs cured the missing W, not the wrong labels.
You know every label you sent, so the check is a set difference:
import re
sent = set(labels_in_prompt) # exactly as you formatted them
cited = set(re.findall(r"\[([^\[\]]+)\]", report))
unknown = cited - sent # flag before a human reads it
Two cleanups belong there too. Each report opens with an empty <think> block, the slot Qwen3 models reserve for reasoning when that mode is enabled (turning it off was not part of our test). And the per-section summaries use slightly different markers than the prompt requests.
What the 95.5% does not tell you
The score proves each reference exists, not that claims are correct. We did not compare each sentence with its excerpt, yet drift turned up anyway. One report gave a paper 25 citations instead of 27, seven times over. Another turned "one eighth" of a cost into "1/4 cost". Ai2 flags the same weakness: the right study, cited for more than it found. Have a person review figures taken from excerpts.
The test was small: 4 questions, 15 OpenAlex abstracts each, inputs around 5,000 tokens. Ai2's quality score (87.0 out of 100, against 77.3 for plain Qwen3-8B) covers computer science questions only. We graded none of the reports, and ran no head-to-head against another model.
For the retrieval side, see our EmbeddingGemma 2 post on multimodal search. The full AstaBrief-8B write-up with the vLLM command, SDK example and per-request timings also has the complete citation tally.
Credit: Ai2's Asta team built, trained, evaluated and released AstaBrief and its prompt, using reports from Anthropic, OpenAI and DeepSeek models, with GPT-4.1 and DeepSeek-R1 as graders. The Qwen team made Qwen3-8B, the vLLM project serves it unchanged, Hugging Face hosts the weights, OpenAlex supplied the abstracts, and an AI Search video pointed us to the release. Our part: the SDK example, the run on our own cloud, the counting. Read the complete AstaBrief-8B report on inferencekey.com before you wire it into your stack.
Top comments (0)