boluo

Posted on Nov 4

Build a Same-Prompt Harness: Compare Kimi K3.1 Pro, DeepSeek V4.1 Flash, GPT-5.1, and Qwen3.8 Max on Meeting Notes

#ai #testing #programming #llm

(Disclosure: AI-assisted. I drafted and edited with AI tools. The same-prompt API runs, the comparison notes, and the conclusions are mine.)

I did not need another model review. I needed a script I could re-run: one fixture, three asks, one compatible endpoint, four ids. On 4 November 2026 that harness ran Kimi K3.1 Pro, DeepSeek V4.1 Flash, GPT-5.1, and Qwen3.8 Max against the same handwritten weekly notes. Below is the recipe, the HTTP gotcha, and where over-completion showed up.

The harness pipe for Kimi K3.1 Pro, DeepSeek V4.1 Flash, GPT-5.1, and Qwen3.8 Max

Four browser tabs mix login walls and upload UX into the score. The pipe keeps only what the comparison needs:

identical user content
identical task wording
swap model only
tick every owner/date against the scribble before anyone else sees the draft

Base URL: https://api.crazyrouter.com/v1. Ids: kimi-k3.1-pro, deepseek-v4.1-flash, gpt-5.1, qwen3.8-max. Callable chat ids: chat models page. Field notes: Crazyrouter docs entry. Measured: interface-side first-pass text. Consumer-app uploads and plugins stay out of scope.

curl https://api.crazyrouter.com/v1/chat/completions \
  -H "Authorization: Bearer $CRAZYROUTER_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "deepseek-v4.1-flash",
    "messages": [{"role":"user","content":"整理下面周会速记……"}]
  }'

Put the key in an environment variable. Kimi K3.1 Pro returned HTTP 400 on one unsupported sampling field; removing it restored a normal reply. Treat vendor error text as truth when you reproduce. A 200 on a short prompt only proves that day’s short-text route for that id.

Fixture design before scoring DeepSeek V4.1 Flash or GPT-5.1

~40 minutes of product standup scribbles: product Yun, engineer Chen, designer Mao. Topics covered a payment-page QA gate, a coupon display bug, an early membership draft, and a post-call ban on publishing the English word overlapping. Gaps were intentional—no P1 id, some todos without owners—so the model had to mark uncertainty or invent.

Three fixed asks:

Clarify one spoken remark; invent nothing.
≤3 conclusions + todos; unknowns = “to confirm.”
Paste-ready table; rewrite overlapping as “stacked on top of each other”; no email; no calendar writes.

Prompts stayed in Simplified Chinese, matching the notes. English here explains method and results; it is not a claim that I re-ran the suite in English.

Weekly notes are a harsh fixture on purpose. A clean table that invents “Chen owns QA” can mis-route work for a week. That failure mode teaches more than a composite Elo.

How I choose among Kimi K3.1 Pro, DeepSeek V4.1 Flash, GPT-5.1, and Qwen3.8 Max outside the harness

I still write three lines first: what small job repeats weekly, how I will know the AI helped, and whether I will spend five minutes on the same pack. Then task → daily product → model id. Company, engine, and app are different layers. I delay paid upgrades until I can name a free-tier blocker (length, caps, export).

Scan by job shape: spoken rewrite for mail/remarks; browsing plus citations for fresh facts; whole-file intake for long PDFs; and a hard split between “writes a snippet” and “touches your file” for code or sheets.

Over-completion notes from Kimi K3.1 Pro versus DeepSeek V4.1 Flash, GPT-5.1, and Qwen3.8 Max

All four first drafts were usable. The gap was fill-in boldness, not prose quality.

Model	First-pass feel	Check first
Kimi K3.1 Pro	Short; keeps “to confirm”	Hard-assigned owners (often slower)
DeepSeek V4.1 Flash	Fast lists	Relative days → calendar dates; unsupported owners
GPT-5.1	External-sync tone; clean tables	Inferred people
Qwen3.8 Max	Smooth Chinese; medium density	Missed uncertainty labels

DeepSeek V4.1 Flash once turned “this Thursday” into a concrete month-day and put Chen on the QA gate—notes only said engineering thought Thursday was possible. GPT-5.1 showed similar fill-in. Kimi K3.1 Pro more often left the hole visible. Qwen3.8 Max read like a calm internal sync.

Routing for this chore: fast editable draft → DeepSeek V4.1 Flash or GPT-5.1; fewer silent inventions → Kimi K3.1 Pro; internal-Chinese tone → Qwen3.8 Max. The human checklist stays mandatory.

Prompt contract that works on GPT-5.1 and Qwen3.8 Max (and the other two)

Stop at “please write the notes.” Lock goal, material, hard limits, and delivery shape. Task-three skeleton (Chinese in the real runs):

From the notes below, write ≤3 conclusions, then a todo table (item / owner / time-or-status / note). Use “to confirm” when unclear; do not guess. Externally write overlapping as “stacked on top of each other.” Do not email. Do not write a calendar.

If the ask is fuzzy, request five clarifying questions first. On revision, name the cell—“row 2 owner → to-confirm”—instead of “make it better.” Completeness beats collecting prompt packs.

Agent, Skill, and MCP after the Kimi K3.1 Pro–Qwen3.8 Max harness

Paste → Markdown is one conversation. Agent-style loops (confirm format, reread, mark gaps, write a file, self-check) need tools that actually exist. A Skill stores weekly house rules—often SKILL.md plus templates. MCP-style connectors attach external systems; protocol ≠ service ≠ permission. None of the four models opened a calendar or sent mail. Chat claims are not actions. Habit first: same-prompt comparison and scribble checks; add Agent / Skill / MCP when a real capability is missing.

Five steps to reuse with Kimi K3.1 Pro, DeepSeek V4.1 Flash, GPT-5.1, or Qwen3.8 Max

Pick one assistant you will open daily.
Use today’s real standup scribble.
Put goal, background, limits, and delivery format in one message.
Ask for one concrete fix on pass two.
Skill-ize repeating rules later; touch MCP or Agent only when calendar/docs automation is real.

Three burns: browsing ≠ verifying owners; long “thinking” ≠ reading an unpasted attachment; one weak draft ≠ brand swap—separate thin notes, thin prompts, and model over-completion first.

Keep list after testing Kimi K3.1 Pro, DeepSeek V4.1 Flash, GPT-5.1, and Qwen3.8 Max

A harness makes the comparison honest. All four can draft; over-completion is the publish risk. Prompt completeness sets the floor. Agent / Skill / MCP are later layers. Eye-checking the scribble usually beats swapping models.

I still paste three sentences before the notes: “Here are this week’s notes. Here is the format. Mark unclear items as to-confirm; do not email for me.” This round used https://api.crazyrouter.com/v1. Web-app uploads and plugins are outside this article’s scope.

DEV Community