Ethan Mercer

Posted on Sep 20 Originally published at cometapi.com

How I’d Evaluate DeepSeek, Kimi, Qwen, and GLM for Coding Agents

#ai #agents

I’d start with DeepSeek V4.1 Flash for general agentic coding, then add candidates based on the work: Kimi K3 for persistent repository agents, Qwen3.8-Max for debugging with documents or visual evidence, and GLM 5.3 for defensive security analysis.

The deciding artifact would be a patch that passes a fixed repository test. Published benchmarks help me choose what to evaluate; they don’t tell me how much correction an agent will need in my environment.

Build the acceptance test before choosing the model

I’d hold the system prompt, repository snapshot, tool schema, timeout, retry policy, and acceptance tests constant. Model-specific reasoning controls should stay at their documented defaults unless those controls are the variable being tested.

My evaluation would separate executable correctness from root-cause analysis, tool reliability, and constraint adherence:

Task	Prompt	Acceptance or measurement
Repository repair	“Fix the failing pagination test without changing the public API. Return a patch and explain the root cause.”	Full test suite passes without a human patch.
Cross-file analysis	“Trace the authentication flow across these files and identify the condition that permits an expired token.”	Correct files and control-flow path are cited.
Agent execution	“Inspect the repository, propose a plan, edit the minimum files, run tests, and stop after two failed attempts.”	Record valid tool calls, retries, and compliance with the stop condition.
Issue triage	“Classify these 100 issues, identify duplicates, and recommend the 10 highest-risk bugs.”	Measure accepted classifications per dollar.

For every run, I’d record pass/fail, human corrections, input tokens, output and reasoning tokens, latency, retries, and total cost.

Price, supported input formats, and context length still matter. I’d use them as deployment filters after establishing that the model can do the work.

Where each candidate earns a trial

These results come from different evaluation settings. Vendors vary their prompts, harnesses, tool scaffolds, and reasoning configurations, so I wouldn’t turn the table into a single leaderboard.

Model and route ID	Published coding evidence	Other reasoning evidence
DeepSeek V4.1 Flash — `deepseek-v4.1-flash`	Terminal-Bench 2.1: 90.6; DeepSWE v1.1: 74.2; NL2Repo-Bench: 65.4; Codeforces rating: 3471	GPQA Diamond: 90.9; MathArena Apex: 65.6
Kimi K3 — `kimi-k3`	TerminalBench 2.1: 88.3; FrontierSWE: 81.2; ProgramBench: 77.8	BrowseComp: 91.2; DeepSearchQA: 95.0
Qwen3.8-Max — `qwen3.8-max`	Terminal-Bench 2.1: 86.6; SWE-bench Pro: 67.7	PaperBench: 93.0; IFBench: 82.8
GLM 5.3 — `glm-5.3`	CyberGym: 84.5%	ExploitBench: 54.4%

DeepSeek: my first general coding evaluation

The official model card reports the coding results above under a max-effort configuration. Repository repair, terminal interaction, codebase generation, and visual debugging are sensible first workloads.

The reasoning results use reasoning_effort=100. That setting belongs in the evaluation record: a max-effort result cannot describe the latency and cost of a lower-effort deployment.

I’d run a second evaluation at the effort level intended for production and compare accepted patches and human correction time. Code style alone would carry little weight.

Kimi: test continuity across a long run

Kimi K3 is the candidate I’d prioritize when an agent needs to preserve a plan across repeated inspection, editing, execution, and recovery.

Its terminal and repository scores support that trial, while BrowseComp and DeepSearchQA provide signals for search-heavy engineering work. These are vendor-reported figures, with settings that aren’t identical to the other candidates.

The useful test is a task containing a failed attempt. I want to see whether the agent recovers while retaining the original constraints inside my actual scaffold.

Qwen: bring the evidence into the test

Qwen3.8-Max is most interesting to me when a bug report includes screenshots, PDFs, diagrams, or video. Its reported terminal, PaperBench, and instruction-following results make those workflows worth evaluating.

The SWE-bench Pro result needs its own consideration. Strong document handling does not guarantee a correct repository repair, and scores from different benchmarks don’t establish a shared performance ceiling.

I’d keep evidence interpretation and patch correctness as separate acceptance checks. The model needs to understand the screenshot and produce a change that passes tests.

GLM: keep the security claim narrow

GLM 5.3’s reported CyberGym and ExploitBench results support a trial for vulnerability discovery and defensive analysis. The source interprets the gap as stronger discovery than reliable exploit completion; the benchmarks still measure different tasks.

I’d begin with defensive code review, attack-surface mapping, and data-flow tracing. Those results don’t establish leadership in ordinary feature development or general repository repair.

Hosted access and downloadable weights are separate decisions

If procurement requires open weights, DeepSeek V4.1 Flash has a published checkpoint and MIT license.

I’d treat Kimi K3, Qwen3.8-Max, and GLM 5.3 as hosted comparison routes until the exact checkpoint and license intended for deployment are independently verified.

There are two dated qualifications to retain:

Qwen3.8-Max had confirmed hosted access, but its downloadable checkpoint and license were not verified for the source article on August 26, 2026.
An open-weight GLM 5.3 release had been announced, while the cited route page still described the public artifact as planned.

“Open weight” means trained parameters are downloadable under a stated license. “Open source” is a broader claim that can involve training code, data information, and reproducibility. I’d check the artifacts and license directly.

Input support also needs route-level verification. DeepSeek V4.1 Flash accepts text and images; Qwen3.8-Max is listed for text, image, PDF, and video. I’d test the exact payloads before documenting production support.

Use one integration, then check the differences

A unified API such as CometAPI makes this comparison easier: the four routes share one API key, an OpenAI-compatible base URL, and a billing workflow.

The base URL is:

https://api.cometapi.com/v1

Keep the request shape, task, repository snapshot, and acceptance tests fixed while switching the model value:

deepseek-v4.1-flash
kimi-k3
qwen3.8-max
glm-5.3

A common interface still requires model-specific checks. I’d record reasoning settings and tool behavior, then retest multimodal payloads and sampling parameters before switching production traffic. Kimi K3, for example, documents fixed sampling behavior and thinking-only operation.

My operational checks would be:

Failure	What I’d inspect
`401 Unauthorized`	Provider-issued API key and the `Bearer` authorization header.
`404 Not Found`	`/v1` in the base URL and the exact current catalog model ID.
`429` or capacity errors	Exponential backoff and capped retries; fallback only to a model that passed the same acceptance test.
Invalid parameters	The route’s documented reasoning and sampling controls.
Unexpected spend	Usage fields, reasoning effort, retries, cache behavior, and time-based multipliers.

Compare spend against accepted work

The source’s pricing snapshot is dated September 14, 2026. These are the listed gateway rates per 1M tokens, with a displayed 20% discount against the official prices.

The example workload contains 100K input tokens and 10K output tokens, excluding cache hits, retries, tool charges, taxes, and account-specific discounts.

Model	Input / 1M	Output / 1M	Example request
DeepSeek V4.1 Flash	$0.12	$0.48	$0.0168
GLM 5.3	$1.12	$3.528	$0.1473
Qwen3.8-Max	$1.60	$4.80	$0.2080
Kimi K3	$2.40	$12.00	$0.3600

DeepSeek has the lowest base cost in this snapshot. It can also receive a 2× request multiplier during 01:00–04:00 and 06:00–10:00 UTC on weekdays, raising the example request to $0.0336 in a matching window.

I’d check live pricing before deployment and calculate:

Cost per accepted task =
(token spend + tool-call spend + retry spend + fallback spend + human review cost)
/
number of accepted tasks

Each spend category should be counted once. A low token rate can lose its advantage through failed patches, repeated attempts, or expensive review.

That is why my initial choice would remain provisional: DeepSeek for broad coding, Kimi for long-running agents, Qwen for document and visual evidence, and GLM for defensive security. Production selection would follow the accepted-work measurements.

References

Originally published at cometapi.com

DEV Community