DEV Community: Karlis Gutans

I automated my refactoring method with GitLab Duo agents. The agent writes the fix, a CI gate decides the merge.

Karlis Gutans — Tue, 20 Oct 2026 12:32:18 +0000

AI writes code fast. The code runs and passes the tests. It is still hard to change.

I refactor a lot of AI-generated code. The same problems come back every time:

Functions grow long and deeply nested.
Files collect responsibilities that belong somewhere else.
The next AI session needs more and more context to make a safe edit.

For the GitLab Transcend Hackathon (theme: Life After Code) I tested how far my own refactoring process can be automated with GitLab pipelines and the GitLab Duo Agent Platform. The result is Liepa refactoring.

My manual method

My method has three steps:

Take the most characteristic keywords of each file.
Check whether those keywords belong to the area of the app where the file lives.
If they do not, the file mixes responsibilities. Refactor it with an AI assistant, one file at a time.

It works. It is also slow, and it depends on me remembering to do it.

What I built

Liepa refactoring has three parts:

liepa scan: a Python scanner. It runs standard checks (function length, nesting, cyclomatic complexity, duplicated function bodies) and gives every file an AI-readiness score from 0 to 100.
A refactor flow: a GitLab Duo agent flow. It scans the project, picks the lowest-scoring file and acts on that one file.
A control dial: a config file that sets how much control is handed to the agent.

The scanner uses the Python standard library only: ast, keyword, collections. No runtime dependencies.

Turning the keyword method into code

The AI-readiness score has seven checks. Six are familiar: context size, function length, nesting, complexity, docstrings and import fan-out. The seventh, keyword_areas, is my manual method.

How it works:

Collect the names in a file: functions, classes, arguments, attributes. Split snake_case and CamelCase into words. Drop Python keywords, builtins and generic words like data or value.
Keep the 7 most frequent words.
The area of a file is its directory.
For every word, find its home area: the directory where the word is relatively most frequent across the whole project.
The penalty is 10 × (keywords whose home is elsewhere) / 7.

This is a real result from the tool's own CLI file:

src/liepa/cli.py  91
`main` has complexity 12 (limit 10); top keywords span 2 areas:
path (src/liepa), config (src/liepa), score (src/liepa), parser (src/liepa),
add (src/liepa/checks), argument (src/liepa), files (src/liepa);
consider splitting the file

One keyword out of seven lives in another area. The penalty is small: 1.4 points. That is correct. It is a hint to look, not a verdict.

The control dial

.refactor-agent.yml lives in the repository:

mode: suggest # suggest | propose | auto

paths:
  include: ["src", "examples/messy/liepa_imin"]

thresholds:
  min_score: 60
  max_function_lines: 50
  max_nesting: 4
  max_complexity: 10

auto:
  require_score_improvement: true
  require_passing_tests: true
  max_changed_lines: 200

The three modes:

suggest: the agent posts one refactoring suggestion on the merge request. No files change.
propose: the agent refactors the file, runs pytest and ruff, re-scans, and opens a merge request for a human to review.
auto: the agent does the same as in propose. Then a CI job decides whether the merge request merges.

The agent never decides the merge

This is the most important design decision. In auto mode the LLM only opens the merge request. A small deterministic job, refactor_auto_merge, checks five conditions:

Only refactorable Python files changed. No tests, CI files, config or flows.
Every changed file scores strictly higher than before.
The change is under 200 added plus deleted lines.
The pipeline is green. GitLab merges only when it succeeds.
The merge request head is the exact commit that was scored.

The gate posts its decision on the merge request. This is the real note from the demo run:

Condition	Result	Details
Only refactorable files changed	pass	1 changed files, all scanned Python files
AI-readiness score improves	pass	`reports.py`: 40 → 51
Change size	pass	124 changed lines, limit 200
Pipeline green	pass	enforced by GitLab
Head SHA unchanged	pass	checked `0dc870fc4e71`

The gate and its limits are read from the target branch, not from the merge request. A refactor branch cannot loosen the rules that judge it.

This was not my first version. Duo Code Review found that the first auto mode trusted the merge request's own config and code. A refactor could have edited the limits that judged it. The fix: run the gate from a git worktree of the target branch.

The demo

I needed a codebase bad enough to give the agent real work. So a Duo Developer agent built liepa-imin: a toy insurance back office with parties, policies, claims, renewals and reports. Its reports.py scored 29: a 92-line function, complexity 20, 6 levels of nesting and 17 imports. Behaviour tests and golden files guard the output.

All three modes ran on the same file:

Mode	What happened	Score
suggest	one comment with three refactoring steps	28 → 28
propose	refactor MR, reviewed and merged by me	28 → 40
auto	refactor MR, merged by the gate with no human review	40 → 51

The auto merge request merged about one minute after it was opened.

What did not work

Triggers need a human. A GitLab flow trigger fires only on a person's action. A CI job, a bot or another flow cannot start it. GitLab staff confirmed this in the hackathon Discord. Every demo run was started by hand from the Duo CLI.
The first auto pipeline was rejected. The gate job needed jobs that existed only in branch pipelines. I moved lint, tests and scan into merge request pipelines.
The sandbox has no PyPI access. The GitLab-hosted sandbox cannot install packages, so propose runs locally for now.
The score can be gamed. Docstrings and split functions raise the score without always improving the design. The size limit and the one-file rule reduce this. They do not remove it.

Why this works

The agent writes, the code decides. An LLM is good at producing a refactor. A small deterministic gate is better at deciding whether it may merge.
Small steps. One file per run keeps every change reviewable, even in auto mode.
Evidence over claims. Every agent session, job and merge request is linked in the README. It is clear what ran and what is still untested.

What is next

Publish the flow and add a trigger, so assigning a reviewer starts it.
Run the dry run and a refused case on a real merge request.
Give the bot token its own least-privilege identity.
Keep running the flow until reports.py reaches 60.

How do you refactor AI-generated code? How much control would you hand to an agent that does it for you?

Give an LLM no facts and ask for a job ad. What does it commit to?

Karlis Gutans — Mon, 05 Oct 2026 08:11:03 +0000

This is a submission for the Kaggle Benchmarking Challenge.

Update, 7 October 2026: the benchmark now runs all 20 briefs. The one-brief results below did not hold up. See Update: 20 briefs at the end.

Job ads are long. The first one I read end to end. After that I scan. I look for the concrete parts: pay, the stack, the team size, where the work happens, what on-call looks like. Everything else is filler I have learned to skip.

I also do not apply because I match every line. I apply when something concrete catches my attention.

More and more of these ads are now drafted by LLMs. So I wanted to know: when a model writes a job ad and is given no facts at all, what does it commit to?

What I Benchmarked

My main goal was to try Kaggle Benchmarks. I already had a scoring method from an earlier project, so I ported it into a Kaggle notebook. In about two days it went from an empty notebook to a public leaderboard.

The task. Every model gets the same brief: a role, a seniority level, and the topics the ad must cover. The brief on the current leaderboard is a junior backend engineer. The ad must cover the service, languages and data stores, code review, team size, location, pay, and the first six months. The brief names no employer and supplies no facts.

The model has three options for every topic:

Invent a specific: "€45,000–55,000", "Go and PostgreSQL", "a team of six".
Leave a blank: "[salary range]".
Write around it: "competitive salary", "a modern stack".

The judge. A fixed judge model (Claude Sonnet 4.5, never a contestant) runs two stages:

Extraction: split the ad into claims, each one an exact substring of the ad.
Classification: label each claim with one tier.
- Concrete: you can quote a particular. A number, a named tool, a place, a cadence.
- General direction: real intent, nothing checkable. "Flexible hours."
- Empty slogan: fits any ad at any company. "Fast-paced environment."

The score is concrete claims divided by all claims, from 0 to 1.

The judge is not trusted on its own. Code enforces the rules after every call:

A claim that is not a verbatim substring of the ad is rejected.
A "concrete" label must quote its particular, and that quote must appear in the claim. If not, the claim is demoted to general direction.
Template blanks like "[salary range]" or "€XX,XXX" are flagged and never count as concrete.
"English" and bare skill nouns like "communication skills" never count as a particular.

Higher is not better. The brief contains no facts, so every concrete claim is made up. The score measures how willing a model is to commit to specifics. Which behaviour you want depends on whether a human fills in the real facts afterwards.

The tier rules come from my earlier project, which scored real job postings. There I measured my own labeling consistency before trusting any model number: Before you report LLM agreement, measure the human twice.

Models Tested

I picked the small, fast tier from each vendor. These are the models a company would plug into an HR tool to draft ads at volume. Cost and speed matter there more than peak quality.

Model	Score	Cost per run	Time per run
Claude Haiku 4.5	0.50	$0.037	29 s
Gemini 3.7 Flash	0.48	$0.062	44 s
Gemini 2.5 Flash	0.37	$0.074	60 s
GPT-5.4 mini	0.35	$0.069	45 s
Grok 4.5	error	—	—

Cost and time are as Kaggle reports them for one run: the ad plus both judge calls. Most of the token count is the judge, not the model under test.

The judge is kept out of the contest on purpose. A model grading its own family's writing is a conflict I did not want to explain.

Findings

What one ad looks like. Gemini 3.7 Flash wrote a 2,528-character ad. The judge extracted 29 claims:

14 concrete
13 general direction
2 empty slogans
0 placeholders

No blanks at all. Every topic got either an invented specific or a soft phrase. It never wrote "[salary range]".

The leaderboard splits into two pairs. Claude Haiku 4.5 and Gemini 3.7 Flash commit to specifics in about half their claims. Gemini 2.5 Flash and GPT-5.4 mini sit around a third.

Do not read it as a ranking yet. The current leaderboard runs one brief, so each score is one ad of roughly 30 claims. For Gemini 3.7 Flash, 14 concrete out of 29 gives a 95% interval of about 0.31 to 0.66. Every model on the board sits inside that interval. The two pairs are a direction to check, not a result.

The judge is unmeasured on these ads. On real postings, an earlier version of the rules reached 86.7% agreement with my hand labels on the concrete/not-concrete boundary. That was a different judge model and older prompts. Nobody has hand-checked this judge on generated ads.

Grok 4.5 failed, and that is by design. The task refuses to publish a score unless every brief is scored. A partial average across a subset of briefs is a different benchmark, and it would sit on the same leaderboard as if it were not.

What surprised me on the Kaggle side:

A task must return a number, bool or dict. Returning a string fails to store.
Reasoning models put <think> text before their JSON. The built-in schema parser then fails, so the judge parses JSON itself.
%choose keeps only the latest run of the main task. Run it once in the notebook, then add models from the task page.
Model names contain /. Sanitize them before using them in file names.

Why model choice matters here. Ads are many and they are long. Getting from an ad to its actual claims takes effort, and that effort goes to a model on both sides. The employer's model decides what gets committed. The candidate's model decides what counts as concrete. Pick the wrong one on the writing side and you get invented salaries. Pick the wrong one on the reading side and "competitive salary" passes as a commitment.

What I would measure next:

All 20 briefs, several samples each. The briefs already cover intern to principal, from frontend to production support. One ad per model is not enough.
A hand check of the judge on a sample of generated claims, the same way I checked it on real postings.
The split per topic. Pay is the interesting one: invented number, placeholder, or "competitive".
Whether the ad attracts the right people. This is the metric that matters most, and specificity is only one input to it. It needs outcome data: who applied, and how many were qualified. That data sits with employers and job boards, not in a public benchmark.

My Benchmark

Kaggle benchmark

Briefs, prompts and judge code are in the notebook. The method behind the tiers is in the repo: github.com/kargut/job-posting-specificity.

When you read a job ad, what makes you apply: the concrete details, or something else entirely?

Update: 20 briefs

The first leaderboard ran one brief. I wrote that it was a direction to check, not a result. So I checked it. I ran all 20 briefs, from intern to principal, with the same judge and the same prompts.

https://www.kaggle.com/benchmarks/karlisgutans/job-ad-specificity-20-what-llms-commit-to/leaderboard

Model	1 brief	20 briefs	Cost, 20 briefs	Time, 20 briefs
Gemini 3.7 Flash	0.48	0.48	$1.31	16 min
Claude Haiku 4.5	0.50	0.37	$0.91	12 min
Gemini 2.5 Flash	0.37	0.28	$1.49	21 min
GPT-5.4 mini	0.35	0.19	$1.19	14 min
Grok 4.5	error	error	—	—

Cost and time cover the full run: 20 ads plus 40 judge calls, as Kaggle reports them.

What changed:

The two pairs are gone. One ad put Claude Haiku 4.5 and Gemini 3.7 Flash level at about half. Over 20 ads, the four models sit on four separate steps, from 0.48 down to 0.19.
Only one model held still. Gemini 3.7 Flash scored 0.48 on one ad and 0.48 on twenty. Every other model dropped. GPT-5.4 mini dropped the most, from 0.35 to 0.19.
The spread is wide. Gemini 3.7 Flash makes almost one claim in two concrete. GPT-5.4 mini makes fewer than one in five. Same briefs, same judge, and a 2.5 times difference in how much each model invents.
The first brief was an easy one to be concrete about. It was a junior backend engineer, and it asked for languages and data stores. That topic invites named tools. A brief for an engineering manager or a technical writer gives a model less to name. This is my reading, not a measured result: I have not yet broken the scores down per brief.
Grok 4.5 failed again. The task publishes no score unless every brief is scored. At least one Grok brief failed in both runs.

What it cost. A full run cost $0.91 to $1.49 per model and took 12 to 21 minutes. Claude Haiku 4.5 was both the cheapest and the fastest.

The lesson. The first post already said not to read one ad as a ranking. The 20-brief run shows why. One ad per model is a demo. Twenty is the start of a benchmark.

Still open:

The spread across briefs, so the gaps between models get an interval.
What each model does with pay: an invented number, a placeholder, or "competitive".
A hand check of the judge on generated ads.

Before you report LLM agreement, measure the human twice

Karlis Gutans — Mon, 14 Sep 2026 10:00:41 +0000

I set out to score how much of a job ad is a real commitment. The number I got is not the interesting part. The interesting part is that I agreed with myself only 75% of the time until I wrote the rule down.

"Competitive salary." "Fast-paced environment." "You will own your work." Delete those sentences and the ad loses no information. Next to them sit real commitments: a salary band, a named database, two days a week in the office, on-call one week in six.

The extract-classify-score loop was a weekend. Getting a number I would stand behind was not.

Code, prompts and the full evaluation report: github.com/kargut/job-posting-specificity

What a claim looks like

Three stages, deliberately separate so each can be evaluated on its own:

Stage	What	How
1 — Extraction	Split a posting into discrete claims	LLM
2 — Classification	Assign each claim a tier	LLM
3 — Aggregation	Scores and rollups	Deterministic Python

Tier 1 — Concrete: contains a number, a named technology, a timeframe, or a falsifiable commitment. EUR 60k. Go and PostgreSQL. team of nine.
Tier 2 — General direction: real intent, no checkable detail. We invest in developer growth. modern stack.
Tier 3 — Empty slogan: would fit any ad for any job at any company. fast-paced. rockstar developer.

Score: tier_1 / all scored claims, per posting. A proportion rather than a count, so a 200-word ad and a 2,000-word ad stay comparable.

On 15 ads, about 4 in 10 claims were checkable — and that number is an overestimate. The gold set is 15 postings, 150 claims, hand-labeled blind, sampled from public Greenhouse job board APIs. No scraping, and no company is named anywhere in the results. Per posting the score runs 0.00 to 0.90, median 0.30, so the metric does separate documents. I am not quoting the 0.407 gold-set mean as a measurement of how specific job ads are. The rest of this post is why that figure is inflated.

Four claims, anonymised, under the rule as written:

Claim	What you can quote	Tier
`EUR 60–75k, band 4`	`EUR 60–75k`	1
`We invest in developer growth`	—	2
`Excellent written and verbal communication skills in English`	`English`	1 — a named language, not a particular about this job
`fast-paced environment`	—	never extracted: Stage 1 drops slogans as atmosphere

The third row is a real gold label. The fourth never reaches the annotator at all. Both are why "about 4 in 10" overstates how much of an ad is a checkable commitment.

Measure the human twice

The standard recipe for an LLM evaluation is to hand-label a gold set and report agreement. I did that. Then, before writing down a single accuracy number, I labeled 36 of the same claims a second time, a day apart, blind to my first pass.

I agreed with myself on 75.0% of exact tiers and 80.6% of the Tier 1 boundary — the distinction the score actually rests on.

The specificity score of that one posting moved from 0.333 to 0.472 between passes. Forty-two percent relative movement, on a document that had not changed a character.

My quality gate was "at least 80% on the Tier 1 boundary". The gate was sitting on the noise floor. A model clearing it would have been indistinguishable from one merely as inconsistent as I am.

The fix was a written rule, not a better prompt

Tier 1 became a mechanical test: quote the particular. A claim is Tier 1 only if you can point at a number, a named technology, a named place, an explicit timeframe, or a quantified policy. Seriousness of the work is not a criterion. "Expert work" and "works closely with the security team" name nothing checkable. Both are Tier 2.

Then I enforced the rule in tooling. The labeler refuses a Tier 1 whose reasoning does not literally quote a substring of its own claim. Run against my drifted pass of 36 claims, the linter flags 34 errors.

A blind 50-claim re-label after the rule:

Measure	Pre-rule (n=36, one posting)	Post-rule (n=50, 15 postings)
Exact tier	75.0%	86.0%
Tier 1 boundary	80.6%	96.0%

The floor moved a lot after the rule. The samples are not matched — 36 claims from one posting under an earlier extraction, versus 50 claims across 15 postings from the current run — so do not treat +15.4pp as a causal estimate. Consistency improved. That is the claim. No model call was involved in producing it.

Same model output, opposite conclusion

With a floor I would actually use, the classifier numbers mean something:

Measure	Value	95% CI
Exact-tier accuracy	76.7%	69–83
Tier 1 boundary accuracy	86.7%	80–91
Tier 1 precision / recall	0.902 / 0.754

Against the pre-rule floor of 80.6%, the model looked six points better than the human. Against the re-measured floor of 96.0%, it sits nine points below.

Same model output. Opposite conclusion. Which floor you pick decides the answer, and most write-ups never publish a floor at all.

Against my second-pass ceiling the model loses; against the labels I later recanted, part of the loss is me. In the blind post-rule pass I changed 7 tiers, and 6 of those 7 landed on the model's answer — without ever seeing the model's output.

Tier 3 never fired

Zero of 150 gold claims landed in Tier 3. The tier built for empty slogans, run on these 15 job ads, found nothing.

Two causes, both upstream of the classifier:

Stage 1 drops slogans as atmosphere. Mission and culture language never reaches the annotator at all. A Stage 1 error is invisible to a Stage 2 metric.
Sharpening one boundary destabilised another. Tier 1 got a mechanical test. Tier 3 kept a prose test. That is the English row in the table above: a quotable token that is not a particular about the job.

The blind re-label makes it sharper. The same annotator, the same written rules, a day later, produced 5 Tier 3 labels in 50 claims. A 0% rate and a 10% rate are both consistent with the taxonomy as written. That is a worse problem than an unused tier, and I shipped it as a documented finding rather than patching it into looking good.

The consequence: on this gold set the score is in practice tier_1 / (tier_1 + tier_2), inflated twice over. Comparisons between postings survive, because the bias runs the same direction for all of them. Absolute values must not be quoted as measurements of slogan content.

A slice that was not a sample

--limit 25 plus jobs[:limit] took an alphabetical slice, not a sample. Every posting pulled from one board had a title starting with "A". Every per-board statistic produced before I found it was invalid. Not an LLM problem; it would have changed a published number.

Cost, from the run artifacts: the prompt is about 61% of all input tokens, 76% at Stage 2, so caching and batch size are the whole story — batching at 3.75 postings per call halved my first estimate.

What the score does not measure

Not honesty. A precise promise can be broken. A vague ad can conceal a good job.
Not quality. Some excellent employers write badly.
Not intent. Most filler is copied from a template by someone who did not choose it.
Not slogan content. The 0.407 mean is an overestimate on two counts, both pushing the same way: missing Tier 3 in the denominator, and false-positive Tier 1s like English.

The method's own limits, stated plainly. One annotator, so there is no inter-annotator agreement, only intra-annotator. Gold tiers sit on model-extracted spans, so an extraction error cannot be counted wrong. 150 claims, so roughly 7 points of confidence interval at 95% — an improvement of a few points is not a claim this set can support. The scope filter matches words rather than functions: I audited three borderline titles by hand, one was a genuine false positive, and the fix went into the rule instead of a hardcoded exclusion list.

What I would tell someone building the same thing

Label a slice twice, blind, before you write down any accuracy number. Your own self-agreement is the ceiling, and it is lower than you think.
Log a warning when two metrics that should differ do not.
Write the boundary rule as a mechanical test, then enforce it in code. The linter that refuses an unquoted Tier 1 was worth more than every prompt iteration combined.
Publish the result that makes you look worse.

Do you measure yourself twice before you report model agreement, or do you only check the model against the model?