DEV Community: ur-grue

FOIA letters are a format, not a vibe — so I made Claude write them properly

ur-grue — Fri, 29 May 2026 11:00:22 +0000

Ask a general-purpose chatbot for a public-records request and you get something that looks like a letter and works like a liability. It misses the statute. It asks for "any and all documents" — the phrasing agencies love to reject. It promises a fee waiver you didn't qualify for. It sounds confident and gets you nothing.

A records request is a format with rules. Cite the right law. Describe the records narrowly enough to be findable and broadly enough to catch what you want. State your fee-waiver basis correctly. Set the response clock. Miss any of those and you've burned weeks waiting for a "no."

I maintain a free, MIT-licensed library of Claude skills for media work, and the FOIA/records skills are some of the most-used. Here's what "properly" means.

What the skill actually does

You give it: the records you want, the agency that holds them, and your jurisdiction. It gives back a filing-ready letter that:

Cites the correct statute — US federal FOIA (5 U.S.C. § 552), a named state public-records act, or the EU/UK equivalent — not a generic "freedom of information" gesture.
Scopes the request so it's specific (date ranges, record types, named offices) instead of the "any and all" phrasing that invites a denial.
States the fee-waiver basis in the language the statute uses — public interest, news-media requester status — only when it actually applies.
Sets the response deadline the law requires, so you know when silence becomes appealable.
Leaves a clean appeal trail — structured so that if you're denied, your next letter writes itself.

Why this beats "write me a FOIA request"

The difference isn't eloquence. It's that the skill encodes the procedural knowledge a beat reporter accumulates over years: which exemptions agencies hide behind, how to pre-empt the "too broad" rejection, when a fee waiver is real. A blank-slate prompt doesn't carry any of that — so it produces a letter that reads well and fails procedurally.

The honest limits

It is not legal advice, and it does not know your specific agency's quirks. Statutes change; local rules vary. The skill gives you a strong, correctly-structured draft — you still confirm the citation for your jurisdiction and adjust for the body you're filing against. It will also tell you when your request is too vague to file, instead of inventing specifics.

The whole library is free. If you file records requests, start with the FOIA skill and the document-analysis skill that pairs with it — feed in what comes back, get a structured read of what's in it and what's missing.

→ github.com/ur-grue/autopunk-media-skills

How do you eval LLM output that isn't code?

ur-grue — Fri, 29 May 2026 10:44:27 +0000

Code has a luxury: it either runs or it doesn't. You write an assertion, you run it, you get a green check or a red one. Most LLM eval frameworks lean on exactly this — assert output contains X, assert valid JSON, assert no error.

Editorial output has no such luxury. A pitch treatment, a show-notes draft, a lede — there's no test that returns true for "a working producer would send this." So how do you evaluate it without a human reading every output, every time?

I had to answer this concretely, because I maintain a library of ~400 Claude skills for media work, and "trust me, it's good" is not a quality bar. Here's the approach.

Two stages: binary first, judgment second

Stage one is the cheap filter — binary assertions. Even for prose, a lot of failure is mechanical and testable: did it produce the required sections? Is the lede one sentence? Did it refuse to fabricate a quote when given no source? These catch the obvious breaks fast, run blind across many inputs, and cost nothing. The library runs thousands of these. The interesting result: the few "failures" were skills correctly refusing to invent content on deliberately thin inputs — the desired behaviour, not a bug.

Stage two is the part that matters for prose — graded judgment. A model scores each output 1–5 across seven dimensions: coherence, relevance, accuracy, completeness, usefulness, format-fit, and one more that does the real work.

The dimension that does the work: Editorial Naturalness

Six of the seven dimensions are standard. The seventh is a hard floor: does this read like a person who knows the medium, or like a model?

This is scored against observable tells, not vibes:

Lexical — the AI vocabulary (delve, leverage, robust, seamless, tapestry).
Structural — the false pivot ("not just X, but Y"), throat-clearing openers, rule-of-three on every line.
Tonal — manufactured enthusiasm, hedging stacks, the apology spiral.
Genre — does it honour the conventions of the format it claims to serve?

A skill can score 5/5 on the other six and still fail. If Editorial Naturalness is below the floor, it doesn't ship. That single constraint is what stops the library drifting into competent-sounding slop.

Why a hard floor and not an average

Averages hide the thing you care about. A draft that's accurate, complete, well-structured — and unmistakably machine-written — would pass an averaged score comfortably. For media work that draft is useless: the audience clocks it in a sentence. The floor forces the failure to surface instead of being averaged away.

The honest limitation

A model grading prose is generous — it tends to like fluent text, including fluent AI text. So the scores are treated as a filter, not a verdict: they catch the clear failures and rank candidates, but the bar for "stable" is deliberately set high (≥ 4.0 with the naturalness floor), and the rubric is anchored on observable tells rather than taste, so two runs roughly agree. It's not perfect. It's a lot better than shipping on feel.

The whole framework — dimensions, thresholds, the banned-phrase list — is open source. If you're evaluating non-code LLM output, take it apart and tell me where it's too soft.

→ github.com/ur-grue/autopunk-media-skills

How to tell if AI wrote it — and how to make it stop

ur-grue — Fri, 29 May 2026 10:10:25 +0000

You can spot it in a sentence. "In today's fast-paced landscape, we're thrilled to delve into game-changing solutions that empower teams to unlock their potential."

Grammatically fine. Recognisably synthetic. And for anyone whose readers notice — journalists, producers, newsletter writers — that register is a liability.

I build a free, MIT-licensed library of Claude skills for media producers. The whole project rests on one idea: AI output for media has to read like a person who knows the format, not like a chatbot. So I had to make "sounds like a human" something you can actually measure, not just feel.

Here is how.

Name the tells

Most AI prose fails in four observable ways:

Lexical — a vocabulary of words real writers rarely reach for: delve, leverage, robust, seamless, cutting-edge, tapestry, realm, navigate, unlock, empower.
Structural — the false pivot ("not just X, but Y"), the throat-clearing opener ("In this article, we will"), the rule-of-three rhythm on every sentence.
Tonal — manufactured enthusiasm, hedging stacks ("may potentially"), the customer-service apology spiral.
Genre — copy that ignores the conventions of the medium it claims to serve. A show-notes draft that reads like a press release. A lede that buries the news.

Once you can name the tell, you can cut it.

Make it a quality gate, not a vibe

In the skill library, every skill is scored on a seven-dimension rubric before it ships. Six dimensions are the usual suspects — coherence, relevance, accuracy, and so on. The seventh is a hard floor: Editorial Naturalness. A skill can score well everywhere else and still fail if its output reads as machine-made. Below the floor, it does not ship.

That one constraint changes how the skills are written. They stop optimising for "complete and correct" and start optimising for "a working professional would send this without rewriting it."

The detox pass

There is also a skill whose only job is to strip the tells: feed it AI-flavoured copy, get back a publishable rewrite plus a before/after table so you learn the pattern. It carries the canonical banned-phrase list — the same list a runtime check uses to flag drafts before they go out.

Worked example. Before:

In today's fast-paced media landscape, our cutting-edge AI-powered solution seamlessly empowers newsrooms to unlock unprecedented efficiencies.

After:

Three regional dailies now use it for pre-publication checks. None of them call the change transformative. All three call it removing a class of small, repetitive edits from the subbing pass.

Shorter. Concrete. Verifiable. No tells.

Why this matters for media work

Readers in media are trained to detect manufactured language — it is the job. Copy that pattern-matches to "AI wrote this" costs you credibility before the argument even lands. The fix is not "write a better prompt." It is treating naturalness as a measurable bar and refusing to ship below it.

The library is free and open source. Browse it, copy a skill into Claude, and watch what changes when the AI tells are gone.

→ github.com/ur-grue/autopunk-media-skills