DEV Community: Christ-loisele Atidegla

Four agent skills I actually use, and the install problem nobody solved cleanly

Christ-loisele Atidegla — Wed, 09 Sep 2026 17:29:17 +0000

Agent skills became a standard faster than most formats do. A directory, a SKILL.md, and since late 2025 every major harness reads the same thing. Claude Code, Codex, Cursor, Gemini CLI and Antigravity all understand the format.

What none of them agree on is where the directory goes. ~/.claude/skills, ~/.codex/skills, ~/.cursor/skills, and so on. Use more than one harness, which most of us do now, and you are either symlinking by hand or keeping copies in sync and losing.

So this is two things: four skills worth having, and an installer that puts them wherever they need to go.

npx github:catidegla/skillbelt add --all

That detects which harnesses are on your machine and installs into each. No clone, no npm install, no dependencies.

The skills

i18n-parity is the one I use most and have not seen elsewhere. It compares translation files across locales and reports keys present in one and missing in another, before the missing key ships as a blank string in production. I write bilingual software, French and English, and this class of bug is invisible in review because the reviewer reads one locale.

laravel-security-review and nextjs-security-review encode the checks I would do by hand on a diff: mass assignment, unvalidated file uploads, server actions missing authorisation, environment variables crossing into a client bundle.

secrets-audit looks for credentials committed where they should not be, with attention to the shapes that get missed because they do not look like an AWS key.

The description rule

There is a validator, and its most important rule is about the description field. It has to say when to use the skill, not only what it is.

That description is the only text an agent sees when deciding whether to load a skill. So "use when reviewing a diff, a pull request, or a controller" pulls its weight, and "a skill for Laravel security" does not.

Where this sits

The installer half is not a novel idea. agent-install does the same job and also handles MCP servers and AGENTS.md, and the skills package is building a general registry for the format. For a package manager for skills, use one of those.

What is here is a curated set of four maintained together and validated as a suite, with installation attached.

# Just one skill, just one harness
npx github:catidegla/skillbelt add i18n-parity --harness claude

# See where everything would go first
npx github:catidegla/skillbelt doctor

https://github.com/catidegla/skillbelt

Le Fa, un système binaire à 256 signes du Bénin, et ce qu'il faut pour l'encoder correctement

Christ-loisele Atidegla — Wed, 09 Sep 2026 17:23:42 +0000

Le Fa est le système divinatoire des Fon du Bénin, d'où je viens. Les Yoruba le pratiquent sous le nom d'Ifá, et les deux partagent exactement les mêmes figures.

La structure est binaire, au sens littéral et non métaphorique.

Le mécanisme

Une figure comporte deux colonnes. Chaque colonne a quatre positions. Chaque position porte soit une marque, soit deux.

Quatre positions à deux états donnent seize colonnes possibles. Deux colonnes côte à côte donnent 2⁸ = 256 signes. Ces seize colonnes sont les seize dù principaux, et les 256 paires forment l'espace complet.

Deux instruments les produisent. Seize noix de palme, les ikin, manipulées en huit passes. Ou une chaîne de huit coques, opele en yoruba et agumaga en fon, lancée une seule fois. Dans les deux cas la colonne de droite est produite en premier, ce qui n'est pas un détail de présentation : cela détermine le nom du signe obtenu.

Il existe une règle dans la méthode aux noix qui est l'inverse de ce que tout le monde suppose, moi compris avant vérification :

Une noix restée dans la main signifie une marque double. Deux noix signifient une marque simple.

C'est le détail le plus fréquemment recopié de travers dans les descriptions rapides de la procédure, donc la suite de tests l'affirme dans les deux sens, là où un refactor ne peut pas l'inverser en silence.

Les relations sont la partie intéressante

Une fois les seize figures écrites comme motifs binaires, une structure apparaît qui vaut le détour pour un programmeur.

Le renversement : retournez une figure. Quatre des seize se lisent identiquement dans les deux sens : Gbé, Yèku, Woli et Di. Les douze autres s'apparient en six couples.

Le complément : échangez chaque marque simple contre une double et inversement. Cela apparie les seize en huit couples sans aucun point fixe, ce qui découle du fait que quatre positions ne peuvent jamais toutes différer d'elles-mêmes.

Ces deux opérations partitionnent le même ensemble de deux manières qui ne coïncident pas.

Le difficile, c'était les données

Je pensais que le code serait le travail. Il fait quelques centaines de lignes de manipulation de bits.

Le difficile a été d'établir ce que sont les figures.

Les sources divergent. Pas sur les mathématiques, que personne ne conteste, mais sur le nom attaché à chaque figure et sur l'ordre de la liste. L'ordre varie selon les régions : celui que j'utilise est donné par fongbebenin.com pour les dù fon et par ileifa.org pour les odù yoruba, et les deux concordent position par position, mais au moins une liste béninoise publiée place Ka en onzième.

La bibliothèque traite donc rank comme un identifiant stable pour l'indexation et précise explicitement que ce n'est pas une affirmation sur la préséance. L'ordre de préséance des 256 dépend des lignées et n'est pas encodé du tout.

Deux figures ont posé plus de difficultés que cela. Deux listes largement recopiées donnent à Sa et Trukpen les figures inverses. J'ai failli publier cette erreur, parce que la vérification évidente passe dans les deux cas : de part et d'autre on obtient quatre figures palindromes et six paires par renversement.

Ce qui tranche, ce sont les relations. Guda renversé donne Sa, et Ka renversé donne Trukpen. Intervertissez les deux et les deux relations se brisent simultanément. La version inversée est donc incohérente avec elle-même, ce qui se démontre sans avoir à décider quelle source est la plus fiable.

Ce qui n'y est délibérément pas

Il n'y a aucun verset et aucune interprétation. Un signe revient avec sa figure, ses noms en fon et en yoruba, et ses relations structurelles, et rien sur ce qu'il signifie.

C'est une ligne tracée volontairement. Le corpus interprétatif est immense, il s'enseigne différemment selon les lignées, et une bonne partie relève d'un savoir qui appartient aux initiés. Le comprimer dans un fichier JSON produirait quelque chose à la fois inexact et présomptueux.

Une application qui a la légitimité de porter l'interprétation peut le faire, en s'appuyant sur l'index ou le nom que la bibliothèque fournit. Le rôle de la bibliothèque est de garantir que la structure en dessous est juste.

L'utiliser

npx @catidegla/fadu cast

La direction la plus utile est la lecture d'un tirage réel plutôt que sa simulation, ce dont une application a réellement besoin :

import { readIkin } from '@catidegla/fadu';

readIkin([2, 2, 2, 2, 1, 1, 1, 1]).fon;   // 'Gbé Yèku'

Une passe qui ne laisse aucune noix, ou trois, est refusée plutôt que transformée en une figure que personne n'a tirée.

69 tests, aucune dépendance, et les fichiers data/du.json et data/signs.json sont livrés pour que les données soient exploitables depuis n'importe quel langage. fadu.

Autant que j'aie pu chercher, il n'existait aucune bibliothèque comparable, dans aucun langage. Cette absence est la raison de son existence, et c'est aussi l'avertissement : il n'y a rien pour vérifier les figures sinon les sources, qui sont donc citées intégralement.

Two published tables disagreed, and the structure settled it

Christ-loisele Atidegla — Wed, 09 Sep 2026 17:22:56 +0000

I was encoding the sixteen principal figures of Fa, the divination system of the Fon of Benin. Sixteen rows, four marks each. Trivial data entry.

I nearly shipped two of them swapped, and the way I caught it is worth writing down, because the obvious check passed.

The disagreement

A search summary handed me a table giving the figure 2212 for Sa and 2111 for Trukpen, where 1 is a single stroke and 2 is a double, read top to bottom.

Another source gave the opposite: Sa is 2111, Trukpen is 2212.

Two published tables, contradicting each other on exactly two of the sixteen. Everything else agreed.

The check that did not help

The sixteen figures have structure you can test. Reverse a figure top to bottom and you get another valid figure. Under that operation the set partitions into figures that read the same both ways, and pairs that map to each other.

So: count them. If one arrangement produced a broken partition, that would settle it.

Both arrangements give four palindromes and six reverse pairs. Four plus twelve equals sixteen, cleanly, either way.

That is the shape of this kind of error. The obvious invariant is a property of the figures, and swapping two names does not change the figures. It only changes which name sits beside which pattern, and the counting test cannot see names at all.

The check that did

Reversal also says which figure each one reverses into, which is a claim about pairs of names and not about the set.

Under the second arrangement:

Guda reversed is Sa
Ka reversed is Trukpen

Under the first arrangement, both of those break at once. Guda would reverse into Trukpen and Ka into Sa, which contradicts every table's ordering of those names.

So the swapped version is inconsistent with itself, and you can demonstrate that without appealing to any authority about which source is more trustworthy.

Then I went and read the primary source properly. Wikipedia's article on Ifá prints two tables, one Yoruba and one for the West African Afa-du set, and they agree with each other and with the second arrangement. Ọ̀sá is 2111. Òtúúrúpọ̀n is 2212.

There is now a test named after it:

test('Sa and Trukpen are not swapped', () => {
  assert.equal(du('Sa').marks.join(''), '2111');
  assert.equal(du('Trukpen').marks.join(''), '2212');
  assert.equal(du('Guda').reverse().fon, 'Sa');
  assert.equal(du('Ka').reverse().fon, 'Trukpen');
});

The last two lines are the ones that matter. The first two would pass if I had transcribed the same mistake into both the table and the test.

What I took from it

A consistency check that passes on both candidates is not a check. Counting palindromes felt like verification and verified nothing about the question I had. Ask of any invariant you rely on which wrong answers it would still accept.

Relations between elements are stronger evidence than properties of the set. The cardinality was identical under both arrangements and the pairing was not. Checks that name specific elements catch attribution errors that aggregate checks cannot.

Search summaries are not sources. The wrong table came from a confident-sounding summary, and fetching the actual article took about a minute. That is the second time in a month reading the primary source caught something a summary had backwards.

Say where the uncertainty remains. The ordering of the sixteen genuinely varies by region, and I could not resolve that the same way, because unlike the figures there is no internal structure that makes one ordering self-consistent and another not. So the library documents that rank is an index and not a claim about seniority, and the seniority ordering of the 256 is not encoded at all.

The library is fadu: the sixteen dù, the 256 signs, both casting mechanics, and no interpretation at all. 69 tests, zero dependencies, and every source cited.

A 256-state binary system from Benin, and what it takes to encode it correctly

Christ-loisele Atidegla — Wed, 09 Sep 2026 17:16:59 +0000

Fa is the divination system of the Fon people of Benin, where I am from. The Yoruba practise it as Ifá, and the two share their figures exactly.

The structure is binary, and I mean that literally rather than as a metaphor.

The mechanism

A figure is two columns. Each column has four positions. Each position carries either one stroke or two.

Four positions with two states each gives sixteen possible columns. Two columns side by side gives 2⁸ = 256 signs. Those sixteen columns are the sixteen principal dù, and the 256 pairs are the complete space.

Two instruments produce them. Sixteen palm nuts, called ikin, worked in eight passes. Or a chain of eight pods, opele in Yoruba and agumaga in Fon, thrown once so all eight land together. Either way the right-hand column is produced first, and that decides the name of the resulting sign.

There is a rule in the palm nut method that is inverted from what everyone guesses, including me before I checked:

One nut left in the hand means a double stroke. Two nuts left means a single.

It is the most frequently miscopied detail in casual descriptions of the procedure, so the test suite asserts it in both directions where a refactor cannot quietly flip it.

The relations are the interesting part

Once the sixteen figures are written down as bit patterns, structure falls out that is worth looking at as a programmer.

Reversal: turn a figure upside down. Four of the sixteen read the same either way: Gbé, Yèku, Woli and Di. The other twelve pair off into six couples. Four plus twelve, and the reversal operation is its own inverse.

Complement: swap every single stroke for a double and vice versa. This pairs all sixteen into eight couples with no fixed points at all, which follows from four positions never being able to all differ from themselves.

Those two operations partition the same sixteen elements in two different ways, and the partitions do not agree. That turns out to be load bearing, as the next section shows.

The hard part was the data

I expected the code to be the work. It is a few hundred lines of bit manipulation and the tests write themselves: assert that sixteen figures cover all sixteen four-position states exactly once, assert 256 distinct signs, assert every name parses back to the sign it came from.

The hard part was establishing what the figures are.

Sources disagree. Not on the mathematics, which nobody disputes, but on which name attaches to which figure and in what order they are listed. Ordering varies by region: the sequence I use is given by fongbebenin.com for the Fon dù and by ileifa.org for the Yoruba odù, and those two agree position for position, but at least one published Beninese list moves Ka to eleventh. Wikipedia's own table carries an editorial note saying the order varies.

So the library treats rank as a stable handle for indexing and says explicitly that it is not a claim about seniority. The seniority ordering of the 256 is lineage-specific and is not encoded at all.

Two figures were harder than that, and they get their own article.

What is deliberately not in it

There are no verses and no readings. A sign comes back with its figure, its names in Fon and Yoruba, and its structural relations, and nothing about what it means.

That line is drawn on purpose. The interpretive corpus is enormous, it is taught differently by different lineages, and a good deal of it is knowledge that belongs to initiates. Compressing it into a JSON file would produce something both inaccurate and presumptuous.

An application that has the standing to carry interpretation can carry it, keyed on the index or the name this gives it. The library's job is to make sure the structure underneath is right.

Using it

npx @catidegla/fadu cast

  Loso Fu
  Ìrosùn Òfún

  │ │    │
   │     │
  │ │   │ │
   │    │ │

  1122|2121   mixed, index 79 of 255
  pods, in order: o o c c c o c o

The more useful direction is reading a physical throw rather than simulating one, which is what an application actually needs:

import { readIkin } from '@catidegla/fadu';

readIkin([2, 2, 2, 2, 1, 1, 1, 1]).fon;   // 'Gbé Yèku'

A pass that leaves no nuts, or three, is refused rather than being turned into a figure nobody cast.

Casting draws from the platform CSPRNG instead of Math.random, which costs nothing and seemed like the minimum for the subject matter.

69 tests, zero dependencies, and data/du.json and data/signs.json ship so the data is usable from any language. fadu.

As far as I can find, there was no comparable open source library in any language. That absence is why it exists, and it is also the warning: there is nothing to check the figures against except the sources, which is why they are cited in full.

Exposer vos modèles Eloquent à un agent IA sans lui donner votre base

Christ-loisele Atidegla — Wed, 09 Sep 2026 17:16:13 +0000

Voici le premier outil MCP que presque tout le monde écrit :

class TicketTool extends Tool
{
    public function handle(Request $request): Response
    {
        return Response::structured(Ticket::find($request->get('id')));
    }
}

Il fonctionne, et il contient trois failles de sécurité distinctes qui ne sont pas visibles dans le code.

Un : il renvoie toutes les colonnes

Ticket::find() vous donne le modèle. Sérialiser le modèle donne tous les attributs de la ligne.

Cela inclut internal_notes. Cela inclut le card_last_four ajouté pour un écran de support. Cela inclut la colonne ajoutée la semaine dernière pour une fonctionnalité pas encore livrée.

C'est la plus discrète des trois parce que rien n'a l'air anormal. Vos vues Blade affichent quatre champs, donc vous pensez au modèle comme ayant quatre champs. L'agent les reçoit tous, et le modèle explique volontiers leur contenu à qui le demande.

Deux : il ignore qui pose la question

Il n'y a aucun utilisateur dans ce code. La requête n'est pas filtrée, donc l'outil récupère n'importe quel ticket par identifiant, pour quiconque peut l'appeler.

Ajouter ->where('user_id', auth()->id()) corrige le cas immédiat et en crée un plus subtil : le filtre vit désormais dans l'outil et non dans vos policies. Quand quelqu'un ajoutera un deuxième outil, une relation ou un scope, il faudra s'en souvenir. On ne s'en souviendra pas.

Trois : rien n'empêche l'énumération

find($id) avec un identifiant auto-incrémenté est une invitation. Un agent qui peut appeler l'outil peut l'appeler avec 1, 2, 3, et continuer. Même une fois la policy ajoutée, la forme de l'échec fuit : un enregistrement inexistant et un enregistrement interdit répondent en général différemment, et cette différence cartographie les identifiants existants.

Ce que fait laravel/mcp, et ce qu'il ne fait pas

Rien de tout cela n'est une critique de laravel/mcp. Il compte 34,5 millions d'installations, il est officiel, et il fait très bien son travail. Son travail, c'est le protocole.

Ce qu'il ne fait délibérément pas, c'est décider ce que votre outil a le droit de renvoyer. C'est une préoccupation applicative, et il serait malvenu qu'un package de protocole la devine.

C'est aussi une préoccupation dont la forme est identique dans toutes les applications, ce qui en fait un bon candidat pour un package plutôt que quelque chose que chaque équipe réinvente un vendredi à 17 h.

Déclarer l'exposition

Ce que j'ai retenu, c'est un attribut sur le modèle :

#[AgentResource(
    fields: ['id', 'subject', 'status'],
    searchable: ['subject'],
    filterable: ['status'],
    relations: ['comments', 'author'],
    maxResults: 25,
)]
class Ticket extends Model {}

C'est toute la configuration, et chaque argument ferme une des trois failles.

fields est une liste blanche. La projection travaille depuis cette liste seule et n'inspecte jamais le modèle pour décider quoi inclure, donc ajouter une colonne ne peut pas élargir l'exposition.

maxResults est un plafond appliqué par-dessus ce que demande l'agent, avec un second plafond global en configuration.

L'autorisation passe par vos policies existantes, enregistrement par enregistrement.

Ce que je défends le plus

Chacun de ces mécanismes échoue en se fermant.

Un modèle sans policy lève une exception. Oublier d'écrire une policy est l'erreur la plus probable de tout le processus, donc le comportement par défaut doit être le refus.

Un get refusé est indiscernable d'un enregistrement inexistant. Les deux renvoient null, donc l'outil ne peut pas servir à découvrir quels identifiants existent.

Les lignes refusées dans une liste sont signalées, pas supprimées. Si un outil retire silencieusement des lignes, l'agent croit la liste complète, puis répond avec assurance à une question à son sujet. Le compte revient donc :

['rows' => [...], 'denied' => 2, 'truncated' => false]

Ne pas écrire les outils du tout

Puisque l'attribut décrit déjà tout ce dont un outil a besoin, les outils sont générés :

php artisan agent-kit:mcp

Une classe d'outil laravel/mcp par capacité déclarée, avec le schéma d'entrée dérivé du même attribut. Le handler généré délègue à la ressource et ne fait rien d'autre, ce que le fichier généré indique en en-tête : toutes les garanties vivent dans la ressource, et tout ce qu'on ajoute à un outil s'exécute en dehors d'elles.

composer require catidegla/laravel-agent-kit

58 tests sur PHP 8.2, 8.3 et 8.4, dont un qui charge une classe générée et la fait passer par le sérialiseur de laravel/mcp. Le dépôt est ici.

Letting an AI agent follow a relation without widening what it can see

Christ-loisele Atidegla — Wed, 09 Sep 2026 17:10:16 +0000

Once an agent can read a model, it wants the neighbours. Show me the ticket, and its comments, and who wrote them.

The naive implementation is ->with($request->get('include')), which is a full database export for anyone who asks nicely. Eloquent will follow any relation name you hand it, and from ticket you can reach user, and from user you can reach orders, and from there everything.

The rule that makes it safe

A relation must not widen exposure.

Expanding comments returns exactly what Comment declares in its own attribute, checked against Comment's own policy, for the same signed-in user. Not the parent's field list, not the parent's policy, and not a special relation mode with looser rules.

So a relation composes two exposures that already existed and never creates a third. Whatever an agent can see through ticket.comments, it could already have seen by asking for comments directly, given a tool for that.

Everything else follows from holding that line.

A relation cannot reach an unexposed model

If the target is not a registered resource, the call is refused:

Ticket declares "notes" as traversable but PrivateNote is not a registered
resource. A relation is not a way to reach a model that was never exposed:
the target carries its own field list and its own policy, and without them
there is nothing to enforce.

This closes the obvious back door. Without it, relations: ['notes'] on one model quietly publishes a table nobody put in the exposure list, and the pull request that did it shows one word.

There is a neat consequence. To make a model reachable through relations only, without a tool that can enumerate it, register it with no abilities of its own:

#[AgentResource(fields: ['id', 'name'], abilities: [])]
class User extends Authenticatable {}

Registered, so exposure stays visible in the one config file where it belongs. No tools generated, so nothing can list your users. Reachable as ticket.author, returning two fields, checked against the user policy.

That falls out of composing two existing ideas, with no "relation-only" concept invented for it.

One level, never two

An expanded record does not itself expand relations.

$resource->get($user, $id, include: ['comments']);

The comments come back. Their relations do not. If the agent needs the next hop it makes another tool call, which is separately authorized and separately recorded in the audit trail.

That is a real constraint and I think it is the right one. Depth limits are the usual answer and they are a slider people turn up. Making each hop a separate call keeps each hop visible, and keeps "what did the agent read" answerable instead of turning it into one enormous nested payload.

Dot notation asking for two hops at once is refused for the same reason.

The two failure modes people forget

A to-many relation needs its own ceiling. A ticket with 4,000 comments is a full table dump reached through a relation. The cap comes from the target's declared maximum, not the parent's, and truncation is reported instead of silently trimmed.

A to-one the viewer cannot see returns null. There is a test for this: Alice reads her own ticket and asks for escalatedTo, which points at Bob. author resolves because she may see herself. escalatedTo comes back null. A declared relation does not make its contents hers to read.

The unglamorous part: N+1

Authorizing per record means a policy check per record, which is unavoidable and correct. Issuing a query per record is not.

Relations are eager loaded, so expanding across a page of results costs one extra query per relation instead of one per row. There is a test asserting the query count, so a future refactor that drops the eager load fails instead of quietly getting slow.

The names are validated against the declared list before they reach with(), so nothing arbitrary is passed to Eloquent.

Catching it before production

Bad relation configuration is caught at deploy time:

$problems = app(Registry::class)->verify();

That reports a relation declared with no method behind it, and a relation pointing at a model that is not registered. Run it in a test and a misconfiguration fails the build.

The audit trail records the ids reached through each relation, because "what did the agent read" gives the wrong answer in exactly the interesting case if it stops at the parent:

agent 41 ticket.get -> ok (1 returned, 0 denied) via comments[2]

catidegla/laravel-agent-kit. 58 tests, PHP 8.2 to 8.4.

An AI agent audit trail that is not a second copy of your database

Christ-loisele Atidegla — Wed, 09 Sep 2026 17:09:30 +0000

After anything goes wrong with an AI agent, the first question is always the same: what did it actually read?

You cannot answer that later if nothing was written down at the time. So you add logging, and the obvious implementation is to log what the tool returned.

That is where it goes wrong.

The log becomes the softer target

An audit trail that stores the rows holds, after a month of operation, a copy of every record any agent has ever read. Reconstructed from tool responses, sitting in a table or a log aggregator, and nobody has written a policy for it.

Your tickets table has a policy. Every read goes through it, per record. The audit log has whoever has log access, which is usually a much longer list including anyone who can search your observability platform.

You built a careful exposure layer and then created an unguarded mirror of everything that passed through it.

Identifiers and field names, never values

The design I settled on records what was reached, not what it contained:

{
  "at": "2026-09-09T14:02:11+00:00",
  "resource": "ticket",
  "operation": "get",
  "outcome": "ok",
  "actor": "41",
  "arguments": { "id": 7 },
  "ids": [7],
  "fields": ["id", "subject", "status", "user_id"],
  "denied": 0,
  "truncated": false
}

From that you can answer the question completely. Which records were exposed, to whom, through which tool, when, and which fields of them. To see the values you go back and read those ids from the source, where the policy still applies and the read is itself audited.

fields is the piece people leave out. Without it you know record 7 was read but not whether that included internal_notes, and the field list changes when the attribute changes.

There is one deliberate exception. arguments records what the agent supplied, including a search term, because "what was it looking for" is half of any incident. That is the agent's input, and it should be read back as untrusted text since a prompt can put anything there.

The distinction the agent must not see

get returns null for both a missing record and a denied one, so ids cannot be enumerated.

The operator needs to know which it was. A run of denials is somebody probing; a run of misses is a broken integration. Those need different responses, and the difference is invisible if both are logged the same way.

So the trail records them apart while the agent still cannot tell:

if ($model === null) {
    $this->record($user, 'get', AuditEvent::MISSING, ['id' => $id]);
    return null;
}

if (! $this->gate->allows($user, $model)) {
    $this->record($user, 'get', AuditEvent::DENIED, ['id' => $id], [], 1);
    return null;
}

Identical to the caller, different in the record the caller never sees. That asymmetry is the point of having an audit log at all, and it was not expressible before, because the old code could not tell the two cases apart internally either.

Refused calls are recorded too. One filter on an unexposed field is a typo. A run of them is probing, and that pattern only exists if failures are written down.

Failing closed, including here

The uncomfortable decision: by default a sink that refuses takes the read down with it.

AuditFailedException: The audit sink refused to record ticket.get, so the
result was withheld rather than served unrecorded.

This is the least popular thing in the package and I will make the argument for it. A trail with silent gaps cannot answer the question it exists for, and the moment a gap opens is exactly the moment it is worth the most to whoever caused it. Serving data you cannot account for is the failure the whole package exists to prevent, so the audit trail should not be the one component that degrades quietly.

It is configurable, because availability is a legitimate thing to want more:

'audit' => ['strict' => false],

That is a real trade and it is yours to make. What matters is that it is a decision and not an accident.

Where it goes

The default writes to a Laravel log channel, so it works on a fresh install with no migration and no table to forget. That is a starting point. Log files rotate, and "what did the agent read three months ago" is eventually a query.

So the sink is an interface. Bind your own and the package uses it:

$this->app->bind(AuditSink::class, YourDatabaseSink::class);

There is also an ArraySink shipped for your own test suite, because the useful thing to assert in an application test is that a tool call produced the record you expected.

The one line version

Log which records were exposed and to whom, and not what was in them. The source of truth already has a policy; your audit log does not.

catidegla/laravel-agent-kit, and there is a test asserting that no field value ever reaches an audit record.

Your Laravel MCP tool returns every column, and three lines of code hide it

Christ-loisele Atidegla — Wed, 09 Sep 2026 17:03:33 +0000

This is the first MCP tool nearly everyone writes:

class TicketTool extends Tool
{
    public function handle(Request $request): Response
    {
        return Response::structured(Ticket::find($request->get('id')));
    }
}

It works, and it contains three separate security failures that are not visible in the code.

One: it returns every column

Ticket::find() gives you the model. Serialising the model gives you every attribute on the row.

That includes internal_notes. It includes the card_last_four somebody added for a support screen. It includes whatever column was added last week for a feature that has not shipped, and the admin_flag that nothing in your UI renders.

This is the quietest of the three because nothing looks wrong. Your Blade templates render four fields, so you think of the model as having four fields. The agent gets all of them, and the model happily explains their contents to whoever asked.

Two: it has no idea who is asking

There is no user in that code. The query is unscoped, so the tool will fetch any ticket by id, for anyone who can reach the tool.

Adding ->where('user_id', auth()->id()) fixes the immediate case and creates a subtler one. The filter now lives in the tool instead of in your policies, so when someone adds a second tool, or a relation, or a scope, the rule has to be remembered again. It will not be.

The check belongs where every other authorization check in the application lives.

Three: nothing stops enumeration

find($id) with an incrementing id is an invitation. An agent that can call the tool can call it with 1, 2, 3, and keep going. Even with the policy check added, the shape of the failure leaks: a missing record and a forbidden one usually answer differently, and that difference maps out which ids exist.

What laravel/mcp is and is not

None of this is a criticism of laravel/mcp. It has 34.8 million installs, it is official, and it does its job well. Its job is the protocol: define a tool, let an agent call it, handle transport and schemas.

What it deliberately does not do is decide what your tool is allowed to hand back. That is an application concern, and it would be wrong for a protocol package to guess at it.

It is also a concern with a consistent shape across every application, which makes it a good candidate for a package instead of something each team reinvents at 4pm on a Friday.

Declaring the exposure instead

What I ended up with is an attribute on the model:

#[AgentResource(
    fields: ['id', 'subject', 'status'],
    searchable: ['subject'],
    filterable: ['status'],
    description: 'Support tickets belonging to the signed in user.',
    maxResults: 25,
)]
class Ticket extends Model {}

That is the whole configuration, and each argument closes one of the three failures.

fields is an allowlist. Projection works from that list only and never inspects the model to decide what to include, so adding a column cannot widen exposure. Anything that is not a scalar after casting is refused, because a nested array arriving from a cast would return contents nobody declared.

maxResults is a ceiling applied on top of whatever the agent asks for, with a second global ceiling in config so no single attribute can raise the limit for the whole application.

Authorization runs through your existing policies, per record, not once per query.

The part I would defend hardest

Every one of these fails closed.

A model with no policy throws. Forgetting to write a policy is the most likely mistake in the whole flow, so the default has to be refusal.

A denied get is indistinguishable from a missing one. Both return null, so the tool cannot be used to discover which ids exist.

Denied rows in a list are reported, not dropped. This one took the longest to get right. A tool that silently removes rows the caller cannot see leaves the agent believing the list is complete, and it then answers questions about it with confidence. So the count comes back:

['rows' => [...], 'denied' => 2, 'truncated' => false]

Truncation is reported for the same reason. The query fetches one row over the limit so the response can say whether there were more.

Exposure is a list, not a scan

One deliberate non-feature: the package does not scan your codebase for the attribute. Exposed models are listed explicitly in config.

With a scan, adding an attribute anywhere publishes a table, and the reviewer of that pull request sees one line in a model file. With a list, they see a change to the application's exposed surface, which is what it is.

Not writing the tools at all

The attribute already describes everything a tool needs, so the tools are generated:

php artisan agent-kit:mcp

One laravel/mcp tool class per declared ability, with the input schema derived from the same attribute, so the agent sees which fields are filterable and what ceiling it will be held to. The generated handler delegates to the resource and does nothing else, which the generated file says at the top, because every guarantee lives in the resource and anything added to a tool runs outside all of them.

It refuses to generate from an exposure that fails verification, and it will not overwrite an existing file without --force.

composer require catidegla/laravel-agent-kit

58 tests across PHP 8.2, 8.3 and 8.4, including one that loads a generated tool class and runs it through laravel/mcp's own serialiser. Repo here.

Zero is singular in French, and five other translation bugs that never throw

Christ-loisele Atidegla — Wed, 09 Sep 2026 17:02:47 +0000

Translation bugs are quiet. Nothing throws, tests pass, and the interface says something slightly wrong to half your users. You find out when one of them tells you, if they bother.

I ship bilingual software in French and English. These are the six that keep happening.

1. The placeholder got translated

The worst one, because the placeholder name looks like a word:

// en
'welcome' => 'Hello :name, you have :count messages',

// fr, translated by someone being thorough
'welcome' => 'Bonjour :nom, vous avez :count messages',

:nom is not a placeholder. Nothing substitutes it, and the user sees the literal characters :nom in the middle of a sentence.

Easy mistake, easy to catch mechanically: compare the placeholder sets on both sides of every key and fail if they differ. Same rule for ICU and next-intl, where {name} stays {name} and only the surrounding text changes.

This one belongs in CI. It has no false positives.

2. Zero is plural in English and singular in French

Count	English	French
0	0 items	0 article
1	1 item	1 article
2	2 items	2 articles

English treats zero as plural. French treats it as singular. Every French interface saying "0 articles" got there this way.

Laravel's MessageSelector already implements the French rule, so trans_choice picks correctly with no work from you. The bug appears when someone builds the string by hand:

// Wrong: the English rule, applied to French.
$label = $count === 1 ? 'article' : 'articles';

// Right.
trans_choice('messages.articles', $count);

// lang/fr/messages.php
'articles' => 'article|articles',

This generalises past French. A ternary on a count encodes one language's rule into your control flow, where no translation file can reach it.

3. ICU categories do not map across languages

On next-intl or formatjs, plural categories are per language and copying the English structure does not work:

{count, plural, =0 {Aucun article} one {# article} other {# articles}}

Arabic has six plural categories. Russian and Polish have three. Missing categories make the formatter fall back to other silently, producing grammatically wrong output with no error anywhere.

If you target any of those languages, read the CLDR plural rules for them.

4. An empty string is worse than a missing key

{ "nav.settings": "" }

A missing key usually triggers fallback to the source locale, so the user sees English. Readable, if not ideal.

An empty string often does not, because fallback logic checks whether the key exists. The user gets a blank button.

So a parity check has to treat empty values as failures instead of as present keys. People forget this case precisely because the key is technically there.

5. French typography is not English typography with accents

Native speakers notice these instantly and reviewers miss all of them.

Narrow non-breaking space before ; : ! ? and inside « ». It is Bonjour !, not Bonjour!. The character is U+202F, with U+00A0 as the widely supported fallback. With a plain space the punctuation can wrap to the next line by itself.
Guillemets for quotations. « comme ceci », not "like this".
Decimal comma, space for thousands. 1 234,56, not 1,234.56. Use Intl.NumberFormat('fr-FR') or PHP's NumberFormatter, never string manipulation.
Lowercase months and weekdays. 7 septembre 2026, not 7 Septembre 2026.
Currency after the amount. 12,50 €, not €12.50.
Sentence case in titles. Paramètres du compte, not Paramètres Du Compte.

A key-by-key diff catches none of these, so a checker is necessary and not sufficient.

6. The string that was never extracted

Keys present in both locales is half the job. A hardcoded string in a template is never missing from a locale file, because it was never in one.

Parity checks compare files against each other, and a hardcoded string is in neither.

What to automate and what to review

Split them by whether a machine can be certain.

Mechanical, belongs in CI, fails the build: missing keys, empty values, placeholder mismatches, plural branch count mismatches. Each is a definite defect with no legitimate reading.

Warns but does not block: extra keys in the target, and values identical across locales. Sometimes a string genuinely is the same in both languages, and failing the build on "Email" being spelled the same way in French trains people to disable the check.

That split matters more than the checks themselves. A parity checker that fails on things which are sometimes fine gets switched off, taking the ones that are never fine with it.

I packaged this as an agent skill rather than only a script, because the second half of the list is judgement rather than comparison. The checker handles the mechanical part:

node skills/i18n-parity/scripts/check-parity.mjs

It auto-detects Laravel lang/{locale}/*.php, Laravel JSON, next-intl messages/{locale}.json and react-i18next locales/{locale}/*.json, exits 1 on anything a user would see, and warns on the rest.

The skill then carries the rules a diff cannot check, so an agent reviewing a translation pull request knows that a ternary on a count is suspect and that Bonjour! is missing a space.

npx github:catidegla/skillbelt add i18n-parity

skillbelt also carries Laravel and Next.js security review skills and a secrets audit, and installs into Claude Code, Codex, Cursor, Gemini CLI and Antigravity, which all read the same SKILL.md format and disagree about which directory it lives in.

What nDCG sees that precision and recall miss

Christ-loisele Atidegla — Wed, 09 Sep 2026 16:56:50 +0000

Two retrieval runs for the same query. Relevant document is D.

Run A:  [D, x, x, x, x, x, x, x]
Run B:  [x, x, x, x, x, x, x, D]

Precision@8 is identical: one relevant document out of eight, 0.125 both times. Recall@8 is identical: you found the one that existed, 1.0 both times. Hit rate is 1.0 both times.

Every set-based metric says these runs are the same. For a RAG system they are not.

Why position is most of the quality

Retrieval feeds a context window, and a context window is ordered and finite.

Take the top 4 chunks and run A includes D while run B does not. Recall@8 was 1.0 and your actual recall at the size you use is 0. The metric measured a depth you never read from.

Even when everything fits, position matters. Models attend unevenly across long contexts, and material buried in the middle of a large window gets used less reliably than material at the top. The practical effect today is that rank order is part of your quality, and a metric that ignores it ignores the part you can most easily improve.

What nDCG does

Discounted cumulative gain gives each hit a value that shrinks with position, using a logarithm so the penalty is steep near the top and gentle further down:

top.forEach((id, i) => {
  if (relevantSet.has(id)) dcg += 1 / Math.log2(i + 2);
});

Position 1 contributes 1 / log2(2) = 1.0. Position 2 contributes 1 / log2(3) ≈ 0.63. Position 8 contributes 1 / log2(9) ≈ 0.32.

That shape matches how people and models use ranked results. The gap between first and second is worth more than the gap between seventh and eighth.

The n is the normalisation. Raw DCG is not comparable across queries, because a query with five relevant documents can score higher than one with a single relevant document without being better retrieval. So you divide by the best achievable ordering for that query:

let idcg = 0;
const ideal = Math.min(relevant.length, k);
for (let i = 0; i < ideal; i++) idcg += 1 / Math.log2(i + 2);

return idcg === 0 ? 0 : dcg / idcg;

Every relevant document first, as many as could fit in k. The result is 1.0 for perfect ordering and comparable across queries with different numbers of relevant documents.

For the two runs above, A scores 1.0 and B scores about 0.32, which is the difference precision and recall could not see.

Which metric answers which question

They are not competing. A report should have all of them.

Metric	The question it answers
`hit_rate`	Did we find anything at all? The floor below which nothing else matters.
`recall@k`	Could the model possibly have got it right? It cannot cite what was never fetched.
`precision@k`	How much noise is in the context window? This is what predicts hallucination.
`mrr`	How far down is the first useful thing?
`ndcg@k`	Is the good material near the top, or merely present?

Read them in that order, and fix them in that order. Hit rate first, since ranking is irrelevant if you are not retrieving anything relevant. Then recall, which bounds everything downstream. Then nDCG, usually where the cheapest wins are, because reranking is a smaller change than reindexing.

Two traps in reporting them

Report k and use the k you actually read. recall@100 is a fine diagnostic and a bad headline if your prompt takes 5 chunks.

Do not average away a metric with missing inputs. A case with no relevance labels has no meaningful recall, and scoring it zero drags the average down for a labelling gap.

All five are in ragbench, computed deterministically with no model in the loop, so the same dataset and predictions give the same numbers on any machine. The nDCG test asserts exactly the property this article is about: a document found first scores higher than the same document found last, while precision and recall report both as identical.

Your eval thresholds will not catch the regression, and here is the arithmetic

Christ-loisele Atidegla — Wed, 09 Sep 2026 16:56:03 +0000

You set a threshold on your retrieval quality. recall@k >= 0.8. The suite runs at 0.94, comfortably above, and stays green.

Then someone changes the chunking strategy, and recall drops to 0.85.

Your gate says nothing, because 0.85 is still above 0.8. You have lost nine points of recall, which is one question in eleven that can no longer be answered, and the check you installed to catch that reports success.

Two different questions

A threshold asks whether the system is good enough. You set it once, from what you believed acceptable at the time, and it rarely changes.

A regression check asks whether this run is worse than the last one. It has no opinion about what good is.

Most eval tooling gives you the first. The second catches the change nobody meant to make, because real systems do not fall off a cliff. They lose two points here and three there, each individually invisible, and a suite with headroom absorbs all of it until someone notices the product got worse and nobody can say when.

The headroom is the problem. The more comfortably you pass your threshold, the more quality you can lose without being told.

What the comparison needs

Three things, and the third is the one people skip.

A stored baseline, somewhere to record what "last time" was. A local SQLite file is enough.

ragbench run --dataset cases.jsonl --exec "python my_rag.py" --label main

A comparison on the branch.

ragbench gate --baseline main --threshold recall@k=0.8

  recall@k       0.5000  -0.5000
  x recall@k fell 0.5000, from 1.0000 to 0.5000

Exit code 1. The pull request is red and you know which metric moved before anyone reads the diff.

A tolerance band, which decides whether the gate survives contact with reality. Retrieval scores move slightly for reasons that are not your change: an embedding service updates, a tie breaks differently, a timestamp shifts an ordering. A gate that fires on that is wrong, and a gate that fires on noise gets disabled inside a week.

So drops inside the tolerance are reported as warnings. The default is one percentage point:

export const DEFAULT_TOLERANCE = 0.01;

Set it from your own observed run-to-run variance. Run the same commit five times, see how much it moves, set the band above that.

The failure mode nobody plans for

What should happen when a threshold names a metric the run never produced?

The comfortable answer is to skip it. The metric is not there, so there is nothing to compare, so pass.

That looks like success and is not. Gate on recall@k, lose the relevance labels from your dataset, and the metric stops being produced, the threshold stops being checked, and the gate reports green forever while checking nothing.

So a threshold on an absent metric fails:

recall@k has a threshold of 0.8 but was not produced by this run

Treating it as passing hides that the check never ran, and the value of a gate is that green means something.

The same reasoning covers the first run. There is no baseline yet, so the gate passes, and it says so out loud instead of reporting a clean comparison that never happened.

Things worth refusing outright

Three dataset rules that are all the same rule.

A case with nothing to check against is rejected. No expected answer, no relevant documents, no required phrases. It passes every metric vacuously and lifts your average.

A duplicate id is rejected. Predictions join to cases by id, so a duplicate makes half your suite vanish without a word.

A case with no prediction scores as empty instead of skipped. Skipping lets a system that answered nothing score identically to one that answered everything correctly.

Each is a way for a suite to look healthier than it is, which is the only real failure mode of an evaluation harness.

npx ragbench gate --baseline main --threshold recall@k=0.8

Deterministic metrics, no LLM judge, no API key, local SQLite history. ragbench.

An LLM judge cannot be a build gate, and it is not about the cost

Christ-loisele Atidegla — Wed, 09 Sep 2026 16:50:07 +0000

Almost every RAG evaluation metric on offer needs a language model to produce it. Faithfulness, answer relevance, context precision: a model reads the answer and scores it.

Those are good metrics. They measure things that are hard to measure otherwise, and for a research sweep or a quarterly quality review, use them.

They cannot gate a build, and the reason people usually give is only half of it.

The half everyone says

A judged evaluation costs money per run. A hundred cases across a few metrics is a few thousand model calls, which is real money on every commit and every branch.

The consequence is that you move the gate. It runs nightly instead of per commit, then weekly, then on a button someone remembers to press. A check that costs a dollar gets run less, and a check that runs less catches things later, which is the property you were buying.

The half that matters more

A judge is not deterministic.

Run the same dataset against the same predictions twice and you get slightly different scores. Not wildly different, but different. That is workable for a report and disqualifying for a gate, because a gate exists to answer one question: did this change make things worse?

Answering it means comparing two numbers. If both carry noise of unknown size, you cannot separate a regression from the measurement. You get a check that fails sometimes for no reason, and the standard response to that is to disable it, usually within a week, usually by someone under deadline pressure.

So the judged metric fails twice: too expensive to run often, and untrustworthy on the difference when it does run.

What is left when you remove the model

More than you would think, and all of it deterministic.

From labelled relevant documents: precision@k, recall@k, MRR, nDCG@k, hit rate. These need a golden dataset and nothing else.

From expected answers: exact match after SQuAD style normalisation, token F1 for partial credit, required phrase presence for when an amount or a date must appear.

Recall bounds everything downstream, so it is the one to be loudest about. The model cannot cite what retrieval never fetched. At recall@k of 0.6, forty percent of your questions were unanswerable before generation began, and no amount of prompt engineering touches that.

Precision measures how much noise is in the context window, which is what predicts hallucination.

The honest limit

One measure in my own tool is a proxy, and the documentation says so.

groundedness is the share of answer content words that appear in the retrieved context:

const answerTokens = normalise(predicted).split(' ').filter((t) => t.length > 2);
const contextTokens = new Set(normalise(contexts.join(' ')).split(' '));

return answerTokens.filter((t) => contextTokens.has(t)).length / answerTokens.length;

That is lexical overlap. It will miss a fluent misreading of a passage that was correctly retrieved, which a judge would catch. It does catch an answer invented wholesale, which is the failure that gets shipped, and it costs nothing and returns the same number every time.

The trade is a cheap proxy running on every commit against an accurate measure running monthly. Which one is the better metric and which one is still switched on in March are different questions.

Missing inputs are not zeroes

One detail decides whether people trust the output.

If a case has no labelled relevant documents, the retrieval metrics for that case are omitted rather than scored zero. Aggregation skips missing values instead of averaging them in:

if (typeof value !== 'number' || Number.isNaN(value)) continue;

A partially labelled dataset should report what it can measure. Scoring the gaps as zero produces a column of failures that describes your labelling rather than your system, and nobody reads a report full of zeroes twice.

Why I wrote one

The platforms are priced for teams. Confident AI runs Free, Starter at $200 a month and Team at $2,000, with Enterprise above that. Braintrust Pro is $249. Galileo Pro is $100. Each has a free tier and each meters it: Confident AI's is two seats, one project and five test runs a week, which a per-commit gate exhausts by Tuesday. (Checked on the vendors' own pages, 9 September 2026.)

The open source libraries, RAGAS and DeepEval, compute good metrics and leave you to build the storage, the comparison and the CI gate yourself. Most solo projects end up with three tools wired together and no gate at all.

npx ragbench gate --baseline main --threshold recall@k=0.8

Local SQLite for history, zero dependencies, nothing leaves your machine. ragbench.

The gate has a second half that matters more than the thresholds, and that is the next article.