This is a collaboration post with Aditya Shukla, Kartik Gupta, and David Tingle from the Georgian AI Lab. We partnered with them to measure how coding agents find and use Expo's agent skills.
You write a skill for your coding agent plugin, try it with a prompt that mentions the feature by name, watch the agent do the right thing, and ship it. That's basically "it worked on my machine" for AI tooling. The problem is you have no idea whether a real user's vague, ordinary request will ever cause the agent to reach for that skill in the first place.
That was the gap we had at Expo. We bundle our agent skills into one plugin for Claude Code and Codex, covering native interfaces, navigation, data fetching, animations, Expo Modules, and our cloud services. We kept adding skills and polishing descriptions, but we had no real signal on whether any of it was landing. So we built an eval harness with Georgian AI Lab, wired it into CI, and used it to find a problem we didn't know we had: our skills were barely getting loaded at all. One targeted fix later, the share of agent sessions that loaded a relevant Expo skill went from 9% to 55%.
Here's how we got there, and what it taught us about testing AI tooling instead of just vibing it.
Direct invocation tells you less than you think
As the people who write these skills, we already know the invocation patterns. Testing the Expo UI skill meant literally typing /expo-ui in Claude Code or $expo-ui in Codex, then checking whether the agent used it well once loaded.
That tells you the skill is useful once it's in context. It tells you nothing about discovery. Real users don't name skills. They say things like "build an iOS todo app with Expo and make it feel native" or "add tabs and a settings screen to this app." Before any guidance inside a skill can help, the agent first has to decide, on its own, that the skill is relevant enough to open. We had no way to measure that decision.
This lines up with something Georgian AI Lab has written about more generally: a final answer doesn't show you everything that happened on the way to it. In Your AI Systems Need an Eval Loop, Asna Shafiq describes how traces let you observe agent behavior, but you need evals to turn those observations into something measurable. We applied the same idea to skills: instead of checking whether a skill works in isolation, test whether an agent finds the right mix of skills while doing ordinary app-development work.
What we decided to measure
The harness takes a product requirements document as a task, runs a coding agent against it, and evaluates both the process (the agent's trajectory and skill usage) and the product (the app it built).
Each scenario in our dataset, whether it's a one-shot app build, a crashing dev build to debug, or a feature to add to an existing app, is annotated up front with what skills we expect the agent to find useful, what evidence should show up if the guidance was actually followed, and which feature primitives an end-to-end task implies.
That gives us three layers of evidence:
- Skill triggering: did the agent load the skills relevant to the task?
- Skill uptake: is there evidence the agent followed guidance it loaded?
- App behavior: does the resulting app build and do what the PRD describes?
These matter separately. A skill that never loads can't influence anything, so triggering has to be measured on its own. A skill can load and get ignored, so uptake is its own check. And an agent can build a working feature from its existing knowledge without touching the skill we expected, which is fine. We don't want to penalize an agent for doing the job correctly just because it didn't reach for our documentation; if anything, that's a sign a skill has been made redundant by model capability. Splitting these into separate layers made it much easier to see where to actually intervene.
The harness starts from product requirements and expected behaviors, runs the task through a coding agent, and keeps both the built app and the agent trace as artifacts. From there it separately checks skill triggering, evidence of uptake in the source tree, and app behavior on iOS.
The harness runs on EAS Workflows and is wired to the Expo Skills repository. Anyone shipping a skill change can add an eval label to their pull request, and the harness runs the chosen scenarios against both main and the PR branch, posting results directly in the PR. Skills are now developed the way we develop code: make a change, run it against a set of scenarios, look at the evidence, decide what to do next.
The traces showed our skills weren't loading
The first thing the traces told us was uncomfortable: the Expo skills we expected the agent to reach for were almost never loaded.
This was easy to miss by eyeballing output, because the apps still looked fine. The agent already knows enough about Expo and React Native to produce reasonable code without our help, so scanning a few files wouldn't reveal that our plugin had contributed nothing. The trace made the gap visible in a way the final answer couldn't.
We had three candidate explanations: the agent might already feel confident enough to skip looking for guidance, the skill descriptions in the frontmatter might not communicate when to use each skill, or the plugin's growing skill count might simply make the whole collection harder to pick through. We started with the easiest fix, rewriting skill names and descriptions, since that's literally what the agent reads when deciding relevance. It helped, but not enough. That pushed us toward the third hypothesis, which we'd suspected for a while.
Giving the agent one door to walk through
Expo's surface area is wide: Expo SDK, Expo Router, Expo Modules, native UI, cloud services. Splitting that into focused skills keeps context usage sane, but it also means the agent has to choose among more than twenty skills before it's done anything else.
So we built [expo-overview](https://github.com/expo/skills/pull/107), an entrypoint skill and router for Expo and EAS tasks. Its description establishes it as the starting point for any Expo, EAS, or React Native request. Inside, a map connects common goals to the more specialized skills that actually contain the relevant guidance, plus a small set of setup rules that apply everywhere, like detecting the Expo SDK version and installing packages with npx expo install. (Sentry and Convex landed on a similar pattern for their own skill plugins, independently.)
The overview skill doesn't reduce the number of skills the agent eventually has to pick from. It fixes a different failure mode: the agent not opening any Expo skill at all. An app-building task already asks a lot of an agent (architecture, dependencies, navigation, state), so adding "choose correctly among 20+ skills" to that pile apparently made it easier to just skip the step entirely. Giving it one clear first hop lowers the activation energy of reaching for a skill at all.
The entrypoint reduces the friction of that first hop, making it more likely the agent reaches for a relevant skill at all.
From 9% to 55%
We ran the PR with the eval label, and the first signal was the agent narrating its own decision:
"I'll start by loading the expo-overview skill, since this task involves building an Expo/React Native app..."
In another run:
"I'll start by loading the Expo overview skill, since it must run first for any Expo/EAS task."
Nobody had named expo-overview in the prompt. The agent recognized an Expo-shaped task, picked the entrypoint, and used the routing info inside it to decide what else to load. Across three sample PRDs, the trace showed consistent chains: expo-overview → expo-project-structure for a notes app, extending to expo-router for one with navigation, and all the way to expo-ui for one with custom UI.
To check whether this held up, we reran Claude Code with sonnet-5-high against 10 greenfield app PRDs (each mentioning Expo but never naming a skill), comparing the plugin before and after expo-overview, three runs each to smooth out variance. In the baseline, about 10% of agent sessions loaded any Expo skill at all. With expo-overview, that jumped to 55% of sessions loading one or more downstream skills, with every session loading the overview itself.
A downstream skill is any non-expo-overview skill. In the baseline, without expo-overview, that's just any Expo skill. Uptake is evidence of the expected pattern showing up in the authored source.
The harder number, Relevant Skill Recall (did the agent load the specific skills we'd mapped to that task), moved from 2% to 18%. Better, but still telling us the agent was usually grabbing only a subset of what was relevant. Looking at individual skills, the ones used most were tied to early, concrete decisions, scaffolding the project (expo-project-structure) or setting up routes (expo-router). In one trace, after loading the overview, the agent said: "Now let's convert it to use Expo Router (needed for tabs navigation) and check the router skill for setup steps."
The misses were just as informative. In one case the agent built RSS and Atom feed fetching entirely from its own knowledge without ever opening expo-data-fetching. In another, it loaded expo-ui for controls but skipped expo-native-ui, even though expo-overview recommends using them together, and the resulting code used Dimensions.get() and React Native's SafeAreaView instead of the patterns that skill would have steered it toward. The pattern across the dataset: agents reach for relevant skills right after reading the overview, then lean on it less as the task goes on, as if the overview is read once rather than treated as a router to come back to.
That's still early signal from a handful of tasks, not a solved problem. But it's a real difference we could measure, which is the thing we didn't have before.
The catalog has a token budget, and your skill description can get cut
Testing in Claude Code made us curious about other agents, so we looked at the open source Codex CLI. After the system prompt, Codex builds a Skill Catalog listing each available skill's name, description, and file path, and that catalog has a fixed total length shared across every installed skill. In the version we checked, the allowance was exactly 2% of the context window: 5,440 tokens out of 272k.
We wanted to know what happens when Expo's skills compete with a bunch of unrelated skills for that budget. Once the catalog exceeds the 5.4k token allowance, Codex starts by truncating descriptions to just their opening words. Push further and descriptions disappear entirely, leaving bare names. Push further still and entire skills get dropped from the catalog.
We saw this in a test environment with 175 skills installed (24 from the Expo plugin). With the 272k context window, every Expo skill was still listed, but only the first 60 characters or so of each description survived.
So as a user's installed skill catalog grows, the agent is choosing among more options while getting less information about each one. The name and opening sentence of a skill description can be the only thing that survives. If you're authoring skills for any agent plugin, test them in a crowded catalog, not just in isolation, because an isolated catalog only tells you if your skill is legible on its own. It won't tell you if it's still visible once 150 other skills are competing for the same 2% of context.
What this changed for how we build skills
A pile of individually good skills doesn't automatically add up to a usable plugin. Past a handful of skills, you need some kind of routing layer, something that tells the agent not just what each skill says but how they relate and where to start.
Testing skills by direct invocation is still useful for checking the contents of one skill, but it can't tell you whether an agent discovers that skill from a normal request. Starting from broad product requirements and reading the resulting traces is what surfaced the actual selection problem. And the environment matters as much as the skill: a crowded, realistic catalog produces different behavior than a clean one with just your plugin installed, so that needs to be part of the eval too.
Finally, triggering is a prerequisite, not an afterthought. We're still improving our uptake checks and the app-level evaluator, because the real goal is a better app, not a loaded skill for its own sake. But if a skill never enters the agent's context, nothing inside it matters for that run.
Where to start
If you're building agent skills or a similar plugin and want to borrow this approach:
- Stop testing by direct invocation alone. Write task prompts the way a real user would phrase them, with no skill names, and read the agent's trace to see what it actually opens.
- If you have more than a handful of skills, build an entrypoint skill that routes to the rest, and measure skill-loading rates before and after.
- Test your skill descriptions inside a crowded, realistic catalog, not in isolation, since token truncation can quietly cut your description down to its first few words.
The full harness and skill source are in the Expo Skills repository if you want to see how the pieces fit together.
This post is based on content from the Expo blog. Follow @expo for more React Native content.
Top comments (1)
Some comments may only be visible to logged-in visitors. Sign in to view all comments.