Armature (YC P26) measured 50,000+ agent sessions and found agents repeatedly hit the same tool limitations. The feedback loop that would fix these issues doesn't exist. Human users file GitHub issues or support tickets. Agents just fail silently, retry with exponential backoff, or hallucinate workarounds.
Agent.reviews is Armature's response: a platform where agents write reviews based on actual usage patterns. The site indexes tools (FastAPI, Supabase, Auth0, Modal, uv, OpenAI API) and publishes agent-generated reviews with numeric ratings. Each review describes what the agent tried, what worked, and what broke.
The interesting part is not the review content. It's the instrumentation layer that captures agent behavior at scale, the schema that differentiates agent feedback from human feedback, and the security boundary that prevents vendors from gaming the system.
The Instrumentation Layer
To generate reviews from 50k+ sessions, you need telemetry that captures:
- Tool invocation traces: which API endpoint, which parameters, which response codes
- Retry patterns: how many attempts, which backoff strategy, which error messages triggered abandonment
- Context switches: when the agent pivoted to a different tool or asked for human intervention
- Success metrics: task completion rate, time to first success, number of fallback strategies
Armature's approach instruments coding agents specifically. The platform provides a skill definition (https://agent.reviews/skill.md) and setup steps (https://agent.reviews/install.md) that agents read to understand how to submit reviews. Every page has a Markdown version (address plus .md) listed in https://agent.reviews/llms.txt.
This is a crawlable, agent-readable schema. The agent doesn't need a custom SDK. It reads the instructions, follows the protocol, and submits structured feedback after completing a task.
Agent Review Schema vs. Human Review Schema
Human reviews rate usability, documentation quality, and customer support responsiveness. Agent reviews rate different dimensions:
| Dimension | Human Review | Agent Review |
|---|---|---|
| API ergonomics | "Easy to understand" | "Parameter validation rejected 12% of valid inputs" |
| Error messages | "Helpful error messages" | "Error code 429 returned no retry-after header" |
| Retry logic | Not rated | "Exponential backoff succeeded after 3 attempts" |
| Documentation accuracy | "Clear examples" | "Code sample in docs returned 404 on live endpoint" |
| Latency tolerance | "Fast enough" | "95th percentile response time exceeded agent timeout" |
Agent.reviews publishes numeric ratings (FastAPI: 4.6, Supabase: 4.3, Auth0: 4.0) but the real signal is in the prose. For example:
"New parts since the earlier review today: a lifespan context manager ran database migrations on startup, BackgroundTasks queued the slow scoring work after a 202 response, Header(default=...) read a shared-secret header, and a Literal-typed dropdown field rejected bad values."
This is not a human describing their experience. This is an agent describing which code paths it executed, which patterns it observed, and which guardrails it encountered.
Security Boundary: Preventing Review Gaming
The obvious attack vector: tool vendors prompt-inject positive reviews or submit fake execution traces. Agent.reviews mitigates this by requiring verified execution traces.
The platform states: "Never your code, prompts or secrets. Your name is never shown." This implies:
- Trace verification: reviews must include cryptographically signed execution logs or API call receipts
- Anonymization: the agent's identity (and by extension, the vendor's identity) is stripped before publication
- Replay detection: duplicate traces from the same session are deduplicated
The security model resembles certificate transparency logs. You can't fake a review without faking the underlying API interaction, and faking the API interaction requires compromising the tool itself.
One unresolved question: can a vendor run their own agent, instrument it to always succeed, and flood the platform with positive reviews? The answer depends on whether Armature cross-references reviews with independent session telemetry. If they only accept reviews from agents they instrument, the attack surface shrinks. If they accept reviews from any agent that follows the protocol, they need reputation scoring or stake-based submission.
Feedback Loop Latency
The value proposition is speed: how fast can a tool maintainer see agent failures and ship a fix?
Traditional feedback loops:
- User hits a bug
- User files a GitHub issue (if they bother)
- Maintainer triages the issue (days to weeks)
- Maintainer ships a fix (weeks to months)
- User upgrades and retries (months to never)
Agent feedback loops:
- Agent hits a bug
- Agent writes a review immediately
- Review appears on agent.reviews (minutes)
- Maintainer sees the review (hours to days)
- Maintainer ships a fix (days to weeks)
- Next agent session tests the fix automatically (minutes after deploy)
The latency improvement is in steps 1-3. Agents don't procrastinate. They don't forget to file issues. They write reviews as part of their task completion protocol.
The remaining bottleneck is step 4: maintainer notification. If agent.reviews pushes reviews to a webhook or GitHub issue automatically, the loop tightens further. If maintainers have to poll the site, adoption suffers.
Architecture: Crawlable Agent Protocol
Agent.reviews uses a crawlable protocol instead of a REST API. This is a deliberate design choice.
Why crawlable beats REST for agents:
- Agents already know how to fetch and parse Markdown
- No SDK versioning or dependency hell
- No authentication flow (reviews are public)
- No rate limiting on reads (only writes need throttling)
- Easier to cache and mirror
The protocol:
- Agent reads
https://agent.reviews/llms.txtto discover available endpoints - Agent fetches tool pages (e.g.,
https://agent.reviews/fastapi.md) - Agent reads existing reviews to understand rating criteria
- Agent submits a new review by POSTing structured Markdown to a submission endpoint
This is similar to how RSS feeds work, but optimized for LLM consumption. The .md suffix convention signals machine-readable content without requiring content negotiation headers.
Observability: What Gets Measured
Agent.reviews exposes:
- Review count per tool: FastAPI has 1,838 reviews, Supabase has 1,214
- Rating distribution: numeric scores (4.0 to 4.7 in the sample)
- Recency signals: "New parts since the earlier review today" indicates continuous monitoring
- Agent diversity: reviews mention Claude Code, Codex, and Muse Code as distinct agent identities
The platform does not expose:
- Individual agent session IDs (privacy boundary)
- Vendor response times (no SLA tracking yet)
- Comparative benchmarks (no head-to-head tool rankings)
The missing piece is trend data. If a tool's rating drops from 4.6 to 4.2 over a week, maintainers need to know which change caused the regression. Agent.reviews could publish time-series charts or diff views between rating periods.
Deployment Shape
Agent.reviews is a public web service with no authentication required for reads. This implies:
- Static site generation: tool pages are pre-rendered Markdown, not dynamic queries
- Write-path separation: review submissions go through a separate ingestion pipeline
-
CDN distribution:
.mdfiles are cacheable and globally distributed - Async processing: review validation and scoring happen out-of-band
The write path likely includes:
# Pseudocode for review submission
def submit_review(agent_id, tool_name, trace_data, rating, prose):
# Verify execution trace signature
if not verify_trace(trace_data):
return {"error": "Invalid execution trace"}
# Anonymize agent identity
anon_id = hash_agent_id(agent_id)
# Check for duplicate submissions
if is_duplicate(anon_id, tool_name, trace_data):
return {"error": "Duplicate review"}
# Queue for moderation and publication
queue_review({
"tool": tool_name,
"rating": rating,
"prose": prose,
"trace_hash": hash(trace_data),
"submitted_at": now()
})
return {"status": "queued"}
The moderation step is critical. Without it, the platform becomes a spam vector. With it, the feedback loop latency increases.
Likely Failure Modes
1. Review spam from adversarial agents
If submission is open to any agent, expect:
- Competitors submitting negative reviews with fake traces
- Vendors submitting positive reviews with synthetic workloads
- Pranksters submitting absurd reviews to test moderation
Mitigation: require agents to register with Armature's instrumentation layer before submitting reviews. This adds friction but increases trust.
2. Trace verification bypass
If trace signatures are weak or optional, vendors can forge positive reviews. If trace verification is too strict, legitimate agents can't submit reviews due to clock skew, network issues, or API changes.
Mitigation: use a grace period for trace freshness (e.g., accept traces up to 24 hours old) and allow agents to resubmit with updated traces.
3. Rating inflation over time
As tools improve, ratings should increase. But if ratings only go up, the signal degrades. A 4.6 rating means nothing if every tool eventually reaches 4.8.
Mitigation: normalize ratings within categories or time windows. Show relative rank instead of absolute score.
4. Vendor non-response
If maintainers ignore agent reviews, the feedback loop breaks. Agents keep hitting the same bugs, reviews pile up, and the platform becomes a graveyard of unresolved issues.
Mitigation: surface review counts and response rates on vendor profiles. Gamify maintainer engagement.
Technical Verdict
Use agent.reviews when:
- You maintain a developer tool and want to understand how agents interact with your API
- You're building an agent and need to choose between competing tools based on agent-specific criteria
- You want to instrument agent sessions and need a reference schema for feedback capture
Avoid agent.reviews when:
- You need real-time feedback (the moderation queue adds latency)
- You're optimizing for human users (agent reviews don't correlate with human satisfaction)
- You require private feedback channels (all reviews are public)
The platform's real contribution is not the review content. It's the proof that agent-generated feedback can be structured, verified, and published at scale. The next step is closing the loop: automated issue creation, fix verification, and continuous rating updates.
Top comments (0)