Sentinel: Test Your AI Visibility Fixes Before You Ship Them

This is a submission for the Hacktoberfest Open-Source AI Challenge Week 2: Side Hustle

What I Built

Sentinel is a weekly "AI visibility" audit for small brands. Shoppers in India now ask an AI assistant "best baby wipes for sensitive skin" instead of scrolling Google. If the assistant never names your brand, you lose that sale and never find out.

Sentinel asks open-weight models the questions a brand's buyers ask, measures how often the brand is named versus its competitors (share of voice), checks whether AI crawlers can read the site, audits the page for SEO and answer-engine readiness, and writes a prioritized fix list.

New this week: a digital twin. It takes the client's real homepage, builds a copy with the recommended fixes applied, and puts each version into the real Google results for the same question (via SerpApi). Then it asks the model the same questions against both and compares. The client sees what the fixes are likely to change before touching their website.

Who pays: small D2C brands and the freelance SEO consultants who serve them. A one-time audit plus fix list, then a monthly re-check. The paid AI-visibility tools in this space cost hundreds of dollars a month, priced for enterprises, not a small Indian hygiene brand.

Demo

First real brand: an Indian D2C hygiene brand (I'll call it Brand X) selling baby wipes, face wipes, tissues, hand wash and home cleaning products.

Baseline (8 buyer questions, 16 AI answers, open model with no page given):

• Share of voice: 0%. Brand X was never named.
• AI crawlers (OAI-SearchBot, ClaudeBot, PerplexityBot, ...): all allowed, so the gap is content, not access.
• 31 fixes found, for example: no FAQ section or FAQPage schema, no question-style headings, a 114-character title, 34 of 103 images without alt text.

Digital twin (homepage inserted at rank 3 in real Google results):

Buyer question	Original page	Fixed page
best natural baby wipes in India	100%	100%
best baby wipes for sensitive skin in India	100%	100%
best refreshing face wipes in India	50%	100%
best toilet paper and tissue brands in India	50%	100%
best kitchen tissue roll brand India	100%	100%
best liquid hand wash brand in India	50%	100%
eco-friendly bamboo hygiene products India	100%	100%
best floor cleaner brand in India	0%	50%
All 8 questions	68.8%	93.8%

Answers that named a competitor but not Brand X fell from 25.0% to 6.2%. The fixed page wins on exactly the questions where the original had no direct answer: face wipes, toilet paper, hand wash and floor cleaner.

Read this as a direction, not a forecast. The page is placed inside the results, so even the original scores far above the 0% baseline; the gap between the two columns is what matters. It's 2 samples per question on one open model (16 answers per side, 0 failed).

Code

Prekshabarjatya / sentinel-ai-visibility

Open-model AI visibility audit for small brands, with a digital twin that tests fixes before you ship them

Sentinel AI: Phase 1 engine

Weekly AI-citation tracker per BRD v1.1. Python 3.10+, only dependency: requests.

Run

pip install requests
python tests/test_engine.py                                   # 8 tests, no keys needed
python -m sentinel.run clients/acme.json --mock               # full cycle with fake engines
export OPENAI_API_KEY=... ANTHROPIC_API_KEY=... GEMINI_API_KEY=... PERPLEXITY_API_KEY=...
python -m sentinel.run clients/acme.json --samples 3          # real engines

Output per client in data/<domain>/: latest_report.md, history.json, audit.jsonl, review_queue.json.

Optimization (needs internet; add --optimize)

python -m sentinel.run clients/acme.json --samples 3 --optimize

Adds to the report and writes data/<domain>/fixes.json:

1. Crawler access (crawlers.py): reads robots.txt; flags blocked OAI-SearchBot, ClaudeBot, PerplexityBot, etc. as P0.
2. Page gap analysis (pagegap.py): for each losing prompt (SoV under 50%), fetches your page and the competitor pages engines cited, and lists what they have that you lack (direct answer, FAQ schema, Organization schema, table, question headings, depth). P1.
3. Fix list (…

View on GitHub

Two commands: sentinel.run for the weekly audit (with --optimize --seo) and sentinel.twin for the digital twin.

A new client is one JSON file (brand, domain, competitors, prompts) plus a _twin_fixes.json file. No code changes.

How I Built It

• Python + requests only. 28 tests run without any API keys, using a mock engine.
• Open-weight models: OpenAI's open-weight gpt-oss-120b on Groq, plus OpenRouter's free router, which picks an open model that's available. Every engine is an OpenAI-compatible endpoint, so swapping models is one environment variable.
• SerpApi fetches the real Google India results for each buyer question. They're cached, so every AI run reuses the same results and doesn't spend extra searches.
• The twin's fairness rules: both versions sit at the same rank among the same real results, and the model is told to use a result only if it actually helps. Fixes may only use facts already on the client's site. No invented "dermatologist-tested" claims.
• Everything is a recommendation. Sentinel never edits the client's site. Fixes have ids, and when one is applied, later weekly runs report the before and after.

What went wrong along the way (and is now handled): free model slugs get retired without warning (switched to OpenRouter's openrouter/free router), free tiers rate-limit hard (paced requests, a configurable delay, failed runs counted rather than hidden), and SerpApi's num parameter returned unrelated results (dropped it and pinned google.co.in).

Why Does Open Innovation Matter?

• The unit economics only work this way. An audit is roughly 100 model calls. On free open-weight endpoints that costs nothing, so I can sell an audit to a small brand at a price it can pay.
• No lock-in. In one afternoon two "free" models disappeared. Because every engine is an OpenAI-compatible endpoint serving open weights, the fix was changing a string, not rewriting the product.
• Can run on the client's own machine. The same code can point at a locally hosted open model, so an agency can audit a client's unreleased pages without sending them to a third party.
• Honest limit: open models aren't ChatGPT or Gemini. Sentinel treats them as a stand-in for consumer AI search, and the report says so on page one.

My First User

I ran it on a real Indian hygiene brand I know well (anonymized here). The first useful feedback came immediately: my first config tracked diapers and sanitary pads, but the brand doesn't sell those yet. I rebuilt the questions around what it actually sells, which is the kind of correction a real user gives that no test catches.

Prior art on DEV

• Measure AI Share of Voice in Python by odeeb, on counting mentions while ignoring "I'm not familiar with X" answers.
• Measuring AI Visibility: A Practical Framework for Reliable Results by benardkemp, on why repeated samples matter.

Sentinel adds the step those stop short of: testing the fix before shipping it.

Prize Categories

• Best Use of SerpApi