DEV Community: Adarsh Singh

SnapSort Lite: My Screenshots Had Timestamps, Not Names

Adarsh Singh — Mon, 26 Oct 2026 03:11:57 +0000

This is a submission for the Hacktoberfest Open-Source AI Challenge Week 3: Slow and Steady.

What I Built

My screenshots folder was becoming a timeline instead of a library.

A filename like Screenshot 2026-10-25 184500.png tells me when I captured something, but not why I saved it. Finding a useful code snippet, document, or webpage meant opening screenshots one by one.

I wanted to automate that repetitive first pass without uploading my screenshots to a cloud service.

So I built SnapSort Lite, a local-first Python CLI that uses open-source Tesseract OCR to extract visible text from screenshots, suggest meaningful filenames, and organize copies into five categories: Code, Documents, Conversations, Web, and Other.

The workflow is intentionally simple:

Select a folder containing screenshots.
Let Tesseract OCR read the visible text.
Generate descriptive filenames and classify the images using keyword rules.
Review the proposed operations.
Apply the changes when satisfied.

The important part: nothing is copied by default.

SnapSort Lite starts in dry-run mode. Only --apply creates organized copies. Original screenshots remain untouched, filename collisions receive numeric suffixes, and a JSON manifest records the operations.

This isn't a magical screenshot-understanding AI. It's a small tool designed to eliminate a boring task while keeping the user in control.

Demo

The video is an edited walkthrough assembled from genuine application outputs, command executions, and filesystem results. It uses synthetic screenshots rather than private user data.

For the demonstration, I generated three screenshots using Pillow: a fictional invoice, a Python code snippet, and a fictional webpage.

Here are the results:

fictional-invoice.png
  → Documents/fictional-data-for-demo.png

fictional-web.png
  → Web/welcome-to-the-fictional-demo-site.png

python-code.png
  → Code/return-f-hello-name.png

The suggested filenames are more informative than timestamps, although they're not always perfect. A filename extracted from a code line, for example, might not be the title a developer would choose manually.

That's one reason I kept the preview step.

Run a dry-run preview:

python snapsort.py ./screenshots --output ./organized

Then apply the plan:

python snapsort.py ./screenshots --output ./organized --apply

The first command previews the operations without copying files. The second creates organized copies and a manifest while preserving the originals.

The local verification run processed all three demonstration screenshots successfully.

You can also explore the demo walkthrough and verification notes.

Code

adarsh-singh106 / snapsort-lite

SnapSort Lite

SnapSort Lite is a privacy-first Python CLI for making small, safe improvements to a screenshot collection. It reads visible text with the local Tesseract OCR engine, proposes a short filename, and copies each image into a transparent category.

Problem

Windows screenshots often accumulate under timestamp-based names. SnapSort Lite turns that timeline into a reviewable first pass without deleting or uploading originals.

Features

PNG, JPG, JPEG, and WebP support
Local OCR with Pillow and pytesseract
Deterministic, filesystem-safe filenames
Rule-based Code, Documents, Conversations, Web, and Other categories
Dry-run preview by default; --apply is required to copy
Never overwrites an existing destination; uses -2, -3, etc.
SHA-256 and OCR details in a JSON manifest
Originals are always retained

Installation on Windows

Install Python 3.10 or newer, then from this folder:

python -m pip install -r requirements.txt

Install Tesseract OCR separately, for example from…

View on GitHub

Repository: https://github.com/adarsh-singh106/snapsort-lite

The project is open-source under the MIT license.

The repository includes the Python CLI, automated tests, synthetic demonstration inputs, documentation, and video production materials.

To get started, clone the repository, install the Python dependencies, and install Tesseract OCR separately.

No API key or hosted inference service is required for the OCR workflow.

How I Built It

The architecture is deliberately straightforward:

Screenshot → Pillow → Tesseract OCR → Filename Generation → Keyword Classification → Dry Run / Apply → Organized Copies

Here's what happens internally:

Image discovery: The CLI identifies supported PNG, JPG, JPEG, and WebP images in the selected directory.
OCR: Pillow loads each image, and pytesseract passes it to the locally installed Tesseract engine.
Filename generation: A promising recognized text line is converted into a short, filesystem-safe filename.
Classification: Deterministic keyword rules select one of five destination categories.
Safe execution: The application prepares a collision-safe copy plan and displays it before writing anything.
Audit trail: When applied, it creates copies using shutil.copy2 and records operations in a JSON manifest with file hashes.

Where the open-source AI fits

The core recognition capability comes from Tesseract OCR, an open-source optical character recognition engine.

Tesseract recognizes text inside images. It isn't a large language model or a vision-language model, and SnapSort Lite's category selection is ordinary Python logic.

I wanted to make that distinction clear rather than market simple keyword matching as sophisticated AI reasoning.

The current version works best when screenshots contain readable text. It cannot reliably understand text-free diagrams, recognize application interfaces visually, or infer the full meaning of an image.

Testing and safety

I separated planning from applying so the tool could show what it intends to do before touching the destination folder.

I also designed the OCR dependency to be replaceable during testing, added collision handling, and made missing Tesseract installations produce explicit errors.

In the latest local verification reported during development, 12 automated tests passed, and the three synthetic demonstration screenshots were processed successfully.

There are still improvements worth exploring: content-based deduplication, better filename selection, and an optional local open-weight vision-language model for screenshots without useful text.

Those are future ideas, not features I'm claiming to have implemented.

Why Does Open Innovation Matter?

Screenshots can contain private conversations, source code, invoices, or account information.

For a tool that processes those images, local execution matters.

Using Tesseract OCR means the recognition step can run on the user's machine without sending screenshots to a third-party inference API.

Open-source components also make the pipeline inspectable and adaptable. Developers can change the naming heuristics, improve the classifier, add OCR language packs, or replace the recognition engine entirely.

The tool doesn't depend on a proprietary API key or a remote model whose behavior might change without notice.

That doesn't mean local processing automatically guarantees security. Users still need to protect their files, review proposed operations, and account for OCR errors.

But it provides a useful foundation: the images and the processing workflow remain under the user's control.

My Agent Session

I used an AI coding assistant to help develop the implementation, automated tests, documentation, and demonstration materials.

The demo video was assembled from actual application outputs and synthetic screenshots, rather than a simulated interactive recording.

I haven't published a DevRelay session, so I'm not including an invented session link.

One thing this project reinforced for me: the most important part of automation isn't always the cleverest algorithm. Sometimes it's making sure the automation behaves predictably and gives users a chance to review its decisions.

The Lazy Payoff

Before SnapSort Lite, organizing screenshots meant opening each image, deciding what it contained, renaming it, and moving it into the right folder.

Now I can generate a first-pass organization proposal with one command, review it, and apply it with another.

In the three-image synthetic demonstration, the application produced categorized copies while keeping the original screenshots intact.

I haven't measured time savings across a large personal screenshot collection, so I won't invent a percentage.

For me, the payoff is simple: fewer repetitive decisions, more useful filenames, and a folder that's easier to navigate.

Slow and steady means giving automation a small, reviewable job before trusting it with something bigger.

The Deployment Was Approved. The Dangerous Change Wasn't.

Adarsh Singh — Mon, 19 Oct 2026 05:17:29 +0000

This is a submission for the Hacktoberfest Open-Source AI Challenge Week 2: Side Hustle.

What I Built

Imagine reviewing a deployment script line by line. Everything looks fine, so you approve it. The script doesn't change afterward. But just before the deployment, someone changes the configuration file it reads:

- "require_human_review": true
+ "require_human_review": false

If the approval check watches only the script, it can miss an edit that changes what the script does. The approved file hasn't changed, but the behavior you approved might have.

That's the problem I explored with ShadowInputs, an open-source Python CLI that records SHA-256 hashes for a reviewed set of repository-local inputs, then checks whether those exact files have changed before a simulated deployment. It is a focused proof of concept, not a general-purpose CI security product.

I started with Md Imran Hossen's October 2026 research paper, When the Approved File Is Unchanged: Authority-Set Integrity for Deployment and Coding-Agent Approvals. The paper examines approvals that bind one script or workflow even though execution depends on additional files. ShadowInputs is my small, reproducible experiment with that idea, not an implementation of the paper's full authority-set model.

Who might pay for this?

Business hypothesis, not proven demand: Small DevOps consultancies and teams maintaining sensitive release pipelines might pay for dependency-mapping audits, custom adapters, or integration of reviewed-input checks into their existing CI workflows. The CLI remains open source under MIT; a genuine commercial offering would require customer interviews, broader dependency discovery, and stronger approval and artifact integrity controls. I haven't acquired a paying user.

Demo

Watch the ShadowInputs video walkthrough on YouTube

The demonstration uses a synthetic GitHub Actions workflow, a Python deploy.py script, and a local policy.json. The workflow and entrypoint are supplied explicitly to the CLI; the prototype does not automatically understand every GitHub Actions step or runtime dependency.

Scenario	Observed CLI result	Exit code
Approved inputs unchanged	`PASS`	`0`
Unrelated documentation edit	`PASS`	`0`
Approved `policy.json` changed while `deploy.py` stayed unchanged	`BLOCK`	`1`
Approved `policy.json` deleted	`BLOCK`	`1`

The local Windows 11 / Python 3.14 verification run reported 20 tests passed, 1 skipped. The skipped symlink test required a Windows privilege unavailable in that environment. These tests validate the prototype's documented behaviors, not production security.

What happened when I tried real Gemma inference?

I used the open-weight Gemma 3 1B model through a local Ollama server to suggest file dependencies that deterministic analysis might overlook. This was a real model call, not a mocked response:

py -m shadowinputs inspect `
  --root .\examples\deployment `
  --entry .github/workflows/deploy.yml `
  --entry deploy.py `
  --model gemma3:1b `
  --output ai-review.json

The static scanner identified the workflow entry, deploy.py, and policy.json. Gemma contributed zero accepted additional file candidates. Its response included a placeholder path (relative/path), which the path validator rejected.

That's not the result I hoped for. But it matters: a successful model call is not proof of successful dependency discovery. In an integrity-sensitive workflow, model suggestions must remain untrusted until they have been checked and reviewed. The deterministic scanner and hash verifier, not Gemma, produced the successful PASS/BLOCK behavior shown above.

Evidence note: These images are rendered cards based on local terminal results, not unedited desktop screenshots. The video is linked separately for the walkthrough.

Code

GitHub: adarsh-singh106/shadowinputs

The implementation is deliberately small:

shadowinputs/core.py: limited Python AST inspection for literal file reads, safe path handling, SHA-256 manifests, and comparison.
shadowinputs/ai.py: local Ollama integration, treating model-suggested paths as untrusted hints.
shadowinputs/cli.py: inspect, approve, and verify subcommands with meaningful exit codes.
examples/deployment/: synthetic workflow, script, and policy file for reproduction.
tests/: checks for drift, missing files, unsafe paths, and other defined boundaries.

To create and verify a manifest from the project root:

py -m shadowinputs approve `
  --root .\examples\deployment `
  --entry .github/workflows/deploy.yml `
  --entry deploy.py `
  --manifest "$env:TEMP\shadowinputs-approved.json"

py -m shadowinputs verify `
  --root .\examples\deployment `
  --manifest "$env:TEMP\shadowinputs-approved.json"

The approve command makes a manifest after the operator has reviewed the intended scope. It does not authenticate that operator or cryptographically sign an approval. Any real deployment integration would need to protect the manifest and bind approval to an authorized reviewer.

How I Built It

The design separates discovery, review, and verification:

Explicit scope and static analysis. The operator supplies entry files. A Python AST scanner finds certain literal file reads, such as Path('policy.json').read_text(). Some unresolved dynamic paths cause the tool to stop rather than claim completeness.
Local open-weight model. Gemma runs via Ollama on loopback, proposing additional candidate paths. The prototype validates those suggestions instead of automatically trusting them. In the reported run it provided no accepted additions.
Operator-reviewed manifest. For the chosen files, the CLI calculates SHA-256 digests and saves a snapshot outside the example repository. That is not a signed approval mechanism.
Deterministic verification. The CLI compares listed paths and their current contents against the manifest. An altered or missing listed file causes the check to fail. It does not itself launch or secure a real deployment.

What the prototype cannot guarantee

It does not identify every input consumed by arbitrary Python, Bash, GitHub Actions, imports, environment variables, or external services.
Files omitted from the manifest are not protected. PASS means only that the listed inputs still match.
An external manifest path alone is not tamper-proof. A real system needs protected storage, access controls, and reviewer authentication.
Inputs may change after verification. This time-of-check/time-of-use gap requires immutable artifacts or controlled execution.
The local Gemma run did not improve dependency coverage in the experiment shown.

These boundaries are part of the project rather than footnotes: someone should be able to understand exactly what a green check does and does not mean.

Why Open Innovation Matters

Deployment source code can be sensitive, and security-related tooling benefits from being inspectable. Using Python, an open-weight Gemma model, and local Ollama inference lets developers experiment without shipping repository snippets to a hosted model API. The static checks and manifests can also be reproduced without depending on a model's answer.

The more lasting open-source value is the reproducible failure case and extensible discovery tooling. Another developer can inspect the hash calculation, contribute a parser for a different deployment language, test a tricky input pattern, or challenge our assumptions. That kind of scrutiny is especially useful when the tool is about trust.

What I'd Test Next

Before claiming that AI helps, I'd assemble real deployment patterns in which the static scanner misses an input, then compare static analysis alone versus static analysis plus Gemma suggestions. I'd measure accepted-candidate precision, dependency recall, and runtime. Until then, the only demonstrated security behavior is the deterministic, manifest-based drift check.

Prize Categories

Best Use of Gemma: ShadowInputs integrates an open-weight Gemma 3 model, served locally through Ollama, as an experimental dependency-suggestion component. I have disclosed its observed result: zero accepted additional candidates in the demonstrated run. This establishes a real integration, not an accuracy improvement or a production-grade AI security system.

ClaimTrail: a 15-minute walk has to include coming back

Adarsh Singh — Mon, 12 Oct 2026 04:53:43 +0000

This is a submission for the Hacktoberfest Open-Source AI Challenge Week 1: Touch Grass.

What I Built

Would anyone open an application just to go outside?

That is the question behind ClaimTrail. I wanted to explore whether AI could give a short walk a purpose without making the walk another screen-based activity. Finding a park is already easy. Having one small thing to notice there is a different starting point.

ClaimTrail turns an available place description or review excerpt into an observation question. You enter your actual starting landmark or address, choose parks, gardens or public squares, and select a complete-trip budget. After choosing a feasible candidate, you read the source and a question, put the phone away, and return to record what you noticed.

MOCKED/SYNTHETIC UI DEMONSTRATION: current application interface; invented starting landmark and mocked service status. No location request was sent.

The intended user is someone with a little spare time: a student between tasks, or anyone curious about familiar surroundings. There is no account, route tracking, photo requirement or leaderboard. The three outcomes are “Saw it,” “Didn't see it” and “Couldn't tell.” A personal observation stays separate from what an online source says.

I have not established that this makes people go outside more often. It is a product hypothesis, not a measured result. The experiment is to make the screen useful briefly, then get it out of the way.

Demo

Watch the captioned synthetic software demonstration on YouTube

Evidence status: the captioned software video and interface captures use mocked provider and model responses. They demonstrate current UI behavior, not a successful current SerpApi-to-Gemma run. An earlier live demonstration exists locally, but it predates the round-trip changes. A fresh real-provider rehearsal remains unverified. No physical outdoor visit is claimed.

MOCKED/SYNTHETIC UI DEMONSTRATION: the returned candidate is fictional. The recommendation shows travel and buffer; the mission view below also exposes the five-minute activity allocation.

MOCKED/SYNTHETIC UI DEMONSTRATION: neither the excerpt nor the selected question came from a live provider/model call during this capture. The question uses the application's actual approved wording.

MOCKED/SYNTHETIC UI DEMONSTRATION: the reflection is simulated, not field evidence. Saving, refreshing and deleting use the real frontend's local-storage behavior.

A video is preferable to exposing the backend publicly. This version is a local, loopback-only application, not a hardened internet-facing API service.

Code

ClaimTrail source and setup instructions on GitHub

adarsh-singh106 / claimtrail

ClaimTrail

Less scrolling. More discovering.

A real listing detail becomes a small outdoor observation question, followed by a private reflection. The world is the experience; the app is preparation and reflection.

New project: October 11, 2026. Hacktoberfest Week 1 deadline: October 12, 12:29 PM IST.

Implemented

Real SerpApi Maps search/detail and walking-directions filtering, bounded review fallback, local Gemma structured output, deterministic grounding/card validation React search-to-reflection flow, browser-local persistence, copy and deletion. A candidate must fit an estimated round-trip budget from the entered starting landmark or address: outbound walk + on-site task + return walk + safety buffer. The provider's resolved origin is not independently verified; do not enter only a city. Walking pace and conditions can differ. Reminders run while the app is active, not as background alarms. See BUILD-LOG for actual verification. No outdoor visit, public deployment or submission is claimed.

Run locally — Windows PowerShell

Prerequisites: Python…

View on GitHub

The application code is MIT licensed. Gemma and upstream place data have their own terms; the repository's license does not relicense either. The README explains the local services, credential setup and limitations.

How I Built It

Keep the model's job small

React and TypeScript provide the field-journal interface. FastAPI handles search, evidence retrieval and mission preparation. SerpApi supplies Google Maps place results, walking estimates and available source text. Ollama runs gemma3:1b locally.

Implementation diagram, not a deployment or live-test result. Location queries leave the machine through SerpApi; reflection notes do not enter the backend pipeline.

The backend filters place categories and travel feasibility before offering candidates. When a user selects one, it checks the selection's origin and budget, retrieves matching place details, and looks for an eligible description. If necessary, one reviews request examines a small bounded set of excerpts. Missing usable evidence means no mission, not an invented fallback.

Gemma selects a supported feature and source quote, then returns structured JSON with an approved observation card. The current choices are deliberately narrow: paths, seating and signs. This is constrained selection, not an open-ended adventure generator.

JSON matching a schema is not enough. The backend also checks provenance and meaning. This excerpt from backend/app/mission.py, lines 93–104, runs after schema parsing:

    source = next((e for e in evidence if e.source_id == proposal.source_id), None)
    normalize = lambda text: ' '.join(text.split())
    quote = normalize(proposal.exact_quote)
    if source is None or len(quote) < 8 or quote not in normalize(source.excerpt):
        raise MissionError('The model quote could not be traced to this place. No mission was created.')
    if proposal.feature not in eligible_features(source.excerpt) or proposal.feature not in eligible_features(quote):
        raise MissionError('The source does not support an allowed observation. Try another place.')
    if proposal.estimated_on_site_minutes != minutes:
        raise MissionError('The model changed the requested activity time. No mission was created.')
    for field, value in CARDS[proposal.feature].items():
        if getattr(proposal, field) != value:
            raise MissionError('The model mixed unsupported observation details. No mission was created.')

The quote must occur in the selected evidence after whitespace normalization. The feature must be supported, the activity time must match, and the card fields must agree. These checks establish a traceable question, not proof that a listing is true today.

Earlier build logs record both an abstention and a mixed-card response being rejected before an accepted local-model result. Those are historical development results, not proof that the revised routing flow has passed a fresh live run.

A fifteen-minute mission has to include coming back

The most important correction was not in the prompt. A destination ten minutes away is not suitable for a fifteen-minute break if the user must also explore and return.

The initial experience treated activity time separately from travel. The current implementation requests outbound and return walking directions independently. It does not simply double the outward estimate or turn straight-line distance into a confident walking time.

It rounds the two estimates upward, reserves activity time, and adds a buffer of at least three minutes. For a fifteen-minute budget, the activity allowance is five minutes. These lines from backend/app/places.py, 97–107, make feasibility a deterministic decision:

    activity_minutes = ACTIVITY_MINUTES[total_minutes]
    buffer_minutes = max(SAFETY_BUFFER_MINUTES, ceil(max(outbound_minutes, return_minutes) * 0.25))
    estimated_total = outbound_minutes + activity_minutes + return_minutes + buffer_minutes
    return {
        'budget_minutes': total_minutes,
        'outbound_minutes': outbound_minutes,
        'return_minutes': return_minutes,
        'activity_minutes': activity_minutes,
        'buffer_minutes': buffer_minutes,
        'estimated_total_minutes': estimated_total,
        'fits_budget': estimated_total <= total_minutes,

Excerpt ends within the returned dictionary; additional metadata and the closing brace are omitted.

Illustrative synthetic calculations, not measured routes: 3 + 5 + 3 + 3 = 14 fits; 10 + 5 + 10 + 3 = 28 does not.

An empty result is better than quietly spending time the user does not have. Even an accepted estimate depends on the entered origin, provider routing, walking pace and conditions. The app cannot certify entrances, access or safety.

Tests needed to follow the browser

The release audit found an embarrassing boundary mismatch: normal searches returned HTTP 422 while 50 backend tests passed. The browser sent total_minutes=15 as query text, but the endpoint used numeric literals. Tests calling functions or omitting the parameter missed that path.

The correction accepts the four textual query values and converts them after validation. HTTP-level regressions now exercise the actual values the browser sends.

Desktop controls also overlapped. Changing an origin or budget could leave old recommendations visible, and late responses could restore stale state. Separate grid rows and revision-bound requests corrected those cases. The return reminder now reserves the return walk plus its buffer; it updates while the app is active, not as a background alarm.

The latest release record reports 72 passing backend tests, 14 passing mocked Chromium tests and a passing production build. Browser coverage includes three widths, stale responses, reminders and reflection persistence/deletion. These are useful checks, not live-provider or outdoor validation.

Why Does Open Innovation Matter?

Local open-weight inference makes the model boundary inspectable. I can change the schema, examine rejected outputs and experiment with the configured model without depending on a proprietary hosted language-model endpoint. Alternative models would still need their own evaluation; changing a name is not evidence of compatibility.

The model is central to mission preparation: an unavailable or rejected response does not silently become a template-only success. But with only three approved cards, a simpler non-AI selector is a fair baseline to compare later. I have not measured whether Gemma improves on it.

Local inference does not make the whole application offline, completely private or free. Starting-point queries reach SerpApi. Discovery needs internet and provider credits; inference needs a running local model. Personal reflections remain in browser storage and can be deleted, but anyone using that browser profile may read them.

There are further limits: provider latency, sparse place coverage, repetitive questions and no verified user adoption. Even request cost needs care: five displayed results can require examining more than five candidates.

My main lesson is that the model should not own promises the application can check. Let it select a question. Make ordinary code enforce the source, time and state boundaries. Whether that question gives someone a worthwhile reason to step outside is the next thing to test.

This is a solo project. AI coding assistance contributed to implementation and tests, and an AI assistant drafted this article. The technical account is based on source inspection and recorded verification; it does not represent independent manual authorship of every line.

Prize Categories

Best Use of Gemma: local gemma3:1b selects the source-linked observation card; deterministic backend validation controls acceptance. Historical inference evidence exists, with the current complete live flow still pending verification.
Best Use of SerpApi: place discovery, independently requested walking legs and retrieved evidence determine which destinations and questions are possible. It is a functional data dependency, not a decorative search call.

Thanks to the maintainers of Gemma, Ollama, SerpApi, React and FastAPI. The distinction I want ClaimTrail to preserve is simple: a source claim, an AI-selected question and a person's observation are three different things.

Explain Aloud: If the Screen Disappears, Does the Answer Still Make Sense?

Adarsh Singh — Mon, 05 Oct 2026 05:41:46 +0000

This is a submission for the Hacktoberfest Weekend Challenge: Build for a Friend

My girlfriend already has a listening workflow. She uses Read Aloud in both ChatGPT and Claude while doing other things — cooking, commuting, winding down after a long day. It works fine for ordinary prose. Then ChatGPT answers a question with a table, or a code block, or a bulleted list with sub-items, and the whole thing falls apart. A three-column table about people, ages, and roles starts sounding like a leaderboard. A Python snippet becomes a stream of colons and underscores. A heading that says "Next Steps" arrives as "Next, small List."

She would skip those answers, or glance at her phone to reconstruct the layout, which defeats the point entirely.

That problem is the reason for Explain Aloud. Not the voice output — she already had that. The missing piece was the structure-aware step that happens before speech.

What I Built

Explain Aloud is a Chrome MV3 extension that attaches to completed ChatGPT responses, extracts their structure into typed semantic blocks, builds a narration plan segment by segment, and sends each segment to Kokoro for in-browser speech synthesis.

It handles five block types: paragraphs, headings, lists, code blocks, and tables. Two modes are available: Literal Mode takes the deterministic path for everything; Natural Mode can route code and table blocks through a local Gemma model running in Ollama for listening-oriented phrasing. The side panel shows playback controls, the narration segments in sequence, and each segment's provenance — whether it came from a rule, from the model, or from a fallback.

The core question guiding every design decision was: if the screen disappears, does the answer still make sense?

Demo

Demo-Video

Demo voiceover generated with ElevenLabs. The Explain Aloud narration heard inside the product is Kokoro.

The extension attaches to a completed response. These model figures are example response content, not measurements of Explain Aloud.

Panel captured in stopped state with a rule-generated segment selected. The "Verified" label is the application's own validation result, not an independent semantic proof. The response identifier has been redacted.

Code

adarsh-singh106/explain_aloud

How I Built It

From DOM to typed blocks

The first step is extraction. When the extension detects a completed ChatGPT response, the content script runs the semantic DOM extractor over it and builds a ResponseIR — a structured representation of the response as a typed block sequence. Paragraphs, headings, list items, code blocks, and tables each carry their own fields. A code block knows its language. A table block knows its headers and cell values. A list item knows its nesting depth.

The extraction is deterministic. No model is involved yet, and no interpretation is made. A Python code block that ChatGPT explicitly labels as Python is extracted with language: "python". An unmarked print(...) is not guessed — it arrives as generic code. Seven regression fixture variants cover this explicitly, including the case where a Python header appears alongside a generic plaintext class. That conservatism matters when the language label will eventually reach a prompt.

What Gemma's job actually is

In Natural Mode, code and table blocks are handed to a local Gemma call through Ollama. The boundary I wanted was specific: Gemma receives the pre-extracted facts, not the raw HTML. For a table, the deterministic parser has already produced the headers, row entities, numeric values, and unit-normalized metrics before the model sees anything. Gemma's task is to choose phrasing, not to read the table.

The model's output is a candidate narration. Candidate narrations pass through conservative block-specific validation before entering the narration plan. Where the validator cannot establish enough support, the system falls back to source-derived deterministic narration. The fallback is always a rule-based transformation of what was already extracted — never an invented summary.

The configured default is gemma4:e4b, with gemma4:e2b as the model-not-found fallback. Which model actually ran in a given browser session was not separately recorded.

The test suite was green. The contract wasn't.

This is where the real engineering tension lives.

The test suite grew substantially during a source-fidelity hardening pass. Parsers were correctly extracting table data. The UI was rendering the right buttons. The pipeline was moving blocks from extraction through to the plan. Green across the board. But none of that guaranteed the listening contract.

The contract is more specific: a table with three people and their ages and roles should arrive as three separate row descriptions, not as a comparative ranking of who is oldest. A code block explained by Gemma should not reverse a comparison or invent behavior. A heading like "Next Steps" should smooth into speech without the word "Steps" collapsing into a list signal.

Passing the fixtures did not prove those things automatically. It took deliberate adversarial review — asking what happens when Gemma returns something plausible-but-wrong — to see where the boundary was missing. That review produced the segment validator and its strict fallback contract:

export interface NarrationSegment {
  id: string;
  sourceBlockIds: string[];
  factIds: string[];
  provenance: Provenance;
  text: string;
  verified: boolean;
  fallbackReason?: string;
  pauseAfterMs: number;
}

Provenance is rule, llm, or literal. Every segment declares where it came from. The inspector in the side panel shows this per segment. If you want to know whether the model touched a narration or a rule produced it, you can see that without digging into logs.

Conservative validation and fallback paths reduce unsupported narration. They do not eliminate it entirely — the current validator is a heuristic safety net, not a formal semantic proof — but they establish a checked path from extraction to voice.

A small table should not become a leaderboard

One concrete example: a table with three rows — Name, Age, Role — for Alex, Sam, and John. The earlier deterministic summary path would lead with comparison framing, emphasizing who was oldest. That preserved the numbers correctly, but it invented a ranking the original response did not contain.

The final patch adds a row-wise path for complete rectangular tables with two to four rows and two to four columns. It ignores relationships between rows and speaks each one directly:

The table has three entries. Name: Alex, Age: 20, and Role: Student. Name: Sam, Age: 21, and Role: Developer. Name: John, Age: 22, and Role: Engineer.

That exact output is what the regression test asserts, character for character. Gemma is not called on this path. Validation does not fall back. The facts are preserved in the order they appear.

Larger or irregular tables stay on the existing path. Merged-cell tables stay on the existing path. The row-wise route applies only where the structure is simple enough to be safe.

This is a test-evidence graphic, not a browser screenshot. The narration is copied directly from the passing regression assertion.

The production WASM failure

Before those listening improvements, there was a more basic failure that showed up only in the production Chrome extension — not in tests, not in the TypeScript check, not in the build.

no available backend found
[wasm] TypeError: Failed to fetch dynamically imported module

Kokoro's underlying ONNX Runtime was trying to import its JavaScript module from a CDN at runtime. The extension build had succeeded. The bundle was present. But Chrome's extension sandbox blocked the external module fetch, and the speech backend could not initialize.

The fix was to bundle the matching ONNX Runtime .mjs and .wasm files as local extension assets, then configure their URLs explicitly before the model loads:

env.wasmPaths = {
  mjs: new URL(ortModuleUrl, globalThis.location.href).href,
  wasm: new URL(ortWasmUrl, globalThis.location.href).href,
};

The packaged files were verified against the installed runtime files by SHA-256. After reloading the extension, actual Kokoro speech was confirmed in Chrome. That confirmation was a browser result, separate from the green build that had preceded it.

This is the distinction the project kept surfacing: a passing test or a successful build is evidence that a specific thing worked correctly, not that the user's experience works. The Kokoro WASM failure would not have appeared in any automated check.

CI, testing, and what the evidence covers

The CI pipeline runs on push to main and on pull requests. It installs dependencies, runs tests, typechecks, and builds the extension — in that order, under permissions: contents: read after a final hardening step.

The verified counts: 191 tests passed across 14 test files. TypeScript compilation passed. The Chrome MV3 production build passed at 24.11 MB. A chunk-size warning remains but is non-blocking.

What those numbers establish and do not establish is worth being direct about:

Check	Recorded result	Boundary
Test suite	191/191, 14/14 files	Fixture and regression coverage. Not a universal faithfulness guarantee.
TypeScript	`tsc --noEmit` passed	Static type checking.
Production build	Chrome MV3, 24.11 MB	Packaging. A large-chunk warning remains.
Real Chrome	Production extension manually tested; Kokoro speech confirmed	Developer-reported browser result.

The narration pipeline uses local Ollama/Gemma and in-browser Kokoro WASM. Complete offline operation was not separately verified. The original ChatGPT conversation also remains a ChatGPT conversation — this extension does not change that service's privacy boundary.

Why Does Open Innovation Matter?

Explain Aloud exists because local models and open browser APIs make the stack possible without a backend. Explain Aloud does not intentionally send extracted response content to an additional cloud backend for narration. Its narration path uses local Ollama/Gemma and in-browser Kokoro. Complete offline operation was not separately verified.

That matters for a use case like this, where the content being narrated is personal conversations. The extension attaches to ChatGPT, which already has its own privacy boundary. Inside the extension's own narration pipeline, the path stays local. Open models and open runtimes make that design available without infrastructure.

My Agent Session

I enabled Entire late in the finalization phase. The permanent checkpoint below captures a Codex documentation session that recorded the final CI review evidence. It does not represent the entire development history of Explain Aloud.

Checkpoint: 2d9f0c278df4
Linked commit: 3aa189b

The Codex review found no shipping blockers. Its only recommendation was adding permissions: contents: read to the CI workflow for least-privilege hardening. That was applied before submission.

Earlier implementation work — including the narration hardening and WASM debugging — happened before this permanent Entire checkpoint was created.

Prize Categories

Best Use of Gemma

Gemma's role in Explain Aloud is bounded by design. It does not read raw HTML or receive an open-ended instruction to "explain this." The deterministic pipeline extracts facts first — table headers and cell values, code structure and language — and then provides those pre-extracted facts to the model with a specific wording task.

The candidate Gemma produces must pass the segment validator before it reaches the narration plan. If it fails, the system falls back to a source-derived deterministic summary and records the fallback reason in the segment's fallbackReason field. The model's output is a phrasing candidate, not the source of truth.

This bounded role — where open local inference contributes without being trusted unconditionally — is the core design argument for using Gemma rather than a simpler rule expansion.

Best Use of Entire

I enabled Entire late in the finalization phase to preserve agent-session provenance against Git history.

The permanent checkpoint 2d9f0c278df4, linked to commit 3aa189b, captures a Codex documentation session that recorded the final CI review evidence. That evidence included the earlier Codex review result of BLOCKERS: None and SAFE TO SHIP: YES, plus the least-privilege permissions: contents: read hardening that followed.

The checkpoint does not represent the broader implementation history. Earlier work — narration hardening, the WASM fix, table narration changes, and the original CI review — happened before this permanent checkpoint was created.

What Entire gives me here is a traceable record connecting the final evidence, agent context, and Git commit at submission time.

Best Use of GitHub Copilot

I used GitHub Actions to automate test, typecheck, and production-build verification on pushes to main and pull requests targeting main, with least-privilege contents: read permissions.

The workflow runs npm test, npm run compile, and npm run build in sequence, giving relevant pushes and pull requests the same automated verification path used during finalization.

Best Use of ElevenLabs

ElevenLabs generated the voiceover for the demo video. It was not used in any part of the running extension. The Explain Aloud application's speech synthesis is Kokoro.js running as WASM inside the Chrome extension, with locally bundled ONNX Runtime assets. ElevenLabs touches only the recorded demonstration.

The problem that started this was small and specific. My girlfriend wanted to listen to ChatGPT answers while doing something else. The built-in Read Aloud features mostly worked — until the answer had structure. Then it stopped making sense.

Explain Aloud's answer to that is to treat structure seriously before it reaches the voice. A table is not a paragraph read aloud. A code block is not a prose sentence with odd punctuation. A heading is a transition, not an item.

The test suite reaching 191 passing tests was a milestone. Hearing actual Kokoro speech in the running Chrome extension — after fixing the WASM module failure that a green build had concealed — was a different kind of milestone. And deciding to speak a three-person table one row at a time, rather than as a ranking exercise, was the kind of decision that only matters if you are actually trying to answer the question:

If the screen disappears, does the answer still make sense?

That question does not have a final answer yet. But it has a more honest one than before.