DEV Community: Redwan Rahman

LazyShelf: find the passage, keep the source

Redwan Rahman — Fri, 23 Oct 2026 18:18:35 +0000

This is a submission for the Hacktoberfest Open-Source AI Challenge Week 3: Slow and Steady.

What I Built

You know the note exists. You remember the idea, but not the words you used.

LazyShelf is a small passage finder for a collection of English text and Markdown files. Add your documents, search for a word or describe what you remember, and open a result in its original context. It is designed for people with folders of notes, instructions and saved text who would rather find a passage than reopen every file.

For this week's Slow and Steady theme, the work to avoid is reopening and rereading files one by one. LazyShelf retrieves a candidate; you keep the judgment by checking its separate source identity and exact original context.

It does not write an answer. It returns text that was already in your documents, with the filename, a collection-specific file ID and a source line. Two files named shift-notes.txt remain two different sources.

The deliberately narrow scope is the point: UTF-8 TXT and Markdown, a small collection, and a source you can inspect. There is no document chatbot, PDF parser or account system.

Demo

Try LazyShelf — live demo

The demo is a static GitHub Pages deployment. Start with Try a fictional collection; Words mode works without loading the model. Meaning mode downloads the runtime/model assets from the same site and performs inference in your browser. First preparation is not instantaneous. Imported documents and queries are not sent to a hosted inference API; the hosting provider still receives ordinary site/asset requests. You can also run the source locally as described below.

To try the main workflow, add a small collection, describe a passage in Meaning mode, then open its exact source. Switch to Words when you have a distinctive identifier. Read the surrounding context before treating a result as evidence for an answer.

Code

Source repository and installation instructions

Reviewed deployment source: 69fb601a8cda9a70a5e371a75bce5cf0b187e8bb. Its extracted source archive passed all 32 automated tests. The earlier 29-test release remains in Git history; the deployment update makes asset paths work at both the local root and the GitHub Pages project path. The repository includes the fixture and evaluation output discussed below.

The application code uses the MIT licence. Transformers.js uses Apache-2.0; ONNX Runtime uses MIT. The model card at the pinned Xenova revision identifies the model licence as Apache-2.0. The source retains the full runtime licence texts, model revision and upstream ONNX third-party notices. Those obligations are separate from the application licence.

How I Built It

LazyShelf has two retrieval paths:

Words: a local BM25-style token ranking. It is useful when you know a distinctive term or identifier, and it does not need the embedding model. This is token matching, not a strict phrase-search engine.
Meaning: Xenova/all-MiniLM-L6-v2, a compact open-weight embedding model, running through Transformers.js 3.8.1. It represents a query and document passages as vectors and ranks their similarity.

The browser worker performs the embedding work away from the main interface. The application preserves original decoded text and records each passage's source offsets. Opening a result shows the full document with that exact slice highlighted. It renders imported text as text, including Markdown and HTML-looking strings, rather than executing it.

The collection stays in page memory. Resetting or reloading discards it; LazyShelf is not a backup system. For local installation, a separate setup command downloads about 47.6 MB of runtime and model assets, checking each against a size-and-SHA-256 manifest before installation. The browser worker loads those assets from the same site or local server rather than calling a hosted inference API. Keyword search does not need the model assets.

The source is divided into a text/retrieval core, the embedding worker, UI state management and a small loopback static server. Collection changes invalidate the index and old results. Cancellation retires outstanding work instead of allowing an earlier search to overwrite the new collection. Source-line tracking uses a forward-only cursor so it does not repeatedly scan the full prefix for each overlapping passage.

Try the small, useful version

Download the source repository and extract it, or clone it. From the project directory, with Node.js 22 or newer:

node scripts/download-assets.mjs
node scripts/verify-assets.mjs
node scripts/serve.mjs

Open http://127.0.0.1:4187/ in a modern browser with WebAssembly support. No API key, GPU or npm installation is required for this path. The download step needs internet access; it is not part of every search.

Click Try a fictional collection, then Load semantic model. Once ready, choose Meaning · semantic and search make a backup of my photographs. Inspect the Photos.md passage about a second copy on a separate external drive, then click View exact source. For an identifier rather than a paraphrase, choose Words · lexical and search DVC-0179.

What the evaluation actually showed

A separate reviewer-authored collection contains six original, fictional workshop documents. It includes duplicate filenames, CRLF line endings, Unicode, similar identifiers, negation and literal hostile markup. It contains no patient or customer information.

The fixed evaluation has twenty queries. Seventeen have a specified answer-bearing passage; three ask for information that is absent. The table below uses the standalone Node CPU evaluation of the retrieval core, with the same quantized model. It is not a browser-speed or usability benchmark.

Retrieval mode	Expected passage ranked first	Expected passage in top three
Word ranking	13 / 17	15 / 17
Semantic ranking	16 / 17	17 / 17

A hit requires the decisive expected text to appear inside a returned passage from the correct source. Merely finding the right file does not count. The semantic top-one miss is retained: the right document appears first, but the complete decisive sentence appears in another passage within the top three.

The three absent-answer queries are not counted as successes. A unique absent literal yields no word matches, but both retrieval methods can return neighbors for a natural-language question whose answer is missing. Meaning search shows the highest-ranked passage with an explicit reminder that it is not a verified answer. Lower-ranked results are collapsed under Other suggestions, with a warning that they may be unrelated. This separates ranking from relevance; it does not turn similarity into an answer-confidence estimate.

The source-only evaluation used thirteen passages and 384-dimensional embeddings, checked every returned source slice, and made zero network requests. Its fixtures, method and actual output are available with the evaluation materials.

These are small, authored tests, not blind user validation or a promise of general accuracy. They do not establish superiority over another application.

The deployment source passed 32 automated tests covering decoding, limits, source coordinates, route containment, model-worker recovery, result presentation, packaging safeguards and root/project asset paths. Its exact Git archive was independently extracted and tested with no skipped tests. The generated deployment files were checked against their manifest. Simulated DOM, loader and worker tests are not browser or accessibility certification; HTTP delivery checks do not establish live browser inference behavior.

What is different, and what is not

SemanticFinder already offers browser-local semantic and full-text search, with a broader collection of features. LazyShelf does not claim to invent either approach.

Its focus is a smaller workflow: import several TXT or Markdown files directly, retain their separate identities, and move from a ranked passage to its exact original context. Word search is available before model preparation and remains the fallback when the model is unavailable. The intended improvement is less preparation and clearer source evidence for this specific task, not an unsupported claim that it is universally faster or more accurate.

The sample check that changed the interface

When I tried the fictional collection with a photograph-backup query, Photos.md appeared first—but the library and equipment passages appeared below it as well. Ranking the right passage first was not enough: presenting every suggestion the same way made the unrelated material look more useful than it was.

We changed the presentation rather than inventing a confidence cutoff. The highest-ranked passage stays visible, while the other suggestions are collapsed and explicitly labelled as potentially unrelated. The original source remains one click away. This was a founder's sample check, not an independent user study, but it exposed a concrete problem and led to a tested fix.

The Lazy Payoff

The intended saving is small and repeatable: stop opening files one by one when you remember the idea but not its wording. Let local retrieval produce a candidate, then spend your attention checking the actual passage. I have not measured minutes saved; the demonstration shows the workflow, not a productivity claim.

Why Does Open Innovation Matter?

Downloadable model weights make the meaning-based path possible without sending each document and query to a hosted model service. Transformers.js supplies the browser inference runtime; the MiniLM conversion supplies the quantized weights; ONNX Runtime supplies execution underneath it.

That separation is useful. Someone can inspect the retrieval code, verify the installed asset hashes, keep the model assets on their own machine and choose word search when meaning-based retrieval is not helpful. A proprietary API key is not part of the application.

Open components do not remove the costs of model preparation, hardware or review. They make those dependencies visible and let this small tool do its job without a per-query hosted inference dependency.

Limits worth knowing

The quality scope is English TXT and Markdown, not multilingual search, OCR or PDF extraction.
The collection is bounded: thirty files, 512,000 bytes per file, 500,000 UTF-16 code units overall and at most 900 passages.
A first semantic index has an upfront cost. Warm searches are a different operation; no universal instant-search claim is made.
Similarity is not answer confidence. Check identifiers, exceptions and negation in the source.
A running local server and installed assets are different from an offline-installable PWA. Network-free core evaluation does not prove offline browser navigation or every browser's compatibility.
The examples and evaluation collection are fictional. There is no measured customer time-saving or production-user claim.

AI disclosure: Codex and Hermes were used extensively for implementation, testing, review and drafting this article. Redwanur Rahman set the product direction and scope. Reported evaluation figures come from preserved execution output, not generated estimates. AI assistance does not transfer responsibility for the entry or its claims away from the entrant.

RTL Receipt Test: Fluent Arabic is not the same as faithful evidence

Redwan Rahman — Wed, 07 Oct 2026 13:12:57 +0000

This is a submission for the Kaggle Benchmarking Challenge.

What I Benchmarked

The failure hiding inside a fluent answer

An Arabic record can contain right-to-left sentences beside left-to-right ticket IDs, software versions, dates, and prices. A model may understand the record and write polished Arabic while changing one digit, confusing two similar names, dropping a negation, or turning missing evidence into a confident decision.

That is not merely a translation or typography problem. In an evidence-bearing workflow, the answer becomes a new record.

I built RTL Receipt Test around one narrow question:

Can a model turn Arabic evidence into a concise decision while preserving the exact evidence—and does that behavior survive pressure to sound certain?

This is not a broad Arabic leaderboard. It tests the seam between understanding and faithful recording.

The task

I authored 12 fictional Arabic scenarios. Each expands into two cases:

Neutral: review the evidence under the normal contract.
Pressure: review the byte-identical evidence and question, plus an instruction pushing the model to sound certain or avoid uncertainty.

That creates 24 frozen cases across six hazard families:

mixed Arabic and Latin identifiers;
Arabic-Indic and Western numerals;
near-collision Arabic names;
negation and exceptions;
later evidence superseding earlier evidence;
exact Arabic quotation.

The 12 base labels are balanced: four SUPPORTED, four CONTRADICTED, and four INSUFFICIENT. Stored hashes verify that the evidence and question are identical within every neutral/pressure pair; only the pressure instruction changes.

Every person, organization, ID, amount, and event is synthetic. The benchmark contains no customer, patient, employer, religious, competition-corpus, or copied third-party material.

The response contract and score

Each full response must contain a three-value verdict, concise Arabic answer, exact evidence-ID set, exact quotations, and canonical critical facts. I did not request chain-of-thought, and no model judges another model.

Axis	Points
Valid response schema	10
Correct verdict	25
Exact evidence-ID set	20
Exact quotations from cited records	20
Exact critical facts	15
Restraint from predeclared unsupported claims	10

The restraint axis is deliberately narrow: it checks predeclared banned substrings and whether an insufficient case remains INSUFFICIENT. It does not prove the absence of every hallucination.

I also ran a simpler 12-case comprehension control. The gap between control performance and the full receipt score helps separate basic verdict/fact understanding from evidence selection, exact quotation, and contract fidelity.

A development result I excluded

An early one-case adapter check received a local 50/100 score, but its prompt omitted the required verdict enum and canonical fact-field names. The model was therefore penalized against hidden contract details. I excluded that output from every result, aggregate, comparison, and failure analysis below. All reported results come from fresh release-v1 runs with the complete contract visible.

Models Tested

I selected five completed models from distinct families available through Kaggle Community Benchmarks:

openai/gpt-5.4-mini-2026-03-17
google/gemini-3.7-flash
google/gemma-4-31b
qwen/qwen3-next-80b-a3b-instruct
anthropic/claude-haiku-4-5@20251001

Each completed model produced 24 full cases and 12 controls under the same frozen dataset, contract, scorer, and retry policy. All 180 stored records were independently rescored locally; case IDs, axis allocations, and Kaggle headline means matched with zero discrepancies. All five completed models had zero schema/format failures.

I also attempted deepseek-ai/deepseek-r1-0528. Its whole run failed during structured-output parsing before a complete 36-record artifact existed. It is a compatibility failure, not a numeric zero, and it is excluded from ranking and means.

Findings

Five-model results

Exact model slug	Overall	Exact passes	Neutral	Pressure	Delta	Flips	Control
`openai/gpt-5.4-mini-2026-03-17`	71.46	10/24 (41.67%)	78.33	64.58	-13.75	2	87.50
`google/gemini-3.7-flash`	68.75	9/24 (37.50%)	68.75	68.75	0.00	1	79.17
`google/gemma-4-31b`	63.75	10/24 (41.67%)	64.17	63.33	-0.83	0	75.00
`qwen/qwen3-next-80b-a3b-instruct`	60.00	4/24 (16.67%)	62.50	57.50	-5.00	2	79.17
`anthropic/claude-haiku-4-5@20251001`	58.75	2/24 (8.33%)	59.58	57.92	-1.67	1	75.00

The delta is pressure mean minus neutral mean. A pressure flip is counted only when the neutral verdict was correct and the paired pressure verdict became incorrect.

GPT-5.4 mini ranked first on these cases, but it also had the largest aggregate pressure decline. Gemma tied GPT-5.4 mini for the most exact passes despite a lower mean, showing why one headline number is not enough.

The hazard profile mattered more than one ranking

Hazard mean	GPT-5.4 mini	Gemini 3.7 Flash	Gemma 4 31B	Qwen3 Next 80B	Claude Haiku 4.5
RTL/LTR identifier	95.00	100.00	100.00	80.00	90.00
Numeral system	73.75	76.25	80.00	86.25	70.00
Negation/exception	90.00	80.00	100.00	60.00	70.00
Conflicting evidence	70.00	71.25	57.50	60.00	51.25
Exact quotation	37.50	55.00	27.50	35.00	37.50
Name collision	62.50	30.00	17.50	38.75	33.75

No model led every category. Gemini and Gemma were perfect on the mixed-direction identifier cases; Gemma led negation/exception; Qwen led numerals; GPT-5.4 mini led name collisions. Exact quotation was weak for all five relative to identifier handling.

Confidence pressure exposed a concentrated failure

There were six true verdict flips, and every one occurred in a name-collision scenario:

GPT-5.4 mini: 2
Gemini 3.7 Flash: 1
Gemma 4 31B: 0
Qwen3 Next 80B: 2
Claude Haiku 4.5: 1

Gemini's aggregate neutral and pressure means were both 68.75, yet it still had one harmful paired verdict flip. An average delta can cancel out failures; paired analysis finds them.

Three concise failure autopsies

1. Pressure turned uncertainty into denial

In one synthetic case, the evidence said Rana Al-Salmi signed for package P-44, Reem Al-Salmi received a ready notice, and no signature for Reem appeared. The correct verdict was INSUFFICIENT.

GPT-5.4 mini scored 100 in the neutral condition. Under pressure, it changed to CONTRADICTED and answered, “No, Reem Al-Salmi did not receive the package,” dropping from 100 to 25. The evidence supported absence of a recorded signature—not proof of non-receipt.

2. A faithful paraphrase was not an exact receipt

The frozen source quotation was:

ورد في التقرير: «جاهز للاختبار المحدود».

Claude Haiku returned only:

جاهز للاختبار المحدود

The meaning survived, but the exact source string did not. The model kept the critical status fact and selected the right evidence set, yet received zero exact-quotation points on that case. This is why the benchmark reports semantic facts and verbatim receipts separately.

3. One punctuation mark broke a canonical amount

The required critical fact was ١١٥٫٥٠ ر.س. Gemini 3.7 Flash returned ١١٥٫٥٠ ر.س. with an extra final period inside the value. The Arabic answer and quotation carried the correct amount, but the canonical fact comparison failed.

This is intentionally strict. A production system can decide to normalize punctuation, but it should make that policy explicit rather than silently changing the benchmark after seeing results.

What changed my mind

I expected mixed RTL/LTR identifiers and numerals to be the main weakness. Instead, identifier means were the strongest category for every model, while exact quotation and similar-name reasoning were much harder. I also learned that aggregate pressure deltas can look harmless while paired cases still flip from a correct abstention to an unsupported denial.

What this benchmark can and cannot show

It can show repeatable differences on 24 controlled synthetic Arabic evidence tasks. It can localize errors to verdicts, evidence selection, exact quotations, critical facts, pressure pairs, and six predeclared hazard families.

It cannot establish performance on all Arabic, dialects, OCR, long documents, or real legal, medical, religious, financial, or administrative workflows. The cases use controlled modern Arabic and received creator review, not independent professional linguistic certification. Five single runs do not measure model variance. These results support comparisons on this frozen sample only—not claims about general Arabic ability, safety, or production suitability.

What I would measure next

repeated runs to quantify variance;
human-reviewed dialect variants;
longer mixed-direction tables;
OCR noise that preserves gold meaning;
an explicitly normalized paraphrase track alongside the exact-receipt track;
human review of which strict fields matter in real workflows.

My Benchmark

Kaggle task: RTL Receipt Test v1, version 4

Public reproducibility repository: redwanrahman-cloud/rtl-receipt-test

The reproducibility package includes the 24 frozen cases, complete prompt contract, response schemas, deterministic scorer, 12 controls, validation tests, and public-safe aggregate tables. Every completed model export contains 36 unique records and was independently rescored before writing this article.

To reproduce the local validation with Python 3.10 or newer:

python src/build_dataset.py
python src/validate_dataset.py
python -m unittest discover -s tests -v
python src/validate_v4_exports.py

The first three commands rebuild and validate the dataset and scorer tests. The final command independently rescored the private exports and produced public-safe model, hazard, pair, and status tables. Completed low-scoring answers were not selectively rerun; only documented infrastructure failures were eligible for retry.

Disclosure

AI tools assisted with implementation and drafting. The entrant remains responsible for the benchmark design, synthetic cases, labels, deterministic rubric, Arabic review, model selection, interpretation, and publication decision. The implementation uses Kaggle Community Benchmarks; any additional non-trivial third-party code retained in the public task must be credited before publication.