DEV Community: Rishiraj Debnath

Why Tesseract Fails on Cursive Handwriting (And How We Built a Two-Tier Vision Pipeline in Node.js)

Rishiraj Debnath — Mon, 09 Nov 2026 11:56:51 +0000

When building Holder—a personal information vault and second brain for screenshots, links, and documents—we ran into a classic computer vision bottleneck: handwritten lecture notes and cursive whiteboards.

If you've ever used classical OCR engines like Tesseract on handwritten notes, you know what happens:

Original handwriting: "A random process X(t) is said to be stationary if all its statistical properties..."
Tesseract output: "Sie xe esp Peo cep... fp yaudom Precen X(t) 2 Ad fe te sfefiono ~y i ike stafistieck Prepertioa..."

Here is why classical OCR fails on cursive handwriting, why sending every image to a multimodal AI model is a bad idea, and how we solved it with an automated two-tier pipeline.

1. Why Classical OCR Fails on Cursive

Classical OCR engines like Tesseract rely on Otsu binarization and character segmentation. They operate under the assumption that individual characters have distinct physical separation or predictable typographic spacing.

In cursive or student lecture notes:

Adjacent letters are connected by fluid, continuous strokes.
Letter heights, slants, and baselines continuously drift.
Letter boundaries do not exist in the traditional print sense.

Because Tesseract has no choice but to segment connected strokes, it chops words into random character fragments. It outputs strings of non-words with low confidence scores (often between 25% and 40%).

2. The Naive Fix (And Why It Fails in Production)

The obvious modern fix is: "Just send every image to a Vision LLM like GPT-4o, Claude 3.5 Sonnet, or Llama 3.2 Vision."

In production, this creates two major problems:

Upload Latency: Vision models take 3 to 8 seconds to process an image. If run synchronously on upload, the user sits staring at an "Uploading..." button unable to save.
Cost & Quota Burn: In a personal vault app, 80% of uploads are clean screenshots, tweets, terminal errors, or website bookmarks. Burning GPU cycles and API quotas on images that classical OCR can read in 800ms for $0 is wasteful.

3. The Architecture: Two-Tier Automated Fallback

To achieve sub-second saving, zero unnecessary API costs, and 100% handwriting transcription accuracy, we built a two-tier architecture:

[ User Uploads Image ] | v [ Save to Storage & Return UI Preview (< 400ms) ] --> User can save immediately | v (Background Queue) [ Tier 1: Fast Local OCR (Tesseract.js) ] | v [ Gibberish & Handwriting Heuristic Check ] | +---> [ Clean / High Confidence (> 80%) ] ----> Save Local OCR Output | +---> [ Gibberish / Handwriting Detected ] ---> [ Tier 2: Cloudflare Llama 3.2 11B Vision ] | v [ Extract Math & Cursive Text ] | v [ Live UI Update via Polling ]

4. The Gibberish Detection Heuristic

How does the server automatically know whether an image needs deep handwriting vision without human intervention?
We implemented a multi-factor lexical and statistical check in Node.js:

function shouldTriggerVisionHtr({ text, confidence, width, height }) {
  // 1. Tesseract confidence threshold (cursive handwriting typically scores < 65%)
  if (!confidence || confidence < 72) return true;
  // 2. Large canvas but sparse text (< 35 chars)
  if (width >= 300 && height >= 200 && text.length < 35) return true;
  const rawTokens = text.split(/\s+/).filter(Boolean);
  if (rawTokens.length >= 6) {
    const alphaTokens = rawTokens
      .map((t) => t.toLowerCase().replace(/[^a-z]/g, ''))
      .filter((t) => t.length >= 2);
    // 3. Valid English word ratio against a dictionary set
    if (alphaTokens.length >= 4) {
      const recognized = alphaTokens.filter((t) => COMMON_ENGLISH_WORDS.has(t)).length;
      const recognizedRatio = recognized / alphaTokens.length;
      // Low dictionary match ratio (< 42%) indicates garbled non-words
      if (recognizedRatio < 0.42 && confidence < 82) return true;
    }
    // 4. Fragmented stroke ratio (typical of chopped cursive: fy, fa, fp, te, fe)
    const shortTokens = rawTokens.filter((t) => t.replace(/[^a-zA-Z0-9]/g, '').length <= 2);
    const shortRatio = shortTokens.length / rawTokens.length;
    if (shortRatio > 0.35 && confidence < 82) return true;
    // 5. Unpronounceable consonant clusters (4+ consecutive consonants)
    const gibberishCount = rawTokens.filter(isGibberishToken).length;
    if (gibberishCount >= 2 && (gibberishCount / rawTokens.length) > 0.20 && confidence < 85) {
      return true;
    }
  }
  return false;
}

5. Tier 2: Serverless Vision Execution

When an image trips the gibberish check, it is dispatched to Cloudflare Workers AI running @cf/meta/llama-3.2-11b-vision-instruct on serverless edge GPUs.

Because multimodal LLMs process images through vision-language transformers, they do not merely segment characters—they understand linguistic semantics and domain context.

In our physics notes example:

Tesseract Word Accuracy: ~10% (confidence: 36%)
Llama 3.2 Vision Word Accuracy: 100% (transcribed formulas, strict sense stationary definitions, and bullet points perfectly)

6. Key Takeaways for Builders

Decouple Ingestion from AI Processing: Never block a user's upload flow on a heavy ML/LLM inference call. Store the file, render the preview, return HTTP 201, and process asynchronously.
Heuristic Routing Saves Budgets: Using classical OCR as a fast, zero-cost classifier allows you to reserve expensive vision models for the 20% of inputs that actually require them.
Lexical Analysis Over Confidence Alone: Confidence scores can be misleading (Tesseract can be 70% confident in a garbled word). Testing output tokens against a vocabulary dictionary provides a reliable signal for handwriting detection.

ZeroDown: Building an AI-Powered On-Call Assistant & ChatOps Engine with Switch & MCP

Rishiraj Debnath — Sat, 07 Nov 2026 17:59:39 +0000

The 3:00 AM Nightmare Every Developer Knows

Your phone buzzes frantically. Multiple Discord channels light up with duplicate messages: "Is the site down?", "Auth is failing!", "Can anyone checkout?".

Before you’ve even rubbed the sleep from your eyes, you're faced with:

Sifting through disconnected logs across cloud dashboards.
Trying to remember who is authorized to restart the gateway.
Running risky manual SSH commands while half-asleep.
Answering anxious teammates across multiple chat platforms.

We built ZeroDown to turn this chaos into a calm, automated, and secure workflow.

What is ZeroDown?

ZeroDown is a full-stack, autonomous on-call incident response and ChatOps engine. It pairs a high-frequency microservice monitoring pipeline with local AI agents via Switch Console and the Model Context Protocol (MCP).

Whenever downtime occurs, ZeroDown:

Detects the outage before your users notice.
Deduplicates teammate alerts across Discord and Telegram into a single active ticket.
Runs AI-driven root cause analysis directly against live HTTP failure traces and subsystem probes.
Enables In-Chat Safe Remediation: Engineers can review diagnostic findings and authorize recovery actions directly inside Discord with a simple command: @on-call-ai oncall approve.

Architectural Deep Dive

ZeroDown was engineered from the ground up as a microservices architecture designed for resilience:

Real-Time Monitor Engine (services/monitor)

A non-blocking probe worker that pings target application health endpoints every 30 seconds. It tracks latency, HTTP status codes, and individual subsystem health checks (e.g. database, auth, redis). When consecutive failures exceed the configured threshold, it automatically opens an incident.

The AI Triage & Diagnostic Engine (services/triage)

When an outage is declared, the triage service collects:

Recent probe failure traces (e.g., HTTP 503, connection timeouts).
Subsystem health checks.
Unverified teammate chat reports from Discord and Telegram.

It structures these inputs and runs root-cause analysis, returning:

Confirmed Facts vs. Hypotheses
Confidence Score
Recommended Action
Strictly Typed Recovery Proposal (manual_operator_action)

Bridging Local AI to Discord with Switch Console & MCP

Traditional chatbots require hosting webhook servers, managing Discord bot websockets, and writing custom gateway adapters.

Instead, we used Switch Console to bridge an OpenCode agent (on-call-ai) directly to Discord and Telegram. We built a custom Model Context Protocol (MCP) server (oncall-mcp.js) running over stdio that exposes safe incident investigation and ChatOps tools to the LLM:

oncall_get_incident: Queries current incident status without requiring UUIDs.
oncall_investigate_incident: Triggers root-cause triage on demand.
oncall_approve_incident_recovery: Validates operator permissions and records signed remediation approvals.
oncall_format_incident_reply: Converts raw JSON diagnostics into Discord-friendly Markdown summaries.
How It Works in Practice: A Real Outage Walkthrough

Here is what an incident looks like in action:

Step 1: Automated Detection
The production database connection drops. The monitor detects three consecutive 503 responses and triggers an incident ticket on the Next.js Dashboard.

Step 2: Teammate Interaction in Discord
An engineer sees an alert in the team’s #general channel and checks in:

Rishi [MLH]: @on-call-ai oncall status

on-call-ai [APP]:
🔴 Active Incident: 3e2b1a66-e329-46b9-8f2d-5fb0b316e2f0
• Application: Demo Application / Campus Card
• Severity: High · Status: Open
• Detected: 3 consecutive probe failures (HTTP 503)
• Investigation: Ready to investigate. Reply with @on-call-ai oncall investigate.

Step 3: In-Chat AI Root Cause Analysis
The engineer requests an investigation:

Rishi [MLH]: @on-call-ai oncall investigate

on-call-ai [APP]:
🔍 Investigation Results:
• Summary: Demo Application is degraded due to database check failures.
• Probable Cause: Internal failure in subsystem: database (unconfirmed).
• Confidence: 75%
• Proposed Action: Restart service process to clear transient connection pool exhaustion.
• To Authorize: Reply with @on-call-ai oncall approve.

Step 4: Direct ChatOps Approval

Rather than logging into cloud consoles or running shell scripts, authorized operators approve the remediation right in the thread:

Rishi [MLH]: @on-call-ai oncall approve

on-call-ai [APP]:
✅ Recovery Proposal Approved
• Incident: 3e2b1a66
• Authorized Action: manual_operator_action (Restart service process)
• Authorized By: Switch On-Call Bot (rishi09509)
• Status: Pending execution by remediation runner.

Key Challenges & Lessons Learned

Clean Failure Fallbacks: During local development and hackathons, remote databases or AI services can occasionally be unreachable. We implemented smart in-memory fallback repositories and deterministic heuristic root-cause inference engines. If OpenCode or Docker is momentarily offline, ZeroDown continues running smoothly without dropping incidents.
UUID-Free Frictionless ChatOps: Nobody wants to type a 36-character UUID on a mobile phone during a 3:00 AM incident. We built smart active-incident resolution so plain commands like oncall status, oncall investigate, and oncall approve automatically infer the ongoing ticket.
Safety First: AI agents should never execute destructive production commands autonomously. ZeroDown enforces human-in-the-loop approvals with cryptographically signed tokens and role-based permissions before any recovery action is authorized.

What’s Next for ZeroDown?

Dynamic Multi-Website Management: Adding a one-click dashboard modal to register and manage multiple URLs dynamically.
Proactive Outbound Alerting: Expanding Switch’s event bus to push unsolicited proactive alerts into bridged channels the second an anomaly is detected.
Automated Canary Rollbacks: Connecting recovery approvals directly to Kubernetes and GitHub Actions deployment runners.
Conclusion

Incident response shouldn't require juggling ten tabs, copying logs, and panicking over terminal windows. By pairing real-time observability with local AI agents via Switch and MCP, ZeroDown brings SRE-level automation directly to where teams already communicate.

Built with Next.js, Node.js, OpenCode, Switch Console, and Model Context Protocol (MCP).

HOLDER: A Personal Vault That Helps You Stop Losing Useful Things

Rishiraj Debnath — Sat, 24 Oct 2026 18:16:12 +0000

This is a submission for the Hacktoberfest Open-Source AI Challenge Week 3: Slow and Steady

What I Built

I built HOLDER, a personal vault for all the digital stuff we save and then struggle to find later.

Think about how often we save a useful link, a YouTube video, a screenshot, a PDF, or a random note. A few days later, when we actually need it, we have no idea where we saved it.

HOLDER is my attempt to solve that problem.

It brings links, YouTube videos, images, PDFs, and notes into one searchable space. You can organize items with folders and favorites, and OCR makes text inside screenshots searchable too.

The goal is simple: save something once and find it when you need it, without going through the same files and tabs all over again.

I also included an AI Assistant entry point to make the vault more useful and help me to search and find anything I need.

Demo

doc-pied-five.vercel.app

Code

GitHub repository:

13507-IN / Doc

Holder — Personal Assistant & Info Vault

Holder is an intelligent personal information assistant and knowledge vault built with Next.js 14, Express.js, MongoDB, and a Chrome Browser Extension.

Store, categorize, search, and converse with all your important information—including YouTube videos, images, web links, code snippets, and private credentials.

✨ Features

📁 Multi-Folder Collections: Organize items into custom folders (e.g. 💼 Brand Assets, 🔒 Private Vault, 🎥 YouTube Tutorials, 🌐 Bookmarks) with emoji icons, color badges, and description notes.
🎥 YouTube Auto Metadata & Lightbox Player: Paste any YouTube link to automatically extract channel metadata, title, and high-res thumbnails, with responsive video playback inside the app.
🖼️ Image Vault & Uploads: Upload local images or paste image URLs, complete with lightboxes.
📄 PDF Document Vault & Viewer: Upload PDF documents or paste direct PDF links, with inline interactive PDF…

View on GitHub

How I Built It

I built HOLDER with a separate frontend, backend, and browser extension.

Frontend

The client is built with Next.js 16 and React 19.

Next.js 16: Application framework

React 19: UI components and interactions

Axios: API communication

lucide-react: Icons

ESLint 9: Code quality and linting

Backend

The server uses Node.js and Express 4.

MongoDB and Mongoose 8: Data persistence

Passport and Google OAuth: Google authentication

JWT and bcryptjs: Authentication and password-related security

Cloudinary: Media storage

Multer: File uploads

Tesseract.js: OCR and text extraction from images

Cheerio: HTML parsing and content extraction

Helmet and express-rate-limit: Security headers and request-rate limiting

The backend connects the different parts of the application, manages stored information, and processes uploaded content.

Chrome Extension

I also built a browser extension using Chrome Manifest V3 and vanilla JavaScript. This provides a browser-side way to interact with the application without building another full frontend.

AI-Assisted Development

I used OpenCode with the Big Pickle model as a coding assistant while developing HOLDER. It helped me work through implementation tasks and iterate on the code.

This was part of my development workflow. The deployed application's OCR functionality is powered by Tesseract.js; Big Pickle is not presented as the model powering the live application.

The core idea behind HOLDER is making saved information easier to find, and Tesseract.js plays an important role in that workflow.

Tesseract.js is an open-source OCR library that uses WebAssembly to bring the Tesseract OCR engine to JavaScript environments. It extracts readable text from images. https://github.com/naptha/tesseract.js

In HOLDER, I use it to extract text from screenshots so that the information inside an image can be searched later.

The workflow is straightforward:

A user saves a screenshot or image.

Tesseract.js extracts the text contained in it.

The extracted text is stored alongside the saved item.

When the user searches for a word or phrase, the application can find matching content without requiring the user to open every image manually.

I built the surrounding application with Next.js 16, React 19, Node.js, Express, and MongoDB. The backend handles data management, while the OCR component helps turn otherwise difficult-to-search images into searchable information.

During development, I also used OpenCode with Big Pickle as a coding assistant to help implement features and work through coding tasks. Big Pickle assisted with development; it is not the model powering HOLDER's live application.

My goal was to combine open-source tools with a practical problem: reducing the effort required to find information that you have already saved.

Why Does Open Innovation Matter?

A personal information vault can contain a lot of context about someone's work, interests, and daily life. That makes control over the AI layer an important design consideration.

Using open-source AI gives developers more freedom to inspect, adapt, and replace components instead of building every AI feature around a single closed provider.

For HOLDER, the important part is making the AI layer useful without making the entire application dependent on one fixed implementation.

I think open innovation matters because useful tools should be adaptable. Developers should have the freedom to understand their systems, experiment with different approaches, and improve them as their needs change.

My Agent Session

The Lazy Payoff

The small tasks are often the most annoying ones.

Searching through old browser tabs, opening multiple folders, checking screenshots one by one, and trying to remember the name of a file all take time.

HOLDER is designed to reduce that repeated effort.

Save the resource, keep it organized, and search for it later. When text from screenshots is indexed through OCR, you don't have to rely only on filenames or memory to find the information again.

I haven't measured a reliable time-saving figure yet, so I won't claim a made-up number. The value is in reducing the effort involved every time you need to retrieve something you saved earlier.

Prize Categories

Render, MongoDB

"AgentGuard", Way to secure your personal data from AI agents

Rishiraj Debnath — Sat, 24 Oct 2026 05:09:18 +0000

A few weeks ago, I had one of those heart-stopping developer moments.
I was tinkering with a local autonomous coding agent running on a small open-source model (Qwen 2.5 via Ollama). I gave it access to a bash tool so it could install dependencies, run test suites, and clean up build artifacts.
Everything was going smoothly until I asked it to clean up some old logs.
In the terminal output, I watched it calmly propose:
bash rm -rf /var/log/mongodb

And right after that, it started poking around files outside its project folder.I hit Ctrl+C so fast I almost snapped my keyboard.
The "Polite Prompt" Trap
Like most developers, my first instinct was to fix it with prompt engineering:
"You are a careful assistant. Please ONLY touch files in ./workspace. NEVER use sudo. NEVER delete system folders. Pretty please."
And you know what happened on the very next run?

It did it again.

Here’s the thing we don't talk about enough: Small, open-source models (3B, 7B, 14B) don't have the massive alignment layers that GPT-4 or Claude 3.5 have. When an open-source agent gets confused, hallucinates, or hits an edge case, your polite system prompt is the first thing out the window.

If an AI decides to read your private banking statement or a .env full of production tokens, you shouldn't have to convince the model to behave.

You wouldn't ask an untrusted web request to "please not do SQL injection"—you use parameterized queries.

So why are we begging AI agents not to nuke our hard drives?

What I Actually Wanted

I wanted a simple rule: The agent can think whatever crazy thought it wants, but before any tool touches my operating system, someone checks its badge at the door.

If the agent decides to read bank_statement.pdf, the system shouldn't plead with the AI. It should intercept the file call, say "Nope, not happening", and ensure 0 bytes of that file are ever read.

That was the spark behind AgentGuard.

How It Works:

AgentGuard is essentially a bouncer sitting between your AI agent and your OS. It does three straightforward things:

[ AI Agent / Ollama / OpenDevin ]
│
│ 1. "Hey, I want to run: cat bank_statement.pdf"
▼
┌───────────────────────────┐
│ AgentGuard Gateway │ ◄── [ Backboard.io ]
└───────────────────────────┘ (Remembers previous blocks)
│ │
[ BLOCK ] [ APPROVE ]
│ │
│ └──► Rings your dashboard bell:
│ "Agent wants to pip install something. Allow?"
▼
Refusal sent back to Agent:
"Access to banking file blocked. Stay in ./workspace."
│
▼
Agent self-corrects:
"My bad! Reading ./workspace/logs/debug.log instead."

Pre-Execution Interception (0ms latency)

When an agent calls a tool (shell command, file read, file delete), AgentGuard inspects the request synchronously before execution.

Trying to run sudo? → Blocked.

Trying to touch .env, id_rsa, or a file matching bank or tax? → Blocked.

Escaping ./workspace with ../../? → Blocked.
The file handle is never opened. The shell command is never spawned.

The 5-Minute Human Review Queue

What about grey-area actions? Say the agent wants to run pip install pymongo.

You don't want to completely block package installs, but you don't want an AI silently pulling unknown wheels from PyPI either.

AgentGuard pauses the agent and puts the request into a Human Review Queue:

You get a notification on a clean web dashboard.

Each request has a 5-minute cryptographic timer so old, stale commands can't accidentally execute hours later.

You click [ AUTHORIZE & RUN ], and it runs inside a controlled sandbox right in front of your eyes.

Giving the Agent an "Immune System" (Backboard.io)

One thing that drove me crazy about local models was amnesia. If you tell an agent "don't delete that folder", it apologizes... and then tries again two turns later.

To fix this, I plugged in Backboard.io for persistent cloud vector memory.

Whenever AgentGuard blocks a command, it writes a short lesson into Backboard:

"Incident: Agent tried to run 'sudo rm -rf'. Blocked by policy shell.destructive_commands."

Next time you start a task, AgentGuard pulls the relevant memories from Backboard and feeds them into the agent's context. The agent literally remembers what got it in trouble yesterday and stops proposing bad commands in the first place.

What the Dashboard Looks Like

I didn't want a clunky terminal-only tool, so I built an industrial, dark-mode cockpit with React and Tailwind:

Top KPI Cards: Live count of blocked attacks, total actions governed, and learned Backboard memories.

Review Queue: Tabs for Pending, Approved, and Expired actions with inline terminal output.

Threat Lab: A built-in sandbox where you can test 1-click red-team scenarios (ransomware simulations, credential exfiltration, path traversal) to see if your local model can break out. Spoiler: 0 escapes.

Making It an NPM Package: Bring Your Own Keys (BYOK)

Right now, you can clone the repo and run it locally with FastAPI and MongoDB.

But I realized that's still too much friction for developers who just want to npm install something and start coding.

So I'm packaging this as a standalone Bring-Your-Own-Keys (BYOK) NPM SDK:

bash
npm install agentguard

You drop in your own Backboard API key (for cloud memory) and your own LLM key (or Ollama endpoint):

typescript
import { AgentGuard } from 'agentguard';
const guard = new AgentGuard({
backboardApiKey: process.env.BACKBOARD_API_KEY,
workspace: './workspace',
});
// Wrap any agent tool call:
const check = await guard.inspect(proposedCommand);
if (check.isBlocked) {
// Feed safety feedback back to the LLM so it self-corrects!
console.log(Blocked: ${check.explanation});
} else {
await guard.execute(proposedCommand);
}

Zero servers to host. Zero background databases to manage. Just pure protection.

Wrapping Up

We are entering an era where AI agents aren't just autocomplete engines anymore—they are active operators on our computers. They write code, manage files, and execute bash scripts.

We have to stop treating AI safety as a philosophical prompt problem and start treating it as a systems architecture problem.

Don't convince the model to behave. Just put a guard on the wire.

If you're building autonomous agents or playing with local models, I'd love to hear how you're keeping them safe. Have you had an AI agent do something terrifying on your machine yet?