DEV Community: Dhrubojyoti Gangopadhyay

Hybrid retrieval with reciprocal rank fusion: notes from a RAG prototype

Dhrubojyoti Gangopadhyay — Mon, 02 Nov 2026 17:18:01 +0000

Why dense and sparse retrieval disagree, why fusing ranks beats blending scores, and what a 20-question evaluation set revealed about a B2B sales-intelligence prototype.

This is a set of notes from a retrieval prototype, the B2B Sales Intelligence RAG repository. It is a prototype: it has fifty unit tests, a twenty-question evaluation set and no deployment. The notes are about the retrieval design, which is the part I would reuse.

The problem with one retriever

A sales team asks questions like "which accounts in the North East mentioned a budget freeze on a call in the last quarter?" The useful evidence lives in CRM notes, call transcripts and account plans. Two retrieval families fail in opposite ways on this:

Dense retrieval (embeddings) understands "budget freeze" and "spending pause" are the same thing. It is poor at exact tokens: account codes, product SKUs, a person's surname, a region string.
Sparse retrieval (BM25) nails exact tokens and dates and is blind to paraphrase.

Sales questions always contain both kinds of signal. So the retriever uses both and then has to combine them.

Why fuse ranks instead of scores

The tempting approach is final = a * dense_score + b * bm25_score. It does not work in practice because the two scores live on different scales and distributions, and the right a and b change with the corpus. You end up tuning a magic number per customer.

Reciprocal rank fusion (RRF) avoids the scales entirely. Each retriever returns a ranked list; a document's fused score is the sum over retrievers of 1 / (k + rank), with k typically 60. A document ranked first by both retrievers wins; a document ranked first by one and absent from the other still surfaces. The prototype applies a weight per retriever on top of that, which is the only knob, and in the evaluation it barely needed moving.

def rrf(ranked_lists, k=60, weights=None):
    scores = defaultdict(float)
    for i, docs in enumerate(ranked_lists):
        w = (weights or [1.0] * len(ranked_lists))[i]
        for rank, doc_id in enumerate(docs, start=1):
            scores[doc_id] += w / (k + rank)
    return sorted(scores, key=scores.get, reverse=True)

Re-ranking the fused list

RRF gives a good top fifty, not a good top five. A cross-encoder re-ranker reads the query and each candidate together and scores relevance directly. It is too slow to run over the whole corpus and exactly fast enough to run over fifty candidates. The prototype uses a small open cross-encoder; the quality jump from fused top-k to re-ranked top-k was the single largest improvement in the evaluation.

The evaluation set

Twenty questions is small, and it was still the most useful artefact in the project. Each question has an expected set of source documents and a reference answer. An LLM judge scores the generated answer for faithfulness to the retrieved context and for coverage of the reference. Retrieval is scored separately with recall at k, which is the number that actually told me whether a change helped.

Three things the set revealed:

Chunk size mattered more than the embedding model. Transcripts needed smaller chunks with overlap; account plans needed whole sections.
BM25 alone beat dense alone on this corpus, because sales questions are full of proper nouns. Fusion beat both.
The judge was lenient on answers that hedged. Faithfulness scoring needs a strict rubric or it rewards vagueness.

What is honest to say about this prototype

The dependency pins did not resolve in CI, so the tests have never run in a pipeline, only locally. There is no deployment and no production traffic. I include it here because the retrieval and evaluation design is sound and reusable, and because the projects page exists to show status honestly rather than to impress.

What I would do in production

Trace every query with the retrieved IDs and scores so failures can be replayed. Grow the evaluation set from real questions, not invented ones. Add a freshness feature, because sales context goes stale in weeks. And keep RRF; in a year of reading about retrieval I have not found a simpler combiner that works as reliably.

Originally published at https://www.dhrubo.shop/writing/hybrid-retrieval-reciprocal-rank-fusion.

What a forward deployed AI engineer actually does

Dhrubojyoti Gangopadhyay — Mon, 02 Nov 2026 17:17:12 +0000

Not a solutions engineer, not a researcher. A working definition of the forward deployed AI engineer role, drawn from doing it inside a company and from reading a hundred job descriptions.

The title "forward deployed engineer" was popularised by Palantir and is now on job boards from Databricks to Sarvam. The AI version of the role is defined inconsistently. Having read about a hundred postings while preparing a job search, and having done the work inside a company since mid-2026, this is my working definition.

The one-sentence version

A forward deployed AI engineer sits with the people who have the problem, scopes a workflow a language model can improve, builds it, deploys it into the tools those people already use, and stays until the numbers move.

Each clause rules something out. "Sits with the people" rules out building from a ticket. "Scopes a workflow" rules out building a chatbot and hoping. "Builds it" rules out pure solutions engineering. "Deploys into existing tools" rules out a demo on a laptop. "Stays until the numbers move" rules out handing over a repo and leaving.

What the week looks like

In my current role at 1XL I own AI adoption across HR, sales, marketing, operations, support, finance and technology. A typical week includes:

Discovery. Sitting with a department head and watching the work. Most "we need AI" requests turn out to be a data-entry problem, an approval-chain problem or a reporting problem. The model is sometimes the smallest part.
Scoping. Writing one page: the input, the output, who approves, what happens when the model is wrong, and the metric. If I cannot write the failure paragraph, the project is not ready.
Building. Usually a small service: a document goes in, structured data comes out, a human reviews, a system of record is updated. Python or TypeScript, a hosted model behind an adapter, a queue if volume needs it.
Teaching. Role-specific curricula and field manuals, because a tool nobody can operate is not deployed. The induction programme for new joiners in the India and UAE offices is part of this.
Infrastructure advice. Which messaging platform, which workflow engine, where the data may live. For us that has meant WhatsApp infrastructure, n8n and agentic-workflow tooling.

What it is not

It is not research. I do not train models and I do not claim to. It is not pure solutions engineering either; a solutions engineer demonstrates the product, a forward deployed engineer changes the customer's process and ships code into it. It is not an internal tools team: the measure of success is a departmental metric, not a shipped feature.

The skills that matter, in order

Scoping judgement. Knowing which 20 percent of a process is worth automating and which 80 percent should be left to a human with a better form.
Integration plumbing. Auth, webhooks, idempotency, retries, logging. Most of the code in a forward deployed project is not AI.
Evaluation. A fixed test set, a judge, and a number that moves. Without it every demo is a success and every deployment is a surprise.
Guardrails. Approval steps before consequential actions. I wrote about how Sahkarmi does this in Approval gates for AI agents.
Explaining. To a finance head who will never read the code, in their vocabulary.

Model knowledge is on the list, but it is fifth or sixth. Providers change monthly; scoping judgement does not.

Why a revenue-operations background helps

I spent sixteen years in enterprise revenue operations before writing software full time. That turns out to be the right preparation for the first two skills on the list. I have been the person whose approval chain was bypassed by a clever spreadsheet, and the person who had to reconcile what the system said against what the dealer received. Operations people can tell when an engineer has never been on the receiving end of their own automation.

How to evaluate a candidate for this role

If you are hiring, look for three artefacts. A deployed system with an approval or review step in it, not a demo. An evaluation set with a number attached. And a document written for a non-technical owner that explains what the system will not do. Résumés list frameworks; these three show judgement.

If you want to see mine, the projects page lists every public system with an honest status, and the résumé is one page.

Originally published at https://www.dhrubo.shop/writing/what-a-forward-deployed-ai-engineer-does.

Approval gates for AI agents: how Sahkarmi decides what a coworker may do

Dhrubojyoti Gangopadhyay — Mon, 02 Nov 2026 17:15:01 +0000

The permission model behind Sahkarmi: sixteen action categories, six that can never be auto-approved, time-boxed delegated authority, and a browser take-over flow that keeps passwords away from the model.

Most agent demos answer one question well: what did the model say? Operations work asks a harder one: what did the system actually do, and who allowed it? Sahkarmi is a desktop platform where AI coworkers browse, file, draft and report on a user's own Windows machine. This article is about the part of it I spent the most design time on, which is the part that stops the agent.

The failure we are designing against

An agent with a browser and a shell can pay an invoice, message a customer, submit a government form, publish a page or delete a folder. Each of those is a one-way door. A chat assistant that hallucinates wastes a minute; an agent that hallucinates a bank transfer costs real money and trust. So the first design rule in Sahkarmi is blunt:

The model proposes. A human approves. The decision is recorded.

Everything below is machinery to make that rule cheap enough that people keep it switched on.

Sixteen categories, six hard gates

Every tool call an agent can make is classified into one of sixteen permission categories in permission-policy.ts: reading files, writing files, browsing, form filling, payments, external messages, publication, shell access, deletions, official forms, and so on. Each category has a default decision: allow, ask, or deny.

Six categories are special. Payments, messages to customers or third parties, publication, shell commands, destructive file operations and official or government forms are hard-coded to ask. The model cannot auto-approve them. The settings UI cannot switch them to allow. A coworker template cannot override them. The reasoning is simple: a policy an agent can talk its way around is not a policy, and a setting a tired user can flip at midnight is not a safeguard.

Delegated authority with an expiry

Asking on every action makes an agent useless for repetitive work, so Sahkarmi has an approval store with two kinds of memory. A specific approval ("yes, send this message") is single-use. A delegated approval ("you may send order confirmations to existing customers for the next thirty days") is scoped to a category and a pattern and carries a hard expiry. Nothing is remembered forever; the store evicts delegated authority at thirty days and plain approvals at seven. Expiry is not a UX nicety. It forces the user to re-read what they have delegated.

The browser take-over flow

The most common place an agent meets a secret is a login page. The naive design is to give the agent the password. Sahkarmi refuses. When a coworker reaches a login or a one-time code, it requests secure input. The sandboxed browser is handed to the human, who signs in with their own hands; the coworker then takes the session back. The model never sees the password or the OTP, which means it can never log them, leak them into a transcript or be tricked into typing them somewhere else.

Each coworker gets its own browser profile, so a coworker that handles supplier portals never shares cookies with the one that drafts marketing posts.

Approvals from your phone

Operations do not wait for someone to be at a desk. The Telegram bridge forwards approval requests to an allow-listed chat and accepts a decision back. The bridge is deliberately narrow: it can approve, reject and report status, and nothing else. A compromised phone can let a pending action through; it cannot start new work.

What this looks like in code

The agent runtime is the Claude Agent SDK, and the hook is canUseTool. Every tool call is routed through the permission policy before execution; the policy returns allow, deny or a pending-approval token. The same policy guards the Codex app-server path and the Agent Client Protocol bridge to OpenCode and Gemini runtimes, so switching models does not change what the system is allowed to do. Tools arrive over the Model Context Protocol, including user-configured MCP servers, which means third-party tools inherit the same gates as built-in ones.

A short honesty note: the gates are enforced in the Sahkarmi host, not in the model. For non-Claude runtimes that call tools through their own loop, the policy is applied at the tool boundary the bridge controls. If a runtime had a side channel the bridge does not mediate, the policy would not see it. That is why the Electron shell is hardened separately: context isolation, sandbox, a strict CSP, Electron fuses and sender-validated IPC on all 209 channels.

Costs

This design has prices. Approval fatigue is real; the fix is good defaults in the 175 coworker templates, so the ask prompts are rare and specific. Delegation expiry annoys power users. The take-over flow adds a minute to any task that needs a login. I think every one of these is worth paying, because the alternative is an agent whose worst day is unbounded.

What I would do next

Three things are missing. Approval requests should show a diff of the effect, not a description of the intent. Delegations should be exportable so an owner can audit what their staff have granted. And the policy needs a formal test suite that proves no path from any runtime can reach a gated category without a token. The test files exist for the policy itself; the end-to-end proof across runtimes does not yet.

Sahkarmi's source, releases and permission policy are public on GitHub. The project page on this site lists what is verified and what is not: Sahkarmi.

Originally published at https://www.dhrubo.shop/writing/approval-gates-for-ai-agents.

Why We Built Sahkarmi: The Open-Source AI Coworker That Never Acts Without Your Approval

Dhrubojyoti Gangopadhyay — Tue, 27 Oct 2026 10:33:13 +0000

By Dhrubojyoti Gangopadhyay (Founder, Sahkarmi)

Website: sahkarmi.site | GitHub: github.com/ganguydhrubo/sahakarmi | License: Apache-2.0

The AI Agent Dilemma: Autonomy vs. Control

Over the past two years, the promise of autonomous AI agents has captured the imagination of every software engineer and startup founder. We were told agents would write our code, debug production incidents, handle customer support, and deploy our cloud infrastructure while we sleep.

Yet in practice, most developers still hesitate to hand full control over to autonomous scripts. Why?

Unpredictable Destructive Actions: An autonomous script with bash access can easily run rm -rf, drop a database table during a migration attempt, or overwrite critical configuration files.
Cloud Lock-in & Privacy Leaks: Sending proprietary codebases, database schemas, and private API tokens to third-party hosted agent clouds poses major enterprise security and data privacy risks.
The Black Box Problem: When an agent works in the background without real-time observability, catching mistakes requires tedious forensic post-mortems after damage is already done.

We asked ourselves: What if autonomous AI agents didn't replace human judgement, but instead operated like trusted colleagues with built-in human verification gates?

That is why we built Sahkarmi (सहकर्मी) — the Hindi word for coworker.

What is Sahkarmi?

Sahkarmi is an open-source desktop team workspace where autonomous AI coworkers run locally on your own machine.

Unlike traditional scripts or chat windows, Sahkarmi's agents are equipped with specialized tools — a sandboxed local terminal, filesystem access, an embedded Chromium browser, and persistent long-term memory — while strictly obeying a single foundational principle:

Zero Unapproved Impact: An agent can plan, inspect files, draft code, and browse documentation autonomously; but the moment it attempts a sensitive action — such as executing a shell command, writing a file, submitting a web form, or executing a transaction — it pauses and presents a clear, interactive Human-in-the-Loop Approval Card.

[ Sahkarmi Human Verification Card ]
--------------------------------------------------
Task: Deploy backend schema migration
Agent: Dev_Backend_Coworker
Proposed Command: python manage.py migrate --database=prod
Working Dir: /home/user/project

[ APPROVE ]     [ REJECT ]     [ REVISE ]
--------------------------------------------------

You maintain complete oversight. You can approve with a single keystroke, reject with corrective feedback, or take over directly.

Under the Hood: The 3-Tier Architecture

1. The Supervised Desktop Interface (Electron + React)

Multi-Agent Tabs: Collaborate simultaneously with specialized coworkers (Code Reviewer, Growth Engineer, System Admin).
Embedded Chromium Engine: Watch your AI coworker navigate websites, test web applications, and take snapshots beside your conversation thread.
Code Diff & Terminal Stream: Real-time syntax-highlighted diffs and streaming terminal stdout/stderr.

2. Local Memory & Routine Engine

Persistent Knowledge Graph: Coworkers remember your project architecture, tech stack preferences, and past decisions across sessions.
Background Scheduled Routines: Define cron-like recurring jobs (check PR reviews every 3 hours, monitor error logs, reconcile ledgers) that trigger autonomously.

3. Sandboxed Tool Execution (Model Context Protocol / MCP)

Local Shell & File Operations: Run builds, tests, git commits, and linters directly on your machine.
Strict Security Boundaries: Passwords, OTPs, and card details are NEVER typed by the agent. If an authentication challenge arises, the agent hands over the browser or requests secure input from you.

Provider Agnostic: Bring Your Own Models

Sahkarmi never locks you into a proprietary LLM API. You can switch models per task:

Local Privacy: Connect to Ollama or vLLM to run Llama, DeepSeek, or Qwen fully offline, with zero cloud data transmission.
High-Reasoning Cloud: Plug in your own API keys for Google Gemini, Anthropic Claude, or OpenAI models.

Getting Started

Sahkarmi is free and open-source under the Apache-2.0 license.

Platform support, honestly stated: Sahkarmi ships a Windows 10/11 (x64) desktop build today. macOS and Linux are planned but are not released yet — installer scripts exist in the repository but are not used for releases.

Download the Windows installer

Grab the latest signed release directly:

sahkarmi.site — download button always points at the current release
Or from GitHub Releases, with a published SHA256SUMS-windows.txt so you can verify the download before running it.

Build from source

git clone https://github.com/ganguydhrubo/sahakarmi.git
cd sahakarmi
bun install
bun run dev

Package managers

WinGet and Scoop manifests are submitted and currently awaiting review by their respective communities. Once those land, installation will also be a one-liner. Until then, please use the direct download above.

Join the Community

We believe the future of software development belongs to collaborative human-AI teams, where humans provide intent and strategic oversight while autonomous coworkers handle the execution grind.

Star the repo: github.com/ganguydhrubo/sahakarmi
Contribute: pull requests welcome for new coworker personas, MCP tool integrations, and UI improvements.

If you try it, I would genuinely like to hear where the approvals model gets in your way — that feedback is what shapes the roadmap.