DEV Community: Gabriel Le Roux

Your Chars-Per-Token Estimate Is Wrong (Mine Was, By Exactly One Context Window)

Gabriel Le Roux — Fri, 17 Jul 2026 07:07:00 +0000

A user saved a long YouTube video. The pipeline pulled the transcript — about 1 MB of text — and sent it to the summarizer, which came back with:

400 Bad Request
This model's maximum context length is 128000 tokens,
however you requested 130700 tokens.

Which is annoying, because the truncation code that exists specifically to prevent this had run. It did its math. It confidently produced a prompt ~2,700 tokens too big.

Two bugs were hiding in that math. Both are the kind that live happily in production until an input arrives that's big enough to find them.

Bug 1: one setting doing two jobs it was never meant to do

My AI capabilities — summarize, classify, tag, extract entities, translate — run through tenant-configurable connectors. Each connector has a maxTokens setting.

The handlers were using that one value for two completely different jobs:

// what I had. both of these are wrong.
var maxOutput = connector.MaxTokens;                 // sent to the model
var maxInputChars = connector.MaxTokens * 4 * 3;     // used to truncate input

That second line is a formula with the structure of a guess. But the deeper problem is the first one, and it's conceptual.

A connector is shared. Its maxTokens might be set high because a chat-style consumer somewhere wants long completions. Feed that same number into a summarization call and you have:

Reserved an enormous output budget the task will never use,
Inflated your input allowance from that number,
Blown clean through the context window,

…for a task that was never going to emit more than a page of text.

The fix: budget per capability, from the two numbers that are actually true

public static class AiTokenBudget
{
    // output is a property of the TASK, not of the connector
    private static readonly Dictionary<string, int> OutputCaps = new()
    {
        ["summarize"] = 4_096,   // nobody wants a 20k-token summary
        ["classify"]  = 4_096,
        ["tags"]      = 4_096,
        ["entities"]  = 4_096,
        ["translate"] = 16_384,  // output scales with input here
    };

    private const int ContextWindow = 128_000;
    private const double CharsPerToken = 2.5;   // see Bug 2

    public static int OutputTokens(string capability) =>
        OutputCaps.TryGetValue(capability, out var cap) ? cap : 4_096;

    public static int MaxInputChars(string capability)
    {
        var reserved = OutputTokens(capability) + 1_000;   // prompt overhead
        var inputTokens = ContextWindow - reserved;
        return (int)(inputTokens * CharsPerToken);
    }
}

Output is capped by what the task needs. Input gets whatever room is left in the real context window. The connector's maxTokens goes back to being what it always should have been: a setting for consumers that need it, not a load-bearing constant for every capability in the system.

Bug 2: real text is denser than prose

The truncation estimate assumed 3.0 characters per token.

English prose runs around 4. So 3.0 feels comfortably pessimistic. It feels like the safe choice a careful person makes.

Then I actually measured the transcript that failed:

1,048,576 chars / 360,335 tokens = 2.91 chars per token

2.91. Below my "safe" floor.

And of course it is. A timestamped YouTube transcript is not prose:

00:14:22.480 --> 00:14:25.120
- yeah, so the, um, the thing is —
00:14:25.120 --> 00:14:27.900
right, exactly, 40%, roughly

Short lines. Numbers. Timestamps. Speaker fragments. Punctuation everywhere. Exactly the kind of text a BPE tokenizer shreds into tiny pieces. At 1 MB, the difference between "3.0, feels safe" and "2.91, measured" is thousands of tokens — which is to say, it was the entire overflow.

The constant is now 2.5, bought by measuring the input that actually failed rather than by picking a rounder number and feeling better about it.

When a heuristic fails, the fix is a datum, not a more confident-sounding guess.

The safety net: halve and retry

Here's the uncomfortable truth sitting underneath both fixes:

Any chars-per-token constant is wrong for some input. Code samples. Dense CJK. Tables. Base64. Some transcript format I haven't met yet. One of them will eventually beat whatever ratio I pick, and I will not know in advance which.

So the last line of defence doesn't estimate at all:

try
{
    return await SummarizeAsync(text, ct);
}
catch (AiContextOverflowException)
{
    // don't re-estimate. just take half.
    var half = text[..(text.Length / 2)];
    _log.LogWarning("Context overflow at {Len} chars; retrying with half.", text.Length);
    return await SummarizeAsync(half, ct);
}

Crude — and that's precisely the virtue. It makes no assumption about why the estimate lost. It converges in a step or two. And it turns "job failed" into "summary of most of the transcript," which for summarization is an easy trade: a slightly shorter input is invisible in the output, while a hard failure is extremely visible in the app.

Three layers, and you need all three

Prevent the systematic error — per-capability budgets, so one shared setting can't size every call.
Narrow the guess — a ratio measured from inputs that actually failed, not one that sounded prudent.
Recover without the estimate — halve-and-retry, which is right even when layers 1 and 2 are wrong.

Estimate conservatively. Verify against reality. Keep one recovery path that doesn't depend on your estimate being right.

The 1 MB transcript summarizes fine now.

The takeaway you can steal

Don't derive your input budget from your output setting. They're different numbers with different owners.
Cap output by capability. A summarizer does not need 16k tokens of headroom.
Measure your chars/token on the inputs that break you. Your content type has its own density, and it is not prose.
Keep a dumb recovery path. The clever one will eventually be wrong.

Don't Build N Pipelines. Build N Adapters to One Pipeline.

Gabriel Le Roux — Wed, 15 Jul 2026 07:06:00 +0000

My read-it-later app had a quiet caste system.

Save a web page and you got the full treatment: clean reading view, AI summary, chunked, embedded, retrievable by the RAG chat.

Save a PDF, a .docx, a screenshot, or a podcast episode and you got... a stored file. A row in a table. A dead end.

Last weekend I fixed that — PDFs, Office documents, images, audio and video all now flow into the reading view and the RAG index. What I want to write up isn't the extraction code (it's mostly library calls). It's the one architectural decision that made the whole thing cheap, and the one guard that keeps the index honest.

The tempting design (don't do this)

The obvious approach: a new pipeline per file type, each with its own result type, each with its own downstream handling.

// the road to hell
public record PdfExtractionResult(string Text, int PageCount, string[] Fonts);
public record DocxExtractionResult(string Body, int WordCount, string[] Headings);
public record AudioExtractionResult(string Transcript, TimeSpan Duration, string Lang);
public record ImageExtractionResult(string OcrText, float Confidence);
// ...and now 5 handlers, 5 mappers, 5 code paths into the indexer

Five times the surface area. Five places to update when the chunker changes. Five subtly different ways for the same bug to manifest.

The design that shipped: one output contract

Every extraction pipeline, whatever it does internally, produces the same shape as the existing page-content pipeline:

public record ExtractionResult(
    string  Markdown,     // the content, as markdown. always.
    int     WordCount,
    string? Summary
);

That's it. PDF text extraction, DOCX paragraph walking, OCR, Whisper transcription — every one of them ends in "here is this thing, as markdown."

The consequence is the whole point: the entire downstream flow needed zero changes. Reading view, summarization, chunking, embedding, retrieval — none of it knows or cares what the file was. A podcast episode enters the vector index through exactly the code path an article does.

On the consuming side, the five per-type result handlers collapsed onto one base class:

public abstract class ExtractionResultHandler<T> : IHandler<T> where T : IExtractionMessage
{
    protected async Task CompleteAsync(Guid itemId, ExtractionResult result, CancellationToken ct)
    {
        await _items.SetReadingArtifactAsync(itemId, result.Markdown, result.WordCount, ct);
        if (result.Summary is not null)
            await _items.SetSummaryAsync(itemId, result.Summary, ct);

        if (string.IsNullOrWhiteSpace(result.Markdown))
        {
            _log.LogInformation("No content extracted for {Id} — not embedding.", itemId);
            return;                              // ← the guard. see below.
        }

        await _bus.PublishAsync(new EmbedItemMessage(itemId), ct);
    }
}

New file type support is now, almost entirely, the question: what turns this into markdown?

The four adapters

Type	Library	What it extracts
Documents	OpenXml	`.docx` paragraphs, `.xlsx` sheets as pipe-separated rows, `.pptx` slides including speaker notes
Images	Tesseract	OCR text
Audio	Whisper	Full transcript
Video	ffmpeg → Whisper	Strip the audio track, hand it to the audio adapter

Two notes worth more than the table:

PowerPoint speaker notes. Most "read a .pptx" implementations give you a wall of bullet fragments. The notes are where the actual argument lives — the slide says "Retention improved 18%", the notes say why, which cohort, and what we're not sure about. Extract them:

foreach (var slidePart in presentationPart.SlideParts)
{
    sb.AppendLine(ExtractSlideText(slidePart));

    var notes = slidePart.NotesSlidePart?.NotesSlide?.InnerText;
    if (!string.IsNullOrWhiteSpace(notes))
        sb.AppendLine($"> Speaker notes: {notes}");   // markdown blockquote
}

Video was nearly free. Once audio worked, video is ffmpeg -i in.mp4 -vn -acodec mp3 out.mp3 and then... the audio adapter. That's the adapter pattern paying rent: the new type didn't need a new pipeline, it needed a converter to an existing one.

The guard that keeps the index honest

One check mattered more than any feature:

If extraction produces nothing, the item is not embedded.

That's the early return in the handler above. Without it, a photo with no readable text enters the index on the strength of its filename, and eventually the retriever hands IMG_4211.jpg to the model as a "source" — at which point the model, being a model, improvises something plausible about a document it has never seen.

That's the retrieval equivalent of confidently citing a blank page, and it is the single fastest way to destroy trust in a RAG feature.

An honest index has to be allowed to leave things out. Three lines of code; the difference between a library and a liar.

The same instinct applies one layer up: if the user asks a question about an item whose extraction hasn't finished, don't call the model at all. Return an honest "I haven't read this yet." The guard against hallucination belongs before the model call, in code that checks whether an honest answer is even possible. No prompt engineering achieves what a precondition does.

Two integration details that cost me an afternoon each

Extensionless PDF URLs. My PDF detection checked for a .pdf suffix. Plenty of PDF links have no extension — every arxiv.org/pdf/2401.12345 URL, for instance. Those saved fine through the PDF path, but also captured the browser's PDF-viewer DOM and queued a page-scrape job that could only ever fail. Fix: probe the content type; don't trust the URL's spelling.

Transcription needs its own connector. Whisper wants an OpenAI-style /audio/transcriptions endpoint. My default gateway for chat-style models (OpenRouter) doesn't serve one. "AI provider" is not one interface — transcription and chat are different sockets even when a single vendor offers both.

The takeaway

Don't build N pipelines. Build N adapters to one pipeline.

The output shape is the contract. Everything upstream of it is replaceable, and everything downstream of it never has to know what happened.

And whatever you build: let the index leave things out.

Cloudflare 403s Your HttpClient. Here's the Four-Rung Ladder That Finally Got the PDF.

Gabriel Le Roux — Tue, 14 Jul 2026 05:01:12 +0000

A user saved a medRxiv paper to my app. PDF extraction failed — not a parse error, a 403 Forbidden on the download itself. The host sits behind Cloudflare, and Cloudflare had decided my server wasn't a person.

This is the normal case now, not the exception. Preprint servers, industry reports, government documents — a growing share of the PDFs people actually want live behind bot detection. A "save this PDF" feature that only works on politely-served PDFs fails on all the interesting ones.

What follows is the four-rung escalation ladder I ended up with. Each rung exists because the one below it failed on a real URL, and the whole thing is sorted by cost, which turns out to be the most important design decision in it.

Rung 1: HttpClient with realistic browser headers

The baseline. A normal .NET HttpClient, a real Chrome user-agent string, the full header set a browser would send.

var request = new HttpRequestMessage(HttpMethod.Get, url);
request.Headers.UserAgent.ParseAdd(
    "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 " +
    "(KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36");
request.Headers.Accept.ParseAdd("application/pdf,text/html;q=0.9,*/*;q=0.8");
request.Headers.AcceptLanguage.ParseAdd("en-US,en;q=0.9");
request.Headers.Add("Sec-Fetch-Dest", "document");
request.Headers.Add("Sec-Fetch-Mode", "navigate");

This still works for most of the web and costs nothing. medRxiv 403'd it without blinking.

Why it fails: because modern bot detection doesn't read your headers. It reads your handshake.

Rung 2: curl-impersonate, because TLS is the real fingerprint

Here's the thing that took me embarrassingly long to internalise:

Your TLS handshake identifies your HTTP client before a single byte of HTTP is sent.

Every client negotiates TLS with a characteristic combination of cipher suites, extensions, elliptic curves, and — crucially — the ordering of all of them. That combination is stable per library and is trivially fingerprintable (JA3/JA4). .NET's SslStream announces "I am a program written in .NET" in the ClientHello. You can spoof every HTTP header you like; you're still holding up a sign.

curl-impersonate is curl rebuilt against the same TLS stack Chrome uses, with the handshake tuned to match byte for byte. You shell out to curl_chrome116 (or whichever build) and you look, at the transport layer, exactly like Chrome.

var tmp = Path.GetTempFileName();
var psi = new ProcessStartInfo("curl_chrome116")
{
    RedirectStandardError = true,
    RedirectStandardOutput = false,   // important — see below
};
psi.ArgumentList.Add("-sL");
psi.ArgumentList.Add("--max-time"); psi.ArgumentList.Add("60");
psi.ArgumentList.Add("-o");          psi.ArgumentList.Add(tmp);
psi.ArgumentList.Add(url);

using var proc = Process.Start(psi)!;
await proc.WaitForExitAsync(ct);

var bytes = await File.ReadAllBytesAsync(tmp, ct);

The implementation detail that bites everyone: do not pipe a binary response through anything that touches strings. RedirectStandardOutput = true plus ReadToEndAsync() will hand you a string, and the moment a PDF's bytes go through UTF-8 decoding they are silently, irreversibly corrupted. Write to a temp file, read it back as byte[].

Rung 3: the same fingerprint, from a residential IP

Some hosts don't care what you sound like, only where you're calling from. Whole datacenter ASNs are blocked outright — AWS, Hetzner, DigitalOcean, all of it.

So rung 3 is rung 2 again, through a residential proxy:

psi.ArgumentList.Add("--proxy");
psi.ArgumentList.Add($"http://{proxyUser}:{proxyPass}@{proxyHost}:{proxyPort}");

Chrome's TLS fingerprint. A residential IP. A full browser header set.

medRxiv 403'd that too.

At which point the lesson lands: you are not going to out-imitate the browser. You have to be the browser.

Rung 4: an actual browser — and the indirection that makes it work

The last rung drives a real stealth browser (I use Browserless; Playwright or Puppeteer with a stealth plugin gets you the same place).

But you cannot just point a headless browser at the PDF URL and grab the response. The naive version fails exactly like the others, because a direct hit on the PDF has no session behind it. The move that matters is indirection:

Navigate to the article page — the HTML landing page, not the PDF.
Let the browser sit through and solve the Cloudflare challenge, the way any visitor does. It gets a clearance cookie.
Then fetch the PDF from inside that page's JavaScript context, so the request carries the cookies the challenge just minted.
Return the bytes as base64, because the query layer between you and the browser only speaks scalars.

// evaluated inside the page, after the challenge has been solved
const res = await fetch(pdfUrl, { credentials: 'include' });
const buf = await res.arrayBuffer();
return btoa(String.fromCharCode(...new Uint8Array(buf)));

Now the PDF request arrives as what it claims to be: a follow-on request from a browser that has already proved itself. That's the difference between imitating trust and acquiring it.

The medRxiv paper that had failed in production for three days came through at 189 KB with a clean %PDF header.

Why the order matters more than the rungs

The ladder isn't "try things until one works." It's sorted by cost.

Rung	Mechanism	Cost	Defeats
1	HttpClient + headers	Free	Naive UA checks
2	curl-impersonate	Free	TLS fingerprinting
3	curl_chrome + residential proxy	Cheap	Datacenter IP blocks
4	Stealth browser	Metered	Full JS challenges

Rungs 1–3 are effectively free. Rung 4 is a paid, metered service with a per-session price. Escalating in cost order means the expensive tier only runs when every cheap option has genuinely failed — and it's quota-checked and usage-recorded like every other paid call.

The expensive path being last is what makes it affordable to have at all. Put the stealth browser first and you have a beautiful, reliable, financially ruinous fetcher.

The single most important line: validate the bytes, not the status code

This is the part I'd tattoo on people if it were legal.

Bot walls routinely return 200 OK with an HTML block page where the PDF should be. Trust the status code and that garbage flows downstream into your PDF parser, which fails later with a baffling error pointing nowhere near the actual cause. You will spend an afternoon debugging a "corrupt PDF" that was never a PDF.

So every rung validates the payload:

private static bool LooksLikePdf(ReadOnlySpan<byte> bytes) =>
    bytes.Length > 4 &&
    bytes[0] == 0x25 && bytes[1] == 0x50 &&   // %P
    bytes[2] == 0x44 && bytes[3] == 0x46;     // DF

Four bytes. A block page fails immediately, at the rung that received it, and the attempt log reads like a story:

[1] plain      → 403
[2] curl_chrome → 403
[3] proxied     → 403
[4] stealth     → 200, %PDF, 189KB ✓

When some future host needs a rung five, that log is where I'll find out what to build.

The general shape

Cheap thing first. Escalate on failure. Validate the payload, not the envelope. Log every attempt.

None of that is PDF-specific — it's the shape of any fetch pipeline that touches the adversarial web. The web you build against in development is cooperative. The web your users actually save from is not.

Build the ladder before you need the top rung.

Turning a Pile of Saved Links Into a Queryable Knowledge Hub (.NET + pgvector)

Gabriel Le Roux — Tue, 30 Jun 2026 19:21:00 +0000

You save things for months — articles, videos, threads that explained exactly the thing you were stuck on. Your library fills up. And then, when you actually need something, you Google it from scratch anyway, because finding it in your own saves feels harder than searching the whole internet again.

That's the gap I wanted to close: turn a passive pile of bookmarks into a knowledge hub you can just ask. Type a question in plain language, get a real answer, with links back to the exact saves it came from.

Here's how it's built.

Disclosure: I'm the founder of SavePosty, a read-it-later app, and this is a write-up of a real feature in our own .NET codebase — not a sponsored post. Every detail below is from production.

The shape of it

The hub is two pipelines that share one index. Ingest runs once per save and builds the searchable representation. Query runs every time you ask a question and reads that representation live.

Nothing here is a separate search product. It's all .NET on the Postgres I already run — which turns out to be the most important design decision.

Storage: the hub lives in Postgres

The popular instinct is to reach for a dedicated vector database. I used pgvector instead — a Postgres extension that adds a vector column type and distance operators (cosine, L2, inner product).

Why it's the right call at my scale:

Every save already lives in Postgres. A similarity search is one SQL query joined on user_id — no second datastore, no sync job, no "did the vector store drift from the relational store?" class of bug.
An HNSW index turns that search from O(n) brute force (seconds at half a million vectors) into O(log n) graph traversal (milliseconds). It's approximate, but "close enough" is exactly what semantic search wants.

The honest tradeoff: the HNSW index is memory-hungry — it wants RAM as your vector count grows. That's fine per user; the real conversation starts at platform scale, and by then I'll be deciding with production numbers instead of a guess.

Embeddings: the provider is config, not code

An embedding is just an array of floats that captures the meaning of a chunk of text. The model that produces them is the one piece I deliberately did not hardcode.

I started on self-hosted Ollama (nomic-embed-text, zero per-call cost) and later moved to OpenAI's text-embedding-3-small. The migration was zero code: the embedding provider is a capability mapping in an admin UI, so I changed Embedding → provider and re-indexed. If prices change or a better model ships, I flip it back the same way.

The ingest details that matter: text is chunked into ~200-word windows with ~40-word overlap (so an idea that straddles a boundary is still retrievable), and an EmbeddedAt timestamp gates visibility — the hub never answers from a half-indexed save.

The agent: it doesn't just retrieve, it acts

A plain RAG chain is a fixed pipeline: embed query → retrieve top-k → stuff into prompt → answer. It works. But I wanted the hub to do things, so I gave it an agent with tools.

The agent has search_my_saves() and create_collection(), and it decides when to use them. So it can skip searching when it already knows, reformulate a vague question, and — the fun part — act on the results: ask it to "group everything I saved about RAG" and it searches, then builds a collection from what it found. Retrieval plus action, from one request.

The cost is real (more tokens, slightly less determinism), so I split the work: a stronger model (claude-sonnet-4-6 via OpenRouter) drives the agent, while a cheap one (claude-haiku-4-5) handles quick one-shot questions for next to nothing.

Streaming: the answer arrives as it forms

Answers stream back over Server-Sent Events, with typed events the UI reacts to.

thinking, text_chunk, citation, collection_created — that's the whole protocol. I chose SSE over WebSockets on purpose: the data flows one direction (server → client), so a stateful bidirectional socket would only add complexity. SSE is HTTP-native, reconnects cleanly, and the typed events are what make the live "thinking" indicator and inline citations possible.

What I'd reach for next

Hybrid search (BM25 + vector) to catch the exact-keyword queries pure semantic search occasionally fumbles.
Semantic chunking instead of fixed windows, for tighter retrieval.
Embedding-cache invalidation so editing a save re-embeds only what changed.

The big lesson: a "knowledge hub" sounds like it needs heavy infrastructure, and it mostly doesn't. Postgres you already run, a swappable embedding provider, an agent with two tools, and a streaming endpoint get you a long way.

I'm building SavePosty in the open. If the internals are your thing, come follow the build — and tell me what you'd do differently in the comments.

From Hangfire to RabbitMQ: Killing Database Polling in a .NET App

Gabriel Le Roux — Tue, 23 Jun 2026 18:58:19 +0000

Hangfire makes you look smart for almost no effort: drop in a NuGet package, point it at the database you already have, and you get a job runner and a dashboard for free. For a single .NET service, it's genuinely hard to beat.

So why did I pull it out of an entire app?

Because I wasn't running one job. I was running all of them through Hangfire — and every single one was hitting Postgres every few seconds asking "anything for me yet?"

This is the story of moving every polling Hangfire job in SavePosty — a read-it-later app I'm building — to RabbitMQ: the wins, the migration pattern that kept it boring, and the one thing you lose. The honest version.

Disclosure: I'm the founder of SavePosty. This is a write-up of a real migration in our own codebase, not a sponsored post — every example below is from production.

What I was running before

Across the app, Hangfire ran all my background work, backed by PostgreSQL — email sends, webhook dispatch, content re-fetch, and the rest. Different jobs, same mechanism: each one discovered work by polling the database on a timer.

Credit where it's due, because the migration posts that pretend the old tool was garbage are lying to you. Hangfire gave me:

Zero new infrastructure — it rides on the Postgres I already run.
A dashboard out of the box: queued, processing, succeeded, failed.
Retry logic in a single attribute.

If you're one service with a small team, honestly, just use Hangfire. You'll be happy.

Why I migrated

I didn't leave because Hangfire was bad. I left because my situation changed.

1. Polling load that scales with the clock, not the work. Every Hangfire job hits Postgres on an interval whether or not there's anything to do. One job is invisible. But I had many jobs, all polling the same database, all the time — constant read load that grows with your poll frequency and your job count, not with actual throughput. You pay it even when the app is idle.

2. Two job mechanisms in one system. One service, PostyFetch, already ran on RabbitMQ. Everything else still polled Hangfire. So I maintained two completely different models for "do work in the background" — two mental models, two runbooks, two places to look when something broke. Moving everything onto the broker collapsed that into one.

A broker also pushes work the moment it arrives instead of waiting for the next poll. For most jobs that latency win is small, but stacked on the two reasons above, the direction was obvious.

The honest tradeoff

No migration is free. If this table makes Hangfire look easy, that's because — for a single service — it is.

	Hangfire	RabbitMQ
Setup	Drop-in NuGet	A broker to deploy & manage
Persistence	SQL table	Queues + dead-letter queues
Dashboard	Built-in	Management plugin
Retry config	Attribute-level	Consumer-level `MaxRetries`
Dev experience	Easy local	Needs Docker / a container
Operational cost	Polling DB load (per job)	Broker RAM + network
Message requeue	By job ID	Queue-level (no per-message)

The case for RabbitMQ isn't "it's better." It's "it gets better with every extra service and every extra job that would otherwise be polling."

The pattern that kept it boring: thin consumers

This is the part that made a multi-job migration safe instead of terrifying.

Each RabbitMQ consumer does as little as possible: receive a message, deserialize it, and delegate to the existing job class. No business logic moves. The code that sends an email or dispatches a webhook stays exactly where it was, still independently testable. The consumer is just a new front door in front of unchanged logic — and I used the identical shape for every job I migrated.

// The consumer is a thin shell. It owns transport, not behavior.
public class SendEmailConsumer : IConsumer<SendEmailMessage>
{
    private readonly SendEmailJob _job; // the SAME class Hangfire used

    public SendEmailConsumer(SendEmailJob job) => _job = job;

    public async Task Consume(ConsumeContext<SendEmailMessage> context)
    {
        // No logic here. Just hand off.
        await _job.Execute(context.Message.EmailId);
    }
}

Two things I got right by doing them first:

Retry parity. The easy thing to get wrong, so I did it before anything else. Hangfire's [AutomaticRetry(Attempts = 0)] became MaxRetries = 0 on the consumer. Get this wrong and you either hammer a failing downstream forever or silently drop messages that should have retried.
No breaking schema change. I left the old HangfireJobId column nullable. Old rows keep their IDs, new rows leave it null, and the data layer never needed a coordinated deploy.

Reliability: a dead-letter queue for every queue

Every job got its own queue and its own DLQ — postysend.sendemail.dlq, postysend.webhookdispatch.dlq, and so on. When a message exhausts its retries, it lands in its DLQ instead of evaporating.

There's a quiet upgrade hiding here too: because jobs now run as consumers, they can publish messages themselves. I moved one job from IBackgroundJobClient to IMessagePublisher, so sending an email now publishes a WebhookDispatch message directly. That kind of service-to-service chaining is awkward in Hangfire and completely natural on a broker.

The thing you actually lose

Here's the part most "we migrated and it's amazing" posts skip.

You lose per-message failed-job inspection. Hangfire's dashboard let me open a single failed job and read exactly what happened. RabbitMQ's management view gives me DLQ counts — how many messages are dead-lettered, not the full story of each one without going and reading the queue directly.

That's a real downgrade. If your debugging workflow leaned on opening one bad job at a time, plan for that gap before you cut over, not after. I now reach for structured logs and DLQ replay instead of a pretty per-job screen, and that was an adjustment.

Should you do this?

Be honest with yourself.

Stay on Hangfire if: you run a single service with a small team, you have no broker, you need the dashboard and per-job inspection, and DB polling load just isn't a problem at your scale. None of those are embarrassing reasons — they're good engineering.

Move to RabbitMQ if: you have multiple services, or enough jobs that polling the database is its own measurable cost; you already run (or can run) a broker; or you want pub/sub and fan-out, which Hangfire fundamentally can't do.

The decision was never about a single job. It was about the system. RabbitMQ wins when you're coordinating many moving parts; Hangfire wins when you're not.

I'm building SavePosty in the open — a faster, smarter home for everything you save. If the internals are your thing, come follow the build. Questions about the migration? Drop them in the comments.