DEV Community: Nemwel Boniface

Best Open-Source OCR and Document Parsing Models for Financial Tables (2026)

Nemwel Boniface — Sat, 17 Oct 2026 14:33:05 +0000

Hello friends! I'm back with a follow-up to my last article on choosing an inference platform for financial documents.

That article was about the where: where your document models run, what they cost, and whether client data ever leaves your environment. This one is about the what: the open-source models that actually sit inside that platform and read the tables. And trust me, when it comes to financial filings, reading tables is a very different job from reading text.

You might think OCR is a solved problem by now. For a paragraph of text? Mostly, yes. For a balance sheet with merged headers, three years of comparatives and negative numbers wrapped in brackets? Not even close. Let me explain.

Disclosure: This article was written in partnership with Superlinked. It compares open-weight OCR and document-parsing options for financial tables and discusses how Superlinked Inference Engine can help teams deploy them.

Introduction: The Bracket That Lied

Picture this. A balance sheet comes back from your OCR pipeline looking spotless. Every cell is filled, every column is aligned, and the person reviewing it gives it a quick glance and moves on.

Except one line used to say (412), and now it says 412.

In accounting, those brackets mean negative. The model saw them, decided they were punctuation, and threw them away. While it was at it, it also read a little superscript footnote marker as one more digit. So a loss became a profit, a number grew a tail, and nothing in the pipeline raised a hand. No exception, no low-confidence warning, no red squiggly line. Just a tidy table that is confidently wrong.

That's what makes financial tables so unforgiving. A single number often depends on things that aren't in the number at all:

The header above it, which might span two or three columns at once.
The column it sits in, where shifting one position means the figure now belongs to the wrong year.
The brackets around it, which decide whether it's money in or money out.
A small line of text above the whole table, like "in millions, except per-share data", which quietly multiplies every number below it and appears in none of the cells.

A model can be brilliant at reading receipts and invoices and still fall over on every one of those. So in this article, I'm ranking open-source OCR models specifically on how much evidence they have for handling tables, and I'll show you how to check them on your own documents so you don't have to take my word (or anyone's leaderboard) for it.

Wait. Do You Even Need OCR?

Look, hear me out on this one before we get to the models, because it's the cheapest win in this whole article.

If you read my last piece, you already know the punchline: most SEC filings don't need OCR at all. 10-Ks, 10-Qs and EX-99.1 earnings releases on EDGAR are HTML, and the main filings carry Inline XBRL since the 2019 to 2021 phase-in. The tagged numbers come straight out of the SEC's XBRL APIs, and the untagged tables come out of a normal HTML parser. Sending those to an OCR model is paying someone to read your handwriting when you already typed the letter.

OCR earns its keep on the documents that only exist as pictures of text: foreign annual reports never filed with the SEC, data-room packs, client uploads and scanned exhibits from before everything went digital.

And here's a step most guides skip: even inside that PDF pile, not every page needs the heavy machinery. Plenty of PDFs are born digital and already carry a text layer. A few lines of Python can sort your pages into buckets before any model sees them:

import re
from pypdf import PdfReader

NUMBER = re.compile(r"\(?\d[\d,]*\.?\d*\)?")

def triage(path: str, min_chars: int = 50, min_numbers: int = 6) -> dict:
    """Sort PDF pages into buckets before any model sees them."""
    buckets = {"needs_ocr": [], "table_candidates": [], "plain_text": []}
    for page_no, page in enumerate(PdfReader(path).pages, start=1):
        text = page.extract_text() or ""
        if len(text.strip()) < min_chars:
            buckets["needs_ocr"].append(page_no)          # no usable text layer: likely a scan
        elif len(NUMBER.findall(text)) >= min_numbers:
            buckets["table_candidates"].append(page_no)   # lots of figures: send to a table model
        else:
            buckets["plain_text"].append(page_no)         # prose: the text layer is enough
    return buckets

print(triage("annual_report.pdf"))
# e.g. {'needs_ocr': [2], 'table_candidates': [1], 'plain_text': [3]}

It's deliberately crude (tune the thresholds on your own documents), but it means the chairman's letter on page 3 never goes anywhere near a GPU, and your table models only see the pages that actually have tables. Keep this triage in mind, because it's going to come back when we talk about running these models in production.

Two Terms You'll See a Lot (and Why They Matter)

Before the ranking, a quick translation of the jargon you'll find on every model card, because the numbers mean very little until you know what they're measuring.

FinTabNet is a benchmark built from tables in real company annual reports. In other words, it's the closest thing we have to a public exam made of the exact kind of tables this article is about. A model with a published FinTabNet score has been tested on balance sheets, not just on academic papers and invoices.

TEDS (Tree-Edit-Distance-based Similarity) is how most table benchmarks grade a model. It turns both the model's table and the correct table into a tree, the same way HTML nests rows inside tables and cells inside rows, and measures how many edits it takes to turn one into the other. A perfect score is 100, or 1.0, depending on who's reporting it. So when you see one model at 96.8 and another at 0.97, they're on the same scale, just written differently.

Here's the catch that I think matters most: TEDS grades the whole table. A model that gets the structure of a 200-cell balance sheet right but drops the brackets from one cell will still score beautifully. A great TEDS score tells you a model understands table structure. It doesn't promise that every sign survived. We'll fix that later with a simple cell-level check.

One more thing: you'll see a lot of models quoting OmniDocBench, a broader document-parsing benchmark. It comes in several versions (the original 2025 release, v1.5 and v1.6), and they're different tests. Comparing a v1.5 score to a v1.6 score is like comparing two people's times on different running routes. Both times are real; they just don't tell you who's faster.

How I Picked and Ordered These Models

My rules were simple, and I'm sharing them so you can argue with the ranking properly in the comments:

Open weights with a licence that allows commercial use, checked on the exact checkpoint. If you can't ship it in a product, it's not on the list.
Finance-table evidence first. A published FinTabNet result beats everything else. Then other table-specific evidence, then whether the model preserves structure and reading order, then licence clarity, then size.
Benchmarks inform the order; they don't decide it. Almost every number on these model cards comes from the people who built the model.
Deployment options don't affect rank. I note them for every model because you'll care, but being easier to host doesn't buy a higher spot.

Alright, let's get into it.

1. Docling: The One You Can Explain to an Auditor

If your tables are going anywhere near an audit, a regulator or a picky finance director, start here.

Docling isn't really a single model. It's a pipeline of specialists: one model (Layout Heron) works out where everything sits on the page, a dedicated table model (TableFormer) rebuilds the rows and columns, and an OCR engine reads the actual characters (RapidOCR PP-OCRv4 in SIE's artifact bundle). Think of it like a small team where everyone has one job, instead of one person trying to do everything at once. When something goes wrong, you can tell which step got it wrong, and that's a luxury you rarely get with end-to-end models.

The evidence is the strongest on this list. The TableFormer paper (arXiv 2203.01017, 2022) reports a FinTabNet TEDS of 96.8 overall, 97.5 on simple tables and 96.0 on complex ones. I'll be honest about one thing: that number comes from the paper, not the current weights, so read it as proof that the approach works on financial tables rather than a guarantee for today's exact checkpoint.

What really earns it the top spot, though, is that Docling rebuilds tables as tables and keeps the reading order intact, so a three-year comparative stays three columns instead of melting into one long list of numbers. As a library, it also records a page number and bounding box for every element. When someone asks "where did this figure come from?", you can point at the exact spot on the exact page. In finance, that answer can matter more than a few extra benchmark points.

It's also refreshingly easy to try:

from docling.document_converter import DocumentConverter

converter = DocumentConverter()
result = converter.convert("foreign_issuer_annual_report.pdf")

print(result.document.export_to_markdown())

for i, table in enumerate(result.document.tables):
    df = table.export_to_dataframe(doc=result.document)
    print(f"Table {i}: {df.shape[0]} rows x {df.shape[1]} columns")

Where it can stumble: on a poor scan, TableFormer can only rebuild structure from whatever text the OCR engine managed to read. A perfect grid of misread digits is still wrong.

The fine print: MIT code. TableFormer weights are CDLA-Permissive-2.0, and Layout Heron and RapidOCR are Apache-2.0, so commercial use is fine. Small enough to run on CPU, served on SIE Cloud as docling, and backed by a project with 68.2k GitHub stars.

2. granite-docling-258M: Tiny Model, Serious Receipts

If you like the Docling approach but want something smaller and simpler, this is your model.

IBM's granite-docling-258M does the whole conversion in one go, turning a page into DocTags, a structured markup that keeps track of what's a heading, what's a paragraph and what's a table. It can also focus on a specific region of a page when you give it a bounding box.

At just 258 million parameters, it has no business carrying the receipts it does. Its model card reports FinTabNet scores of 0.97 for structure and 0.96 including content, at 150 dpi, compared with 0.82 and 0.76 for its predecessor, SmolDocling. That makes it one of only two models on this list with published results on real financial tables, and the other one is number one.

Why it's not first: it's a single model, so there are fewer places to look when a table goes sideways, and that score comes from its own publisher. One benchmark from the people who built it is a great reason to test it. It's not a reason to skip testing. The good news is that at this size, testing is about as cheap as it gets.

The fine print: Apache-2.0. Open weights that you serve yourself, on a modest GPU or even locally.

3. MinerU2.5-Pro: For Notes That Refuse to Fit on One Page

Every annual report has them: the notes to the accounts that start on page 112 and finally end on page 115, with the header row only printed once at the very top. If those are your nightmare, this is your model.

But first, a warning, because this one catches people out. Read the licence on the exact checkpoint:

opendatalab/MinerU2.5-Pro-2604-1.2B is Apache-2.0, so commercial use is fine.
The earlier MinerU2.5-2509-1.2B is AGPL-3.0, which is a very different conversation with your legal team.
The MinerU toolkit code has its own custom licence, separate from both.

Same family name, three different situations. Pin the checkpoint, and you'll sleep fine.

Once that's sorted, the toolkit handles tables that run across pages, which is exactly where multi-year notes break. Remember the segment table from my last article that lost its header row at a page break? This is the first model I'd point at it.

The publisher's numbers are ambitious: 95.69 on OmniDocBench v1.6, up from 92.98 for MinerU 2.5, a 5.54-point jump in table TEDS, and a 1.39-point lead over the second-placed model across five table benchmarks. It also claims to beat GLM-OCR, PaddleOCR-VL-1.5 and larger general models. Two things keep it at number three: those are vendor numbers, and v1.6 is a benchmark version the same publisher proposed.

The fine print: Apache-2.0 for the 2604 checkpoint. 1.2B parameters, served on your own GPUs or through SIE's self-hosted catalog.

4. PaddleOCR-VL-1.5: For Scans That Have Seen Things

Older exhibits are rarely scanned nicely. Pages come in tilted, curved near the spine or photocopied one too many times. If that sounds like your archive, this is the model to try.

PaddleOCR-VL-1.5 is a compact 0.9B model from PaddlePaddle that reads a whole page end to end, tables included. Its card specifically calls out tolerance for scans, skew and warped pages, and like MinerU, it can merge tables across pages and recognise headings that continue onto the next page. It even does seal recognition, which you'll almost never need for filings, but hey, it's there.

Its publisher reports 94.5 on OmniDocBench v1.5 and claims first place overall and on tables. You'll notice a pattern here: nearly every publisher claims first place somewhere. That's not a criticism; it's just why rule 3 exists. That v1.5 number can't be lined up against MinerU's v1.6 number as though they ran the same race.

Why fourth: solid table support, but its biggest strength is coping with ugly scans rather than finance-specific evidence.

The fine print: Apache-2.0. 0.9B parameters, for your own hardware, and listed in SIE's self-hosted catalog.

5. GLM-OCR: Neck and Neck With PaddleOCR

GLM-OCR from Z.ai (0.9B, according to Z.ai) reports 94.62 on OmniDocBench v1.5, first overall at its release. PaddleOCR-VL-1.5 reports 94.5 on the same version and also claimed first at its release.

A 0.12-point difference is well inside the noise of a single benchmark run. If anyone tells you one of these is clearly better than the other based on that gap, smile politely and run both on your own pages.

It sits one spot lower for a practical reason: layout detection is handled by a separate model, PP-DocLayout-V3. That isn't a flaw, but it's one more thing to deploy, version and keep in step. And if you read my Celery post-mortem, you know how I feel about two things that have to stay in sync by hand.

The fine print: MIT weights, plus Apache-2.0 for the layout model, and the card asks you to comply with both. Runs on Linux with NVIDIA GPUs, directly or through SIE's self-hosted catalog. 1.8M downloads on its card, so you're in good company.

6. dots.ocr: Every Number Comes With a Home Address

If your product needs to highlight exactly where on the page a number came from, this one is worth a look.

dots.ocr does layout and reading in a single model (a 1.7B language base, roughly 3.0B parameters in total). Its standout feature is the output: for every element on the page, it returns a bounding box, a category and the text, and tables come back as HTML. That's ready to plug into a review screen or an audit trail without writing a pile of glue code.

Why sixth: its benchmark claims are on the original 2025 OmniDocBench, a different test from the v1.5 and v1.6 numbers above, so there was nothing comparable to rank it against. That says more about the benchmarks than the model.

The fine print: MIT, on both the model card and the repository. The project has moved to dots-studio/dots.ocr, and old links redirect. About 3B parameters on your own hardware.

7. LightOnOCR-2-1B: The Bouncer at the Door

Putting LightOnOCR last on a table list might look harsh. It isn't. It's just not a table model.

LightOnOCR-2-1B is a 1B page-to-text model, and LightOn reports state-of-the-art results on OlmOCR-Bench while being about nine times smaller than comparable models. What its card doesn't have is any table-specific evidence, and it doesn't return coordinates for text (its -bbox variants only locate embedded images).

So don't use it as your table reader. Use it as the bouncer at the door. Remember the triage script from earlier? Pair it with LightOnOCR for a fast, cheap pass over a big archive of scanned pages, find the ones that actually contain tables, and only send those to Docling or MinerU. That's the same instinct as my PaceTracker article: don't do expensive work until something genuinely needs it.

The fine print: Apache-2.0. 1B parameters, on your own hardware or on SIE Cloud at $2 per 1,000 pages (listed 2026-09-30).

Didn't Make the Cut: olmOCR-2

olmOCR-2 was on my shortlist until I read its own benchmark, where Tables scored 48.3, its weakest category. On a list about tables, the model ruled itself out.

So, Why Not Just Pay for a Hosted API?

Fair question. Hosted services exist, they work, and some of them are very good:

Hosted option	Listed price	What you get
Azure AI Document Intelligence (Layout)	$10 / 1,000 pages (US East)	Structured tables, nothing to choose
Amazon Textract (Tables)	$0.015 / page (US West, Oregon)	Native to AWS, zero operations
Mistral OCR 4.1	$4 / 1,000 pages	Strong table handling, per Mistral's own claim
Google Document AI (Layout Parser)	$10 / 1,000 pages	Layout parsing inside GCP

Prices as of 2026-09-30.

When a hosted API is the better call:

You're all-in on one cloud, your documents are allowed to go there, and you want zero operations.
You don't have the time or appetite to compare models, and "good enough, out of the box" genuinely is good enough.
Your volumes are small enough that convenience costs less than an engineer's afternoon.

When open models win:

You want to choose the model that fits your tables, instead of accepting someone else's choice.
A client's data-room pack is contractually not allowed to leave your environment, which in financial work comes up far more often than people expect.
You want to swap models as better ones come out, without re-integrating every time.

Running the Bake-Off Without Losing Your Weekend

Ok, ok, I know seven models is a lot. And here's the uncomfortable truth: the only honest way to choose between them is to run several on your own PDFs. Done the usual way, that means seven different runtimes, seven sets of dependencies and a lot of "works on my machine." That's the kind of side quest that quietly eats a sprint.

This is exactly where Superlinked's Inference Engine (SIE) shines, and honestly, it's the reason the bake-off becomes realistic at all.

SIE is an inference layer built for task-specific models, and it runs the same engine three ways:

SIE Cloud: fully managed and priced per use. Superlinked runs the GPUs.
Self-hosted: the same engine under Apache 2.0 in your own Kubernetes cluster and VPC.
Local: for development and quick experiments.

My favourite part is how boring it makes model comparison, in the best way. The SIE Python SDK calls docling, LightOnOCR-2-1B, PaddleOCR-VL-1.5, MinerU2.5-Pro and GLM-OCR through the same extract method. Five integrations become one loop:

# Illustrative only: the shape of the bake-off, not SIE's exact client API.
# Check the SIE docs for client setup, the extract signature and full model IDs.
CANDIDATES = [
    "docling",            # SIE Cloud
    "LightOnOCR-2-1B",    # SIE Cloud
    "MinerU2.5-Pro",      # self-hosted catalog
    "PaddleOCR-VL-1.5",   # self-hosted catalog
    "GLM-OCR",            # self-hosted catalog
]

for model_id in CANDIDATES:
    output = client.extract(model=model_id, document="hardest_50_pages.pdf")
    save_for_scoring(model_id, output)

The two cheapest candidates don't even need a GPU on your side. On SIE Cloud (as of 2026-09-30):

Model on SIE Cloud	Price	10,000 pages
docling (to markdown)	$0.397 / 1,000 pages	$3.97
docling (OCR profile)	$1.86 / 1,000 pages	$18.60
LightOnOCR-2-1B	$2 / 1,000 pages	$20

Under four dollars to run Docling over ten thousand pages. That's cheaper than the coffee you'll drink while reviewing the results.

MinerU2.5-Pro, PaddleOCR-VL-1.5 and GLM-OCR run from the self-hosted catalog. A small heads-up: several of them need a Linux machine with an NVIDIA GPU, so your MacBook gets to sit this round out.

What I like even more is what happens after you pick a winner. On SIE, OCR, field extraction, embeddings and reranking all share the same client. The table reader you chose lives right next to the extractor that turns cells into schema fields and the reranker that powers search. Nothing gets re-wired. And the day a client says "this data stays in our VPC", you move the same engine with the same model IDs inside their walls. No rewrite, no "the self-hosted version has different models," no drama.

That's the difference between a model you tested and a model you can actually put into production.

Don't Trust Your Eyes: Score the Cells

Remember the balance sheet from the beginning? It passed the eyeball test. That's precisely the problem. And as we saw with TEDS, even a great benchmark score can hide a flipped sign.

So once your bake-off produces results, grade them the boring, reliable way. Take a handful of real tables, write down the correct value, row header, column header and sign for each cell, and count how many each model gets exactly right. Count sign flips separately, because those are the errors nobody spots on screen.

Here's a small scorer to get you going:

import re

DASHES = {"-", "—", "–"}

def normalise(cell: str, scale: int = 1) -> float | None:
    """Turn a financial-table cell into a signed number in base units."""
    text = cell.strip()
    for symbol in ("$", "€", "£", ","):
        text = text.replace(symbol, "")
    text = re.sub(r"[¹²³⁴⁵⁶⁷⁸⁹⁰*†‡]+$", "", text).strip()  # trailing footnote markers
    if text in DASHES:
        return 0.0  # a dash in a statement usually means nil
    negative = text.startswith("(") and text.endswith(")")
    text = text.strip("()").strip()
    try:
        value = float(text)
    except ValueError:
        return None
    return (-value if negative else value) * scale


def score_cells(truth: dict, predicted: dict, scale: int = 1) -> dict:
    """Compare cells keyed by (row header, column header). Exact match or it doesn't count."""
    exact = sign_flips = missing = 0
    for key, true_raw in truth.items():
        expected = normalise(true_raw, scale)
        got = normalise(predicted.get(key, ""), scale)
        if got is None:
            missing += 1
        elif got == expected:
            exact += 1
        elif expected is not None and abs(got) == abs(expected):
            sign_flips += 1
    return {
        "cell_accuracy": round(exact / len(truth), 3),
        "sign_flips": sign_flips,
        "missing": missing,
    }


truth = {
    ("Revenue", "FY2025"): "3,180",
    ("Revenue", "FY2024"): "2,947",
    ("Net income (loss)", "FY2025"): "(412)",
    ("Net income (loss)", "FY2024"): "118¹",
}

predicted = {
    ("Revenue", "FY2025"): "3,180",
    ("Revenue", "FY2024"): "2,947",
    ("Net income (loss)", "FY2025"): "412",   # brackets read as punctuation
    ("Net income (loss)", "FY2024"): "1181",  # superscript read as a digit
}

print(score_cells(truth, predicted, scale=1_000_000))
# {'cell_accuracy': 0.5, 'sign_flips': 1, 'missing': 0}

Half the cells are wrong, one of them has the opposite sign, and on screen the table looks absolutely fine. Fifty of your ugliest pages scored like this will tell you more than any leaderboard on the internet.

Summary and a Useful Cheat Sheet

If your problem is...	Try this first	Then compare with
Tables you need to defend to an auditor	Docling	granite-docling-258M
You want the smallest model with finance-table evidence	granite-docling-258M	Docling
Notes and tables that span several pages	MinerU2.5-Pro	PaddleOCR-VL-1.5
Skewed, warped or old scans	PaddleOCR-VL-1.5	GLM-OCR
You need coordinates and table HTML for a review UI	dots.ocr	Docling
A huge archive you need to sort cheaply first	LightOnOCR-2-1B	Then route tables to Docling or MinerU
Running all of the above without seven pipelines	SIE	—

What This List Doesn't Cover

In the spirit of honesty, a few things this article deliberately leaves out:

Turning tables into structured fields. Reading a table correctly is step one. Mapping "Net income (loss), FY2025" into your schema, with the right period and scale, is a separate job for extraction models, and it deserves its own article.
Independent benchmarks. Almost every number here comes from the model's publisher. They're useful for building a shortlist, not for crowning a winner.
Your documents. No ranking, including mine, knows what your ugliest PDF looks like. Only a bake-off does.

Conclusion

There's a Swahili proverb that sums up this whole article: Mtaka cha mvunguni sharti ainame. Whoever wants what's under the bed has to bend down.

There's no shortcut to the right OCR model for financial tables. Leaderboards will get you a shortlist, but you still have to bend down, pull out your ugliest PDFs and see who actually reads the brackets correctly.

My shortlist for you: parse XBRL and HTML first, and triage your PDF pages before anything touches a GPU. For the tables that are left, test Docling and granite-docling-258M first, bring in MinerU2.5-Pro or PaddleOCR-VL-1.5 for tables that span pages or scans that have seen better days, and use LightOnOCR-2-1B to sort a big archive cheaply.

And to make that bending down a lot less painful, run the whole bake-off on SIE. Docling and LightOnOCR are a per-page API call on SIE Cloud, the rest are in the self-hosted catalog behind the same extract method, and the winner can go straight into production, on SIE Cloud or inside your own VPC, without anyone rewriting a thing.

Final Thoughts & Conversation

The most surprising thing about this space right now is how good the open options have become. There are commercially licensed models with real table evidence behind them, some small enough to run on a laptop. The hard part has shifted from finding a capable model to proving which one works on your documents, and that's a much better problem to have.

What's the worst financial table you've ever had to extract? A note that ran for four pages, a scale line that quietly multiplied everything by a million, or brackets that vanished into thin air? Which model finally got it right? I'd love to hear your horror stories in the comments!

A Quick Side Note:

I regularly write about backend performance, AI infrastructure, document pipelines and real-world engineering trade-offs. If you enjoy deep dives into systems that fail loudly instead of quietly getting things wrong, I'd love to connect and chat!

You can find me on LinkedIn, follow my latest technical thoughts on Twitter/X, or check out my open-source projects and previous articles on GitHub.

I hope this was useful information for you. See you in my next article!

Best AI Inference Platforms for Financial Document Processing in 2026: Why Superlinked SIE Tops My List

Nemwel Boniface — Thu, 15 Oct 2026 13:51:07 +0000

A production incident usually gets my attention faster than an invoice. But in financial document processing, an invoice can be a warning of its own: you're paying a model to do work that a simple parser could have handled.

There's a Swahili proverb I keep coming back to whenever I see a team rush into a technical decision: Haraka haraka haina baraka, hurry hurry has no blessing. In this case, the expensive mistake is picking a document-AI service because it says "PDF" on the landing page, sending every filing through it, and checking the bill only when finance comes knocking.

Let's look at how to choose an inference platform for financial documents without paying to process things that were never hard to read in the first place.

Introduction: Turning Clean Markup Into Pixels (and Back Again)

Picture a team building a pipeline to read US annual reports. They sign up for a document-AI service, pay per-page rates, and feed it every 10-K they can find.

Here's the punchline: those 10-Ks were born-digital HTML. Every figure on the face of the financial statements was already tagged in Inline XBRL. The service dutifully took perfectly clean, machine-readable markup, rendered it into pixels, and then used OCR to turn those pixels back into text. That's like printing an email so you can scan it and email the scan.

Meanwhile, the one source that genuinely needed document AI, a foreign issuer's annual report published only as a PDF, was the part that broke. A multi-page segment table split across pages and lost its header row, so every number below the page break was now an orphan with no column to belong to.

Two problems, one root cause: the team picked a tool before they understood which of their documents were actually hard. So before we rank anything, let's fix that.

Before You Pick a Platform: Do You Even Need Document AI?

This is the part most "best platform" listicles skip, and it's the part that saves you the most money.

EDGAR primary documents are HTML, and have been Inline XBRL HTML since the 2019 to 2021 phase-in. EX-99.1 earnings releases are HTML too. For tagged statement figures, you don't need OCR at all. You need an HTTP client:


import requests

# The SEC asks for a descriptive User-Agent with contact details.
HEADERS = {"User-Agent": "Your Name your.email@example.com"}

cik = "0000320193"  # Apple, zero-padded to 10 digits
url = f"https://data.sec.gov/api/xbrl/companyfacts/CIK{cik}.json"

response = requests.get(url, headers=HEADERS, timeout=30)
response.raise_for_status()
facts = response.json()

revenue = facts["facts"]["us-gaap"][
    "RevenueFromContractWithCustomerExcludingAssessedTax"
]["units"]["USD"]

annual = [
    fact
    for fact in revenue
    if fact.get("form") == "10-K"
    and fact.get("file")
    and fact.get("end")
]

# Sort by filing date instead of assuming the last item is latest.
annual.sort(key=lambda fact: fact.get("file", ""))

latest = annual[-1]
print(latest["end"], latest["val"])
# Period end date and value in base units (dollars, not millions).

No pixels, no per-page fee, and the values come back in base units, so nobody has to guess whether "394,328" means thousands or millions.

There is a catch, and it matters. The companyfacts and frames APIs only return entity-wide facts on standard taxonomies. Company extension tags, segment facts and member facts live in the filing's own iXBRL, which is still HTML you can parse. Same story for untagged content like MD&A tables and press-release exhibits: parse the HTML tables.

So where does a document-AI platform actually earn its fee? Two places:

PDF-only sources. Non-US annual reports, data-room documents, client uploads, and paper-era scanned exhibits.
The chain after the text exists. Pulling fields with their period and scale, embedding passages for search, and reranking whatever retrieval returns.

Notice that the second one has nothing to do with OCR. That's the part people forget, and it's exactly where a single-API platform starts to pull ahead.

The Scorecard: Four Questions That Actually Matter

I ranked every platform against one question, broken into four parts:

Coverage: Does it give me PDF OCR, field extraction, embeddings and reranking behind one API?
Price transparency: Does each step have a published per-unit price I can put in a spreadsheet before I run anything?
Model choice: Can I pick (and swap) the model at each step, or am I stuck with whatever the vendor chose?
Data residency: Is there a path to run the exact same thing inside my own VPC for client documents that must never leave?

One distinction before the ranking: these platforms don't all start from the same place. Some are document-processing APIs, some are ingestion platforms, and some are infrastructure for serving models you package yourself. I'm comparing them against the same practical requirements, but that doesn't make their outputs or operating models interchangeable. The right choice still depends on which parts of the pipeline you want the platform to own.

Financial data makes question 4 non-negotiable more often than you'd think. Client notes, portfolio data and data-room documents frequently come with contractual "this stays in your environment" clauses, and "we'll figure out self-hosting later" is how a three-week project becomes a six-month security review.

1. Superlinked Inference Engine (SIE): The Whole Chain, One Client

SIE is an inference engine built for task-specific models, and the first thing worth understanding is that it runs three ways with the same engine underneath:

SIE Cloud: the managed service. Superlinked runs the GPUs, you call an API, you pay for what you use.
Self-hosted: the same engine under Apache 2.0, deployed into your own Kubernetes cluster in your own VPC, with Terraform modules for AWS, GCP and Azure. Here, you operate the cluster.
Local: the same models on a single NVIDIA or Apple Silicon machine with one command. Great for prototyping on your laptop before anyone approves a GPU budget.

That "same engine, same model IDs, three deployment modes" property is the quiet superpower here. The pipeline you prototype locally is the pipeline you cost on SIE Cloud, which is the pipeline you deploy into your VPC when a client contract demands it. No rewrite, no "the self-hosted edition has different models," no surprises.

Coverage of the Chain

The SIE Cloud task menu lists OCR, Doc to Markdown, Doc field extraction, Structured output, Named entities, Search and Rerank, each with per-model prices. For a filings pipeline, that's every step after "the PDF arrived" on one client.

Here are a few of the listed prices (as of 2026-09-30):

Task	Model	Price
Doc to Markdown	docling	$0.397 per 1,000 pages
OCR	LightOnOCR-2-1B	$2 per 1,000 pages
Field extraction	Qwen3.8-27B (FP8)	$0.72 per million tokens (in and out)
Rerank	Qwen3-Reranker-4B	$61.89 per million pairs
Rerank (smaller)	Qwen3-Reranker-0.6B	$6.63 per million pairs

That reranker price looks scary until you do the math. A top-50 rerank is 50 query-document pairs, so:

50 pairs x ($61.89 / 1,000,000) ≈ $0.003 per query   (4B reranker)
50 pairs x ($6.63  / 1,000,000) ≈ $0.0003 per query  (0.6B reranker)

Less than a third of a cent per query for the big one. I've seen teams spend more than that on the logging for the same request.

Swapping Models Is a String Change

This is the part I genuinely love. Because every step lives on the same client, changing your OCR or extraction model is a model ID change, not a migration. That makes the single most valuable experiment in document processing, "which model actually handles our ugly PDFs best?", cheap to run.

Conceptually, your pipeline config ends up looking like this:

# Illustrative only: the shape of the pipeline, not SIE's exact client API.
# Check the SIE docs for the real client calls and full model IDs.
PIPELINE = {
    "doc_to_markdown": "docling",
    "ocr": "LightOnOCR-2-1B",          # try PaddleOCR-VL-1.5, MinerU2.5-Pro or GLM-OCR self-hosted
    "extract": "Qwen3.8-27B-FP8",
    "rerank": "Qwen3-Reranker-4B",      # drop to the 0.6B if quality holds on your eval set
}

Remember that foreign issuer's report with the segment table that lost its header row? This is how you fix it: run the same 50 nasty pages through three OCR models by changing one string, compare the output, and keep the winner. SIE's model catalog adds self-hosted OCR options like PaddleOCR-VL-1.5, MinerU2.5-Pro and GLM-OCR, so you're not limited to one vendor's opinion of what a table looks like.

Costing It Before You Run It

Published per-model prices let you estimate parts of the monthly bill before a single page hits a GPU. This small example calculates Doc-to-Markdown parsing and reranking only; it does not include OCR, field extraction, embeddings, infrastructure, or other operational costs. Treat it as a starting point for a cost model, not the full pipeline bill.

# Partial estimate only: parsing + reranking.
# Add OCR, extraction, embeddings, infrastructure, and other costs separately.
PRICES = {
    "docling_per_1k_pages": 0.397,
    "rerank_4b_per_1m_pairs": 61.89,
    "rerank_06b_per_1m_pairs": 6.63,
}

def monthly_cost(pdf_pages: int, queries: int, top_k: int = 50, reranker: str = "rerank_4b_per_1m_pairs") -> dict:
    parsing = pdf_pages / 1_000 * PRICES["docling_per_1k_pages"]
    reranking = queries * top_k / 1_000_000 * PRICES[reranker]
    return {"parsing": round(parsing, 2), "reranking": round(reranking, 2), "total": round(parsing + reranking, 2)}

print(monthly_cost(20_000, 100_000))
# {'parsing': 7.94, 'reranking': 309.45, 'total': 317.39}

print(monthly_cost(20_000, 100_000, reranker="rerank_06b_per_1m_pairs"))
# {'parsing': 7.94, 'reranking': 33.15, 'total': 41.09}

(Field-extraction tokens depend heavily on your schema and page density, so measure those on a sample and add them in.)

That's the kind of number you can put in front of a manager on day one. Try doing the same thing with a vendor whose pricing page says "Contact Sales."

Compliance

For the managed service, Superlinked lists SOC 2 Type 2 along with US or EU data residency. For anything that can't leave your environment at all, you self-host the same engine in your VPC, with the same model IDs. The code is in the open-source SIE repository, so your security team can actually read what they're approving.

My honest take: SIE is the only platform on this list that ticks all four boxes. OCR, extraction, embeddings and reranking on one API. A published price for every step. Open weights you choose and swap by model ID. And the same engine on SIE Cloud or in your own cluster under Apache 2.0. For a financial document pipeline, that's the production path.

A Quick Reality Check on Per-Page Pricing

Here's what 10,000 PDF pages costs at listed rates across the platforms in this article. Important caveat: these tiers don't all produce identical output (plain OCR text versus layout-aware structure versus form fields), so treat this as a sense of scale, not a benchmark. Benchmark on your own PDFs.

Platform / Tier	Listed Rate	10,000 Pages
SIE: docling (Doc to Markdown)	$0.397 / 1k pages	$3.97
SIE: LightOnOCR-2-1B (OCR)	$2 / 1k pages	$20
Azure Read	$1.50 / 1k pages	$15
Azure Layout	$10 / 1k pages	$100
Google Enterprise Document OCR	$1.50 / 1k pages	$15
Google Layout Parser	$10 / 1k pages	$100
Textract Detect Document Text	$0.0015 / page	$15
Textract Tables	$0.015 / page	$150
Textract Forms	$0.05 / page	$500
Reducto Parse	$10 / 1k pages	$100
Unstructured (pay-as-you-go)	$0.015 / page	$150

Prices as listed on 2026-09-30; Azure figures are US East and Textract figures are US West (Oregon), and other regions differ.

The price difference is worth investigating, but don't treat these rows as interchangeable. Doc-to-Markdown conversion, OCR, and layout-aware parsing can produce different outputs, and the right choice depends on what the downstream pipeline needs. If Markdown is sufficient for your documents, SIE's listed docling rate is an attractive starting point. If you need reliable table structure or form fields, benchmark that output directly against your requirements before comparing the price tags.

The Rest of the Field

To be fair to everyone else, these are all real, capable products. They just answer my four questions differently.

2. Azure AI Document Intelligence

Formerly Form Recognizer, now listed as Azure Document Intelligence in Foundry Tools. Its standout is the Layout model: tables come back with structure and there's no model to choose. Containers are available, and Azure's pricing page notes container pricing matches the cloud service. Embeddings (Azure OpenAI) and a semantic ranker (Azure AI Search) live in the same tenant and the same bill.

The trade-off: the chain is assembled from several Azure services, each with Azure's models. If you're an all-Azure shop, that's comfortable. If you want to pick the model at each step, it's limiting.

3. Google Cloud Document AI

A set of hosted processors on GCP: Enterprise Document OCR, Form Parser, Layout Parser, Custom Extractor, plus a classifier and splitter. The Layout Parser returns chunks along with layout, so part of your retrieval prep arrives pre-done. Form Parser and Custom Extractor list at $30 per 1,000 pages up to one million pages.

The trade-off: embedding and ranking are separate Vertex services, and processing and billing are on GCP. Model your tier mix carefully, because sending every page through a custom extractor costs many times what OCR does.

4. Amazon Textract (with Bedrock)

Inside an all-AWS estate, Textract is zero-ops and the security review is short because everything shares the same account and IAM. Pair it with Bedrock Data Automation for structured output from a "blueprint" at $0.040 per page for 30 fields or fewer.

The trade-off: there's no self-hosted edition, and watch the Forms tier: at $0.05 per page it's thirty-three times the plain text rate. Flip it on for every page of a 300-page annual report and you'll feel it.

5. Reducto

One API for Parse, Extract, Split, Classify and Edit, plus Studio, a visual interface a non-engineer can use to inspect results. It's strong on complex tables with little setup.

The trade-off: on-prem and VPC deployment are Enterprise-only, you configure processing options rather than the underlying models, and embedding and reranking happen elsewhere.

6. Unstructured

An ingestion and ETL platform with an Apache 2.0 open-source library. It states 40+ connectors and 45+ file types, which makes it genuinely useful upstream of an inference layer if you're pulling from shared drives, data rooms and object stores.

The trade-off: you choose from the hosted models it integrates rather than arbitrary open models, reranking sits elsewhere, and in-VPC deployment is on the Business plan.

7. Baseten

Managed GPU inference for models you package yourself, billed per GPU minute (an H100 is $0.10833 per minute, about $6.50 an hour) or per token on Model APIs.

The trade-off: it's not priced per page and it doesn't ship document models; you package and operate them. It's the right call when you're bringing your own model and want to run it yourself.

(Runtimes like vLLM, SGLang and TEI often sit underneath platforms like these. They're engines, not rivals, so I haven't ranked them.)

Summary and a Useful Cheat Sheet

Platform	One API for OCR → Extract → Embed → Rerank?	Published Per-Unit Prices?	Choose Your Models?	Self-Host in Your VPC?
Superlinked SIE	Yes	Yes, per model	Yes, open weights by model ID	Yes, Apache 2.0
Azure Document Intelligence	Across several Azure services	Yes	Azure's models	Containers
Google Document AI	OCR/layout yes; embed and rank on Vertex	Yes	Google's processors	No (runs on GCP)
Textract + Bedrock	Across two AWS services	Yes	AWS's models	No
Reducto	Parse and extract only	Standard plan yes	Processing options only	Enterprise only
Unstructured	Up to embedding; no rerank	Yes	Integrated hosted models	Business plan
Baseten	Bring your own models	Per GPU minute / token	Yes, you package them	Enterprise

What I'd Tell My Past Self

A few honest lessons, in the spirit of not pretending anyone gets this right on the first draft:

Sort your documents before you sort your vendors. HTML and XBRL first, PDFs second. The cheapest page to process is the one you never send to a model.
Price the whole chain, not just the OCR. Extraction, embedding and reranking are recurring costs that scale with queries, not pages. A platform that looks cheap at step one can get expensive at step four.
Build a nasty-PDF eval set early. Fifty of your ugliest pages, multi-page segment tables included. If swapping a model is a string change, there's no excuse not to test three of them.
Decide your data-residency story on day one. It's much easier to choose a platform that self-hosts the same models than to retrofit "this can't leave our VPC" six months in.

Frequently Asked Questions

Do I need document AI for 10-K and 10-Q filings?

For the tagged statements, no. The SEC's companyfacts and frames APIs return entity-wide facts on standard taxonomies in base units. Company extension tags and segment or member facts need the filing's own iXBRL, which is HTML you can parse. Document AI earns its cost on PDF-only sources and on the extraction and retrieval steps after that.

Why not just keep everything in Azure?

You can, and Azure assembles the chain from Document Intelligence, Azure OpenAI and AI Search. SIE puts OCR, extraction, embedding and reranking on one API with open weights you choose, on SIE Cloud or in your own VPC. One client plus a catalog of open models is why I'd run a filings pipeline on SIE.

What about IFRS filers and data that can't leave our environment?

Foreign private issuers filing 20-F or 40-F appear in the XBRL APIs under the ifrs-full taxonomy, but their home-market annual reports are often PDF-only, which is exactly where OCR earns its keep. For client notes or portfolio data that must stay put, run the same SIE models self-hosted in your VPC or locally; the deployment documentation covers the Terraform and Helm paths.

Conclusion

The expensive mistake in financial document processing is rarely the model. It's sending the wrong documents to it, paying per page to read HTML that was never hard to read, and then discovering that the one PDF that actually mattered broke on a table nobody tested.

SIE fixes the parts that a platform can fix. It covers OCR, field extraction, embedding and reranking behind one API, on SIE Cloud or self-hosted on your own GPUs under Apache 2.0. The open weights are the document models you actually put into production, and you swap them by model ID. And because every model has a published price, you can cost the whole chain against your own page and query counts before you run a single thing.

Haraka haraka haina baraka. Slow down long enough to look at your documents, and the right platform picks itself.

Final Thoughts & Conversation

Financial documents are a perfect example of a problem where the tooling question is easy and the thinking question is hard. Which of your documents are actually difficult? Which steps scale with pages, and which scale with queries? Which data can never leave your cluster? Answer those first, and the platform choice becomes almost boring, which is exactly how infrastructure decisions should feel.

Have you built a pipeline over 10-Ks, IFRS reports or messy client PDFs? Did you get burned by a per-page bill or a table that split across pages and took its headers with it? I'd love to hear your horror stories and your favorite OCR models in the comments below!

A Quick Side Note:

I regularly write about backend performance, AI infrastructure, system reliability, and practical engineering trade-offs from the real world. If you enjoy deep dives into document pipelines, defensive engineering, or building things that don't quietly burn through a budget, I'd love to connect and chat!

You can find me on LinkedIn, follow my latest technical thoughts on Twitter/X, or check out my open-source projects and previous articles on GitHub.

Thanks for reading, and see you in the next one!

Silent Data Loss in Celery + RabbitMQ on Azure Container Apps: A Queue Registration Post-Mortem

Nemwel Boniface — Sun, 02 Aug 2026 10:53:00 +0000

Hello friends! I'm back with another real-world debugging story from the trenches.

If there's one goal every backend engineer should have,, it's building systems that fail loudly instead of quietly dropping data while returning 202 Accepted. Today, we're diving into a production incident that took days to track down-not because the code was wrong, but because cloud infrastructure and container entry points had a silent disagreement.

Grab your coffee, and let's go hunting for ghost queues.

Introduction

One of the first lessons every engineer learns moving from a monolith to microservices is that consistency stops being free. In a monolith, a write either commits inside your transaction or it doesn't; there's one database, one process, one truth. Split that same write across services and you trade that guarantee for eventual consistency: the write will happen, just not necessarily right now, and not necessarily in the same process that accepted the request.

Message queues are the workhorse of that tradeoff. A typical pattern looks like this:

Client → API (producer) → Message Broker → Worker (consumer) → Database

The API accepts the request, publishes a message, and immediately returns 202 Accepted. Somewhere downstream, milliseconds later, usually, a worker process picks up that message and actually does the work. This is a genuinely good pattern: it decouples slow work from request latency, it lets you scale producers and consumers independently, and it gives you a natural retry boundary if something downstream is briefly unavailable.

It also has a failure mode that almost never shows up in tutorials: what happens when the consumer isn't actually listening, and nothing tells you?

This is the story of a bug that took days to run down, precisely because every individual piece of evidence pointed somewhere reasonable and wrong. The application code was correct. The message broker was healthy. The container was running, healthy, and logging normally. And yet, for a specific subset of operations, every single write vanished - accepted, acknowledged, never persisted, and never once producing an error anywhere in the stack.

The root cause, when we finally found it, wasn't in our code at all. It was a single stale field on a cloud platform resource that nobody had thought to check.

The Architecture, Briefly

The system in question is a fairly standard FastAPI + Celery + RabbitMQ setup, deployed as three services on Azure Container Apps:

A Frontend/API service that accepts HTTP requests and enqueues Celery tasks for anything that writes data.
A Worker service running Celery, consuming from RabbitMQ and writing to Postgres.
A Data service behind the worker, doing the actual persistence.

Celery's queueing model is simple on paper. Each task gets a name and, usually, its own dedicated queue:

# worker/tasks/order_tasks.py
from celery import shared_task
from worker.main import app

@app.task(
    name="orders.tasks.register_order",
    bind=True,
    queue="orders.tasks.register_order",
)
def register_order_task(self, command):
    order_service = self.resolve(IOrderService)
    return order_service.register_order(command)

Declaring @app.task(...) registers the task in Celery's Python-level task registry the moment the module is imported. That's step one. Step two, the one that actually matters for whether messages get processed, is telling the worker process which queues to consume from, at startup:

# docker/worker.dockerfile
ENTRYPOINT ["celery", "-A", "worker.main", "worker", \
            "--loglevel", "info", \
            "-Q", "orders.tasks.register_order,orders.tasks.edit_order,orders.tasks.import_order_row"]

That -Q flag is the entire contract. If a queue name isn't in that comma-separated list, the worker never opens a consumer for it, even if the task is fully defined, fully imported, and fully capable of running. Celery and RabbitMQ don't complain about this. Nothing complains about this. A producer publishing to an unconsumed queue succeeds silently, because RabbitMQ (correctly, by design) auto-declares the destination queue on first publish regardless of whether anyone is listening.

That single asymmetry, task registration and queue subscription being two independent things that happen to usually move together, is where this entire incident lived.

The Symptom

The bug looked exactly like a straightforward, if odd, application defect:

POST /consumables
{"name": "Test Item", "code": "TEST001"}

→ 202 Accepted
{"task_id": "8f3a...", "data": {"id": "e8942ca9-...", "name": "Test Item", "code": "TEST001", ...}}

A plausible-looking response, complete with a freshly generated ID. Everything about it looked like success.

GET /consumables/e8942ca9-...
→ 404 Not Found

The record simply never existed. Not eventually. Not after a retry. Not after ten seconds, or ten minutes. This affected exactly four entity types out of seventeen in the system, consistently, at 100% loss, every single write, every time, forever.

No exceptions. No 5xx responses anywhere in the chain. No entries in a dead-letter queue (there wasn't one). No metric crossed a threshold, because there was no metric measuring "messages published to a queue with zero consumers." From the outside, four specific operations across the whole API had quietly stopped doing anything, and the system was structurally incapable of telling anyone.

Chasing Ghosts: What We Ruled Out

Given the symptom - accept, then silently vanish- the natural first suspects are the ones every distributed-systems postmortem reaches for:

A duplicate-write race condition. Ruled out: this happened on the very first, uniquely-coded write, no concurrency involved.
RabbitMQ policies applying unexpected arguments to specific queues (this broker was shared across several unrelated services, so a stray wildcard policy from someone else's team was a real candidate). Checked directly in the RabbitMQ management UI - zero policies configured, user or operator.
Per-vhost or per-user resource limits silently rejecting new queue declarations once some threshold was hit. Checked - none configured.
Stale queue/exchange metadata - the theory that these specific queues had been declared at some point with incompatible arguments (different durability, different type) and were now permanently rejecting redeclaration. This one felt promising, until we tried the most direct possible test: restarting the entire RabbitMQ broker. A full broker restart wipes any transient inconsistency and forces every queue to be redeclared from scratch by whoever connects next. The bug survived it, completely unchanged. Theory dead.
The application code itself. This is the one that eventually broke the case open, precisely because it kept coming back clean. We took the exact -Q queue list from the deployed Dockerfile, spun up a disposable RabbitMQ container locally, and started the worker against it with the identical code:

docker run -d --name repro-rabbitmq -p 5680:5672 rabbitmq:3.13-management-alpine

celery -A worker.main worker --loglevel=info -Q "$(cat queue_list.txt)" &

grep "exchange=" worker_boot.log | wc -l
# → all 67 expected queues bound cleanly, zero errors

Every single queue, including the four that were failing 100% of the time in production, bound perfectly against a clean, local broker running the exact same code. Twice we ran this experiment, from two different angles, and twice the code came back innocent.

At this point we had eliminated the broker, the policies, the limits, and the code. That doesn't leave many places left to look, which was itself the clue we were missing.

The Breakthrough: Ask the Container What It's Actually Running

Every diagnostic tool we'd used up to that point application logs, celery inspect registered, the RabbitMQ management UI, even the Dockerfile itself- describes what should be true. None of them describe what the running process was actually invoked with. There's a difference, and on a container orchestration platform, that difference is exactly where a bug can hide indefinitely.

Azure Container Apps supports exec'ing directly into a running replica:

az containerapp exec \
  -n worker-service \
  -g my-resource-group \
  --command "/bin/bash"

And once inside, there is exactly one place that tells you the unfiltered truth about how PID 1 was started:

cat /proc/1/cmdline | tr '\0' '\n'

The output was not what the Dockerfile said it should be.

# What the Dockerfile's ENTRYPOINT specifies:
celery -A worker.main worker --loglevel info --autoscale 5,1 -Q orders.tasks.register_order,orders.tasks.edit_order,...,orders.tasks.import_order_row  [67 queues total]

# What /proc/1/cmdline actually showed:
celery --app=worker.main worker --loglevel=info --autoscale=10,1 -Q orders.tasks.register_order,orders.tasks.edit_order,...  [39 queues total]

Different flag syntax (--app= vs. -A), a different autoscale value, and a drastically shorter -Q list, missing every import-row queue and every queue for the four failing entities entirely. This wasn't a formatting quirk. It was a completely different command than the one baked into the currently-deployed image.

Root Cause: The Silent Platform-Level Override

Azure Container Apps lets you set an explicit command and args override directly on the Container App resource, independent of whatever ENTRYPOINT/CMD is baked into the image. If that override is set, it wins unconditionally, regardless of what the image itself specifies.

At some point, months earlier, someone had set this override, most likely when the app was first provisioned, before several of the entities (and their import queues) in this codebase existed. Since then, every deploy had used a CI/CD step that looked like this:

az containerapp update \
  --name worker-service \
  --resource-group my-resource-group \
  --image "myregistry.azurecr.io/worker-service:${GIT_SHA}"

Only --image. Never --command, never --args. Every single deploy since the override was first set had faithfully shipped new code - and then silently ignored it, because the platform kept running the old, frozen command instead of the new image's own entrypoint. The Dockerfile was a complete, accurate, up-to-date source of truth for a command that had never actually run in production.

Confirming and fixing this took two steps. First, read the override back out in full, since a value this long is too easy to corrupt by hand-editing a form field in a portal UI:

az containerapp show \
  -n worker-service -g my-resource-group \
  -o yaml > worker-app.yaml
# edit the containers[0].args block directly
az containerapp update \
  -n worker-service -g my-resource-group \
  --yaml worker-app.yaml

Then - and this is the part worth remembering- a redeploy alone does not fix a bug like this. New revisions of a Container App still boot the same overridden command unless the override itself is changed. Every restart, every scale event, every new revision from that point forward had been re-running the identical stale command. The fix isn't "redeploy"; it's "stop overriding the image's own entrypoint," or at minimum, keep the override in lockstep with the Dockerfile by hand, which is itself a second, ongoing source of risk.

Data Loss and the Eventual Consistency Trap

Here's the part that matters beyond this one platform quirk: this failure mode is a direct, sharp-edged consequence of how "eventual" consistency actually works.

Celery's default publisher behavior auto-declares the destination queue when a message is sent, whether or not any consumer exists (task_create_missing_queues, on by default). That's normally a convenience; you don't have to pre-provision every queue by hand. But it also means a producer with a perfectly correct queue name will happily publish into the void forever. RabbitMQ doesn't reject the message. It doesn't expire it by default. It just sits there, Ready, unconsumed, for as long as nothing is listening, which in this case turned out to be months.

# RabbitMQ management UI, Queues tab
Name                                   Ready  Unacked  State
orders.tasks.register_order            5      0        running

Five messages, sitting there, 0.00/s deliver rate, indefinitely. That number is the whole bug, visualized: five real user actions, accepted by the API, and then never acted on again.

"Eventual consistency" is a promise that a write will eventually be reflected - it is not a promise about how long eventually is allowed to be, and it says nothing at all about what happens if the thing that's supposed to make it happen structurally never shows up. When the consumer side of an async pipeline is silently missing, "eventually" quietly becomes "never," and the system has no vocabulary for telling you the difference. A 202 Accepted and a permanently-lost write look identical from the caller's side, because in both cases the only promise actually made was "we'll get to it", never "we did it."

A related discovery made the same point from a different angle. While chasing this, an unrelated-looking bug turned up: certain list endpoints intermittently returned an empty 204 No Content even though the records genuinely existed. The real cause was a data-mapping layer raising a validation error on a NULL field, which the internal HTTP client silently converted into a generic "not successful" response with no status code preserved, so what was actually a 500 on one internal hop surfaced to the end user as an innocent-looking "no data here." Different bug, same underlying lesson: any layer that swallows the specific reason for a failure and reports a generic, plausible-looking result is doing your future self a disservice. Silent degradation is worse than a loud failure, because a loud failure gets fixed.

What We'd Do Differently

A few concrete takeaways, in rough order of how much they'd have saved us:

Verify the running process, not just the source of truth. /proc/1/cmdline (or your platform's equivalent, docker inspect, kubectl get pod -o yaml, and check the actual container spec, etc.) tells you what's actually executing. A Dockerfile, a Helm chart, an IaC template - these are all descriptions of intent. Only the running process tells you the truth, and on any platform that supports a command/args override independent of the image, that intent and that truth can diverge invisibly.
Avoid dual sources of truth for a container's startup command. If the Dockerfile's ENTRYPOINT fully specifies how to start the process, don't also maintain a platform-level override of the same thing. Pick one. If a platform-level override is unavoidable for some reason, treat any deploy that doesn't also touch it as suspect.
Instrument the gap, not just the endpoints. An unconsumed-message-count alert on your broker (Ready > 0 and deliver/get ≈ 0 for more than N minutes, per queue) would have caught this in minutes instead of days. Application-level health checks and 200-OK dashboards are blind to this entire failure class by construction, the API was never unhealthy; the worker was never crashing. The problem lived entirely in the space between two healthy-looking components.
Write a queue-parity check. A five-line script that statically extracts every queue= from your task decorators and diffs it against whatever's actually passed to -Q at deploy time would have caught this before it ever reached production, and would catch the next instance of "we added a new entity and forgot to wire its queue" too, which, notably, is exactly what happened a second time later in this same incident, for twelve other queues nobody had thought to check.
Don't trust "it's already correct" claims made against source files alone. More than once in this incident, a piece of code being correctly committed was mistaken for that code actually running in production. Those are different facts, and conflating them is exactly the trap that let this bug hide as long as it did.

Why This Bug Was So Difficult to Find

Looking back, what made this incident particularly difficult wasn't that any individual component was broken; it was that every component appeared to be working correctly when viewed in isolation.

At different stages of the investigation, each layer gave us evidence that suggested the system was healthy:

The API accepted requests and returned 202 Accepted as expected.
Producers successfully published messages to RabbitMQ.
RabbitMQ accepted and stored every message without errors.
Celery itself was healthy and running normally.
Azure Container Apps reported healthy containers and successful deployments.
Application logs showed no exceptions or crashes.
Our deployment pipeline completed successfully, giving us confidence that the latest code was running.

None of those observations were incorrect. In fact, every single one was true.

The problem was that the worker process was consuming only a subset of the queues because of a stale Azure Container Apps command override. Every component behaved exactly as it had been configured to behave—the configuration itself was the problem.

This is what made the incident so deceptive. We naturally tend to investigate individual systems in isolation:

Is RabbitMQ healthy?

Is Celery running?

Did the deployment succeed?

Is the application code correct?

The answer to every one of those questions was yes.

The real failure existed between those systems, in the assumptions that connected them.

That's one of the hardest lessons I've learned about distributed systems: many production incidents aren't caused by a single failing component; they emerge from the interaction between components that are each behaving correctly according to their own configuration.

When that happens, traditional health checks, deployment success indicators, and application logs can all tell a perfectly consistent, but incomplete story. Solving the problem requires shifting your focus from individual services to the contracts and assumptions that connect them.

Conclusion

None of the individual technologies here were at fault. Celery did exactly what it was told. RabbitMQ did exactly what it was told. The application code, when we finally proved it beyond doubt, was correct the entire time. The failure lived in the seam between the deployment platform and the container image, a place none of our existing tooling was looking, because none of our existing tooling had a reason to distrust it.

That's the real lesson of eventual consistency in a microservice architecture: decoupling a producer from a consumer buys you resilience against the consumer being slow. It buys you nothing against the consumer being silently absent, and the more hops your architecture has, the more places that absence can hide. The fix, in the end, wasn't a code change at all. It was asking the one question none of our monitoring had ever thought to ask: is the thing that's running actually the thing we think we deployed?

If you've hit something similar- a queue, a topic, a subscription that looked perfectly configured everywhere except in the one place that mattered- I'd genuinely like to hear about it in the comments.

Final Thoughts & Conversation

Distributed systems give us incredible scale and decoupling, but they trade away obvious failure modes. When your health checks, application logs, and deployment pipelines all report 100% green, the hardest bugs to solve are the ones hiding in the seams between your platform configuration and your code.

Have you ever ran into a bug where an infrastructure override or an auto-declaring broker silently ate your data? How do you handle queue parity and container command parity in your CI/CD pipelines? I'd love to hear your horror stories and prevention strategies in the comments below!

A Quick Side Note:
I regularly write about backend performance, queue architectures, system reliability, and real-world debugging post-mortems. If you enjoy deep-diving into distributed systems, defensive engineering, or API architecture, I'd love to connect and chat!

You can find me on LinkedIn, follow my latest technical thoughts on Twitter/X, or check out my open-source projects and previous articles on GitHub.

Thanks for reading, and see you in the next post-mortem!

Rails Authorization Beyond Models: Securing Dashboards and Service Controllers with CanCanCan

Nemwel Boniface — Fri, 06 Feb 2026 08:44:00 +0000

Hello friends! I’m back with more Ruby on Rails concepts I refreshed this week to stay productive and avoid the social media "doom scroll." 2026 is the year we kill the bad habits hindering our potential, and for me, that means trading the doom-scroll for the docs.

Introduction - Authentication and Authorization. What are these?

Authentication and authorization are often confused, and many people think they are the same, but they are not. Similarly, their ease of implementation is often confused. You might think they are easy to integrate. But are they really? Let me explain.

Authentication, which is verifying who you are, is the easy part. A quick Devise gem install and a migration, and your system knows its users.

Authorization, which decides what they can do, is where the mess begins. Most of us start with CanCanCan basics: you add load_and_authorize_resource to a PostsController, and magic happens. Admins edit, users read. Life is good.

But what happens when you have a controller that doesn't map neatly to a database table? Think about a DashboardController or a SearchController. There is no "Dashboard" table in your database.

Suddenly, the magic breaks and reality sinks in. You start getting confusing error messages or, even worse, silent security gaps that only appear once a user accidentally accesses classified data.

In this article, we’ll explore how to handle authorization when you need to step outside the standard model framework, specifically using tools like authorize_resource class: false to keep your "non-model" controllers secure.

The Foundation: The Rulebook (Ability.rb)

Before we touch our controllers, we have to understand the Source of Truth. In CanCanCan, your app/models/ability.rb file is the rulebook. Everything we do in the rest of the app is just asking this one file a single question: "Is this user allowed to do X to Y?". Let me show you the anatomy of one below:

class Ability
  include CanCan::Ability

  def initialize(user)
    user ||= User.new # guest user

    if user.admin?
      can :manage, :all
    else
      # Scenario A: Database Model Permission
      # Users can manage their own posts
      can :manage, Post, user_id: user.id

      # Scenario B: Abstract Permission
      # Users can read the dashboard, even though there is no Dashboard model
      can :read, :dashboard 
    end
  end
end

Pay close attention to Scenario B. Notice how we defined that permission using a symbol (:dashboard) instead of a class name like Post. In Rails, we are used to everything being linked to a database table. But here, we are telling the rulebook that "Dashboard" is just a concept, a place the user can go. Understanding this shift from Classes to Symbols is the key to solving our future "Model Not Found" headaches.

The "Secure by Default" Mindset

Before we look at specific tools, we need to set a baseline. In a professional application, you shouldn't have to remember to add security; security should be there by default. You should only have to turn it off when you explicitly mean to, in very rare situations. In your ApplicationController, I recommend adding this:

class ApplicationController < ActionController::Base
  # Throw an error if a controller action runs without checking authorization first.
  check_authorization unless: :devise_controller?
end

Think of this as your Authorization Alarm System. If you create a new controller and forget to define any permissions, Rails will yell at you with a CanCan::AuthorizationNotPerformed error, which is much better to have your app crash during development because you forgot a security check than to have it silently expose private data in production. We skip devise_controller? because Devise handles its own "Who are you?" logic, and we don't want to get locked out of the login page!

Now that the alarm system is armed, let me show you how we satisfy it in different scenarios below:

Scenario 1: The Standard CRUD Resource (This is the Happy Path)

Imagine you have a Product model and a standard ProductsController to handle the basics: looking at a list, viewing a single item, or editing details. The Solution would be to add the load_and_authorize_resource at the top of the controller. This will then tell CanCanCan: _"Hey, look at the controller name (Products), find the matching model (Product), load the data into a variable (like@product`), and check the Rulebook to see if this user is allowed to be here."_

You are to use this ideally 90% of the time if your controller maps directly to a database table and performs standard RESTful actions. Example:

The beauty here is Clarity. By moving the data loading and the security check to a single line at the top, your controller actions stay "pure." They only focus on what happens after the door has been opened.

Scenario 2: The "Service" Controller (When there is no Database Model)

This is where the confusion usually starts. You have a controller that performs a service-like a DashboardsController, a SearchController, or a ReportsController. If you check your database, you will find that there is no "Dashboards" table. Dashboard here is just a page that aggregates data from other places.

If you try to use ur favourite load_and_authorize_resource here, CanCanCan will go looking for a Dashboard model, fail to find it, and crash your app because it does not exist. The solution is using authorize_resource class: false.

This tells CanCanCan: "I need to secure this controller, but stop looking for a matching database class. Just check if the user is allowed to access the concept of this controller.". When you set class: false, CanCanCan converts the controller name into a symbol (e.g., DashboardsController becomes :dashboard) and checks your Rulebook for that specific symbol. This is best used whenever your controller represents a concept, a process, or a service rather than a specific row in a database table.
Example (Recall our Ability.rb had can :read, :dashboard.):

If a user tries to access this page without the can :read, :dashboard rule in their ability.rb, they’ll get an "Access Denied" error. With this, you get the same level of security as a standard model, but without the database baggage.

Scenario 3: The Complex or Expensive Action

Sometimes, checking permissions at the very start of a request is too "blunt". Imagine you have an action that kicks off a heavy background job or calls an external API that charges you per request. You ideally don't want to waste any CPU cycles or money until you are 100% sure the user has the right to be there. The solution is the explicit inline authorize!

This is the manual approach, and instead of letting the controller handle security automatically at the top, you take the wheel and tell CanCanCan exactly when and what to check inside the method itself.

Situations when you are supposed to use this include:

When your logic is too dynamic for the standard helpers.
When you need to do some "cheap" setup (like checking params) before doing an "expensive" authorization check.
When your action handles multiple different types of objects at once.

Example usage:

By using authorize! manually, you ensure that your expensive code is protected by a high-security gate. If the user isn't authorized, the method stops right there and raises an exception before the HeavyReportingJob ever gets queued.

Summary and a useful Cheat Sheet

Choosing the right tool comes down to two questions: "Does this match a database table?" and "When do I need the check to happen?"

Scenario	Mapped to DB?	Tool to Use	The "Why"
Standard RESTful (Posts, Users)	Yes	`load_and_authorize_resource`	The magic bullet. Loads data and checks rules automatically.
Service/Concept (Dashboard, Search)	No	`authorize_resource class: false`	Authorizes against a symbol so Rails doesn't look for a missing class.
Complex/Expensive Actions	Maybe	Explicit `authorize!` in the action	Gives you total control. Best for guarding expensive API calls or background jobs.

Conclusion

Doom-scrolling gives you a dopamine hit that lasts five seconds. Solving a tricky authorization bug gives you a secure, professional application that stays protected while you sleep.

By moving past the basic "magic" and mastering tools like class: false and explicit authorize! calls, you stop fighting the framework and start making it work for your specific needs. In 2026, we aren't just building apps that "work"; we are building apps that are secure, intentional, and architected for the long haul.

Next time Rails "yells" at you about a missing model, don't reach for your phone to scroll. Reach for your ability.rb and check if you're trying to authorize a class that doesn't exist.

Do you prefer the "magic" of automated authorization, or are you a "manual control" developer who likes to see every security gate in your code? I’d love to hear your horror stories about AccessDenied errors in the comments!

A Quick Side Note: While I’m heads-down building this search engine, I’m also looking to pitch in on new projects. If you’re looking for a mid-level Rails engineer with solid JavaScript experience who prioritizes security, authorization logic, and building defensive, scalable systems, I'd love to chat. You can find me on [LinkedIn/Twitter] or check out my other work here.

I hope this was useful information for you. See you in my next article.

What Happens When a User Deletes Their Account? A Guide to Rails ActiveRecord dependent: Strategies

Nemwel Boniface — Tue, 20 Jan 2026 06:25:43 +0000

Hello friends! I'd like to show you Ruby on Rails concepts I refreshed this week to Avoid Social Media Doom Scrolling.

Introduction

We’ve all been there. You’re at the end of a high-intensity coding session, or maybe you're in the middle of a Pomodoro break. Your brain is fried, and the easiest "hit" is a social media feed scroll. Suddenly, you're deep into doom-scrolling, watching phenomenal creators like Simon Squibb ask people, "What’s your dream?" (He is honestly my favourite creator right now, and his videos are raw and authentic and a whole vibe! Who’s your favorite creator lately?)

This week, I decided to fight the scroll. I closed the "Trending" tab and opened the Rails Documentation instead. My goal was to master ActiveRecord Association Dependencies.

If you're building a user-centric app, like the "Google-style" search engine I’m currently architecting, you quickly realize that choosing how a record "dies" is just as critical as how it lives. In a search engine that tracks saved queries, leads, and API keys, our strategy has to balance two things: User Privacy (deleting personal data) and Business Continuity (preserving lead data for analytics).

Here is how I structured my models to ensure my database doesn't become a "ghost town" of orphaned records.

1. The Clean Sweep: `dependent: :destroy`

In my app, when a user deletes their account, I want their Saved Searches to vanish. These are private and hold zero value once the owner is gone.

I chose :destroy because it’s the "polite" way to say goodbye. Unlike a cold database delete, :destroy instantiates each record and triggers its specific callbacks. If you have logic that says "Notify the search engine to de-index this query when deleted," this is the only way to ensure that "cleanup" logic actually executes.

class User < ApplicationRecord
  has_many :saved_searches, dependent: :destroy
end

2. The Bulk Cleanup: `dependent: :delete_all`

Sometimes, you don’t need a polite goodbye; you need raw speed. Imagine a user has millions of Audit Logs. Calling :destroy would try to load every single log into Ruby memory, a surefire way to crash your server or hang your database.

:delete_all skips the Rails middleware entirely. It sends a single, sharp SQL command directly to the database. This, however, has a Trade-off: If your AuditLog has a before_destroy hook that cleans up a file on Amazon S3, it will not run. Use this only for "passive" data where speed is the priority over side effects.

class User < ApplicationRecord
  # If we had millions of log entries, we didn't need to process
  has_many :audit_logs, dependent: :delete_all
end

3. The "Legacy" Handover: `dependent: :nullify`

This was the trickiest one for me to grasp, but it’s vital for B2B apps. In our search engine scenario, if a User generates a Lead and then deletes their account, the business still needs that Lead record to calculate conversion rates.

If I delete the lead, I break the business analytics. By using nullify, the user_id on the Lead becomes NULL, but the record itself stays on the "shelf." The user’s privacy is respected (their name is gone), but the business value is preserved.

class User < ApplicationRecord
  has_many :leads, dependent: :nullify
end

4. The Soft Block: `dependent: :restrict_with_error`

I want you to think of this as the "Safety Valve." Sometimes, a user cannot be allowed to leave yet. In my app, if a user has Active API Keys, letting them delete their account would leave external integrations in a broken state.

Instead of a hard crash, :restrict_with_error adds a message to the errors object. This allows me to tell the user in the UI: "You can't close your account while you have active API keys. Please disable them first." It’s a UX-friendly way to protect critical system resources.

class User < ApplicationRecord
  # "You can't leave us while you still have an active API Key!"
  has_many :api_keys, dependent: :restrict_with_error
end

5. The Hard Guard: `dependent: :restrict_with_exception`

This is the ultimate safety net for financial data. I use this for Billing Subscriptions. If the system attempts to delete a user who still has an active billing record, Rails will raise a hard ActiveRecord::DeleteRestrictionError.

This isn't for a "nice UI message." This is for your "Security-First" logic, a last line of defense to ensure your financial integrity is never compromised by a stray user.destroy call.

class User < ApplicationRecord
  has_many :billing_subscriptions, dependent: :restrict_with_exception
end

Short Recap

When a user leaves, not all data should be treated the same. Privacy, performance, and business continuity pull in different directions. Pick :destroy when cleanup must run, :delete_all for raw speed, :nullify to keep business artifacts, and :restrict_with_* to block dangerous deletions.

Conclusion

Doom-scrolling gives you a dopamine hit that lasts five seconds. Solving a database integrity problem gives you a stable application that lasts years.

By strategically picking between :destroy, :nullify and :restrict, I’ve ensured that my system handles user churn like a pro. My advice? Next time you’re bored, don't check the "Trending" tab. Check your schema.rb. There is always a relationship that could be handled more gracefully.

Which dependent strategy do you find yourself reaching for most often? Are you a "Clean Sweep" developer who keeps a pristine database, or do you lean heavily on :nullify to protect business analytics? I’d love to hear how you handle the "afterlife" of your data in the comments!

A Quick Side Note: While I’m heads-down building this search engine, I’m also looking to pitch in on new projects. If you’re looking for a mid-level Rails engineer with solid JavaScript experience who cares about building defensive, scalable systems, I’d love to chat. You can find me on [LinkedIn/Twitter] or check out my other work here.

I hope this was useful information for you. See you in my next article.

Architecture Backwards: Engineering a Self-Defending System Before the UI Arrives

Nemwel Boniface — Fri, 09 Jan 2026 07:28:31 +0000

Happy New Year!

As we kick off 2026, many of us are starting new projects or refining old ones. This year, I’ve decided to challenge the "industry standard" of MVP development. Instead of racing to build a login page, I’m spending my time in the trenches of the core engine.

Introduction

In the rush to launch a Minimum Viable Product (MVP), the standard advice is almost always: Start with the User. Build the login page, the profile, and the "Join Now" button. We often prioritise Authentication and Authorization, then move to the core logic, the actual reason the product exists, nearly as an afterthought.

This approach works, and it’s the industry standard. But what if I tell you there’s a more resilient way to architect a system? One that isn't as glamorous or visually aesthetic, but is infinitely more stable?

Lately, I’ve been building a new system where I’ve intentionally flipped the script. I am working backwards, starting with the "brain" of the engine rather than the face of the app. I am two weeks into development, and I haven't even touched a User module. Instead, I’ve spent my time building Self-Defence Mechanisms and Telemetry.

Hear me out, while authentication and authorisation in your system is vital, we have battle-tested gems like Devise and CanCanCan that can be dropped in later and will solve this for you in minutes. The core health of your system, however, cannot be "dropped in" as an afterthought and therefore needs a complete system architecture shift. Here is why the most scalable systems are built from the inside out using what I call the Sensor & Suitcase pattern, which I will introduce in the second section below.

1. The "Horizontal" Nature of Users

When working with a modern Ruby on Rails application, the User resource can be treated as a horizontal layer. I want you to think of the User layer as the front door to your house. It’s how people get in, and it’s what they see first. But before I invite guests over, I want to make sure the foundation is solid and the electricity is wired correctly. Because Rails emphasises Convention over Configuration architecture, adding a user_id to a record later is often just a simple migration away. I have the expensive luxury of treating "Identity" as a pluggable module because the framework and the core language, Ruby, are designed for developer happiness. I am actually happy working with this stack for the project.

However, when we talk about the core engine, the part that actually does the work, this should be seen as vertical. If that engine is fragile, it won't matter how many users you have or how beautiful your login page is.

If your core system cannot handle the "real world" problems that come with scale, such as race conditions, API rate limits, or cache stampedes, the system will collapse under the weight of its own success.

1.1 The Tortoise and the API

This might feel like a re-invention of the story of the Hare and the Tortoise. The "Hare" approach is to race toward a UI, a "Join Now" button, and a beautifully designed user dashboard. It looks fast, but it’s tripping over every unhandled exception along the way.

By building "Backwards," I am the Tortoise. I am building a system that knows how to protect itself first. I am ensuring that when the "Hare" (the User) finally arrives, the ground beneath them is solid, monitored, and reinforced.

2. Building the "Defensive By Design Perimeter."

Before a single user is able trigger any request through my new system, I needed to know that the system will protect its most expensive resource: External API Credits!

I am making use of what is called the Sensor & Suitcase pattern. By implementing the Sensor & Suitcase pattern, we shift the architecture from being reactive (fixing things when users complain) to proactive (handling failures before the user even notices).

Here is why this "Backwards" approach is a game-changer for your system:

1. Total Visibility (The Sensors): Most developers fly blind until an error log hits their inbox. By using ActiveSupport::Notifications to broadcast every “heartbeat”, you create a "Black Box" recorder for your app. Remember that a heartbeat is any user-defined activity and can be something like a user requesting a resource from my system. This way, you aren't just guessing how the system performs; the system is telling you exactly where it feels strain in real-time.

# The following snippets of code and all that follow are  
# intentionally simplified to communicate architectural  
# intent rather than production-ready implementations.
# Implementation of the 'Sensor' within the service layer
def fetch_resource(query, location)
  # Wrapping the core logic in a 'Sensor' block
  ActiveSupport::Notifications.instrument('<Module name>::external_call', {
    query: query,
    location: location
  }) do |payload|
    result = <Module name>::Client.execute(query, location)

    # Enrich the sensor data with metadata for real-time telemetry
    payload[:status] = result.status
    payload[:duration_ms] = result.duration

    result
  end
end

2. De-risking Through Isolation (The Suitcase): External APIs are often the "chaos variables" of web development. By packing those risks into SolidQueue background jobs, you decouple your app’s success from an outsider's failure. If a third-party service hangs for 10 seconds, your user doesn't see a spinning loading icon; they see a finished task because your "Suitcase" is handling the heavy lifting in the background.

# The 'Suitcase' - Isolating external volatility from the main request thread
class <Module name>::IngestJob < ApplicationJob
  queue_as :external_api

  # Defensive retries with exponential backoff to handle provider downtime
  retry_on <Module name>::ExternalError, wait: :exponentially_longer, attempts: 3

  def perform(signal_id)
    signal = <Module name>::Signal.find(signal_id)

    # Perform the 'heavy lifting' away from the user's eyes
    <Module name>::Ingestor.new(signal).call
  rescue StandardError => e
    # Log the failure for the 'Black Box' recorder
    <Module name>::Telemetry.record_failure(e)
    raise e
  end
end

3. The "Snappiness" Factor: In a Rails 8 monolith, speed is often about what you don't do during the request-response cycle. This pattern strips the main thread down to its bare essentials, ensuring your UI stays lightning-fast while the complex logic executes safely behind the scenes.

The Core Benefit here is that you aren't just building a feature; you are building a self-healing perimeter. Your system defends its own performance metrics, regardless of how the outside world (or the user) behaves.

3. Solving the "Thundering Herd" Problem

One of the most satisfying parts of this "Backwards" approach was preparing for a "Thundering Herd" before it ever happened. This refers to the moment when multiple requests hit your system at once for the same resource.

Usually, this is where most systems start to show off their glamorously covered-up cracks. In a situation where 10 users request the same data at the exact same millisecond, a standard app might fire off 10 expensive, redundant calls to an external API.

However, by focusing on the "Self-Defence" layer first, I solved this before a single user ever signed up on my system. I used atomic increments in Redis and PostgreSQL row locking to make the system smarter. Now, when those 10 requests hit at the same time, the system identifies the "herd" and performs one expensive API call, while the other nine requests simply wait for a heartbeat to share that single result.

Instead of 10 separate stresses on the engine, it’s just one. This is the difference between a panicked crowd and an organised queue.

# Utilizing PostgreSQL Advisory Locks to organize the 'Herd'
def find_or_create_resource(query, location)
  # Create a unique lock key based on the specific search parameters
  lock_key = Digest::SHA256.hexdigest("#{query}:#{location}")

  # Ensure only ONE process handles the expensive external call
  <Module name>::Database.with_advisory_lock(lock_key) do
    # Re-check cache inside the lock; the first worker filled it already
    return cached_result if cached_result.present?

    # This is the single 'Tortoise' worker performing the expensive API trip
    execute_expensive_external_call(query, location)
  end
end

4. The Admin-First Dashboard

Because I haven't spent a single minute working on a "User Profile" page, I had the time to build something much more valuable: a Telemetry Dashboard.

Right now, this dashboard isn't for a customer; it’s for the system (and, okay, it's for me to geek out on until I go live! xD). Instead of looking at user avatars or bios, I am looking at the "System’s Efficiency." I’ve built a metric I call the Vault Ratio. This metric tracks how many requests are served from the "Vault" (my local, optimized data) versus how many require a slow, expensive trip to an external API.

By seeing these numbers move in real-time, I can tune the engine while it’s still on the workbench. I’m not guessing if my caching strategy works; I’m watching the "Vault" grow stronger every day. By the time the first user signs up, they aren't entering a construction site; they are stepping into a fully furnished house.

# The core logic behind the Telemetry Dashboard
def today_snapshot
  date = Date.current
  {
    discovery_requests:  <Module name>::Metrics.read('discovery.requests', date),
    cache_hits:          <Module name>::Metrics.read('discovery.cache_hits', date), # Local Rails cache
    vault_hits:          <Module name>::Metrics.read('discovery.vault_hits', date), # Persistent optimized data
    external_calls:      <Module name>::Metrics.read('discovery.external_calls', date), # Expensive credit usage
    signals_created:     <Module name>::Metrics.read('signals.created', date)
  }
end

The Lesson I have learned so far: Embrace the trenches

In the past, I was always afraid of staying in the backend trenches for too long. There is a constant pressure to "show progress" through UI and buttons. But if there is one thing I’ve learned from systems that I didn’t architect well in the past, it’s this: Don’t be afraid to stay in the back-end longer than it feels comfortable.

Remember the story of the tortoise and the hare. The slow, intentional pace you take now is what prevents the dreaded 3:00 AM headaches down the line.

When you build an engine that is self-defending and observable from day one, adding the "User" front-facing layer isn't going to be a stressful launch; it will be just a normal Tuesday. This approach gives you the confidence to sleep soundly, whether your system is handling 10 users or 10,000 users concurrently.

Conclusion

As a parting shot and a last word of advice from an engineer implementing best software practices that help global businesses thrive, to build software that helps businesses truly thrive, we have to prioritize the "brain" over the "face." Focus on your system’s core logic and bulletproof it first. Make it exist, make it resilient, and then make it pretty.

The users might not see the Redis locks or the background workers, but they will certainly feel the stability.

Is the "Sensor & Suitcase" pattern something you or your team would consider for your next project? I’d love to hear how you handle the "Defensive" side of your builds in the comments!

A Quick Side Note: While I’m heads-down building, I’m also looking to pitch in on new projects. If you’re looking for a mid-level Rails engineer with solid JavaScript experience who cares about building defensive, scalable systems, I’d love to chat. You can find me on [LinkedIn/Twitter] or check out my other work here.

I hope this was useful information for you. See you in my next article.

Building a Custom Audit Trail in Ruby on Rails Without PaperTrail

Nemwel Boniface — Thu, 21 Aug 2025 11:14:21 +0000

Introduction

Auditability in a system is a key feature, and having a way in which you can do that in your application will definitely help a lot. In your system, you need to know who did what, who created what, updated what, deleted what, and when this was done. Please note that for deletes, it is not a hard delete but a “soft” delete option whereby you have a boolean field maybe called “Archived,” which is set to false by default and set to true when you press to delete it (Archive it).

While I know there exist gems that can do this for you, such as the papertrail gem, you may want to reduce the number of dependencies in your application by coming up with your own custom maker checker. This involves adding fields in your database table, such as created_by, deleted_by, modified_by, etc. Look, hear me out, this may sound counterintuitive to add a created_by field, but the aim of making a maker checker is to add all the needed fields.

Prerequisites

Ruby v 3+
Rails V 7/8
Rails app with Devise for Authentication
Some experience with Rails MVC

Let's get started:

Assume that you have a table called modules. To add the auditable fields, run the following migration to create a migration file that we will update below:

rails g migration AddAuditFieldsToModules

Update your migration file to look like the one shown below:

class AddAuditFieldsToModules < ActiveRecord::Migration[7.2]
  def change
    change_table :modules do |t|
      # Audit fields
      t.references :created_by, type: :uuid, foreign_key: { to_table: :users }
      t.references :modified_by, type: :uuid, foreign_key: { to_table: :users }
      t.references :deleted_by, type: :uuid, foreign_key: { to_table: :users }
      t.datetime :deleted_on

      # Archive status
      t.boolean :archive_status, default: false, null: false

      # Indexes
      t.index :archive_status
      t.index :deleted_on
    end
  end
end

Run rails db:migrate to update your Schema file.

The next step would be to create a Current model inside app/models/current.rb, which inherits from ActiveSupport::CurrentAttributes and update it as follows:

class Current < ActiveSupport::CurrentAttributes
  attribute :user
end

The Current class file is needed to provide a thread-safe way to access the current user (or other request-specific attributes) throughout your application, particularly in models where you normally can't access controller-level information like the current_user devise helper method. By using ActiveSupport::CurrentAttributes, it creates a globally accessible but request-specific storage that your Auditable concern can use to automatically track who created, modified, or deleted (In our case, a delete is actually an archive) records. This maintains clean separation of concerns while ensuring your audit trail always captures the correct user, whether the action originates from controllers, background jobs, or other parts of your application. The Current pattern is the Rails-recommended way to handle this cross-cutting concern without polluting your models with controller logic.

Our next step will be creating an auditable concern module inside app/models/concerns/auditable.rb file and add the following code to it:

module Auditable
  extend ActiveSupport::Concern

  included do
    # Associations
    belongs_to :created_by, class_name: 'User', optional: true
    belongs_to :modified_by, class_name: 'User', optional: true
    belongs_to :deleted_by, class_name: 'User', optional: true

    # Scopes
    scope :active, -> { where(archive_status: false) }
    scope :archived, -> { where(archive_status: true) }

    # Callbacks
    before_create :set_created_by
    before_save :set_modified_by
  end

  # Instance method for archiving
  def archive!
    update(
      archive_status: true,
      deleted_by: Current.user,
      deleted_on: Time.current,
      modified_by: Current.user
    )
  end

  # Instance method for unarchiving
  def unarchive!
    update(archive_status: false, modified_by: Current.user)
  end

  private

  def set_created_by
    self.created_by ||= Current.user
    self.modified_by ||= Current.user
  end

  def set_modified_by
    return unless !new_record? && Current.user
    self.modified_by = Current.user
  end
end

Ok, ok, I know that this may seem like a lot, but we are mostly done now, you can relax.

The Auditable concern provides automatic tracking of record authorship and changes through a complete audit trail system. It adds three user associations (created_by, modified_by, deleted_by) to any model that includes it, along with scopes for filtering active/archived records. Using Rails callbacks, it automatically stamps records with the current user (via Current.user) during creation, updates, and soft deletion (archiving). The concern also provides explicit archive! and unarchive! methods that handle soft deletion while maintaining referential integrity and a complete change history. This creates a self-contained audit system that tracks the complete lifecycle of records while keeping the implementation DRY and consistent across different models. The soft delete functionality preserves data by setting an archive_status flag rather than physically removing records.

In case you may still need a way to perform a hard delete, you can extend the functionality by adding this to your Auditable concern. Please note that this may be counterintuitive, as the record will no longer be there. So, setting the "deleted_on" or "deleted_by" will not be there as the record will be gone.

# Callbacks
    before_destroy :set_destroyed_by

  def set_destroyed_by
    self.deleted_on = Time.current
    self.deleted_by = Current.user
    self.archive_status = true # Archive when destroyed
  end

Inside our module.rb model file, we shall add the following to first include the auditable module, including all its functionality, adding it into our module model class, and using the scope to create a query shortcut that automatically filters for non-archived records.

# Auditability concern
  include Auditable

Our setup is mostly complete, and you do not need to make any changes to your controller logic and The only place you need to make changes is in your destroy action, as shown below, which instead of performing an actual hard delete, calls the archive method, which we defined in our auditable concern earlier:

def destroy
    @module.archive! # Soft delete / archive
    respond_to do |format|
      format.html { redirect_to modules_url, notice: "The Module '#{@module.name}' was successfully deleted." }
      format.json { head :no_content }
    end
  end

So, why are we re-inventing the wheel?

So, why would someone want to use this approach as opposed to using pre-existing gems such as PaperTrail?

Eliminating dependencies caused by using gems. While some gems are very useful, some may stop receiving updates, which may mean, further down the line, you may be forced to migrate to a custom auditability form or move to another gem
It is in line with the native Rails way of doing things; you probably don't need those gems if you can implement it custom. Such a custom approach is generally faster than using PaperTrail by upto 2-3 times, reducing the amount of “magic” happening in your code.
There is complete transparency of what is going on, what is being collected, and how everything works. This is unlike when using a gem, and there is some “magic’ that happens behind the scenes, and you do not fully understand how it works in the first place.
To have control over the data you want to collect/ track in your system. While some gems are very useful for a quick fix for data collection, some may be designed to collect an excessive amount of data, which may not be needed and may definitely increase the database costs. Customization allows you to collect the least amount of data, only the ones that you need, and nothing more.
This approach incorporates query efficiency in situations where you would like to know who did what; you just get it in the table columns instead of doing complex queries, making your application data load faster.
For scenarios where you are using cloud storage, this will help you to avoid using cloud offerings such as AuditLog, which charges per event. This in place saves on costs for the company.

When it is better to use existing solutions:

While the pros of a custom approach are undeniable, there exist scenarios where you may need to use pre-existing gems as opposed to a custom solution like ours:

It may be a regulatory Requirement for the company. For example, healthcare systems bound by HIPAA may require complete object versioning with cryptographic signing of all changes.
In situations where you need to perform cross-model Search in your system. This occurs when you need to search audit trails globally.
In a situation where you need a Full Change History in your system. Gems such as PaperTrail have reify for undo functionality
When a separation of concerns is needed. This is where the production database and the audit database are in two separate databases, keeping everything neat and maintainable.

Areas for Improvement (What Could Be Better)

While this approach is clean and avoids extra dependencies, there are several considerations to keep in mind if you want to make it production-ready:

Sometimes your system performs certain tasks, such as background jobs, where in such scenarios, there usually isn’t a current_user value, and it will be nil, which would make auditing hard. In such situations, assigning such changes to a dedicated System user account when no explicit user is available would be a safe workaround. For example:

def set_created_by
    # Assign the system user if no current user is present
    user = Current.user || User.find_by(email: 'system@yourdomain.com')
    self.created_by ||= user
    self.modified_by ||= user
  end

When working with Rails APIs and you are exposing some of the audit fields, such as created_by or modified_by, directly through JSON APIs, you may unintentionally leak personal information (e.g., user IDs, emails). From a compliance standpoint, you may need to consider whether those fields should be public, anonymized, or restricted to admin roles.

Alternatives & Extensibility

Just as Rails has some best practices for using the framework effectively, below are my best practices rules, which I highly recommend considering when choosing to have a custom audit mechanism for your application, may it be a custom audit tooling or a ready-made software:

It is best to focus on capturing the changes, and not just actors: While I am aware that my current approach only records who changed a record, but not what they changed, I would recommend extending the functionalities with a changes_log JSON column or an associated audit_logs table that stores the changes made.
Embracing database-level auditing: databases such as PostgreSQL offer native auditing solutions using triggers or extensions such as pg_audit, which are battle-tested solutions that can help you to log all operations at the database layer.
KISS (Keep it simple, stupid) and choose a hybrid approach of just recording “who/when” directly in your model (just as in my demonstrated examples), combined with a dedicated audit log table or external log system for mission-critical changes.

Auditing may feel daunting at first. However, knowing what amount of data you want to collect is the first step. Don’t sweat it.

Conclusion

The choice ultimately depends on your application’s specific requirements. However, using the custom approach as I have demonstrated is an excellent fit for most mid-sized applications, prioritizing maintainability and performance.

If you need a lightweight, Rails-native auditing functionality, this approach works perfectly well. However, if you need full versioning, support for cryptographic trails, or cross-model searching capabilities, it would be better to stick with PaperTrail or database-level solutions.

I hope this was useful information for you. See you in my next article.

How to Automate Daily Task Emails in Rails using Whenever and Cron

Nemwel Boniface — Wed, 16 Jul 2025 06:49:51 +0000

Background

I have been working on a task assignment system where one of the tasks I need to do is to send everyone at the end of the work day stats of what tasks they had at the start of the work day and what tasks they have been able to close and if there are any new tasks that they were assigned during that work day. Initially, I created a manual method to send emails to them by using a button that I had to click at the end of each workday. This approach worked perfectly well. However, in situations where I was not there physically to click the button to send out the daily tasks reports to all users, this quickly became an issue for me, and coming up with a way to automate this process is no longer an option, but mandatory. This is where the Ruby Whenever gem comes in.

Introduction: Whenever Gem

Whenever Gem is a Ruby DSL(domain-specific language) for managing Unix-style cron jobs. It translates your Ruby syntax into actual crontab entries that your system can use and understand. As a result, it works reliably on Unix-based systems (Linux, macOS). Still, it is not compatible with Windows unless you use a Unix-like environment (such as WSL) or adapt your jobs to Windows Task Scheduler.

The Gem provides a clear syntax for writing and deploying cron jobs. A cron job is a time-based job schedule that allows users to automate tasks by scheduling commands or scripts to run at specific times/ intervals. The scheduled tasks are known as cron jobs and are all defined in a configuration file known as the crontab. A cron is a daemon that runs in the background and checks the crontab file for scheduled tasks to run. Remember that a daemon is a computer program that runs in the background, performing tasks without direct user interaction.

What are schedulers?

Schedulers are components that trigger certain tasks at specific intervals (hourly, daily, weekly, etc). Schedulers operate in the background, outside of the request/ response lifecycle (You do not run them when a user visits your website).

While Rails does not have a native (inbuilt) scheduler for arbitrary cron-style jobs, we have options for using Background jobs using ActiveJobs + Sidekiq. Or asynchronous work and making use of external scheduler tools (such as whenever + sidekiq-scheduler) for repeating jobs.

Prerequisites:

Rails V7
PostgreSQL database
Ruby V 3+
Preferred web browser
Visual studio code
Some experience with Rails MVC architecture

Installing the Whenever Gem

Installing the whenever Gem and configuring it to work is fairly straightforward in Ruby on Rails. To install the whenever gem, add the following to your GEMFILE and run bundle install to install the package to your application.:

gem 'whenever', require: false

Optionally run the following in your terminal:

bundle add whenever

Then run the command wheneverize . to create the schedule.rb file inside the config/ directory. This step is very crucial as the schedule.rb file is where you define your cron jobs using Ruby DSL syntax (e.g., every :day, at: '5:05pm'). Without this file, the whenever gem has no instructions to generate crontab entries.

If you would like to follow along with me through the article, I created this GitHub repository which you can clone to test out the setup.

Real world practice: Sending out an email every weekday at 5:05 pm

This is a relatable situation where you may want to automate the process of sending out emails every workday at 5:05 pm. To do this, we will first create a mailer. In your terminal, head over and type:

rails g mailer DailyMailer

This generates a file in the app/mailers/ directory called daily_mailer.rb, which we are going to open so that we can edit it by adding the following to it:

Class DailyMailer < ApplicationMailer
  def daily_email
    mail (to: "user@gmail.com", subject: "My update"
  end
end

The next step will be to create a view inside the app/views/daily_mailer/ directory called daily_email.html.erb adding this line: <h1>Hello! This is your daily update at 5:05 pm </h1> to it.

We shall then create a Rake task inside the lib/tasks/ creating the file called email.rake, using the command touch lib/tasks/email.rake, and open it to edit it as follows:

namespace :email do
  desc "Your daily email"
  task send_daily_email: :environment do:
    DailyMailer.daily-email.deliver_now
  end
end

We shall now run the command: rails email:send_daily_email, to test out our current setup.

How the Automation Flow Works

Before we test locally with Letter Opener, let’s visualize the full workflow:

Key Steps showcased in the flowchart include:

schedule.rb which defines the cron schedule in Ruby (e.g., "Weekdays at 5:05 PM").
crontab system converts this into a background process.
Rake Task which executes your custom logic (e.g., querying tasks).
Mailer which formats and sends the email.
Letter Opener (Dev) / SMTP (Prod) which handles delivery of your email.

Integrating the email send automation with the Whenever Gem

We are now going to ensure that after every work day at 5:05 pm, users can receive an email sent to them. Open the config/schedule.rb file and inside it add:

every :week, at: '5:05 pm', on: [:monday, :tuesday, :wednesday, :thursday, :friday] do
  rake "email:send_daily_email"
end

To be able to check the output, run the command whenever in your terminal. In case nothing happens on your end, not to worry. It means that you do not have any SMTP server configured, and this is where we will resolve it in the next section of the article, the letter_opener gem.

Local testing using the letter_opener gem

In case you do not have an email SMTP setup locally, we will use the letter_opener Gem that catches any email that we want to send. Instead of trying to connect to an SMTP server, it renders the email in our browser and is an amazing tool for testing, as it will show your email like an HTML page for you. This means that:

We do not have to configure an email server
We can see and test emails quickly
We do not send any actual emails over the internet

If you are following along upto here, right now when we try to send the email by connecting to localhost:25, it fails because we do not have an SMTP server configured locally. But with the letter opener gem configured, when our app tries to send an email using the mail() or deliver_now(), letter opener intercepts it and saves it as an HTML page. Then it automatically opens the page in your browser for you. The letter opener gem inline allows you to test your mailer easily, with no error messages, no ECONNREFUSED, and you can review your email layout. Amazing right?

How to use the letter opener gem

You will add the gem 'letter_opener' to your Gemfile in the development group, and run bundle install. Open the config/environments/development.rb file and edit it to add the following configurations:

config.action_mailer.perform_deliveries = true
config.action_mailer.delivery_method = :letter_opener
# Don't care if the mailer can't send.
config.action_mailer.raise_delivery_errors = false

# Disable caching for Action Mailer templates even if Action Controller
# caching is enabled.
config.action_mailer.perform_caching = false

config.action_mailer.default_url_options = { host: "localhost", port: 3000 }

These settings configure Action Mailer to actually send emails in development mode, preview them in the browser using the letter_opener Gem, suppress delivery errors, disable caching for email templates, and generate proper URLs using localhost:3000 as the base. Amazing right?

Once this has been done, run: rails email:send_daily_email in your terminal. If you have followed along so far, your default email browser will be opened, and you will see something similar to what I have below.

Up to this point, it will load the email in the browser as expected. The application will no longer attempt to connect to localhost:25 (SMTP port) and will instead open your browser.

Up to where we are, we have been able to test out our setup, and we all validated that it works perfectly fine.

Notes about the letter_opener gem:

Is not for production, it is a tool for development testing alone
Helps you test and view emails quickly without sending them explicitly over an SMTP network
Fixes the connection refused error for local testing

Considerations while using the whenever and the letter_opener gems

When you update the schedule.rb file, and you run the command whenever, Rails will automatically assume RAILS_ENV = production by default. So, when you run whenever without specifying any flags, it shows you the schedule tasks for production and not for development. This may mean that you are not going to see any output and you may think that your setup is all wrong. Luckily, we have a few work a rounds for this and below are some options to use when testing out in development mode. See attached screenshot below as an example.

Remember that running the whenever command is essential because it converts your human-readable schedule written in Ruby (in schedule.rb) into raw cron syntax that your system understands. Without this step, your defined jobs won’t actually get scheduled (We mentioned that whenever doesn't run jobs itself. It simply tells your operating system when and how to run them by updating the crontab). If you skip this step, your background jobs will never run, no matter how well your Rake tasks or mailers are written.

To fix this, we have a few options, starting with the temporary option, which is running the command shown below which will allow you to preview the schedule for development environment.

whenever --set environment=development

The other solution will be:

whenever --update-crontab --set environment=development

To explicitly set the environment in the schedule.rb file at the top of the schedule.rb file, you will add the following:

set :environment, "development"
set :output, "log/cron.log"

After adding this to our schedule.rb file, we will need to run the command whenever --update-crontab With the flag --update-crontab because running whenever alone simply previews what the generated cron entries would look like but it does not install them. To actually schedule your tasks, you must run whenever --update-crontab, which writes those entries into your system's active crontab.

To confirm whether this was successful run crontab -l in your terminal. If you were s, you should see something like:

Always remember that: the whenever Gem does not send emails itself, it just tells your OS (using cron) to run the command at a specific time. In our case, we want it to run every time at 5:05 pm every workday. The cron service will run RAILS_ENV=development bundle exec rake email:send_daily_email

Conclusion

Technology exists to help us work faster and focus on what truly matters. Let it handle repetitive, automatable tasks like sending daily emails at specific times so you can focus on building smarter systems.

By using the whenever gem together with letter_opener, we’ve implemented a powerful and practical scheduling pattern. With this setup, we’ve removed human dependency from a critical workflow and learned how to safely test scheduled jobs in a development environment without the need for a live email server.

This one marks the end of the step by step guide on automating tasks using Whenever Gem and Cron. What will you be automating next in your systems? I would love to hear about that.

I hope this was useful information for you. See you in my next article.

Stimulus.js Explained: A Beginner’s Guide for Ruby on Rails Developers

Nemwel Boniface — Mon, 10 Feb 2025 07:26:10 +0000

Introduction

Hello friends!

We all have to agree on one thing, JavaScript is the king of the web. It is very true that it is the undisputed language that powers over 98.7% of websites, making it the dominant language for web interactivity. You can get so far with plain HTML and CSS when working on your websites. You will however want to have some reactivity on your website. Imagine without JavaScript in your website, you may not be able to implement features such as changing content on the DOM.

As Ruby on Rails developers, we often prefer server-side rendering for speed and SEO benefits and at all costs, we try to reduce the reliance of JavaScript on our applications and rely on server-side rendering of content and sending it to our views to be shown to the client side. This has multiple advantages as this approach allows for pages to load quickly and be indexed by search engines without requiring JavaScript to render content. This is however still limiting and thus need for JavaScript to add that pop to our application reactivity.

This is where Stimulus.js comes in. It is a lightweight JavaScript framework that adds interactivity to HTML designed to enhance static or server-rendered HTML by connecting JavaScript objects to elements on the page using simple annotations. Let this not be confused with Turbo, Turbo focuses on enhancing client-side interactivity by handling page updates, while Stimulus is more focused on managing client-side behavior and adding interactivity to HTML elements.

This article will give you a general overview of Stimulus, covering its features, explain why Stimulus is useful, especially in Ruby on Rails, and give a comparison with other libraries such as React or Jquery. In a future article, we will cover how to set up and use Stimulus in your Ruby on Rails applications.

Stimulus JS deep dive

We earlier mentioned about JavaScript objects. In Stimulus, these JavaScript objects are called controllers. Stimulus uses data-controller attributes to continuously monitor the pages. For each attribute, Stimulus looks at the attribute’s value to find a corresponding controller class, creates a new instance of that class, and connects it to the element. I would like you to think of it this way: just like the class attribute is a bridge connecting HTML to CSS, Stimulus’s data-controller attribute is a bridge connecting HTML to JavaScript.

Stimulus JS core concepts

Stimulus JS builds on controllers with three key concepts: actions, targets, and values. Let’s discuss the further:

1.Actions, which connect controller methods to DOM events using data-action attributes. Actions allow you to specify what action to take when eg a button is clicked. As an example, assume that you have a controller called hello_controller that has a method called greet, and you would like to connect that greet method to an HTML button, your action would look similar to this:

<button data-action="click->hello#greet"> Greet </button>

where the data action is a click event and that maps it to the greet method in the hello controller.

2.Targets, which locate elements of significance within a controller. Assuming that you have an input field and you would like when you click the button, the value that is in the input field will be captured and that value will be used somewhere else, this is where the data-<controllerName>-target attribute comes in. For example, we have our input field, we may want to reference this value as follows:

// Our views
<div data-controller="greet">
  <input type="text" data-greet-target="name" placeholder="Enter your name">
  <button data-action="click->greet#sayHello">Say Hello</button>
  <p data-greet-target="output"></p>
</div>

// Hello controller
import { Controller } from "@hotwired/stimulus";

export default class extends Controller {
  static targets = ["name", "output"];

  sayHello() {
    this.outputTarget.textContent = `Hello, ${this.nameTarget.value}!`;
  }
}

In the code shown above, the Stimulus controller "hello" defines the nameTarget and the outputTarget to reference the HTML input and paragraph. It then updates the paragraph with a greeting message when the button is clicked.

3.Values, allow you to store and update dynamic data within a Stimulus controller using the data-* attributes. They automatically sync between the HTML and JavaScript, making it easy to manage dynamic data inside Stimulus controllers without manually handling DOM updates. A good example is using values to create a counter that increments or decrements when a button is clicked.

Phew! That was almost a lot of technical content on Stimulus JS, right? Ok, let's take it easier now.

Why anyone should be using Stimulus.js in their applications

We have so far seen what Stimulus is, and had an understanding of its core features. But why should you want to use it anyway? Let’s explore more below:

Can help you with basic interaction enhancement. Take an example of when you want to implement a toggle button functionality in your website.
Form input handling. A great example is the sample I shared earlier when covering the stimulus js concepts (Actions, targets, values).
Fast and lightweight. As compared to similar libraries such as React, it is lightweight and minimal making sure that your application does not experience any lags due to heavy packages.
It has a seamless integration with Ruby on rails. Stimulus is designed to easily work with Ruby on Rails, especially alongside Turbo, and does not affect the existing backend structure.
Has automatic element identification - uses data-targets to reference elements within a controller, reducing the need for manual document.querySelector calls and keeping JavaScript neatly organized.
Stimulus allows for progressive enhancement of your web applications' client views. It works on top of server-rendered pages, thus ensuring that the core functionality remains intact even if JavaScript is disabled, This improves accessibility and SEO of the web pages.

Despite using Stimulus.js showing real tangible benefits to our applications, they also have some serious considerations that are worth factoring in when using Stimulus over other libraries such as React or Vue.

Stimulus may not be well suited for applications that are very complex and need single page (SPA) functionalities. If your app requires complex state management, virtual DOM updates, or client-side routing, you may need a framework like React or Vue.

The fact that Stimulus is tightly coupled with HTML and relies heavily on the data-* attributes in HTML, leads to having the code become cluttered if not managed properly. This is especially true in large applications.

While this might not be necessarily true for everyone, however, the syntax that Stimulus uses is not really intuitive at first glance and requires additional reading to try and have a grasp of the syntax to understand the moving parts. This extra layer of complexity might not really suit everyone and they may find it hard to start working on it quickly.

One of the determining factors of the use of a tool is usually the community behind it. Unlike other libraries such as React, Stimulus has a relatively smaller community backing it which would bring a challenge in finding resources or even help when you run into a challenge. You however need not worry because I am here to bridge that gap! 😄

Conclusion

Just like any tool out there, it has its strengths and its weaknesses. In my own opinion, Stimulus is an amazing tool that I believe everybody should be using especially in their Rails applications to add some reactivity to their pages. Stimulus features such as reusable components make it an attractive tool allowing you to define a feature once and reuse it multiple times allowing you to stick to programming principles such as DRY and KISS.

In my next article, we shall go deeper into the technical bits covering a practical demonstration of how to use Stimulus JS in your Ruby on Rails applications.

That wraps up this introduction to Stimulus. I hope you found it insightful, and I look forward to sharing a hands-on guide in the next article.

Mastering Code Reviews: The Ultimate How-To Series

Nemwel Boniface — Mon, 28 Aug 2023 10:43:29 +0000

Introduction

Hello friends!

Picture this, assigned a certain amount of features to implement. how will you know that the code submitted by them is functioning according to the given project requirements? You check it! This is where the code review process kicks in.

This will be a multiple series where I aim to teach you all you need to know about code reviewing. In this article, I will teach you what we mean by code review, why code review is important, what are the qualities of a good code review, what makes up a good code reviewer, list down some drawbacks associated with the code review process, and wrap it up with a conclusion. In a later article, I will do a hands-on tutorial on how to make use of the GitHub review features available to perform a code review.

At the end of the series, you will be more equipped to start making reviews for your peers and add code reviewing as a skill under your belt.

Sounds interesting right? Let's dive straight in!

What do we mean by Code review?

Code review in software engineering means the systematic, collaborative, and iterative process in which one or more developers examine code written by team members scanning for errors in the code to assess whether that submitted code does exactly what it is supposed to do as per the provided project requirements.

Code review is collaborative since it involves at least two people, the author of the code and another person who reviews the code. It is systematic since to perform a meaningful code review, there needs to be a set standard followed to approach the code review process. It is also iterative since if there are errors found, the code will be sent back for revisions and when done put back into review. This process is repeated until the code satisfies the provided project requirements.

Why is the code review process important?

Code review is one of the most important steps in the software development lifecycle. So what are the advantages that you will accrue from doing code reviews?

For your junior developers in your team, code review facilitates the transfer of knowledge and helps them to learn faster. This is because as they collaboratively receive feedback on their code from their peers or the seniors, they can learn new ways of doing something or why implementing features differently is better.
It serves as a bug detection avenue that might have been missed by the written tests for the project. This is because the code reviewer will generally be someone who is knowledgeable and has some understanding of the future implications of not re-writing a code block and rectifying it during the code review process.
We often have the "Make it work optimize later". Code review helps to eliminate code that is unreadable or unnecessarily complex making it unmaintainable. It is here where that complex code will be rewritten when you are provided with suggestions of how to better approach an issue. This will in turn make the code readerable and maintainable.
Code review ensures consistency of the whole code base. This is because different organizations have different ways in which they structure their code. This is always enforced during the code review process where any code that is not up to the company's standard is returned to the author and changes are required.
Code review enhances the interaction and collaboration of the different team members. This is because different people might be the ones to review your code and in this way, you get to interact with different people within the organization.

What are the qualities of a good code review?

We have to acknowledge that anyone can give a code review. However, we have to ask ourselves, was the review given of quality? Below are some of the pointers for a good code review:

One that is provided promptly. It would not be good if you had to wait for a long time to receive a review of your code. Remember that time is of the essence when creating software and you would be required to also start working on another feature which requires that this feature be approved.
One that is thorough and does not miss some of the important aspects. It would hurt if you received few changes to make in the first review and when you submit your code for a re-review, the code reviewer introduces new issues that were not mentioned in the first review. One that is constructive and does not deviate from the project requirements. The reviewer needs to raise issues in the code that are actionable and not guessed issues.
One that uses respective language and tone and acknowledges your efforts so far. We all make mistakes and the best you can do is to be respectful to someone when you are suggesting changes. This would make the one who is to make the changes respected and acknowledged and will make the changes with enthusiasm.
One that offers a learning opportunity. This depends on the kind of changes that will be requested and the approach that will be used by the reviewer which will either help the junior developer learn by offering constructive feedback or break them by adding unrealistic changes to be made.

What makes up a good code reviewer?

We can all see why code reviews are very important and by now, you might be itching to get your hands dirty reviewing your friend's code. But wait! Do you have the qualities required to become a good code reviewer? Let's see if you possess any of the below qualities. A good code reviewer is one who:

Has strong communication skills. Remember that when performing a code review, you will be explaining to someone, "This code is great, but it has some issues. Here is a suggested way to fix it, and here is why you should consider making this change".
Has a strong technical understanding of the underlying technologies being used. You cannot make a code review for something that you do not understand and this is why to become a good code reviewer, you need to have mastery of the underlying technologies to be used.
Is empathetic and patient. The truth is making changes to your code is hard. Most of the time, the author might not be able to implement exactly how it is supposed to be implemented and this is where being empathetic and patient comes into play. You will be better able to accommodate everyone even the least experienced developers in your team.
Is one who can think critically. Making a code review involves assessing the code. It is up to you as the code reviewer to assess whether this submitted code is optimum or might require changes. This is where the skill of critical thinking comes into play.

What are some drawbacks associated with the code review process?

While it is evident that the code review process does have many advantages, it is not always bliss, and like everything, there are drawbacks related to performing code reviews. They include:

Sometimes the code review process can be very time-consuming. This is based on the fact that there could be multiple revisions which take time to both implement changes and re-review.
The code review process could lead to conflicts between team members. When one of the team members feels that the other party is intentionally requesting changes even though the project works perfectly fine that could lead to a lot of conflicts between team members.
Sometimes the code review process could be subjective and might not be objective. Remember that the code review process relies on what someone thinks about a specific code and might not be as objective as someone would want it to be.
With the possibility of making numerous revisions of the code, this would significantly slow down the development lifecycle making it take longer to implement features than they should have taken. This has a direct effect on cost since the longer a project takes to be completed the higher the cost involved in its development.

Conclusion.

From my discussion above, we can see why code reviews are important. We also flipped the other side of the coin and saw the cons that come with the code review process. Would you be willing to do a code review for your peer or suggest it to your team members? Please let me know in the comments section bel

This brings us to the end of the first part of this series. After reading this article, are you able to define what code review is, what are the advantages of performing code reviews, how you can distinguish whether the code review you received was good or not, and for those who are already excited about code reviews, the qualities that would help make them good code reviewers?

Part B of this series will include a practical tutorial teaching you how to perform reviews making use of the GitHub review features. Stay tuned!

If you read till this point, I send a big thank you.

That is it for today's class. I hope this was useful information for you. See you in my next article.

Mastering Technical Writing: A Step-by-Step Guide for Beginners

Nemwel Boniface — Sun, 18 Jun 2023 16:43:03 +0000

Introduction

Hello my esteemed reader! This article is well suited for beginners in technical writing but will still benefit seasonal writers who would like to take their writing skills to the next level. This will be a multiple series where I aim to take you from point A where you know nothing about technical writing to point Z where you write meaningful articles that people actually want to read.

In this short article, I will explain what technical articles are, discuss the characteristics of a good article, extensively explore steps that you can take to write a good article, and conclude with a summary as I answer the question, "are technical articles limited to software related articles alone?".

What do we mean by "Technical articles"?

Technical writing. A huge word that most people get wrong and only associate with software-related articles and publications. However, today I am here to demystify this misconception and explain to you my reader why technical writing is not restricted to software-related articles alone.

For us to understand what this is, we need to have an understanding of what the two terms technical and writing mean. Technical means "relating to a particular subject, art, or craft, or its techniques." anywhere mentioning software? no. The term Writing encompasses the process of documenting ideas, thoughts, or facts. Therefore combining the two, technical writing is the process of conveying complex and specialized information clearly and concisely.

What are the indicators of a good article?

Have you ever read an article and something about it spelled out quality? Well, quality is one subjective term. However, in technical writing, we have standardized indicators of what a quality article is that I will list below:

An article with a purpose and has a target audience. This is an article that clearly defines the scope of the specific article that is being written and has an identified audience beforehand.
An article that combines the written text with visual aids such as images, gifs, and screenshots to further explain a specific concept shedding more light on it by using visuals to arouse more interest.
An article that makes use of relatable examples and analogies. Research has it that when hard concepts are accompanied with an analogy that the readers can relate to, they end up remembering that concept by relating it to the analogy that was used to explain it.
An article that is consistent throughout and has accurate information. Nothing hurts more than an article whose flow seems inconsistent and whose content is misleading.
An article that has been proofread and is free of grammatical and spelling errors.

What are the steps to follow when writing an article?

Anyone can write an article. However, not everyone is able to write a good article. Luckily enough with the right preparation, we can all be able to write resourceful artiles and this will be what will separate your article from the vast majority of sub standard articles published online following the steps highlighted below:

Preparation - they say, given 10 hours to cut down a tree, use 7 hours of that time sharpening your axe and 3 hours to cut it down. This is the step where you get to understand what you are going to write it and why you actually need to write it. It is in this stage where you choose the medium which you will use to relay your thoughts through your article.
Research - before you even start to write about an article, you need to get an understanding of the topic at hand and do an assessment of whether this is something that you are capable of writing about. Here is where you get your hands dirty busy going deep that topic/ domain to get an understanding of it before writing about it.
Organization - with extensive research, comes a monolith of data that will not make sense as information. You need to sit back, organize your thoughts and see what bits of all this raw data combined will lead to a meaningful information for your article.
Once you have your thoughts organized, Writing the article is the next logical step. It is in this step where you jot down your thoughts either on paper or on an online editor that offers grammar and spelling checks such as Grammarly.
The last and final step is to proofread your work. In this step I advice sitting on your article for a few days and asking a friend to proof check your work and this will allow you to have a third eye/ opinion that will help you see if your article has a good thought/ logical flow.

Conclusion

From the steps listed above, it is evident that technical articles are not restricted to software related articles alone. A technical article can be any article written for a specific domain, be it hospitality, engineering or medicine all qualify as technical articles. Therefore when someone says that other articles are not technical articles, please direct them to this article where they will also get enlightened.

This brings us to the end of the first part of this series. After reading this article, are you able to recognize a well researched and written article? Please do let me know in the comment section. In the next article, I shall walk you through how you can develop an appropriate writing style in technical writing and how you can structure your article.

If you read till this point, I send a big thank you.

That is it for today's class. I hope this was useful information for you. See you in my next article.

Implementing Custom Slugs with FriendlyID in Rails: A Step-by-Step Technical Guide

Nemwel Boniface — Mon, 29 May 2023 07:38:21 +0000

Introduction:

Have you ever wondered how applications like Instagram or Twitter use your username as the unique identifier in your browser URL and not the ID of your account?

In the development of applications, the use of eg usernames instead of the ID is called slugging. In programming, slugs are the human readable unique identifying parts of a web address typically at the end of the URL (Uniform Resource Locator).

Using custom slugs for our applications offers many advantages that I will discuss later in the article. Implementing them in our applications is relatively easy and luckily for us who use Ruby on Rails, we get the advantages of using a mature stable framework where many functionalities have solutions that were already implemented for us.

In Ruby on Rails, we have a gem called friendly_id that makes the process of generating human-readable slugs simple by allowing us to replace ids in our URLs with slugs. Using friendly_id gem lets our applications work with friendly URLs eg https://www.my_app/users/nemwelboniface instead of https://www.my_app/users/1.

Typically, the implementation of custom slugs in Rails involves leveraging gems like FriendlyID that streamline the process and offer additional functionalities like resolving slug conflicts and automatically generating easily readable slugs.

Within this article, we shall delve into a comprehensive step-by-step walk through that demonstrates how to implement custom slugs in Rails with the assistance of the FriendlyID gem. My guide will encompass essential aspects such as installation, configuration, migration, and code generation, enabling a seamless integration of custom slugs into your Rails application. By the conclusion of this tutorial, you will possess the necessary knowledge and tools to craft personalized and user-friendly URLs tailored to meet your specific needs.

Let's dive in and unlock the power of custom slugs in Rails!

Prerequisites:

Rails V7
PostgreSQL database
Ruby V 3+
Visual studio code
Some experience with Rails MVC architecture.

Project set-Up

To set up the custom slugs functionality in our application, we need to add the friendly_id gem to our Gemfile and run bundle install.

gem "friendly_id"
bundle install

If you would like to follow along, I created a new branch called friendly_id in this repository where you can follow along with me. If you would like to know how to create a new Rails application, kindly have a look at the first part of this article I wrote.

Our next step will involve adding a slug column to our desired table. In our case, it will be the Users table. Therefore we shall run the following line in our terminal which will create a migration file that adds a slug column to our Users table.

rails g migration AddSlugToUsers slug:uniq

We shall also need to generate the friendly configuration file and a new migration with the following command:

rails generate friendly_id

We are now free to run the migrations to update the schema file with the new columns added by the migration files created.

rails db:migrate

We are going to head over to our Users model inside app/models/user.rb file and add the following:

class User < ApplicationRecord
  extend FriendlyId
  friendly_id :name, use: :slugged
end

With the line for "extend FriendlyId" we are extending the functionalities of the User model with the methods that are provided by the FriendlyId module. This is what allows us to use the "friendly_id" method in the next line.

With the line "friendly_id :name, use: :slugged" this is the line that configures the friendly_id gem for the user model. This is where we specify that the name attribute of the user model should be used as the basis for generating the slugs. The :slugged option indicates that the friendly_id gem should use the slugged module to generate the slug.

Making use of the above means that when a new user record is being created, the friendly_id gem will generate a slug based on the name attribute. The slug is then stored inside the slug column in our user's table.

In addition to the above, let us head back to our user's model inside app/models/user.rb file and create a to_param method. Our User model shall now be like this with all the changes made:

class User < ApplicationRecord
  extend FriendlyId
  friendly_id :name, use: :slugged

  def to_param
    slug
  end
end

This method is necessary when using the friendly_id gem to ensure that our slug is used in the URL instead of the default ID. What it does is overrides the default behavior of Rails when generating URL's for model instances.

By defining the to_param method and returning the slug value, you are instructing Rails to use the slug attribute when generating the URL's for instances of our model.

We are almost done now. We need to make some slight adjustments to our controllers. Head over to our users controller in app/controllers/users_controller.rb file and inside the show action and set_user method, replace this:

@user = User.find(params[:id])

with:

@user = User.friendly.find(params[:id])

What is happening here is we are using the friendly_id gem to find a user's record based on its slug. The params[:id] value is passed in as the argument to the find method. Basically using the friendly.find, the user record will be found based on its slug attribute instead of the ID which is the default behavior of ActiveRecord.

The last step involves running the following in our Rails console:

User.find_each(&:save)

Note that running the above line is only necessary if you're adding friendly_id to an already existing Rails application and you need to generate slugs for already existing users.

What that line does is it iterates over each user's record and calls the save method on them. This triggers the saving process and generation of slugs for each user based on the friendly_id configuration in our user model.

This step is not necessary if you setup friendly_id before creating users in your application. If you find having to run the line in your console application tedius, you have an option of creating a rake task that can be used to automate that process for you. However, this is beyond the scope of what we were to learn in this article. I will however consider writing an article about rake tasks upon request from someone in the comments section.

Pros and cons of using custom slugs

We have covered the technical setup and I believe that so far you have an idea of why having custom slugs in your application is a good thing. Below I will list a few advantages that I have found from using custom slugs in my own applications:

It has helped me to improve the SEO of my applications ten fold as search engines tend to give more weight to keywords in the URL when determining the relevance of a page.
Using custom slugs gives you a human readable URL that gives more context and describes what thet specific webpage you are viewing contains. This rings true especially when creating blogs where you can have th title of a blog as the slug for a blogs show page.
Using custom slugs has given my applications alot of flexibility as the slugs are more flexible and meaningful than numeric ID's. Having keywords in the slugs has really helped users of my applications to identify and remember the contents of webpages that they visited from my application.
Persistence and a sense of security in my application as unline numerical IDs are sequential and predictable by revealing th order in which records were created, having the custom slugs give me a sense of security as they remain consistent even if the record is moved or re-ordered in the database.

Despite using slugs showing real tangible benefits to our applications, they also have some serious considerations which are worth factoring when using custom slugs over the default ID's. You need to ensure that the slugs are unique, they can handle situations where the title or the attribute that is used for the slug is changed and also properly handling URL redirections when slugs are modified.

Luckily for us, in Rails all the above concerns are already handled by the friendly_id gem where the process of handling these edge case is simplified. This is one of the many advantages we get from using a mature and stable framework like Ruby on Rails for development of our applications.

Conclusion

Would you like to use friendly_id in your applications after reading my short article? Please let me know in the comments section below.

This one marks the end of the Step-by-Step Guide: Implementing Custom Slugs with FriendlyId in Rails article guide.

That is it for today's class. I hope this was useful information for you. See you in my next article.

DEV Community: Nemwel Boniface

Best Open-Source OCR and Document Parsing Models for Financial Tables (2026)

Introduction: The Bracket That Lied

Wait. Do You Even Need OCR?

Two Terms You'll See a Lot (and Why They Matter)

How I Picked and Ordered These Models

1. Docling: The One You Can Explain to an Auditor

2. granite-docling-258M: Tiny Model, Serious Receipts

3. MinerU2.5-Pro: For Notes That Refuse to Fit on One Page

4. PaddleOCR-VL-1.5: For Scans That Have Seen Things

5. GLM-OCR: Neck and Neck With PaddleOCR

6. dots.ocr: Every Number Comes With a Home Address

7. LightOnOCR-2-1B: The Bouncer at the Door

Didn't Make the Cut: olmOCR-2

So, Why Not Just Pay for a Hosted API?

Running the Bake-Off Without Losing Your Weekend

Don't Trust Your Eyes: Score the Cells

Summary and a Useful Cheat Sheet

What This List Doesn't Cover

Conclusion

Final Thoughts & Conversation

Best AI Inference Platforms for Financial Document Processing in 2026: Why Superlinked SIE Tops My List

Introduction: Turning Clean Markup Into Pixels (and Back Again)

Before You Pick a Platform: Do You Even Need Document AI?

The Scorecard: Four Questions That Actually Matter

1. Superlinked Inference Engine (SIE): The Whole Chain, One Client

Coverage of the Chain

Swapping Models Is a String Change

Costing It Before You Run It

Compliance

A Quick Reality Check on Per-Page Pricing

The Rest of the Field

2. Azure AI Document Intelligence

3. Google Cloud Document AI

4. Amazon Textract (with Bedrock)

5. Reducto

6. Unstructured

7. Baseten

Summary and a Useful Cheat Sheet

What I'd Tell My Past Self

Frequently Asked Questions

Do I need document AI for 10-K and 10-Q filings?

Why not just keep everything in Azure?

What about IFRS filers and data that can't leave our environment?

Conclusion

Final Thoughts & Conversation

Silent Data Loss in Celery + RabbitMQ on Azure Container Apps: A Queue Registration Post-Mortem

Introduction

The Architecture, Briefly

The Symptom

Chasing Ghosts: What We Ruled Out

The Breakthrough: Ask the Container What It's Actually Running

Root Cause: The Silent Platform-Level Override

Data Loss and the Eventual Consistency Trap

What We'd Do Differently

Why This Bug Was So Difficult to Find

Conclusion

Final Thoughts & Conversation

Rails Authorization Beyond Models: Securing Dashboards and Service Controllers with CanCanCan

Introduction - Authentication and Authorization. What are these?

The Foundation: The Rulebook (Ability.rb)

The "Secure by Default" Mindset

Scenario 1: The Standard CRUD Resource (This is the Happy Path)

Scenario 2: The "Service" Controller (When there is no Database Model)

Scenario 3: The Complex or Expensive Action

Summary and a useful Cheat Sheet

Conclusion

What Happens When a User Deletes Their Account? A Guide to Rails ActiveRecord dependent: Strategies

Introduction

1. The Clean Sweep: dependent: :destroy

2. The Bulk Cleanup: dependent: :delete_all

3. The "Legacy" Handover: dependent: :nullify

4. The Soft Block: dependent: :restrict_with_error

5. The Hard Guard: dependent: :restrict_with_exception

Short Recap

Conclusion

Architecture Backwards: Engineering a Self-Defending System Before the UI Arrives

Introduction

1. The "Horizontal" Nature of Users

1.1 The Tortoise and the API

1. The Clean Sweep: `dependent: :destroy`

2. The Bulk Cleanup: `dependent: :delete_all`

3. The "Legacy" Handover: `dependent: :nullify`

4. The Soft Block: `dependent: :restrict_with_error`

5. The Hard Guard: `dependent: :restrict_with_exception`