DEV Community: Aryan Saxena

One judge gave everything a 4: building a hackathon judging portal that survives its own data

Aryan Saxena — Fri, 02 Oct 2026 11:08:52 +0000

Thirty judges. 126 ballots. One judge, jdg_07, scored every criterion of every project a 4.

Standard deviation: exactly 0.

That one judge is why a hackathon portal of mine ended up with a ridge-regression normalizer. This is my write-up for the Hackathon Raptors Dogfood challenge: every team builds the same product, the portal Raptors would run its own events on, in 72 hours. I built Raptors, a self-hosted submission and judging portal (FastAPI, SQLite, server-rendered Jinja), and most of the interesting decisions in it were forced by the data.

TL;DR

Built Raptors, a self-hosted hackathon submission and judging portal (FastAPI, SQLite, server-rendered Jinja) for the Hackathon Raptors Dogfood challenge.

A judge who scores everything 4 breaks the textbook per-judge z-score, so I built a two-stage calibration: shrunken scale, then a ridge additive model.

Calibration cut the spread between judges by 40.7%, but I also show what that does not prove, and how sensitive the result is to my parameter choices.

I claimed only the tiers the acceptance script can verify: T1 and T2, 7/7 PASS.

Why one flat judge matters: a plain average treats his 4s as real evidence, and on a small event a few judges like him decide the podium. So calibrating judges stopped being a nice-to-have and became the core product requirement. The rest of this post is how I got there, and just as much, what I can and can't claim about it.

Repo: AryanSaxenaa/raptors (MIT). docker compose up --build gives you a seeded portal.

Read the grader before writing a line

The brief ships a spec, a fixture file and an acceptance script, run.py. Before building anything, I read all four closely. INTEL.md in the repo is a forensic reading of all four, written before any implementation code. PLAN.md records each engineering decision in the same shape: what most teams will do, why that is weak, what we do instead.

Reading the grader first paid for itself immediately. These are the traps the obvious build walks into:

The one worth repeating outside this challenge is the permission check. A hidden resource (404) or an empty list (200) is not a denial. The peer-scores endpoint returns a real RFC 9457 peer_scores_denied with a 403 and writes a row to the audit log. The rest are specific to this grader: the gallery renders all 41 projects in arrival order, because the suite only reads page one.

The architecture decision that shaped everything

The suite greps the raw HTTP bytes of the gallery. A client-rendered gallery looks perfect in a browser and fails that check, so the gallery is server-rendered Jinja. That one constraint set the shape of the whole app:

Public reads are HTML. Every write is JSON through /api/*, with no second, privileged form-post path. If the UI can do something, curl can do it through the same endpoint.
Auth is one opaque token (df_session cookie or Bearer), because the suite attaches a cookie and never logs in.
One process, one SQLite file in WAL mode. No Redis, no Postgres. I wrote down when I would move to Postgres: measured need, not because it sounds production-grade.

The Docker gotcha: a port that answers on an empty database

If the server binds its socket before it seeds the database, there is a window where the app answers requests with no data in it, which is how a gallery check fails on a perfectly healthy container.

So bootstrap (schema and fixtures) runs before the socket binds, and /api/health doesn't mean "process alive". It reports seed counts and requires them to match the fixtures and the schema_meta.version:

{"ok":true,"schema_version":"3","schema_ok":true,
 "counts":{"projects":41,"judges":30,"scores":126},
 "fixture_counts":{"projects":41,"scores":126}}

The scoring algorithm that doesn't survive the data

The obvious approach is a mean per project. On these fixtures, it is vulnerable to large judge effects:

The highest judge mean is 4.22. The lowest observed judge mean is 2.00, from jdg_01, who submitted only one ballot, so it says little about how harsh that judge really is.
Loads run from 1 ballot to 11 per judge.
Eight projects got 2 reviews and four got 5.

A plain mean treats a generous 4 and a harsh 4 as the same evidence. The textbook fix is a per-judge z-score, and it breaks twice. It divides by zero on jdg_07. For a single-ballot judge the spread is undefined, so an implementation has to choose a fallback. If that fallback is a zero spread, the normalized score becomes 0.0 and the observation contributes no discriminative information. It is also poorly suited to this assignment structure, because judges were restricted by track and so did not necessarily score comparable samples.

What shipped is two stages in pure Python with no solver:

Shrunken scale. Each judge's spread is blended with the pooled spread, weighted by their ballot count: σ̃ = (nσ + kσ_pool) / (n + k), with k = 5. At k = 5, a judge needs more than five ballots before their own spread gets more weight than the pooled spread. As long as the pooled spread is nonzero, σ̃ is nonzero, so a flat judge like jdg_07 cannot cause a divide-by-zero (the code falls back to a scale factor of 1 if every ballot in an event were identical). Each judge's scores are then rescaled around a shrunken mean. For jdg_07 that gives every project they scored the same value, so their ballots say nothing about which of those projects is better. That is a choice I made: a judge whose scores contain no within-judge variation doesn't get to discriminate between projects. It doesn't mean their ballots have no effect on the ranking.
Ridge additive model. score = μ + project effect + judge bias, fitted by alternating least squares on the cells that exist. The penalty is the same for every judge, but the data's pull against it is not. With 11 ballots, the data can overcome the judge penalty. With one ballot, the penalty pulls that judge's bias strongly toward zero, because one observation cannot separate "generous" from "good project".

On the published fixtures, the spread between judges falls from 0.4127 to 0.2449, a 40.7% reduction:

But is the new ranking better, or just different?

This is the question that matters, and a smaller spread alone doesn't answer it. 29 of the 40 ranked projects change position after calibration. Small Meadow and Flat Meadow each drop 9 places and Flat Relay rises 6:

So I fit the same estimator on a labelled synthetic panel: the same (judge, project) cells as the fixtures, scores generated from planted project effects and judge biases. Because the truth is known, I can score the recovery directly:

Check on the planted panel	Result
Correlation, recovered vs true project effect	0.872
Correlation, recovered vs true judge bias	0.963
Kendall τ vs true ranking, raw means	0.649
Kendall τ vs true ranking, calibrated	0.671

The gain on that one planted panel is modest (τ from 0.649 to 0.671), but one panel is one random draw. So I repeated the experiment over 200 random seeds, with the same missing cells as the fixtures. Treat this as a simulation check that the estimator recovers what was planted, not as empirical validation on real judging data:

Synthetic panel, 200 seeds	Mean τ, raw mean	Mean τ, calibrated	Calibrated better in
Matches the model's assumptions	0.540	0.698	199 of 200 seeds
Judges apply different scales to project effects	0.549	0.693	198 of 200
Project effects include a nonlinear quadratic term	0.512	0.663	199 of 200

The single panel above understated the gain. Across the three scenarios, the mean τ improvement ranges from 0.144 to 0.158. These are three separate scenario results, not one pooled estimate. I still treat this carefully, because the planted panels are variations on the same additive idea the estimator assumes.

What this does not prove

Calibration is not ground truth. No one knows the true best project of a real event, and the planted-data test only shows the method recovers effects I planted.
The synthetic panels are friendly to the estimator. The judge-scale and quadratic variants are mild misspecifications, not an adversarial test.
The 40.7% figure measures spread, not accuracy. It is a drop in how spread out the judges' averages are, and says nothing about whether the ranking is more accurate.
Identifiability is limited. Assignment is restricted by track and isn't random, so judge and project effects are only weakly separated for judges with few ballots. That is why the portal flags any project with fewer than 3 reviews as low confidence, and prints a rough standard error next to each score (the model's overall residual spread divided by the square root of the project's review count). It is a warning signal, not a confidence interval.
k = 5 and the ridge penalties are design choices, not discovered constants. They come from reasoning, not a search. At k = 5, a judge needs more than five ballots before their own spread gets more weight than the pooled spread (the median load is 3). The judge penalty is larger than the project penalty because a judge with few ballots is confounded with whatever they happened to be assigned.

Here is how much the shrinkage constant and the ridge penalties matter on the real fixtures (τ against the default calibrated ranking):

Setting	τ vs default	Top 3 unchanged	Top 10 overlap
shrinkage k = 1	0.938	yes	10 of 10
k = 3	0.972	yes	10 of 10
k = 5 (default)	1.000	yes	10 of 10
k = 10	0.982	yes	10 of 10
k = 20	0.959	yes	9 of 10
judge penalty 0.5	0.903	yes	10 of 10
judge penalty 1	0.946	yes	10 of 10
judge penalty 3 (default)	1.000	yes	10 of 10
judge penalty 12	0.954	yes	10 of 10
project penalty 0.25	0.990	yes	10 of 10
project penalty 1 (default)	1.000	yes	10 of 10
project penalty 3	0.977	yes	10 of 10

Across the parameter values I tested, the top 3 never changes, and the top 10 changes by one project at most (at k = 20 the overlap is 9 of 10). That is evidence about these values, not a proof that the ranking is invariant to the parameters. The middle of the table does move a little. A lighter judge penalty actually scores better on the synthetic panels (mean τ 0.728 at 0.5, against 0.698 at 3), so the default isn't the optimum among the tested synthetic penalty values. I kept the heavier penalty because of the confounding argument above, and I'd rather say so than pretend the number was optimised.

What it changes for an organizer

On the real fixtures, calibration swaps the top two and puts a different project on the podium.

prj_37 replaces prj_10 in the top 3, and the top-10 membership is the same.
Eight projects move five or more places, and 29 of 40 move at all.
Dropping jdg_07's three ballots entirely changes the order of 18 projects, by up to 10 places (τ 0.941 against the full ranking), with the same top 3. This also changes how many reviews some projects have, so it is not a pure measure of that judge's influence, but it does show that a flat judge still moves ranks through the ridge stage even though they can't tell their projects apart.
With prj_41 included in the ranking, the duplicate would rank 8th, a top-10 spot. With the duplicate excluded, as shipped, the original prj_07 ranks 27th.

Reproduce the proof with python -m dogfood.normalize --proof --fixtures fixtures.json, and the tables above with the script at the end of this post. The repo was frozen for judging before I wrote the sensitivity analysis, so the script isn't in it.

The part of the spec I thought was simple: "judges can't see each other's scores"

The deadline

"Closed events refuse submissions" sounds like one if. But if you validate the body first, a malformed payload gets a 422 instead of "closed". The deadline check now runs before the JSON is parsed, and tests/test_deadline.py throws malformed JSON at a closed event to prove it.

The UI isn't the control

The role matrix is 5 actors by 5 capabilities, 25 cells. The brief's checker probes three. All 25 are enforced in one ROLE_MATRIX in security.py and walked over HTTP in tests/test_role_matrix.py. Every denial is a problem document with a machine-readable code, and it writes an authorization.denied row to a hash-chained, append-only audit log. Hiding a button isn't isolation. curl is:

$ curl -i -H "Cookie: df_session=<judge_b>" "localhost:8080/api/judge/scores?judge=jdg_01"
HTTP/1.1 403 Forbidden
content-type: application/problem+json
{"code":"peer_scores_denied","status":403,
 "detail":"scores belonging to judge jdg_01 are not visible to you"}

The duplicate that took 4 of 126 review slots

The fixtures contain a duplicate submission: prj_41, a copy of prj_07 (Dry Harbour). The duplicate submission accounts for 4 of the 126 ballots, and the two copies together account for 9. Three judges (jdg_19, jdg_21 and jdg_26) reviewed both copies.

The rule is that the duplicate is flagged, excluded from the ranking, and its ballots still count for calibration. Dropping them would bias the judges who saw both copies. The assignment algorithm also skips known duplicates when topping up reviews, so scarce judge attention isn't spent on something that won't be ranked.

What I refused to claim

The brief lets you claim a tier only if you can prove it, and says that overclaiming is the one thing that costs points. run.py verifies T1 and T2, so .dogfood.toml claims T1 and T2, and the report closes with claimed T1 T2, verified T1 T2. Quadratic community voting, signed records, an embeddable gallery, bulk import/export and the full API (T3 and T4) are built, but I didn't claim them because the checker can't verify them. I also skipped pairwise judging on purpose: the brief says to pick one hard bonus and nail it, so I picked normalization.

How I worked

I used Cursor for the implementation, under rules I set: the spec is the contract, run.py is the grader, the reading of the checks goes in INTEL.md and PLAN.md before any application code, and everything is verified over HTTP. I claimed T1 and T2 only, and the report reads 7/7 PASS. About 80 pytest cases pass (25 of them are the role matrix), plus a black-box smoke suite of about 190 checks.

Audits and tests caught the real late bugs, and each fix has a test: the open and email voting modes returned 403 because the ballot endpoints still demanded a judge permission, an organizer could publish results while voting was still open, a participant could move a project onto another event's track, and the default gallery mixed two events.

What I took from it

The hard part of a judging portal isn't accepting scores. It's deciding what those scores are allowed to mean when the data is sparse, biased, duplicated, probed by a script, and graded by software. Every decision I'm proud of in this project is about that question, and the ones I'm least sure of, such as k = 5, I've tried to show rather than hide.

Try it

git clone https://github.com/AryanSaxenaa/raptors
cd raptors
docker compose up --build
# in another terminal
python run.py .dogfood.toml

Walkthrough

A silent 80-second walkthrough of the portal, with background music:

Appendix: the sensitivity script

Save it as sensitivity.py in the root of a clone of the repo and run python sensitivity.py. It only reads fixtures.json and imports dogfood.normalize.

sensitivity.py

"""Sensitivity of the calibrated ranking to k_shrink and the ridge penalties.
Save this file in the root of a clone of the repo and run:  python sensitivity.py
"""
import json, random, statistics, sys
from pathlib import Path
ROOT = Path(__file__).resolve().parent
sys.path.insert(0, str(ROOT / 'src'))
from dogfood import normalize as N
fx=json.load(open(ROOT / 'fixtures.json'))
scores=fx['scores']
keys=sorted({k for s in scores for k in s['criteria']}); w={k:1.0 for k in keys}
obs=[N.Observation(s['judge'],s['project'],N.weighted_value(s['criteria'],w)) for s in scores]
pids=[p['id'] for p in fx['projects']]
EXCL={'prj_41'}
def rank(res,method):
    r=[p for p in res.projects if not p.excluded and p.raw is not None]
    r.sort(key=lambda p:(-(p.score(method)),p.project_id)); return [p.project_id for p in r]
base=N.compute(obs,all_project_ids=pids,excluded_project_ids=EXCL)
raw=rank(base,'raw'); cal=rank(base,'additive_ridge')
print('ranked',len(cal))
print('top3 raw',raw[:3],'cal',cal[:3],'same set top3',set(raw[:3])==set(cal[:3]))
for k in (3,5,10):
    print('top',k,'overlap',len(set(raw[:k])&set(cal[:k])),'of',k)
pos=lambda l:{x:i for i,x in enumerate(l)}
pr,pc=pos(raw),pos(cal)
print('moved>=5 places',sum(abs(pr[x]-pc[x])>=5 for x in cal),'moved>=1',sum(pr[x]!=pc[x] for x in cal))
# duplicate check: does excluding vs including change top 10
inc=N.compute(obs,all_project_ids=pids,excluded_project_ids=())
ci=rank(inc,'additive_ridge'); print('dup rank if included', ci.index('prj_41')+1 if 'prj_41' in ci else None, 'prj_07 rank', cal.index('prj_07')+1)
def tau(a,b):
    ids=[x for x in a if x in set(b)]
    return N._kendall_tau(N._ranks([-pos(a)[x] for x in ids],ids=ids),N._ranks([-pos(b)[x] for x in ids],ids=ids))
print('--- sensitivity on real fixtures: tau vs default calibrated ranking')
for k in (1,3,5,10,20):
    r=rank(N.compute(obs,all_project_ids=pids,excluded_project_ids=EXCL,k_shrink=k),'additive_ridge'); print('k',k,round(tau(cal,r),3),'top3',r[:3]==cal[:3],'top10 overlap',len(set(r[:10])&set(cal[:10])))
for lj in (0.5,1,3,6,12):
    r=rank(N.compute(obs,all_project_ids=pids,excluded_project_ids=EXCL,lambda_judge=lj),'additive_ridge'); print('lambda_judge',lj,round(tau(cal,r),3),'top3',r[:3]==cal[:3],'top10 overlap',len(set(r[:10])&set(cal[:10])))
for lp in (0.25,1,3):
    r=rank(N.compute(obs,all_project_ids=pids,excluded_project_ids=EXCL,lambda_project=lp),'additive_ridge'); print('lambda_project',lp,round(tau(cal,r),3),'top3',r[:3]==cal[:3],'top10 overlap',len(set(r[:10])&set(cal[:10])))
print('tau raw vs default',round(tau(cal,raw),3))
# synthetic multi-seed, well-specified and misspecified
pj=sorted({s['project'] for s in scores}); jd=sorted({s['judge'] for s in scores})
def synth(seed,mode,k=5,lj=3,lp=1):
    rng=random.Random(seed)
    P={p:rng.gauss(0,.55) for p in pj}; B={j:rng.gauss(0,.8) for j in jd}
    S={j:(rng.uniform(.5,1.6) if mode=='scale' else 1) for j in jd}  # per-judge scale
    o=[]
    for s in scores:
        j,p=s['judge'],s['project']
        if mode=='scale': v=3.5+B[j]+S[j]*P[p]+rng.gauss(0,.3)
        elif mode=='skew': v=3.5+B[j]+P[p]+rng.gauss(0,.3)+ (0.5*P[p]**2)
        else: v=3.5+B[j]+P[p]+rng.gauss(0,.3)
        o.append(N.Observation(j,p,min(5,max(1,v))))
    res=N.compute(o,all_project_ids=pj,k_shrink=k,lambda_judge=lj,lambda_project=lp)
    sc=[x for x in res.projects if x.raw is not None]; ids=[x.project_id for x in sc]
    tp=N._ranks([P[i] for i in ids],ids=ids)
    return N._kendall_tau(tp,N._ranks([float(x.raw) for x in sc],ids=ids)), N._kendall_tau(tp,N._ranks([float(x.additive_ridge) for x in sc],ids=ids))
for mode in ('well-specified','scale','skew'):
    a=[synth(s,mode) for s in range(200)]
    print(mode,'raw tau mean',round(statistics.fmean(x[0] for x in a),3),'cal tau mean',round(statistics.fmean(x[1] for x in a),3),'cal>raw in',sum(x[1]>x[0] for x in a),'/200 seeds')
print('--- synthetic (well-specified, 200 seeds) by params')
for k,lj in [(1,3),(5,3),(10,3),(5,0.5),(5,12)]:
    a=[synth(s,'well-specified',k=k,lj=lj) for s in range(200)]
    print('k',k,'lj',lj,'cal tau',round(statistics.fmean(x[1] for x in a),3),'raw',round(statistics.fmean(x[0] for x in a),3))

Built for DOGFOOD 2026 by Hackathon Raptors.

Porting price-parser from Python to Rust: the hard part was proving it still lied the same way

Aryan Saxena — Wed, 12 Aug 2026 16:10:05 +0000

I went into this port thinking the Rust part would be the hard part.

It was not.

The hard part was taking a small Python library that parses real-world price
strings and proving that the Rust version still behaves like the original,
including the awkward parts I would normally want to clean up.

This was my Code Resurrection 2026 / Port Mortem submission for Track D:
Python to Rust.

Project repo: https://github.com/AryanSaxenaa/PriceMortem

Source repo: https://github.com/scrapinghub/price-parser

What I ported

The source project, scrapinghub/price-parser, extracts price amounts and
currencies from raw text.

Simple example:

from price_parser import Price

Price.fromstring("$12.99")

That looks easy until you remember what price text looks like in the wild:

$12.99
12,99 €
USD 1,299.00
1.299,00 EUR
Free
Price unavailable
-$10
1e3 USD

A lot of parser behavior lives in those edge cases. Some of it is intentional.
Some of it is just what the original library already does.

For the hackathon, I did not want to ship a Rust parser that merely looked
reasonable. I wanted evidence that it matched the Python package.

The architecture I ended up with

The repo has three layers:

src/core.rs                 pure Rust parser logic
src/lib.rs                  PyO3 compatibility adapter
src/main.rs                 standalone Rust CLI
src/currencies_generated.rs source-derived currency table

The important split is between core.rs and lib.rs.

The parser core is Rust. It does not call into Python. The PyO3 layer exists so
the original Python tests can import price_parser.Price without being
rewritten.

That gave me two ways to run the port.

As a standalone Rust CLI:

cargo run -- '$12.99'

Output:

{"amount":"12.99","amount_text":"12.99","currency":"$"}

And as a Python-compatible package for the preserved upstream tests:

maturin build --features extension-module
python -m pytest

This was a useful compromise. The actual parser is in Rust, but the old test
suite can still exercise it through the public Python API.

The test suite was the oracle

The original pytest file is preserved in the port repo:

tests/original/test_price_parsing.py

Its recorded SHA-256 is:

dfb5d2ea7b7aff485ca89becd5a0090dac9f74f9e6b7c3c08a67cbb22e0bad32

That hash matters because editing the source tests would make the proof much
weaker. If a port only passes after the tests are adjusted, the port might be
right, but the evidence is no longer clean.

Current result against the Rust-backed package:

1061 passed, 134 xfailed

The 134 xfailed cases are not hidden failures from the port. They are
expected failures already marked by the upstream test suite. I kept them as-is
instead of rewriting them away.

I also kept native Rust tests for fast feedback:

7 passed, 0 failed

Those tests are useful, but they are not the main proof. The main proof is the
unchanged upstream test file passing against the Rust implementation.

Differential testing caught the things examples do not

Unit tests are good at checking known examples. Parsers need more than that.

I added differential testing that runs the same inputs against:

the original Python package
the Rust port through the compatibility adapter

Then it compares the structured outputs.

The latest 60-second fuzz run:

351 cases, 0 divergences

The fixed differential corpus includes cases across:

currency symbols
ISO currency codes
Unicode spaces
malformed prices
locale-style separators
negative values
scientific notation
empty strings

The useful part of this was not just the final zero-divergence number. It was
the process. Every mismatch forced a decision:

Is the Rust implementation wrong?
Is the source behavior surprising but intentional?
Is this an upstream quirk that the port has to preserve?

For a behavioral port, "I can write this cleaner in Rust" is not automatically
a good reason to change behavior.

What broke

The most annoying issues were around ambiguity.

For example, separators are not just formatting. In one string, a comma may be
a decimal separator. In another, it may be a thousands separator. Sometimes the
currency appears before the amount, sometimes after it, and sometimes the input
contains text that only sort of looks like a price.

Currency matching also had some rough edges. The source has behavior around
currency symbols, ISO codes, and ordering that is easier to preserve once you
treat the source as the oracle instead of trying to redesign the rules from
memory.

The PyO3 boundary added another class of issues. The Rust core naturally wants
to produce Rust data. The original Python API returns Python-facing objects and
decimal.Decimal values. I kept Decimal conversion at the adapter boundary so
the Rust parser stayed independent while Python comparisons still matched what
the original tests expected.

The performance result was not flattering

The PyO3 compatibility path is slower than the source on hot-loop calls.

Current benchmark summary:

Rust/PyO3 adapter total time ratio: 9.76x slower than source
Rust/PyO3 adapter p99 ratio: 7.51x slower than source
Rust/PyO3 startup: 0.669s
Source startup: 1.346s
Rust/PyO3 RSS: 21.27 MB
Source RSS: 21.88 MB

That is not the number I would choose for a marketing page.

But it is the number I got from measuring the public Python-compatible path,
including extension calls and Decimal conversion. Hiding that would make the
submission weaker, not stronger.

The better next benchmark would separate the pure Rust CLI path from the PyO3
adapter path. For this submission, behavioral equivalence mattered more than
making the benchmark look good.

The decision I would take back

I should have added the standalone CLI earlier.

At first, the project leaned heavily on the PyO3 path because that was the best
way to run the original tests unchanged. That made sense for equivalence, but
it made the port look too Python-shaped from the outside.

Adding this helped:

cargo run -- '$12.99'

It shows the parser core can run as Rust without importing Python. The PyO3
adapter is still important, but now it is clearly a test and compatibility
layer, not the whole product.

What I would improve next

If I kept working on this after the hackathon, I would do five things:

Increase Rust core coverage beyond the current line coverage.
Add more fuzz inputs from real scraped ecommerce pages.
Benchmark the pure Rust CLI separately from the Python adapter.
Publish a small Rust crate API around the core parser.
Keep the Python adapter optional for compatibility testing.

Final result

The final port has:

preserved upstream tests with a recorded SHA-256
unchanged pytest parity through a PyO3 adapter
a standalone Rust CLI
native Rust tests
differential fuzzing
benchmark disclosure
coverage reporting
zero unsafe Rust blocks in the project code

The main lesson was simple: generating a port is not the finish line.

For this kind of rewrite, the real work is building enough evidence that the
new implementation deserves to be trusted.

I Built a Figma Plugin to Kill Translation Bugs Before They Reach Developers — Here's Every Wall I Hit

Aryan Saxena — Mon, 23 Feb 2026 14:20:45 +0000

The Moment That Started This

Picture this: You are reviewing a pull request. The developer built the UI exactly as designed. Every pixel matches the Figma mockup. It looks perfect.

Then QA opens it in German.

The "Jetzt kaufen" button text overflows its container. The navigation bar wraps to two lines. The hero section headline is clipped mid-sentence. A modal dialog has text bleeding into the close button.

A bug ticket gets filed. Against the developer.

But the developer did nothing wrong. They built exactly what the designer handed them. The designer just never checked what happens when "Buy Now" becomes "Jetzt kaufen" (40% longer), or when "Settings" becomes "Einstellungen" (130% longer).

This is not a code bug. It is a planning gap.

I kept running into this pattern across projects, and I kept thinking the same thing: why are we discovering these problems after development, when they should be caught during design?

So I built LingoAudit — a Figma plugin that translates your designs into multiple languages, generates localized copies of your screens, and highlights every text box that will break. All inside Figma, before a single line of code is written.

It took me down a rabbit hole of sandbox restrictions, CORS nightmares, and typography destruction that I never expected. This is the full story.

What LingoAudit Actually Does

Before diving into the technical chaos, here is what the finished product looks like:

Open the plugin inside any Figma file
Select a frame (or an entire page)
Pick target languages — German, Arabic, Japanese, Hindi, whatever you need
Click Scan

The plugin then:

Extracts every text node from the selected frames
Sends them to the Lingo.dev translation engine in batched chunks
Clones your entire frame for each target language
Injects the translated text into the clones while preserving all formatting
Measures every text box against its container
Highlights overflows with severity indicators (Critical / Warning / Safe)

Your original design is never touched. You get side-by-side translated copies showing exactly where your layout breaks.

The results panel categorizes everything so the designer can fix layouts immediately — before any developer writes any code.

The Tech Stack

Here is what powers LingoAudit under the hood:

Layer	Technology	Role
Plugin Sandbox	TypeScript	Reads/writes the Figma document, clones frames, measures text
Plugin UI	React + TypeScript	API key management, locale selection, results display
Translation	Lingo.dev SDK	AI-powered localization engine
Bundler	Webpack	Dual-target build (sandbox + UI iframe)
Styling	Vanilla CSS	Figma-native dark theme

The critical thing to understand about Figma plugins is that they run in two completely separate contexts:

┌─────────────────────────────────────────────────────────┐
│  SANDBOX (code.ts)                                      │
│  - Can read/write Figma layers                          │
│  - Has NO fetch, NO DOM, NO window                      │
│  - Runs in a QuickJS-like restricted realm              │
│                                                         │
│              ↕ postMessage (async, serialized)          │
│                                                         │
│  UI IFRAME (index.tsx)                                  │
│  - Has fetch, DOM, React                                │
│  - CANNOT access Figma layers                           │
│  - Subject to browser CORS policies                     │
│  - Origin is "null" (not http://localhost)              │
└─────────────────────────────────────────────────────────┘

Every single interaction between these two halves happens through serialized postMessage calls. There is no shared memory, no shared state, no function calls across the boundary.

This architecture is the source of almost every challenge I faced.

Wall 1: The Network Is Unreachable

My first approach was obvious: call the Lingo.dev API from the UI iframe using fetch.

// This seems reasonable, right?
const response = await fetch("https://engine.lingo.dev/translate", {
  method: "POST",
  headers: {
    "Authorization": `Bearer ${apiKey}`,
    "Content-Type": "application/json"
  },
  body: JSON.stringify({ texts, targetLocale: "de" })
});

It failed instantly. The browser blocked it.

Here is why: Figma plugin iframes have a null origin. When you send a request with custom headers (like Authorization) from a null origin, the browser fires a CORS preflight OPTIONS request. The target server needs to respond with Access-Control-Allow-Origin: * (or specifically allow null). Most APIs do not do this — and they should not, because allowing null origins is a security risk.

What I tried:

Attempt	Result
Direct fetch to `engine.lingo.dev`	CORS preflight blocked
Routing through the Figma sandbox's `fetch`	Sandbox has no `fetch` API
`corsproxy.io`	403 Forbidden
`thingproxy.freeboard.io`	DNS resolution failure
`proxy.cors.sh`	Required paid API key
`api.allorigins.win`	Mangled POST body

Five proxies. Five failures. Each one for a different reason.

What actually worked:

The sixth proxy, cors.eu.org, properly handled both the OPTIONS preflight and the POST body passthrough. I configured the Lingo.dev SDK to route through it:

const lingoDotDev = new LingoDotDevEngine({
  apiKey,
  apiUrl: "https://cors.eu.org/https://engine.lingo.dev"
});

And added it to Figma's allowed domains in manifest.json:

{
  "networkAccess": {
    "allowedDomains": [
      "https://cors.eu.org",
      "https://fonts.googleapis.com",
      "https://fonts.gstatic.com"
    ]
  }
}

The lesson: If you are building a Figma plugin that calls any third-party API with auth headers, you will need a proxy. Plan for it from day one. Ideally, deploy your own (a 15-line Cloudflare Worker would do it). A public proxy works for a prototype but is not suitable for production since all traffic passes through a server you do not control.

Wall 2: Replacing Text Destroys Typography

This one was painful to discover. Here is a simplified version of what I expected to work:

const clone = textNode.clone();
await figma.loadFontAsync(clone.fontName as FontName);
clone.characters = "Jetzt kaufen"; // German translation

It works perfectly — until you hit a text node with mixed formatting.

In Figma, a single text node can contain bold, italic, different sizes, and different colors all within the same string. Think of a heading like "Welcome to LingoAudit" where "LingoAudit" is bold and green.

When you read node.fontName on such a node, Figma does not return a font. It returns a special symbol: figma.mixed. And if you try to set node.characters without first setting a uniform fontName, Figma throws an error.

The obvious fix — force the entire node to one font, then replace text — works but annihilates every bit of styling the designer applied. The bold disappears. The colors reset. The design looks broken.

My solution: Character-Level Style Mapping

Before replacing any text, I extract the formatting of every single character:

const styles: StyleRecord[] = [];
for (let i = 0; i < originalLen; i++) {
  styles.push({
    fontName: node.getRangeFontName(i, i + 1),
    fontSize: node.getRangeFontSize(i, i + 1),
    fills: node.getRangeFills(i, i + 1),
  });
}

After injecting the translated text, I proportionally map those styles onto the new string:

for (let i = 0; i <= newLen; i++) {
  // Map position in new string to position in original string
  const originalIndex = Math.min(
    Math.floor((i / newLen) * originalLen),
    originalLen - 1
  );
  // Apply the style from that original position
  node.setRangeFontName(rangeStart, i, styles[originalIndex].fontName);
  node.setRangeFontSize(rangeStart, i, styles[originalIndex].fontSize);
  node.setRangeFills(rangeStart, i, styles[originalIndex].fills);
}

Is the mapping perfect? Not mathematically. If the translated text is dramatically shorter or longer, the style boundaries stretch or compress. But in practice, headings stay bold, colored words stay colored, and the design looks intentional rather than broken.

Wall 3: Font Loading Freezes Everything

Figma enforces a strict rule: before you can modify any text on a node, you must call await figma.loadFontAsync(fontName) for the exact font that node uses.

My initial implementation loaded fonts inside the per-node loop:

// Slow: loads fonts redundantly for every node
for (const textNode of allTextNodes) {
  await figma.loadFontAsync(textNode.fontName as FontName); // blocking
  textNode.characters = translatedText;
}

On a frame with 300 text nodes using 8 different fonts, this makes up to 300 async calls — most of them redundant duplicates loading the same font over and over. Figma's UI completely locks up for 5-15 seconds.

The fix — batch parallel loading:

async function prepareFontsForNodes(nodes: TextNode[]): Promise<void> {
  const uniqueFonts = new Set<string>();

  for (const node of nodes) {
    if (node.fontName === figma.mixed) {
      // Extract every unique font from mixed-style nodes
      for (let i = 0; i < node.characters.length; i++) {
        uniqueFonts.add(JSON.stringify(node.getRangeFontName(i, i + 1)));
      }
    } else {
      uniqueFonts.add(JSON.stringify(node.fontName));
    }
  }

  // Load ALL unique fonts concurrently
  await Promise.all(
    Array.from(uniqueFonts).map(f =>
      figma.loadFontAsync(JSON.parse(f) as FontName).catch(() => {})
    )
  );
}

One pass to collect. One Promise.all to load. The per-node loop then runs without any font loading at all. On a 300-node frame, the difference is the gap between a 12-second freeze and near-instant execution.

The .catch(() => {}) is deliberate — if a designer uses a local font that is not installed on the current machine, the load will fail. Instead of crashing the entire plugin, we catch it silently and fall back to Inter when we actually set the characters. The user still gets their translated screen; the font just defaults gracefully.

Wall 4: The Overflow Measurement Was Lying

Figma text nodes have a textAutoResize property that dictates how the bounding box behaves:

Mode	Behavior
`WIDTH_AND_HEIGHT`	Box grows in both directions to fit
`HEIGHT`	Box keeps its width, grows downward
`NONE`	Box is completely fixed, text clips

For overflow detection, I clone the node, inject the translated text, and compare the resulting dimensions to the original. My initial approach for fixed (NONE) boxes was to temporarily set the clone to WIDTH_AND_HEIGHT to see how wide the text "wants" to be.

The problem: This unwraps multi-line text into a single horizontal line. A paragraph that fits perfectly when wrapped was being reported as 200% overflow because it was measured as one infinitely wide line.

// Wrong: measures single-line width, not real overflow
clone.textAutoResize = "WIDTH_AND_HEIGHT";
overflowAmount = clone.width - originalWidth; // Wildly inaccurate

The fix: Lock the clone's width to the original, set it to HEIGHT mode, and measure vertical growth:

// Correct: keeps width locked, measures downward expansion
clone.resize(originalWidth, originalHeight);
clone.textAutoResize = "HEIGHT";
overflowAmount = clone.height - originalHeight; // Accurate

If the clone grows taller than the original fixed box, that is your real overflow — the text needs more vertical space than the designer allocated.

Wall 5: Large Files Crash the API

A production design file can have thousands of text nodes. Sending all of them to a translation API in a single HTTP request results in either a 413 Payload Too Large response or a timeout.

The fix — automatic chunking with progress tracking:

const CHUNK_SIZE = 50;

for (let i = 0; i < texts.length; i += CHUNK_SIZE) {
  const chunk = texts.slice(i, i + CHUNK_SIZE);
  const content: Record<string, string> = {};
  chunk.forEach((text, idx) => { content[`str_${idx}`] = text; });

  const translated = await lingoDotDev.localizeObject(content, {
    sourceLocale: source,
    targetLocale: target,
  });

  // Map results back to the master array
  chunk.forEach((text, idx) => {
    results[i + idx] = translated[`str_${idx}`] ?? text;
  });

  // Report progress to UI
  onProgress(Math.min(((i + chunk.length) / texts.length) * 100, 100));
}

Each completed chunk fires a progress callback. The React UI uses this to drive a smooth progress bar. Instead of staring at a frozen screen for 30 seconds, the user sees incremental progress across both chunks and locales.

The Non-Destructive Cloning Architecture

Early versions of the plugin modified the original design — adding red stroke borders around overflowing text. Users hated it. Nobody wants a tool that alters their source of truth.

The final architecture clones the entire selected frame for each locale:

for (const locale of Object.keys(byLocale)) {
  for (const root of rootNodes) {
    const clone = root.clone();
    clone.x += xOffset; // Place next to original
    clone.name = `${root.name} - ${locale.toUpperCase()}`;

    // Walk both trees in parallel
    const origTexts = walkTextNodes(root);
    const cloneTexts = walkTextNodes(clone);

    for (let i = 0; i < origTexts.length; i++) {
      const result = resultMap.get(origTexts[i].id);
      if (result) {
        applyTextWithMixedStyles(cloneTexts[i], result.translatedText);
        if (result.isOverflow) highlightNode(cloneTexts[i]);
      }
    }
  }
  xOffset += totalRootWidth;
}

The key insight is walking the original tree and the cloned tree simultaneously. Since clone() preserves the tree structure, the i-th text node in the clone corresponds to the i-th text node in the original. We use the original node's ID to look up translation results, then apply them to the clone.

The result: the designer's original frame is untouched. The translated copies appear neatly to the right, with overflow highlights only on the clones.

What I Would Do Differently

Deploy my own proxy from day one. I wasted hours testing five different public CORS proxies. A 15-line Cloudflare Worker would have solved it permanently in 10 minutes.

Cache translations locally. Right now, scanning the same frame twice re-translates everything. A Map<string, string> keyed by hash(locale + sourceText) stored in figma.clientStorage would eliminate redundant API calls and make re-scans nearly instant.

Handle component instances smarter. Figma components and instances have their own text override system. My walker treats them as regular nodes, which works, but a smarter implementation could translate at the component master level and propagate to all instances automatically.

The Impact

The core thesis held up: translation overflow is a design problem that should be caught during design.

With LingoAudit, a designer can test 10 languages on a complex screen in under 30 seconds. Without it, the same check requires manually copy-pasting translations, creating duplicate frames by hand, and visually inspecting each one. That workflow takes hours per screen and is so tedious that nobody actually does it — which is exactly why these bugs survive into production.

The plugin handles:

Mixed-style typography preservation across translations
Right-to-left languages (Arabic, Hebrew) with automatic text direction
Large files with thousands of text nodes via chunked batching
Missing fonts with graceful fallback instead of crashes
Non-destructive workflow that never touches original designs

Try It Yourself

The project is open source: github.com/AryanSaxenaa/Figma-Plugin-Lingo

To run it locally:

git clone https://github.com/AryanSaxenaa/Figma-Plugin-Lingo.git
cd Figma-Plugin-Lingo
npm install
npm run build

Then in Figma: Plugins > Development > Import plugin from manifest and select the manifest.json.

You will need a Lingo.dev API key to use the translation features.

Resources

Figma Plugin API Documentation
Lingo.dev SDK
i18next — If you need runtime translation in React apps
Unicode CLDR — The standard for locale data

Built with Lingo.dev, React, TypeScript, and a mass of patience for Figma's sandbox.