DEV Community: Yash Vardhan Shukla

The AI that hallucinated a bug into my zero dependency project

Yash Vardhan Shukla — Mon, 31 Aug 2026 15:23:38 +0000

Repo: https://github.com/Yash-vs9/bindery
Demo video: https://youtu.be/9wFCPcWEZzM
@partnerships_raptors

I built bindery for the Zero Dependency Hackathon: a tool that turns a folder
of Markdown into a full documentation site, complete with live reload, full
text search, PDF export, and hand rendered diagrams. One binary. Go standard
library only. Empty manifest, no require block, nowhere for a dependency to
even hide. The whole premise of this event is that half of today's code gets
written by an AI that confidently invents package names that don't exist, and
that the registry those hallucinated packages point at added something like
four hundred and fifty thousand malicious entries last year alone. My plan
was to prove the opposite was possible. Build something genuinely useful using
only what the language already gives you, no imports, no excuses, and see how
far that actually gets you.

What I didn't expect was that the AI helping me build it would hallucinate
something far more interesting than a fake package name. Over one long
weekend it hallucinated a protocol constant, misread a standard library
upgrade's silent behavior change, drew a graph with an arrow pointing at
nothing, and then, near the very end, wrote a test that flagged its own
correct code as a security hole. Every single one of those looked completely
fine on the surface. Every single one compiled. Every single one passed the
tests that existed at the time. Catching them, one at a time, ended up being
the actual story of this project, more than the feature list ever was.

The bug that would have failed silently forever

Bindery's live reload works over a WebSocket I wrote by hand, because Go's
standard library has no WebSocket implementation at all, none, not even an
experimental one. The opening handshake needs one very specific ingredient: a
GUID defined in RFC 6455, concatenated with the client's Sec WebSocket Key,
hashed with SHA1, and base64 encoded into the response. Get one character of
that GUID wrong and nothing crashes. No error. No stack trace. No panic to
grep for in a log file. The socket just quietly, politely, never opens, and
you sit there refreshing a browser that never reloads, with absolutely no clue
why, because every layer above the handshake is working exactly as designed.

That's exactly what happened. The AI wrote the GUID from memory and it was
wrong. Not obviously wrong, not gibberish, not a string that looks like a
placeholder. It had the right shape, the right length, the right hyphen
pattern of a UUID, and it sat right next to a code comment claiming it was
"transcribed directly from the RFC." It wasn't. One character had drifted
from the front of the final group to the back of it, the kind of transposition
a human proofreader skims straight past because the overall silhouette of the
string looks correct.

The only reason this got caught at all is that I refused to accept "it
compiles" as evidence and insisted on a test that pinned the actual output
against the worked example the RFC itself publishes, including the
intermediate SHA1 digest, not just the final base64 string. That test failed
on its very first run. And here's the part that actually worried me: when it
failed, I went back to check the "correct" value from memory too, and my own
recollection didn't match either. Neither the code nor my own head could be
trusted at that moment, which meant there was exactly one thing left to do,
which was go fetch the actual RFC text and check both of us against it like a
referee. The code was wrong. My memory of what should replace it was also
wrong in a different way. Only the primary source settled it.

That felt like the entire hackathon compressed into one bug. An AI can write
code fast and with total, unwavering confidence. It genuinely cannot tell you,
from the inside, when that confidence is misplaced, because a hallucinated
constant reads exactly the same as a correct one. The only real defense is a
test that checks against ground truth published somewhere outside the model's
own head, not a test that checks whether the output merely looks plausible.

When the standard library itself quietly broke me

Go 1.27 shipped a new package, encoding/json/v2, and I reached for it to
serialize bindery's search index because it's newer and faster and the
hackathon's own capability notes flagged it as a fresh, real standard library
answer worth using. What none of us caught until much later is that the old
encoding/json sorted map keys before writing them out as JSON, quietly, as an
implementation detail nobody ever had to think about, and the new package
does not carry that behavior forward. It writes map keys in whatever order
Go's runtime happens to iterate them internally, and Go deliberately
randomizes that order between runs, on purpose, specifically so nobody
accidentally depends on it.

Bindery's search index is, structurally, a map of words to the list of pages
containing them. So every single time I rebuilt the site, that JSON came out
with its keys in a different byte order. Same words, same postings, same
content in every meaningful sense, completely different bytes on disk. That
kind of bug is invisible right up until the exact moment you try to prove
something stronger than "it looks right," which is precisely what one of this
event's bonus challenges asks for: build the same source twice, hash both
outputs, and they had better match exactly.

I wrote a test that built the identical search index nine times in a row and
compared every byte of the result. It failed on the very first run. The
compiled binary itself was perfectly reproducible, byte for byte, hash for
hash. The data that binary produced when it actually ran was not. A check that
only hashes the executable would have printed a clean green REPRODUCIBLE and
sailed straight past a real, silent hole in the actual claim being made. The
fix turned out to be a single line, a Deterministic option the new package
happens to expose if you know to look for it, but finding the gap in the
first place meant refusing to treat "reproducible" as a property of the build
system alone, when it's really a property of every single piece of output
that build system ever touches.

Looking at the actual picture instead of trusting the code

Bindery renders diagrams straight out of Markdown fences, written in a small
subset of Mermaid's syntax, but with zero JavaScript shipped to the browser
and zero external layout library involved. It's a hand written graph layout
algorithm underneath: assign every node a layer based on the longest path to
it, order the nodes within each layer to reduce how many lines cross each
other, then draw boxes and arrows as raw SVG text, no image library required
anywhere in the chain.

The first version compiled cleanly, passed every unit test I had written for
it, and quietly produced a diagram where an arrow pointed directly at nothing,
floating in empty space. A loop in the graph, something as mundane and common
as "the file watcher notices a change, does its work, and goes back to
waiting for the next one," completely broke the layering, because my layer
assignment logic had quietly assumed the graph could never contain a cycle,
and just kept pushing the same node deeper and deeper on every single pass
whenever one actually existed.

None of my tests caught this, for a very specific and slightly embarrassing
reason: not one of them actually rendered the SVG and looked at it. They
checked node counts. They checked edge counts. They checked that specific
strings like "marker-end" showed up somewhere in the output. All of that
passed while the picture itself was visibly, obviously broken to anyone who
bothered to open it. I only found the bug because I exported that SVG to a
PNG and actually looked at it, the exact same way any real human being would
look at it before ever putting it in a demo video for other people to judge.
And once I started actually looking instead of just asserting, I found a
second problem sitting right next to the first one: an edge that spanned more
than one layer was being drawn as a perfectly straight line directly through
the boxes sitting between its two endpoints, which is completely correct
coordinate math and completely wrong to look at.

This is the pattern I kept running into over and over across this whole
project. Correctness according to the tests I had already thought to write is
not the same thing as correctness. Sometimes the only honest verification
step is opening the actual output the way a real person eventually will.

Three Windows bugs, and none of them were logic bugs

Once the core parser was solid, I set up continuous integration across Linux,
macOS, and Windows, because a judge running your submission on the wrong
operating system and hitting a wall is entirely on you, not on them. Windows
failed. Not once, but three separate times, on three genuinely different,
completely unrelated causes, and looking back at all three together taught me
something I didn't expect going in: every single one of them was a hidden
assumption baked into a test, never a real defect in what bindery actually
does.

The first failure took down an entire suite of end to end tests all at once,
every single one with the identical error, "executable file not found."
Windows requires a literal dot exe extension to execute a binary at all, full
stop, no exceptions, and my test harness built the shared test binary with a
bare name and no extension whatsoever, because that's simply what works
without a second thought on Linux and macOS. The fix was two lines checking
the current operating system. The lesson underneath it was bigger: an entire
category of failures, every single test in that file, traced back to one
completely wrong assumption about what "runnable" even means on a different
platform.

The second failure was subtler and, honestly, more interesting to me. A test
asserted that a missing directory produced the literal Unix error string "no
such file or directory" on the program's stderr. Windows reports the
identical underlying failure with entirely different wording, something like
"the system cannot find the file specified," because that's simply how its
own error reporting is worded at the operating system level. Bindery's actual
behavior was completely correct on both platforms the entire time: right exit
code, empty stdout, a real and useful error message on stderr. My test was
quietly checking English phrasing rather than the actual behavior it claimed
to be verifying.

The third was the strangest of the three. Windows checks out Git repositories
with a setting called autocrlf turned on by default, which silently rewrites
every LF line ending to a CRLF pair inside anything Git considers a text
file, including Go source files, including the exact multi line raw string
literals my code embeds for the theme's CSS and JavaScript. Without an
explicit gitattributes file forcing LF everywhere regardless of platform,
those embedded byte for byte strings could have silently gained an invisible
carriage return before every newline purely because of which operating system
happened to check the repository out, an entirely different compiled binary
built from an identical git commit depending solely on which machine cloned
it.

Three genuinely different root causes, three real fixes, and not one line of
bindery's actual logic changed for any of them. What changed, every single
time, was an assumption quietly baked into a test or a build script that
nobody had ever bothered to question because it had simply never been wrong
before that specific afternoon.

Sourcing real data instead of letting anything guess

The PDF export needed accurate character widths so that lines of text
actually wrap correctly instead of running off the edge of the page or
leaving awkward gaps. The tempting shortcut here, the one I almost took, was
to just ask for typical widths for a handful of common fonts and trust
whatever number came back with total confidence. Given everything above, I
didn't trust that shortcut for one second longer than it took to think of it.

Instead I went and fetched the actual published Adobe Font Metrics files for
Helvetica, Helvetica Bold, Helvetica Oblique, and Courier, the same core
fourteen fonts every single PDF reader on the planet is contractually
guaranteed to already have installed, parsed the real character width tables
directly out of those files, and cross checked a handful of well known values
by hand against numbers I could actually verify independently, like the fact
that Helvetica's capital M is exactly eight hundred and thirty three
thousandths of an em wide, and Courier is uniformly six hundred thousandths
across every character because it's a monospace font by definition. A test
now asserts both of those specific facts on every single run, which means a
corrupted metrics table fails a build loudly instead of silently producing a
subtly misaligned document that only a human proofreading a PDF would ever
notice, months later.

The exact same discipline showed up again chasing the very last percentage
point of CommonMark conformance. Bindery sat at six hundred and fifty one out
of six hundred and fifty two official spec examples for a while, one single
stubborn failure away from a perfect score, and the one remaining case needed
full Unicode case folding, specifically the capital sharp S character folding
down to two lowercase letters rather than one. Go's standard library only
implements simple case folding, one character to one character, by design,
so there was no shortcut sitting there waiting to be called. I generated the
actual exception table directly from the Unicode Consortium's own published
CaseFolding.txt file, rather than trusting anyone's memory, mine or the
model's, of which few dozen code points actually need special handling. That
last one percent turned into a genuine, documented standard library gap
instead of a fudged number, and bindery now passes all six hundred and fifty
two.

Catching my own test lying to me

Near the very end of the weekend, continuous integration's fuzz testing found
an input that supposedly broke bindery's HTML escaping entirely: a raw HTML
tag containing a literal angle bracket sitting inside one of its own
attribute values. My own property based test flagged it immediately as a
serious injection vulnerability, exactly the kind of finding that makes your
stomach drop a little right before a submission deadline.

Except it genuinely wasn't a vulnerability at all. CommonMark explicitly
requires that raw HTML written directly by a document's own author pass
straight through completely untouched, no escaping applied whatsoever,
because that is precisely what "raw HTML support" means as a documented
Markdown feature, and it's the identical behavior every other fully
conformant Markdown renderer on earth exhibits, the same behavior already
certified by that six fifty two out of six fifty two conformance number. My
test was the thing that was wrong, not bindery's renderer. It had been
scanning the entire rendered page for anything shaped even remotely like an
HTML attribute, instead of checking only the small, specific handful of
attributes bindery itself actually constructs from untrusted user input, like
an href or a src.

I did not simply loosen the check and quietly move on with my day, because a
test you can no longer fully trust is meaningfully worse than having no test
at all sitting there giving you false confidence. I deliberately, temporarily
disabled the real escaping function on purpose, confirmed the newly narrowed
test still correctly caught a genuinely leaked angle bracket when escaping
was actually broken on purpose, and only then put the original code back and
reran absolutely everything from scratch. Trusting the fix meant proving it
both ways, not just one.

What all of this actually proves

None of the four bugs above were "the AI is simply bad at writing code." The
code compiled cleanly every single time. It passed whatever tests already
existed at that moment every single time. That is precisely what makes this
particular failure mode so genuinely dangerous: confident, fluent,
well formatted, plausible looking code that is subtly wrong in a way that
looks, on the surface, exactly like code that is subtly right. A hallucinated
constant does not announce itself. A silently changed default in a standard
library upgrade does not announce itself. A layout algorithm that quietly
assumes away cycles does not announce itself either, right up until you
actually look.

The fix was never simply "trust the AI less" as some vague, generalized
posture. It was building layers of verification that check against something
genuinely outside the code itself: an RFC's own published worked example, the
same search index built nine separate times and compared byte for byte, an
actual rendered picture opened and looked at with human eyes, a real escaping
function deliberately broken on purpose and used as a canary, official
published Unicode data pulled directly from its actual source rather than
recalled from memory by anyone, human or otherwise. Every single one of those
catches happened because I refused to accept "it compiles and the existing
tests pass" as the actual finish line, and kept pushing one layer further
each time.

Bindery ends up at six hundred and fifty two out of six hundred and fifty two
on the official CommonMark conformance suite, with a completely empty
dependency manifest, and a reproducible build proven two entirely separate
ways, the compiled binary and its actual data output both. But if I'm honest,
the number I'm most genuinely proud of out of this whole weekend isn't six
fifty two. It's four. Four real, distinct bugs, caught not because I somehow
wrote flawless code on the first attempt, but because I built a project whose
entire founding premise forced me to stop trusting anything I hadn't gone and
independently verified myself, including the AI helping me, including my own
memory, including the very tests I had written to protect myself from exactly
this.

That's the actual lesson sitting underneath a zero dependency hackathon, once
you've lived inside one for a full weekend. It was never really about the
packages. It was about learning, the hard way, four separate times in one
weekend, what it actually costs you to trust something you never bothered to
check.

If you want to see any of this for yourself rather than just take my word for
it, the full source is public at https://github.com/Yash-vs9/bindery, and
there's a five minute walkthrough of it actually running, live reload, search,
the diagrams, the PDF export, and the reproducible build check all included,
at https://youtu.be/9wFCPcWEZzM.

The test suite never knew it was running on Rust

Yash Vardhan Shukla — Mon, 03 Aug 2026 10:39:55 +0000

Port Mortem hands you seventy two hours and a dare. Take a real library written in one language, rebuild its guts in another, and prove the new version behaves exactly like the old one. The proof is where it gets interesting. You do not get to write cozy little tests that flatter your own code. You run the original project's own test files against your port, and you keep your hands off those files the entire time. If you touch them, you lose.

I took Track D, Python to Rust, and I picked grantjenks/python-sortedcontainers.

If you have shipped Python for money you have almost certainly used it without thinking twice. It gives you a list that stays sorted as you add to it, a dict whose keys iterate in order, a set with order, and it does all of that in pure Python while keeping pace with things written in C. That last claim is the hook. There is no C extension hiding inside it. The speed comes from a data structure, not from escaping the language. I wanted to see what actually happens when you do escape the language.

The trick the library plays

Most people reach for a balanced tree when they hear "keep it sorted." sortedcontainers does something dumber and faster. It keeps a list of smallish lists. New values get slotted into whichever sublist they belong in, and when a sublist grows past twice a load factor it splits in half. When one shrinks too far it merges with a neighbor. Each sublist stays short enough that a plain insert into a Python list, memory copy and all, beats the pointer chasing you would pay in a tree. This is square root decomposition, and it is the whole engine.

The part that made me want to port it is positional indexing. sl[5000] on a hundred thousand element sorted list has to find the five thousandth element in sorted order, fast. A BTreeSet in Rust's standard library cannot do that. It has no idea how many elements sit to the left of any node. sortedcontainers builds a small binary tree of subtree sizes on the side, flattened into one flat array, and walks it in logarithmic time. That index is clever and finicky and exactly the kind of thing that breaks in quiet ways if you get the arithmetic slightly wrong.

So I rebuilt it. One Rust crate, pure safe code, #![forbid(unsafe_code)] at the top so the compiler physically refuses to let me cheat. The list of lists became a Vec<Vec<T>>. The index tree became a Vec<usize> with the same layout the original flattens into memory. There was one place the original computes an offset using math.log, and floating point at exact powers of two is a coin flip you do not want in your addressing math, so I used integer bit tricks instead and got the same answer without ever touching a float.

Making Python forget

Here is the constraint that shapes everything. The tests do import sortedcontainers. My module is a Rust extension called sortedcontainers_rs. I am not allowed to edit a single test to fix that mismatch.

The answer turned out to be small. A tiny shim directory named sortedcontainers that re-exports my Rust classes, dropped onto the Python path ahead of everything else. When the test suite says import sortedcontainers, Python finds my shim first, and the shim quietly hands back Rust. The test suite spends its entire run believing it is exercising the library it has always known. It never finds out. That is the part I keep grinning about. Two hundred and ninety six tests, written years ago by someone who had never heard of my project, all passing against code they were never meant to see.

The container that ate itself

Every port has one bug that steals an evening. Mine was a test that builds a set, then adds the set to itself.

temp.add(temp)

In Python this is fine. The set now contains a reference to itself, and when you print it, the standard library notices the loop and prints an ellipsis instead of recursing forever. Cute. Harmless.

In my Rust binding it detonated. add had borrowed the set so it could mutate it. Then, to figure out where the new element belonged, it compared the element against the ones already inside, and the element was the set, so the comparison reached back in and tried to borrow the same set a second time. Rust's borrow rules do not care that this is clever. You cannot hand out a second borrow while a mutable one is live. The whole thing panicked with a message about a value already being mutably borrowed.

The fix taught me something about writing bindings. The mistake was holding the borrow across a call back into Python. So I stopped doing that. add now grabs what it needs, lets go of the set completely, lifts the ordered list out into a local variable, does the comparison heavy insertion there where nothing is borrowed, and only then puts everything back. The container can contain itself now, print its little ellipsis, and move on. Same fix pattern showed up three more times before I trusted it.

When a subclass has to arrive from nowhere

The other one that cost me real time was quieter and nastier. There is a constructor detail in the original where SortedList(key=something) does not give you a SortedList at all. It hands back a SortedKeyList, a different class, on the fly. A factory hiding inside a constructor.

Rust bindings cannot do that directly. The moment Python decides which type it is building, that decision is made, and the binding does not get to swap the answer for a different class. I chased pure Rust solutions for a while and every one of them lied to the type system in a way that broke something else. In the end the honest place to put the dispatch was a thin Python layer, with the key list set up to inherit through a diamond so that every identity check the tests make still holds. isinstance is happy, the exact type check is happy, and constructing it the wrong way still raises the same error the original raises. It is not the fix I wanted. It is the fix that is true.

The number I did not want to publish

Everyone loves a benchmark where the new thing wins. I have those. Indexing into the sorted list is about four times faster in Rust, because it is pure integer tree walking with nothing crossing back into Python. Cold import is roughly ten times faster. Inserts are a little faster.

Membership is slower. x in sl runs about thirteen percent behind the pure Python version.

I sat with that for a minute before deciding to lead with it rather than bury it. The reason is not mysterious once you see it. Every comparison during a membership check has to cross from Rust back into Python to ask two objects which one is smaller. Pure Python never leaves the interpreter, so it never pays that toll. My Rust core pays it on every single compare. The crossing is the cost. A tree or a fancier layout would not save me, because the bottleneck is the border, not the algorithm. So the number stays in the README, in bold, next to the wins. A benchmark you can trust is worth more than a benchmark that flatters you.

By the numbers

Tests, the whole point of the exercise:

296 of 296 passing. Zero test files edited. Zero skipped. Zero marked as expected failures. The suite is provably byte for byte identical to the version pinned at kickoff.

Fuzzing, our port against the real library, comparing results after every operation:

1,649,853 randomized operations across list, set and dict. Zero divergences.

Speed, measured on a SortedList of 200,000 elements, CPython 3.13.1, p99 latency:

operation	original	this port	change
indexing `sl[i]`	1583 ns	375 ns	4.2x faster
insert `add`	1875 ns	1417 ns	1.3x faster
membership `x in sl`	833 ns	959 ns	0.87x, slower
cold `import`	7.38 ms	0.75 ms	9.8x faster

The indexing win is even wider at the median than at the tail. Typical sl[i] drops from around 1166 ns to 125 ns, close to nine times faster, because the whole operation is integer arithmetic in Rust and never once asks Python a question. Membership loses for the exact opposite reason. It cannot take a step without asking Python which of two objects is smaller, and every one of those questions is a trip across the border.

Safety, the part the compiler guarantees rather than the part I promise:

Zero unsafe blocks in the core. Not zero by discipline. Zero by #![forbid(unsafe_code)], which means the crate will not compile if I ever slip.

Hunting for a bug that was not there

One of the bonus objectives is to find a real latent bug in the original library through differential testing. I wanted it badly. I built an oracle, a deliberately stupid and obviously correct model, a plain list kept sorted the slow honest way, and I threw millions of randomized operations at both it and the real library, comparing every result. I aimed at the corners where bugs like to hide. Inverted range bounds. Empty ranges. Negative step slices. Non monotonic keys and how ties order. Pickle round trips.

It found nothing. The library is ten years old and it shows. Every single case came back clean.

I thought about how to spin that into a finding anyway, some tiny documentation nitpick dressed up as a bug. Then I did not. Reporting a clean result honestly is the whole ethic of a project like this. A library surviving tens of millions of adversarial checks without a single divergence is a real result, and pretending otherwise would poison everything else I was claiming. So the write up says plainly: no bug found, and here is exactly how hard I looked.

Proving I did not cheat

The one accusation that could sink this project is "you edited the tests." So I made that accusation impossible to make. At the very start I took a cryptographic hash of every original test file and froze it. The submission carries a single kickoff hash that is bound to those exact files at those exact paths. Anyone can run one script and watch it confirm that the tests are byte for byte what they were on day one, and that they match the real upstream library at the pinned commit. Not similar. Identical.

That hash turned into a small design constraint later, in a good way. When I was tempted to reshuffle folders to match a template more neatly, I realized moving the test directory would change the manifest, change its hash, and break the frozen pin. The provenance proof outranked the tidy folder. So the folder stayed where it was, and I wrote down why. Integrity you can verify beats structure that merely looks right.

What I actually built

Strip away the story and here is the residue. One safe Rust core, zero unsafe code, driving four containers through one shared element type. The original test suite passing in full, two hundred and ninety six out of two hundred and ninety six, with the files provably untouched. Millions of fuzzed operations with zero divergence from the real thing. Honest performance numbers including the one place I lose. And a decision log with seventeen entries explaining every place I chose to differ and why.

The line I keep coming back to is the one Port Mortem opens with. Languages die and code does not. You can take an idea that someone poured years into, written in a language that will eventually fall out of fashion, and carry it forward into new ground without losing what made it correct. The proof that you carried it faithfully is not your word. It is the old code's own tests, running green against something they were never built for, none the wiser.

Built for Port Mortem 2026, the 72 hour porting hackathon run by Hackathon Raptors (@partnerships_raptors ). The full project, the original test suite with its kickoff hash, the differential fuzzer, and the benchmark harness are all in the repo.

Repo: https://github.com/Yash-vs9/Port_Mortem