DEV Community: Kaung Zin Hein

AI Video Needs CI More Than Better Prompts

Kaung Zin Hein — Mon, 20 Jul 2026 20:49:27 +0000

Here I'll share what I built for Track 2 of the Global AI Hackathon Series with Qwen Cloud, and what it taught me about testing things you can't see.

My expensive little problem

I generated a five-second video clip. The first frame looked perfect.

Then I watched the other four seconds.

The camera drifted left. My brief said hold static.

Here's the part that irritates me: I found out after I paid. On Qwen Cloud's Wan models a clip is a fixed five seconds of premium quota whether it's usable or garbage. The bill doesn't care.

Text models don't work like this. You read the answer, you see it's wrong, you move on. The artifact and the verdict show up together and being wrong costs you a few hundred tokens.

Video fails in motion. The failure lives in frames you haven't looked at yet.

So you pay first, and find out second.

But it gets worse...

The obvious build for this track is a premise-in, episode-out generator. I started there. Within a couple of days I had something that produced a demo reel.

But demo reels hide the interesting failures.

Every bad take that reaches your final render is money you already spent. And I kept asking myself a question that had nothing to do with generation:

Can the system tell whether the video obeyed the brief, before it spends premium budget on it?

That question has a name in software. It's CI.

And almost nobody is running it on generated video.

So why bother? Don't we just prompt better?

That was my first instinct too. Write better prompts, get better clips.

But a prompt is not a contract. It's too easy to rewrite, stretch, and forget as an artifact moves between models. You can't assert against a prompt. You can't fail a build with one.

Before I burned a week on this bet, I wanted to know it wasn't just a slogan I liked the sound of. So I counted. Twice.

Test 1 — the simulated field. I ran 15 independent LLM brainstorms over this track's public brief. Two model families, 14 personas, none of them seeing my repo or each other. Around 150 ideas.

All 15 produced a generator.

Zero produced a way to check whether the generated episode came out right.

Two of them even emitted the phrase "CI for AI-generated video" verbatim — and then bolted it onto a generator anyway.

Test 2 — the observed field. Two days before the deadline I read the eight most substantial shipped entries in this track's public GitHub field. Every one routes its quality checks through a token-billed model call. Six of the eight import no computer-vision library at all. None ships a closed assertion vocabulary. None can gate a video it didn't itself generate.

Then I found this line in the config of a shipping logline-to-video pipeline:

DAILIES_QC = False   # off = faster pipeline

Someone built a review stage. Named it dailies. Shipped it switched off. For speed.

That's the whole problem in one line.

When the gate lives inside the generator's codebase, the gate is the first thing traded away for throughput.

Time to build the tests instead

In film production, dailies are the screening where the crew watches yesterday's footage before more money is spent shooting on top of it. It's the daily quality gate.

So that's the name. Dailies. The review stage that can't be quietly switched off, because it doesn't belong to the pipeline it judges.

It reads frames, not generator internals. Point it at any mp4 from any model.

Here's how it works.

You write what a shot has to do. Every take gets measured against it before it ships. A take that misses gets one bounded repair attempt. A take that still misses never reaches your cut.

A spec is a short program in a closed assertion DSL — ten sentence types across three tiers. The compiler translates each sentence into a call in a shared check library, and rejects anything outside the grammar before a single token is spent.

Send it something it doesn't recognize and parse_assertions raises. Send {"type": "vibe_check"} to the MCP server and it comes back isError, not a guess.

That rejection is the literal compile error in "CI for generated video."

Assertion type	Tier	Compiles to
`duration_between`	A · deterministic	clip length within bounds
`brightness_range`	A · deterministic	mean-luma within bounds
`flicker_below`	A · deterministic	inter-frame luma std under threshold
`scene_cuts`	A · deterministic	HSV-histogram cut count at or under max
`camera_motion`	A · deterministic	optical-flow pan direction
`palette_deltae`	A · deterministic	dominant-color ΔE to a brand palette
`subject_present`	0 · pre-render still	subject recognizable in the t2i still
`identity_consistent`	B · advisory	VLM: same identity across frames
`action_completed`	B · advisory	VLM: the briefed action visibly completes
`title_card_present`	B · advisory	VLM: a title / text card is visible

Notice there's an organizational idea hiding in that table.

Right now, the person who defines "correct" — brand, legal, marketing — is coupled to the person who operates the generator. They're joined by one overworked reviewer eyeballing every clip.

The stakeholder writes these assertions once, in plain language. Every shot gets tested against them automatically.

Spec-driven development, for video.

The tiers are boring on purpose

Not every check deserves the same budget. So the cascade runs cheapest-and-most-certain first.

Tier-A is deterministic OpenCV. Duration, brightness, flicker, scene cuts, optical-flow camera motion, palette ΔE. Zero tokens, so it runs on every take.

It's deliberately unglamorous. OpenCV has no idea whether your story works. But it cheaply catches a huge amount of broken video, and it never has an opinion it can't defend with a number.

Measured over 25 iterations: all six checks on a 5-second clip cost a median 317 ms of CPU. Nothing else.

Tier-B is Qwen-VL, and it's advisory. A VLM is good at "does this appear to match the brief." But a model judgment is softer evidence than a pixel measurement — so Tier-B flags for the human and never blocks promotion.

Deterministic checks are the foundation. Model-graded judgment sits on top. Never underneath.

Tier-0 is a still-image pre-screen. Before any motion is generated, a wan2.1-t2i-plus still renders at roughly 1/25th of a video's cost, and qwen-vl-plus gets asked one question: is the briefed subject even in the frame?

If a prompt can't render its own subject as a still, paying for motion just buys you the same lesson at 25× the price.

Only after surviving all that does a shot earn promotion:

premise → script + specs (qwen-plus) → compiled assertion checklist
        → Tier-0 still pre-screen (wan2.1-t2i-plus)
        → [human review gate — the one checkpoint, before any video spend]
        → drafts (wan2.1-t2v-turbo)
        → Tier-A CV (deterministic, zero tokens) + Tier-B VLM (qwen-vl-plus, advisory)
        → bounded prompt-repair + retake (qwen-plus)
        → promote passing shots (wan2.2-i2v-flash, anchored on the approved frame)
        → narration, one voice per character (qwen3-tts-flash)
        → ffmpeg assembly → certified episode, with sound

A rendered architecture diagram was a required deliverable anyway, so here's the same thing drawn formally. One convention worth knowing: red is reserved for the two paths that matter most — a spec rejected before any spend, and a blocking Tier-A fail.

Two human roles that today collapse into one exhausted reviewer. The stakeholder owns "correct" and authors the spec once. The operator is the only human in the loop at runtime, at the single gate before any video is paid for.

Level 2 answers the question the hackathon actually asked — how Qwen Cloud connects to the frontend, backend, and store:

That was the design.

Then I ran it for real, and reality started correcting me.

The rule that kept saving me: let the model choose, never let it spell

Before the war stories, one design decision that turned out to matter more than anything else I did.

That closed vocabulary started as an assertion-layer detail. It ended up being the rule the whole system runs on.

There are three places where a Qwen model makes a real decision in Dailies. In all three, it emits a choice from a set the server owns — never the artifact itself.

Where	The model emits	The server produces	The failure it prevents
Assertions	a `type` from a closed set + typed params	a call into the check library	a check that means something other than what it says
Pipeline topology	run parameters (`premise`, `pack`, `max_shots`, `custom_checks`)	the canonical node/edge graph	a structurally malformed run
Narration casting	a `speaker` — a character name	a voice from a probed roster	a silent hole in a certified episode

That middle row became my headline demo.

You type a request in plain language. qwen-plus calls a build_pipeline_graph tool with tool_choice="auto". The server deterministically expands those parameters into the graph the run executes, rendered live in React Flow off the same 2.5-second poll the dashboard already uses.

So "an agent wired the pipeline" is a claim I can make without flinching. The model authored the run. It could not have emitted a broken graph, because it never emitted topology at all.

This is the difference between validating the model's output and making invalid output unrepresentable.

Validation is a check you can forget to run. Or run in only one of two code paths — which I did, and I'll get to that.

A parameter contract is a shape the bad value cannot fit through.

The day my green tests lied to me

Every end-to-end test I'd written used synthetic clips. A fake Wan client returning generated mp4s so I could exercise the pipeline for free.

That validates control flow. And nothing about the API's contract.

A fake client can't reject a request the real service would reject.

So I built a fixtures mode. Real Wan client, real Tier-A, real Tier-B, real assembly — with only the two non-deterministic text stages pinned to fixed prompts. (Prompts are cache keys, so a pinned run replays identically at zero quota.)

Then I ran it cold, against the real API, for the first honest time.

It found four bugs.

Seventy-nine passing tests had missed every single one.

Bug	Why 79 passing tests missed it
`wan2.2-t2v-plus` *rejects `1280720`** — so every premium promotion the project had ever made had silently failed	`_promote` treats a failed promote as "keep the passing draft," so a rejected final and a healthy skip produced identical state. The premium tier — my entire wedge — had never actually run.
The budget governor asked the cache about the wrong resolution key for finals, calling every cached final "fresh"	Under judge mode that would have refused free replays during judging — the one thing the live URL depends on.
The ledger wrote an empty note for both a no-op and a hard failure	Which is precisely how the first two hid. The audit trail didn't audit.
`state.json` was write-only — snapshots atomically written, never read back	Every redeploy silently discarded every run a viewer had made, while my architecture diagram promised state persists across restarts.

Sit with that first row for a second.

My entire pitch is that generated video ships unverified because nobody runs the claim against the artifact.

And my own premium tier had been dead for the project's whole life. In a repo whose thesis is a conformance gate.

The gate caught its own maker.

I fixed all four in one commit, each with a regression test I verified by reverting the fix and watching the test fail. But this is the line I wrote in the log afterward, and it's the one I keep coming back to:

Mocks test the code you wrote. Fixtures test the assumption you made. This bug lived in the gap.

A mock returns what you told it to. So a suite of mocks can only ever confirm your own mental model. It re-asserts your assumptions back at you, in green.

The real contract lives at the boundary with the external service. The only test that touches it is one that actually calls the thing.

If you're integrating any generative API — spend the tokens on one real end-to-end run early. It's the cheapest bug-finder you'll ever build.

There was a fifth of the same family, and it's the one I'd hand to a test-design skeptic.

subject_present — my Tier-0 check — was declared, marked blocking-class, and evaluated by nothing. All three runtimes wired it to lambda spec, still: [], while the pipeline dutifully generated and billed a still for every shot, then stored the empty list it got back.

A green suite couldn't catch that. Because of one line:

take.passed = not [r for r in results if not r.advisory and r.status is Status.FAIL]

A check that returns no result and a check that passes are the same empty list.

Wiring it to a real qwen-vl-plus read costs 325 tokens per shot to avoid a 5-second premium clip. I measured it in both directions — present subject and absent — because a check that only ever answers PASS is not a check.

The receipt that made me believe in the thing

Run 3e1f628d4acf, shot 0, contract camera_motion: static.

The turbo draft passed cleanly at optical-flow magnitude |v|=0.30.

Then the premium wan2.2-t2v-plus render came back drifting left at |v|=0.92, and failed the same contract the draft had passed.

The gate did exactly what it exists for. Rejected the premium clip. Certified the cheaper draft.

Working as designed — but now my episode was stuck with the lesser render.

So I built targeted repair.

Tier-A doesn't just say a check failed. It keeps the per-frame series behind each measurement, so a failure carries a time window. Here it placed the drift at 0.4s–3.6s.

That window gives you an anchor: the last good frame, at 0.2s. A frame-anchored Wan model (wan2.2-i2v-flash) re-renders from that frame forward, Tier-A re-verifies, and a passing patch re-cuts the episode.

	before (premium, take 1)	after (patch)
`camera_motion`	FAIL — drifting left, `\	v\
flicker / brightness / duration / cuts	pass	pass

Cost: 5 video-seconds on the i2v free-tier pool, plus one {% raw %}qwen-plus repair call.

The premium look, recovered under contract, by the same gate that rejected it.

My best idea was only half right

That receipt made me greedy.

If anchoring on a good frame rescues a broken clip, why not anchor everything?

The reasoning felt airtight. Draft, repair, and final were three independent rolls from the same prompt — three separate lotteries — so a certified episode could look like three different films of the same script.

Anchor the retake on the last good frame. Anchor the promotion on the frame the human approved. Every take of a shot inherits one continuous look.

Same primitive, generalized. I shipped it.

Then I ran it on real Wan output. It was wrong in two specific, measurable places.

1. An anchor can't fix a defect that has no "before."

Shot 1 asserts camera_motion: right. Its first draft came back static — |v| = 0.005 against a 0.4 threshold — with Tier-A localizing the failure to fail_window_s [0.0, 5.33].

The entire clip.

There's exactly one frame before that window. Frame 0. And anchoring the retake there pinned the exact staticness the retake existed to remove.

The anchored retake measured |v| = 0.112. Still a FAIL.

A fresh t2v roll on the same repaired prompt reached |v| = 0.745 'right' and passed.

2. An anchor frame carries composition. It does not carry motion.

This one I did not see coming.

I took that approved rightward pan — |v| = 0.745 'right' — and promoted it by anchoring at 0.1s.

The premium final came back panning the other way. |v| = 6.15 'left'. Failed re-verify.

A single frame encodes framing, lighting, wardrobe, palette. It encodes nothing about which direction the camera was travelling when that frame was taken.

So the model just invents one.

The rule those two measurements produced is now code, not prompt advice:

Anchor to preserve. Re-roll to change.

Concretely: _anchor_for_retake returns None when the failure window opens at t = 0, so a whole-clip defect re-rolls from the corrected prompt instead. And a shot whose contract asserts camera motion skips promotion entirely — the clip that satisfied the contract is the clip that ships.

That second consequence is my favourite number in the whole project. Look at the ledger for a certified three-shot batch:

Stage	Calls	Billed
`drafting` (`wan2.1-t2v-turbo`)	4	20 video-s
`promoting` (`wan2.2-i2v-flash`, frame-anchored)	2	10 video-s

Three shots certified. Two promotions.

The missing one is the shot with the motion contract.

So verification here isn't only rejecting bad output. It's deleting a generation the run would otherwise have paid for, because measurement proved that generation could only make things worse.

A quality gate that reduces spend is a much easier thing to sell than one that adds to it.

Both findings cost one clip each to discover. 10 seconds out of a 600-second budget.

And notice which half of my idea survived. "Anchor for visual continuity" was right. "Anchoring beats re-rolling" was an unexamined extension of it, and it took two real generations to pull them apart.

When a primitive works beautifully in the case you built it for, the temptation is to extend it by resemblance.

Every bug I found was the same bug

Here's the postscript, and it's the most uncomfortable finding in my log.

The promotion path and the manual re-render button computed their anchors in two separate places.

pipeline.py learned the t=0 rule. patch.py had its own anchor_second, which clamped happily to 0.0.

So for another full day, the button in my UI cheerfully offered — and the endpoint faithfully performed — precisely the move I had just measured as useless.

No test caught it. And the reason is worth stating plainly:

Both surfaces agreed with themselves.

Each was internally consistent. Nothing in the suite asserted they agreed with each other. It only surfaced because someone finally clicked the button.

There's a third instance lurking in my deploy runbook. The budget governor wraps gen_video_fn — but promotion now renders through patch_video_fn, deliberately unwrapped so anchored work draws the separate i2v pool instead of competing with drafts.

Correct decision. Unintended consequence: fresh_final_cap no longer bounds a live run.

The safety rail didn't break. The thing it guarded moved out from under it.

Now line up every bug this project has found:

· a fallback that returns the success value on failure
· a missing result indistinguishable from a passing one
· two code paths deriving the same value independently
· a wrapper whose subject moved

Not one of them failed loudly.

Every single one presented as success.

Which is, uncomfortably, the exact thing my product exists to catch. Generated video ships unverified because a bad clip and a good clip both look like output.

Giving it a voice (and the roster the API won't tell you)

Wan's video models return silent clips. So my certified episode was silent.

Listing the models available to my key turned up 149 of them, including the qwen3-tts-flash voice family. A 107-character line synthesizes in 2.4 seconds.

Easy win. Except for two things.

The twelve-hour episode. The returned WAV is streamed, so its header declares an effectively infinite length. A ~7-second clip reports 44,739 seconds to both wave and ffprobe.

Anything that trusts that header to size the output produces a half-day-long file.

My assembler survives because -shortest bounds every segment by its video, and a paired apad stops a short narration line from truncating the video the other way. Verified: a 5.00s clip muxed with that 44,739s-declaring WAV comes out at exactly 5.00s.

Never trust a container's self-reported duration for a stream. Bound it by something you do know.

The roster you can't list. Narration shipped with one hardcoded voice, so every character sounded identical.

Giving a cast distinct voices means knowing which voices this account may actually use. And the API won't say:

400 InvalidParameter: Invalid voice specified, the requested voice does not exist
                      or is not licensed for use—please select a supported voice.

That message enumerates nothing.

And "not licensed for use" means availability is an account property, not a model property. So the roster has to be probed, not read.

I fired 20 candidate names at qwen3-tts-flash. Got 200-with-audio for all 20:

Cherry · Ethan · Nofish · Jennifer · Ryan · Katerina · Elias · Jada · Dylan · Sunny · Li · Marcus · Roy · Peter · Rocky · Kiki · Eric · Serena · Chelsie · Aiden

Which brings back the rule from earlier. My script agent emits a speaker — a character name. The server casts the voice from that probed roster.

The model cannot emit an invalid voice, because it never emits a voice at all.

That closure earns its keep here more than anywhere. Because this failure is silent: an unlicensed voice degrades that shot to silence rather than failing the run. A certified episode would just have a hole in it, and nothing would say so.

One more detail that's load-bearing. Casting is by order of first appearance, fixed at scripting time — not by hashing the character's name.

A hash can collide and hand two characters one voice with nothing to signal it. An ordinal is equally deterministic, so a re-run casts identically and every narration cache key still hits.

That key is sha1(model|voice|text). Unstable casting would silently re-synthesize every line on a replay that was supposed to be free.

Making it survive the judges

A gate is only real if someone can click it. And generated media is slow and quota-sensitive — exactly the wrong properties for a live demo.

A content-addressed cache. Identical (model, prompt, seed, size) requests replay from disk for free:

	billed video-seconds	wall clock	shots certified
Cold (2 uncached finals)	10 s (~$3.00 est.)	207 s	3/3
Warm (everything cached)	0 s ($0.00)	6 s	3/3

Re-certifying real 1080p video costs zero video quota. Across the entire life of this project, my honest end-to-end runs have spent 35 video-seconds. Total. Ever.

A real deployment. The backend runs on an Alibaba Cloud Simple Application Server in us-west-1 — nginx serving the SPA on :80, proxying /api to a uvicorn app, bind-mounted volume so cache and run-state survive restarts. A push to main triggers a GitHub Actions job that SSHes into the box, rebuilds, and gates on the health check.

A catalog deliberately kept off the critical path. Finished runs publish into a Postgres 18 sidecar with media in a private Alibaba OSS bucket, addressed by content hash.

It's additive and flag-gated (CATALOG_ENABLED, default off). Live runs stay on the in-memory store. A run gets mirrored only once it has finished. Publishing tolerates a dead database or unreachable bucket and never raises.

Why? Because a storage outage must not fail a run that already passed its gates. The conformance verdict is the product. Archiving it has a lower right to fail.

One thing bit me immediately, and it's a direct consequence of my own determinism: media paths are many-to-one against content.

Deterministic runs produce byte-identical episodes. So ~40 project paths collapsed onto 13 objects.

That's the cache working as designed. It also means the path→hash mapping needs its own media_paths table — not a column on the object, where it would keep only the last path written and 404 every other one.

The deploy had its own small war stories. A plan upgrade that grew the cloud disk but not the partition inside it — grow the partition first, then the filesystem, because a filesystem can't stretch past its partition. And the discovery that this box is Alibaba Cloud Linux 3, RHEL-family, so it's dnf, not apt.

The kind of thing you only learn by actually putting compute on the cloud, instead of calling its APIs from your laptop.

Why this isn't just a demo

The engine has no coupling to the generator, because its checks take frames, not generator internals.

So it lifts out. It's exposed as an MCP server — the same primitive coding agents already speak — runnable as a package:

uvx --from '.[mcp]' dailies-mcp

ListTools returns two tools, and the pair is the point:

run_shot_tests — zero-token, deterministic Tier-A CV, runs on any mp4. The model-agnostic claim, made executable. It reports.

patch_clip — localizes the first blocking failure in time, anchors a Wan i2v regeneration just before it, re-verifies, and reports patched only if the retake passes. It acts, and spends one generation to do it.

Watching a Qwen agent drive this loop was genuinely satisfying. It calls the free tool. Gets back a real FAIL — camera static, |v|=0.00. Explains it in plain English. Then offers the repair — naming it correctly as keyframe-to-video generation, which it only knows because ListTools handed it patch_clip's description.

It never fires it, because its system message forbids the tool that spends quota.

A gate an agent can read is a gate an agent can act on.

Both ends of that loop are mine. Dailies is an MCP producer, and the client consuming it is a Qwen-Agent Assistant with my own server in its mcpServers block.

That same engine is reachable three ways — a native function-calling tool, a Qwen-Agent custom skill via @register_tool, and the MCP server — over one shared core that imports neither openai nor qwen_agent.

Let me be honest about the edges, because the whole ethos here is not claiming what the engine can't do.

Assertions are per-shot and evaluated whole-clip. There's no audio transcript check, no OCR, no time-windowing (a "logo in the first 3 seconds" check), no episode-level concept yet.

When a rule needs a modality the vocabulary lacks, the compiler omits it rather than approximating it with a type that means something else. Known gap: that omission is currently silent.

Those are roadmap, not shipped features.

What Qwen Cloud actually gave me

The integration splits along a line the platform draws for you, and picking the right side per capability is most of what "sophisticated use" means here.

The OpenAI-compatible endpoint (dashscope-intl.aliyuncs.com/compatible-mode/v1) fronts everything text- and vision-shaped. qwen-plus for scripting, repair, and both function-calling loops. qwen-vl-plus for Tier-0 and Tier-B verdicts.

The native DashScope task API handles the media models, because generation is long-running and the OpenAI-compatible shape has no vocabulary for it. wan2.1-t2v-turbo drafts, wan2.2-i2v-flash frame-anchored finals and repairs, wan2.1-t2i-plus Tier-0 stills, qwen3-tts-flash narration.

Video generation is mandatorily asynchronous — create with X-DashScope-Async: enable, then poll. And it carries a gotcha I'm very glad I found on day one, at zero cost:

HTTP 200 on create does not mean the request was valid.

Validation surfaces asynchronously, on the first poll. So retries must branch on the polled task_status, never the POST's status code. (The image endpoint, confusingly, validates synchronously and returns a 400.)

One more finding I'd reuse anywhere: the video endpoints accept a base64 data: URI in place of an image URL. That's what my entire repair feature depends on, because a frame extracted on a 127.0.0.1 box has no public URL and now needs no upload step.

And notice what's missing from that model list. The deterministic tier.

Tier-A is OpenCV. It calls nothing. The Qwen surface gets spent on generation and judgment — never on measurement.

Which is the whole reason measurement can run on every take.

Key Takeaways:

Test the artifact, not the prompt. Orchestrating models is only half the problem, and it's the easy half. The hard half — the one almost nobody demos — is deciding when not to call the expensive model, and being able to prove with a number that the thing you didn't render was one you were right to reject.
A mock can only confirm your own mental model. It returns what you told it to, so a suite of mocks re-asserts your assumptions back at you in green. The real contract lives at the boundary with the external service. If you're integrating any generative API, spend the tokens on one real end-to-end run early. Cheapest bug-finder you'll ever build.
Let the model choose, never let it spell. Don't validate the model's output — make invalid output unrepresentable. Ask what the model actually needs to decide, which is almost always far less than what you're asking it to emit. A closed vocabulary, a parameter contract, a probed roster: three shapes a bad value can't fit through.
Generalize by measurement, not by resemblance. My best idea worked beautifully in the case I built it for, so I extended it everywhere by analogy. Two real generations proved half of it wrong. The failure mode isn't that the primitive is bad — it's that you've silently changed what you're asking it to preserve.
Watch for the bugs that present as success. A fallback returning the success value. A missing result that reads as a passing one. Two code paths deriving the same value independently. A wrapper whose subject moved. None of these fail loudly, and green tests love them.

So, here's the recap:

· AI video fails in motion, after you've already paid for it
· The fix isn't better prompts — it's a spec you can assert against
· Deterministic checks first, model judgment on top, never underneath
· Verification doesn't just reject bad output; sometimes it deletes spend
· Every bug worth finding looked like success first

Dailies is open source (MIT), built for Track 2 of the Global AI Hackathon Series with Qwen Cloud, and deployed on Alibaba Cloud.

Go test your generated video like you test your code.

Happy shipping!

How to contribute to a large codebase?

Kaung Zin Hein — Thu, 03 Oct 2024 03:19:56 +0000

A step-by-step guide on contributing to open-source projects with thousands of lines of code

As a budding open-source contributor, I was at a loss when I found a lack of resources on how to get started. Other than a few good guides, most of the advice is general and, often, shallow. I felt like the concrete steps were brushed off and left to the readers to figure out for themselves. Which, in a sense, makes for a great learning experience. At the same time, I felt overwhelmed by the number of projects on GitHub, by the number of issues in major projects.

Even for a developer with years of experience, approaching a new codebase takes domain-specific knowledge and patience. On top of that, errors thrown when navigating through the code maze may not even be due to a lack of skills - poorly documented build guides, platform incompatibility, etc., may explain why the repository you just cloned doesn’t work.

With that said, a pre-requisite before diving into a large codebase is to get familiar with contributing to small-to-medium-sized repositories. The best projects to contribute to are ones you’re most knowledgeable about, ones that you’re most familiar with.

But here’s the catch. A lot of what we use daily may not be a simple library that solves one thing. It may well be a full-blown framework or a full-stack application. That’s why it’s important to have honed in on the foundational concepts and the semantics of your language. I don’t mean mastery - which is a continuous pursuit, rather than an accomplishment - I’m referring to an attempt.

Your attempt at trying to master something. For example, I wrote a few npm modules to better familiarize myself with the native Node.js APIs. Why Node.js? Because my goal was/is contribute and learn from other web technologies built on the Javascript ecosystem. I’ll talk more about writing utilities in another guide.

For now, here’s my approach when dealing with a large open-source repository:

1. Use the tool so you understand, what some call, the business problem.
How is it helping customers or developers?

The ideal project to contribute to is one whose API you’ve been using regularly. Whose functions and classes you know well enough to integrate it into your applications and other projects. Familiarity with the public implementations lend itself to understanding the internal code better. If it’s a project you’re completely new to but are willing to dig into, the “Getting Started” guide from the official docs is a great starting point.

2. Start with the what.

Reading a codebase in general terms (the main functions, the entry point, the layers, etc.) is a good practice. But what’s even more effective? A targeted approach.

Trying to solve an issue and following the code trail that the problem leads to, may not cover as many lines of code as passively reading, but it puts to test your understanding of a small subset of the repository. Here's a task for you: open an issue you’ve run into (ideal) or pick up a manageable issue. This approach, compounded over time, issues, and PRs, can lead to deeper understanding.

It’s tricky to define ‘manageable’ because not every project owner assigns ‘good-first-issue’ or ‘up-for-grabs’ labels. And depending on the project, even issues with those beginner-friendly tags may still seem daunting. Which brings us to the next step.

3. Analyze how existing modules and tests are set up.

Take those as reference in laying out the structure of your code and tests. For example, when adding a new plugin, study how other integrations are written and copy the outline of the corresponding tests.

Doing so ensures that you’re following best practices that are specific to the repository at hand and staying in sync with other contributors’ code style.

4. Write down the build and test process.

Note-taking doesn’t get the credit it deserves especially in the software development circle. That’s because note-taking is a skill many haven’t mastered yet, or not willing to try at all.

Instead of looking for patterns, some mark everything down, quickly leading to burnout. Building and testing are processes at the core of most projects. They are the steps you take after making any sort of change. If you’re dealing with a repository more than once (not a YOLO contribution), it’s worth the time to jot down exactly how a change in the code is tested.

Likewise, in a mono-repo, you don’t want to waste time trying to build and test every package when the changes you’ve made only affect one or a few. But wait, can’t the compiling and testing for one package be done via a script each? Probably yes. But when you’re fiddling with more than a single package and their workspace dependencies, the processes tend to involve a couple of steps.

The practice is beneficial in 2 main ways:

A) When you return to an issue after taking a break, all the steps you’d taken last time are right before you. So, you can focus on the problem-solving part immediately, instead of trying to recall what you did the last time for building/testing/modifying configs.

B) When tackling a similar issue, you can follow the same process, instead of figuring out the steps from scratch. This saves time! Which matters a lot when your goal is to ship quality features/fixes as quick as possible.

5. Apply forensics on the commit history of a particular block of code.

When figuring out why a piece of code is implemented, reviewing not only the related PRs, but also the surrounding discussion (comments, reviews, linked issues/PRs) leaves clues - often outright solutions. This step helps you learn about the thought process and the decisions that were formed by other contributors before you arrived. For example, you can check why the author introduced a new utility function or how a bug was fixed.

6. Take feedback from maintainers/major contributors to heart.

No one’s more familiar with the codebase than those who’ve spent months and years working on it. Try to fix the pain points that they’ve brought up, especially in a code review.

On the other hand, don’t be too sorry to point out their misunderstanding of the intent of your code or to state your doubt on their arguments. Open-minded senior devs are willing to elaborate and are, well, open to feedback themselves.

Summary with Actions

1. Use the product/library/framework.
Action: If it’s not a tool you’re already familiar with, try it out to get the hang of it. Start from the official docs.

2. Follow a bottom-up approach.
Action: Open or pick up an issue. Analyze the necessary code files and functions as you try to solve the issue.

3. Look at the structure of existing modules and tests.
Action: Analyze and apply a similar (or sometimes the same) structure to your code.

4. Write down the build and test process.
Action: Record how exactly you compile and test the code involved in your PR.

5. Learn from the past.
Action: Review the commit history and the related conversations, code reviews, and comments.

6. Accept constructive feedback.
Action: Review and try to resolve the code reviews from senior contributors. Don’t hesitate to explain any misunderstanding.

Happy contributing!

Filtering in ExpressJS

Kaung Zin Hein — Fri, 16 Aug 2024 23:29:16 +0000

How I implemented the filter feature in my volunteer management application

This article is part of the series explaining the principles and algorithms implemented in my capstone web application project. You can check out the app here. Or view the source code.

Introduction

A major ingredient for modern management applications (be it content, e-commerce, customer, or financial) is the filter feature, alongside sorting and searching. Here’s how I’ve implemented it in my web app. It filters volunteer events in the backend, according to the filter category sent by the client.

" width="800" height="406">

The goal here is to build a custom filter feature using what Express can offer, without depending on any third-party library. So, I decided to make use of the request-response lifecycle and the “locals” property of the response.

Following the modern MCV architecture for ExpressJS applications, I’ve refactored the previous code by separating business logic from the controller. This logic is contained in service functions. Let’s take a look at the filter controller first.

Filter events controller

This controller is an array of request handlers, each wrapped in an express-async-handler function to manage asynchronous errors. Its length depends on the number of filter categories that are stored as constants. As of now, these are:

Event date (date)
Event venue (venue)
Events with open positions (isOpen)
Events that are happening in the future (isUpcoming)

Plus, the service function of the last handler is responsible for curating the filtered results (event IDs) from the above 4 filters and sorting each result based on how many filter categories it falls under. So, I’ve got an array of 5 request handlers.

Let’s examine the request handler for the event venue category (venue).

Filter Events by Venue

const filterEventsHandler = [
  //venue filter
  asyncHandler(async (req, res, next) => {
    if (!objKeysIncludes(req.body, FILTER_OPTIONS.VENUE)) {
      return next();
    }

    const { venue } = req.body;
    res.locals.filteredVenue = await filterEventsByVenue(venue);

    next();
  }),

//...more handlers

Inside the handler, there’s a helper function that checks whether or not req.body (sent from the client) includes the category “venue”. This function serves as a form of data validation for the user input.

const objKeysIncludes = (obj, checked) => {

    if(!(obj instanceof Object)){
        throw new Error('Obj param must be an object')
    }

    return Object.keys(obj).includes(checked)
}

If it’s included, the value of the venue is extracted (e.g., {venue: “External”}), after which it’s processed by the service that filters events by venue. The result is an array of event IDs, which are then stored in res.locals property. The data/variable stored() in res.locals is not shared between requests as they belong only to the current request-response cycle. If the client didn’t include the “venue” filter, this request handler is skipped by returning the next function.

if (!objKeysIncludes(req.body, FILTER_OPTIONS.VENUE)) {
      return next();
    }

    const { venue } = req.body;
    res.locals.filteredVenue = await filterEventsByVenue(venue);

    next();

Even after storing the filtered result in res.locals, next has to be called, moving on to the following handler that checks for another filter category.

Final Handler

After repeating this process for all categories, we arrive at the last handler. Its service function (filterEvents) sorts all the filtered event IDs from res.locals, attaches the filter tags that each event belongs to, and returns a sorted array of events based on the number of tags.

  //sorted filter
  asyncHandler(async (req, res) => {

    const {
      filteredResultsByKey,
      filteredAllIds,
      idsWithTags,
      sortedIdsWithTags,
    } = filterEvents(req.body, res.locals);

    return res.status(200).json({
      filteredResultsByKey,
      filteredAllIds,
      idsWithTags,
      sortedIdsWithTags,
    });
  }),

Let’s see how it works.

The values of the properties in res.locals stored by the preceding filter handlers are assigned to respective variables. Object destructing is not used because the client may not have selected all filter categories, making these variables undefined.

const filterEvents = (filterObj, resLocals) => {

  const filteredVenue = resLocals.filteredVenue;
  const filteredDate = resLocals.filteredDate;
  const filteredIsOpen = resLocals.filteredIsOpen;
  const filteredIsUpcoming = resLocals.filteredIsUpcoming;

//...
Then, the filteredResultsByKey object is declared which will house the name of the filter category (e.g., venue, isOpen) as the key, and the value of the corresponding filtered result as the property. See the example below, where each event ID qualifies as the selected venue.

// {"venue": [id1, id2, id3]}

let filteredResultsByKey = {};

Object.keys(filterObj).forEach((filterKey) => {
    switch (filterKey) {
      case FILTER_OPTIONS.DATE:
        filteredResultsByKey = {
          ...filteredResultsByKey,
          [filterKey]: filteredDate,
        };
        break;

      case FILTER_OPTIONS.IS_OPEN:
        filteredResultsByKey = {
          ...filteredResultsByKey,
          [filterKey]: filteredIsOpen,
        };

        break;

      case FILTER_OPTIONS.VENUE:
        // filteredResultsByKey[filterKey] = filteredVenue

        filteredResultsByKey = {
          ...filteredResultsByKey,
          [filterKey]: filteredVenue,
        };

        break;

      case FILTER_OPTIONS.IS_UPCOMING:
        filteredResultsByKey = {
          ...filteredResultsByKey,
          [filterKey]: filteredIsUpcoming,
        };

      default:
        break;
    }
  });

Next, the property values of filteredResultsByKey are turned into a 1-D array (from a 2-D array). This service function returns an array of event IDs, each with only one occurrence, which is stored in the filteredAllIds variable.

 const filteredAllIds = filterArrSortLoose(
    Object.values(filteredResultsByKey)
  );
//filterArrSortLoose helper
const filterArrSortLoose = (twoDArr) => {

  if (!twoDArr.every((arr) => Array.isArray(arr))) {
    throw new Error("each elem must be an arr - an arr of arr");
  }

  const oneDArr = twoDArr.flatMap((arr) => arr);

  const matchIds = [];

  oneDArr.forEach((id) => {
    if (!matchIds.includes(id)) {
      matchIds.push(id);
    }
  });

  console.log("matchIds from loose filter: ", matchIds);
  return matchIds;
};

Filter Tags

Finally, it’s time for the most challenging task yet. Assign filter tags to each event ID. To do so, we need to relate the filteredResultsByKey object and filteredAllIds array, resulting in an idsWithTags array.

//filteredResultsByKey
{
filterKey1 : [id1, id2, id3],
filterKey2 : [id3, id1, id4]
}

//filteredAllIds
[id1, id2, id3, id4]

//filteredResultsByKey + filteredAllIds -> idsWithTags
[
 {id1: [{filterKey1: filterValue}, {filterKey2: filterValue}]
 {id2: [{filterKey2: filterValue}]},
 {id3: [{filterKey1: filterValue}]}.
 ...
 }
]

Remember that an event ID can appear under more than one filter category. In this case, we assign all the associated filter tags to the event ID object.

In the following nested loop, the outer loop iterates over all filtered IDs, and the inner loop runs through each entry of the filteredResultsByKey object.

  filteredAllIds.forEach((id) => {

    Object.entries(filteredResultsByKey).forEach(([filterKey, result]) => {

      const [_, filterKeyVal] = Object.entries(filterObj).find(
        ([key, _]) => key === filterKey
      );

//...

For each event ID, we must find the corresponding filter category it belongs to by iterating through each entry in the filteredResultsByKey and cross-checking with req.body (filterObj).

If the key from the req.body equals the current key of filteredResultsByKey, we access the value of the key (e.g., in {date: “2023–12–31”, 2023–12–31 is the value). This filter category value (filterKeyVal object) and the filter key are appended as a filter tag for each qualifying event ID.

If the current event id (outer loop) is included in a property array of filteredResultsByKey (inner), there are two possible scenarios:

The event ID object is already included in the idsWithTags array, which means the event qualifies for more than one filter category.
The event ID object is not included yet.

Both scenarios are determined by finding in the idsWithTags array an object whose eventId property has the same value as the current event id. The responsible helper function returns an index number if found, and -1 if not found.

  if (result.includes(id)) {

        const isEventIdAlrExists = elemObjPropValIncludes(
          idsWithTags,
          "eventId",
          id
        );
  //...

//helper to determine if current event ID is already in idsWithTags
const elemObjPropValIncludes = (arr, propKey, checkedPropVal) => {


    //propKey must exists in all objs in the array
    if(arr.some(obj => (!Object.hasOwn(obj,propKey)))){

        throw new Error(`all elem obj in the arr must contain propKey: ${propKey} : as a property key`)
    }


    const included = arr.findIndex(obj => obj[propKey].toString() === checkedPropVal.toString())

    return included
}
Dealing with the second scenario is simple. We just push an object with “eventId” and “filterTags” entries into the parent array.

 idsWithTags.push({
            eventId: id,
            filterTags: [{ [filterKey]: filterKeyVal }],
          });

For the first case, however, there’s an extra step involved. We must map through the idsWithTags array until we arrive at the current event. Then, reassign that event ID object with the additional filter category entry it belongs to.

  const bufferArr = idsWithTags.map((event) => {

            if (event.eventId === id) {

              event = {
                ...event,
                filterTags: [
                  ...event.filterTags,
                  { [filterKey]: filterKeyVal },
                ],
              };
            }

            return event;
          });

          idsWithTags = bufferArr;

After assigning tags to each event, we must sort them in descending order based on the number of tags. At the end of the request-response cycle, this final handler returns sortedIdsWithTags to the client. It’s now the front-end’s job to manage how the events with filtered tags will be displayed.

That’s it! That’s how I’ve implemented filtering in my Express API using the nature of the request-response cycle and res.locals property.

The key takeaways I got from coming up with this feature are:

Request-response cycle
Modular middleware and the role of next()
Data Validation
Higher Order functions
Filtering logic!

Further improvements in the future:

Use Map and Set for a simpler data structure
Implementing Bloom’s filter algorithm
You can interact with the app live here. Or view the code on my GitHub repo.

Why write a library?

Kaung Zin Hein — Fri, 03 May 2024 04:01:54 +0000

This is a blog version of the talk I gave at TorontoJS Lightning Talks event on Apr 30, 2024.

My technical pet peeve

This irritates me - typing a variable’s name beside the variable in the log statement. It isn’t that big of a problem for just a few variables here and there. But many variables multiplied by many files in your app? You do the math.

That's when I realized writing my own logging library could save me some headaches.

The rant continues...

Now, who enjoys commenting out or deleting log statements before pushing their code to production? I really don't.

Fortunately, there are tools available to solve this such as the debug library which is based on the core NodeJS logging utility.

We already have a solution. So why bother? Let’s take a look at my personal use case:

More often than not, a variable is logged from within nested function calls, and across multiple files. When I wanted to quickly check from where a variable is being logged (that happens after you’re debugging and have called it a day and you return to it later), I would write the metadata (the function name) next to the log variable.

Instead of a quick check, this process became quickly tedious. Here was a use case that i hadn't found any JS library addressing (as far as I know).

Time to modify the wheel!

So, I decided to write a module that logs the value of the variable alongside its name without having to type the name manually. And I'll share with you how I did it from scratch.

Of course, this can be done using console.log() with the object destructing { } property.

But what console.log() lacks is the ability to print the function scopes of the logged variable in a developer-friendly way. Here's the output:

By developer-friendly, I mean anyone seeing this log message in their terminal can immediately tell the function scope of the variable being logged. The order of function scopes is parsed from the call stack provided by the NodeJS stack trace API. Hence the name, scope-logger.

Note: “Object.” just means the outermost function is invoked from the global object/topmost level of the file.

Here's the recipe on how to write a library by studying others (in pseudo-code!):

Steps.length = 4

Step 0: Just build it!

“Zero” not only because we are programmers, also because it’s one fundamental step before taking any further. You must be building the Minimum Viable Product (MVP) as soon as possible, or at the very least, a small chunk of it.

You’re not striving for perfection. You just need a working version.

Step 1: Start small

As a reference, pick a small library with similar use cases to your idea.
Here’s my criteria for selecting one:

Popular: not because it’s the new kid on the block, but because it’s been well-maintained for a long while with a large, active user-base.
Well-tested: because quality is important - look for integration/e2e tests and unit tests.
Challenging enough for analysis: you don’t wanna be tackling the Express framework or React on your first go. Pick a library that is considerably smaller, which you can determine by the following factor.
Number of dependencies: one way to tell if a library is not too challenging to be used as study source is based on the production dependencies count. The fewer the better. For example, I chose debug because it only has 1 dependency (ms), while the rest of the code relies on core NodeJS modules - which is exactly what I was looking for - to learn how to build a library from scratch, not off the shelf libraries with many external deps, which in turn are based on more deps. There you go, dependency hell.

Step 2: Break it down smaller

After you’ve picked a library, break it down even smaller. Study how the main functions work by replicating their behaviors: isolate those functions, provide input, and see the output.

This is one of the few times that you should allow yourself to fall into rabbit holes, and really understand the in’s and out’s of each line of code in a function.

For example:

This is the formatArgs() function of the debug library. By isolating it, I learned about ANSI colour escape codes, and where and how the properties of one debug instance is used (namespace, useColours, and this.colour)

Step 3: Publish

After you’ve studied other approaches and built your own module, all there's left to do is hit publish, right? Well, not before you complete these three things.

Semantic Versioning: for every update (major, minor, or patch) made, increment the version number according to semantic versioning.
Changelog/history: track and display the updates you’ve made so far - the version number, the date you published it, and a brief description of those changes.

Documentation: You cannot expect other developers to magically figure out how to use your library without detailed descriptions of the features. It also serves as a roadmap of what you’ve built. You can either dedicate a whole website for it or simply fill in the README.md file.

Note: I left out tests because we should be writing them as the library is being built, not just before hitting publish.

Recipe Recap

Zero: you build a working version.
One: start small by picking a small but challenging enough library to learn from
Two: analyze that library by breaking it down into even smaller pieces
Three: after attaching semantic versioning, changelog, and documentation, hit publish.

Breaks are Great!

These steps are in a loop, repeated until it’s time for a break, which is just as important as the preceding steps.

As a rule of thumb, after publishing your major or minor version, or several patch updates in a row, take your well-deserved break!

Taking a break is great for discovering more use cases as you use the library/module you just made in other projects.

As you might’ve guessed, I did use scope-logger to develop other modules. By doing so, I found features that I really needed, not just stuff that I thought would be cool to have.

When a particular use case keeps popping up, that’s my cue to update the library with a new feature to address it.

Key Takeaways:

Solve your own problems first: As a budding developer, how do you go about determining what projects to pursue? The idea for this library occurred to me as I was building projects for my portfolio. With limited industry connections and programming know-how to cater to a target audience, I felt stuck. But there lurked a hidden opportunity. If you're in the same boat, try this: ditch the latest trends, and double down on understanding the nuts and bolts of one technology, to tackle a problem that's currently bogging you down.
Reinvent the wheel, sometimes: And to solve your developer problem, contrary to conventional advice, you must reinvent the wheel to understand in-depth how the wheel is built. In the long haul, the time and effort put into studying a great library, and building one using core modules and fundamental programming principles, is what’s gonna make you stand out.
Depth && Breadth (not versus): With that said, depth is not the superior path. Tread both. The breadth part involves exploring other libraries, studying their differences, and observing patterns that show up repeatedly.

So, I invite you to try this project idea - create a library by studying great examples. As slow as it may seem, your programming craft will grow exponentially.