DEV Community

Shimona Carvalho
Shimona Carvalho

Posted on Edited on

Harnesses! The other kind

Lately I've been noticing how much more I'm leaning on building test harnesses that I would never have bothered building before.

I have three test harnesses now that I am using to verify my software and part of the reason here is when you have LLMs writing code you want to really improve your verification game. Another thing is that things that might have required a prohibitive amount of effort to test are now no longer that much effort with the help of LLMs.

So I have 3 harnesses I've added on to the usual vite/jest/pytest type testing I do on my site and apps.

  1. An upload harness - this is an e2e reliability rig that tests my upload mechanism. Part of my app requires uploading lots of image files to an S3 bucket using pre-signed URLs that will later be added to photo albums. These are reasonably big files which may take a long time, the user might get bored and close the browser window or switch to another browser window, their network might be slow, the s3 bucket might go down intermittently.

    So to mimic this variety of errors happening while the browser is uploading upto 100 large files, I made a harness that injects a variety of errors such as

    • S3 bucket stops responding
    • S3 presigned urls expire or fail
    • Transactions time out once
    • Transactions retry and get exhausted
    • Responses get dropped
    • Latency injection

    It deliberately does not mock the upload pipeline. Instead it drives the real UI - file input, image grid, retry buttons - against a real backend and a real S3-compatible store. This helps test the connections between browser, presigned storage PUTs, batching and confirmations, Celery tasks, and the database.

    It's playwright, it's slow, I don't run it in CI, but anytime we touch anything near the upload pipeline, it gives me so much peace to run this and ensure it is still passing.

  2. A celery orchestration harness.

    I have celery jobs that first run image analysis on all the images and then combine all that data to run album level analysis to cluster and build the album. Once that's all done, I run an llm to generate blurbs for the album. So I really don't want a single job failure somewhere to stall this whole workflow.

    Enter a synthetic orchestration harness. It spins up a throwaway Postgres + Redis in Docker, runs the production Celery app with workers and starts the tasks over 100 photos. The job dependencies are the same as in the prod app, but the actual tasks are synthetic, if that makes sense. They don’t actually do image analysis.

    I seed random faults (errors, delay, hang, child exit) and also have a recovery process in celery beat, so if things stall (which they should with the seeded errors) the beat process should kick them back into completing the job, even with imperfect data.

    The goal is to verify the coordination layer - delivery, retries, "chords", retry-safe sends, recovery - which unit tests can't really test. Again, not in CI, but a great way to enforce reliability.

    Let's just say I would never have built this without LLMs, just a lot of effort that I wouldn't have had time for, certainly not as a solo dev.

Top comments (0)