DEV Community: Choon-Siang Lai

Recreating the Matrix Rain with Pygame: Manual Fades and the Transparency of Code.

Choon-Siang Lai — Fri, 19 Dec 2025 19:24:31 +0000

Earlier in the year, we explored Pygame by writing a simple tic-tac-toe game. The exercise unexpectedly yielded a reusable mini framework. Despite being far from “production-ready”, it is still sufficiently good for smaller-scale experiments. One of the possible projects mentioned towards the end of the article was a remake of the video intro with the Matrix code rain effect. Even though I phrased it as a joke at the time, we are diving into it in this article.

A cute cartoon generated by my editor Gemini

Returning to the Source: A Baby-Steps Approach

We are taking a baby-steps approach in this remake project, as I am rather tied up with my full-time work commitments. In this installment, we are only focusing on the fade-out animation and will continue working towards a full recreation in the future. The fade-out effect in the animation was made possible through the use of jQuery’s .fadeOut() method, which fades an element out in a specified duration.

How do we do it in jQuery, one may wonder? The snippet below shows how it is being done in the context of a web application.

<!-- html -->
<p id="target">Lorem ipsum dolor sit amet</p>

<script>
  // jQuery
  $("#target").fadeOut()
</script>

Deconstructing the Magic: Recreating .fadeOut()

Now that we know how it’s done with jQuery, the logical next step is to start defining the problem. Revisions to the framework may be required if there’s a need to support the specific effect. Actual implementation comes a bit later, after drafting a proper plan. Read on to follow along.

What is the problem we are solving today?

We want to make it possible to render a character on the screen, and repeatedly re-render the character with a different alpha value until it reaches 0. Each character is expected to be rendered over a pygame.Rect and we are assuming there are no overlaps in our application. Additionally, the alpha value decreases linearly over the specified duration, which should keep the implementation simple.

a screen recording of the desired outcome

Next, we figure out what is needed from the framework. In order to trigger a re-render at a consistent interval, we need to post an event in our rendering loop. Let’s call it FRAME_NEXT, and add it to the enum of SystemEvent alongside the INIT event we saw in the tic-tac-toe project.

That yields an updated display_update function, as shown below:

async def display_update(
    screen_update: asyncio.Queue, clock: pygame.time.Clock
) -> None:
    clock.tick(60)

    updates = []

    with suppress(asyncio.queues.QueueEmpty):
        while element := screen_update.get_nowait():
            updates.append(element.position)

    pygame.display.update(updates)

    # Trigger new FRAME_NEXT event
    pygame.event.post(pygame.event.Event(SystemEvent.FRAME_NEXT.value))

Printing a character to the screen is similar to how we draw a circle or a cross in the tic-tac-toe project. First, we draw a pygame.Rect, then we draw the rendered text surface object over it. Start by rendering the character, and we get the corresponding text_surface and the rect enclosing it:

import pygame.freetype

font = pygame.freetype.Font("/path/to/font.ttf", font_size)

text_surface, rect = font.render('a', (red, green, blue, alpha))

The self-explanatory font_size, red, green, blue, alpha are all integers, and the latter four are in the range of [0, 255].

Given both the text_surface and enclosing rect, we can print them to the screen at any desired location as follows:

# optionally we can move the rect
rect.topleft = (x, y)

# first draw the backing rect
pygame.draw.rect(application.screen, (0, 0, 0), rect)

# secondly draw the rendered character
application.screen.blit(text_surface, rect)

We can finally assemble two functions, one to create an initial print, and another to update the region with the same character but with a different alpha value:

@dataclass(frozen=True)
class CharElem(Element):
    value: str = " "
    font: pygame.freetype.Font | None = None
    color: tuple[int, int, int] = (0, 0, 0)
    alpha: int = 0

async def char_create(
    application: Application,
    value: str,
    size: int,
    color: tuple[int, int, int],
    alpha: int,
    position: tuple[int, int],
) -> CharElem:
    assert len(value) == 1 and isinstance(value, str)

    font = pygame.freetype.Font("/path/to/font.ttf", size)

    surface, rect = font.render(value, (255, 255, 255, alpha))

    rect.topleft = position

    pygame.draw.rect(application.screen, (0, 0, 0), rect)
    application.screen.blit(surface, rect)

    return CharElem(position=rect, value=value, font=font, color=color, alpha=alpha)

async def char_update(application: Application, elem: CharElem, alpha: int) -> CharElem:
    assert elem.position
    assert elem.font

    surface, _ = elem.font.render(elem.value, elem.color + (alpha,))

    pygame.draw.rect(application.screen, (0, 0, 0), elem.position)
    application.screen.blit(surface, elem.position)

    return replace(elem, alpha=alpha)

Updating a drawn character returned by char_create is done through a handler listening to the FRAME_NEXT event we defined earlier. An extension to the usual element event handler is hence needed to hold two extra pieces of data: the duration and start time. Fortunately, this is trivial with the use of __call__ dunder:

@dataclass
class FadeOutHandler:
    duration: int
    start_time: int = field(default_factory=lambda: monotonic_ns() // 1_000_000)

    async def __call__ (
        self,
        event: pygame.event.Event,
        target: CharElem,
        application: Application,
        logger: BoundLogger,
        **detail: Any,
    ) -> None:
        current = monotonic_ns() // 1_000_000
        elapsed = current - self.start_time
        remaining = 1.0 - (elapsed / self.duration)
        alpha = int(max(255 * remaining, 0))

        elem = await char_update(application, target, alpha)

        asyncio.create_task(screen_update(application, elem))

        if alpha > 0:
            asyncio.create_task(
                application.delta_data.put(
                    DeltaUpdate(ApplicationDataField.ELEMENTS, target, elem)
                )
            )
        else:
            asyncio.create_task(
                application.delta_data.put(
                    DeltaDelete(ApplicationDataField.ELEMENTS, target)
                )
            )

Some points worth noting regarding the snippet:

As with the .fadeOut(duration) offered by jQuery, we define the duration in milliseconds
We remain somewhat stateless by storing just the start_time, and we calculate the alpha value at any time by comparing it to the percentage of remaining time.
After alpha reaches 0, remember to remove the element reference from the application

The Versatility of __call__: A Python Developer's Secret Weapon

Registering the event can then be done through a helper function:

async def fade_out(application: Application, elem: CharElem, duration: int) -> CharElem:
    return await add_event_listener(
        elem,
        SystemEvent.FRAME_NEXT.value,
        FadeOutHandler(duration),
    )

Are we done yet?

One last piece of the puzzle is the click event handler, where a character is shown and eventually fades out into oblivion wherever the user clicks on the screen.

async def handle_click(
    event: pygame.event.Event, target: Application, logger: BoundLogger
) -> None:
    elem = await fade_out(
        target,
        await char_create(target, "a", 20, (255, 255, 0), 255, event.pos),
        1000,
    )

    asyncio.create_task(screen_update(target, elem))
    asyncio.create_task(
        target.delta_data.put(DeltaAdd(ApplicationDataField.ELEMENTS, elem))
    )

We are almost done, though I am skipping the boilerplate bootstrapping code as it is rather similar to the tic-tac-toe application. Basically, the code sets up the window, paints the background colour, and registers the click event. For those interested, do check the GitHub repository for the complete application.

GitHub - Jeffrey04/kitfucoda-intro2

The Cracks in the Framework: Scalability and Loops

Some expansion to the framework was already done when I was building the tic-tac-toe game earlier. Apart from some small syntactic changes made to the API, I was pleasantly surprised to find the overall framework design was left untouched. That’s good news, no?

A revision to the DeltaOperation was done previously to consolidate all changes to the Application object into one queue. It was a sensible improvement to the original implementation, but I still don’t quite like the complexity of the queue consumption. Sooner or later scalability will become an issue we want to address when the need to draw hundreds of elements at a time arises.

Furthermore, another pain point occurs in the event dispatching loop. Ideally, we want a quick lookup that is close to O(1). Unfortunately, we are looping through elements followed by events on every cycle. Performance is definitely taking a hit as we are running everything concurrently and we are wasting so much time looping needlessly in lookup operations.

What’s next then? I suppose you may wonder where we are headed, if you are still following thus far.

Apparently the refactoring that was done to the DeltaOperation somewhat resembles the Actor model. Further research and experiments are needed to see how it fits the design. From what I know so far, the model offers a cleaner separation of data storage and retrieval logic from the rest of the application. Hopefully that possibility of clean separation allows some independence. This can be helpful as it enables us to further split different parts of the application to run in parallel. This would be challenging, given how Pygame is not even thread-safe.

That means the next installment should focus on the optimization instead of continuing the momentum of building the next feature — the chain of characters raining down the screen.

Visibility, Implementation, and Identity

Just to recap, we achieved the goal of building an almost equivalent to jQuery’s .fadeOut() effect that works with our mini Pygame framework. An exact copy is never the goal, considering the rendering technique is different and we are synchronizing the change with the frame rate. Implementing the gradual decrease of alpha value really does take some effort in designing and planning. This is truly an interesting learning journey and I hope you benefit from following along.

It really does make you appreciate all the work done to provide the functionality through a simple method call.

The reimplementation in this low-level setting leads to a personal reflection related to recent news that deeply saddens me. There seem to be ongoing raid operations targeting men-only wellness centres. In one of those operations, a video was leaked across social media where the arrested patrons could be seen unclothed. A list with enough identifying information sans the name was also circulated.

Every line of code written in the low-level setting is very explicit and corresponds to a visible consequence. This kind of transparency prompted a revelation where, as a gay man, it is crucial to increase the visibility of members of the sexual minority. There are a lot of us in society working earnestly and also actively taking part in our civil duties. It is important to recognize that everyone in society is multi-faceted. As long as no harm is caused, one’s private life should remain private. Judging a person solely on visiting a wellness centre is simply unfair. Even the authorities fail to bring the patrons to court due to lack of evidence in one of the raids. Unfortunately, some still faced repercussions at work, especially the public servants.

Hopefully you gain insights while reading the articles published here on KitFu Coda, which is my ultimate goal as a fellow engineer. My sexuality indeed doesn’t matter to the content, but it is the best way I could think of to voice my support and solidarity with those who are still suffering the consequences to this day.

Lastly, thank you for reading, and I shall write again soon.

This article was written with editorial assistance from Gemini to refine the language and structure. Please note that while the AI helped catch errors, all project ideas, code algorithms, and personal perspectives are my own original work.

The Concurrency Choreographer: Making BLE’s Async Data Sync with OBS

Choon-Siang Lai — Tue, 25 Nov 2025 02:53:18 +0000

“What can we do with asynchronous programming?”, This was one of the questions I received in my talk on asynchronous programming that I delivered in Pycon MY. While I briefly mentioned building a chatbot that day. I believe the project we are discussing today should resonate more. During my spare time (however rare it is these days), I stream my gameplay session to document my journey. One of the widgets in my streaming session is a heart rate monitor, and it has been broken for some time.

Cute cartoon on the topic generated by Gemini

Seeking the Open Standard: From Fitbit’s Lock-in to Garmin’s BLE

I was doing some spring-cleaning for my OBS setup, and one component caught my eye — the heart rate monitor. For a time, it was a novel way to convey a streamer’s psychological state in a numerical form. Coincidentally, I was already wearing a fitness tracker: Fitbit Versa 3 and I decided to give it a go. Fortunately, there was a way to capture heart rate through the somewhat limited proprietary API, and have it streamed to my computer. Problem is, the fitness (not so) smartwatch died.

Funnily enough, a family member bought an almost identical Fitbit Sense a few months earlier and her watch is still functioning as of now. In a way the watch died rather prematurely. On the other hand, I was out of a job, and yet that didn’t translate to more gaming time as I was juggling with multiple things at the time. Consequently, the replacement only came much later, not long after I started my current job. Instead of getting another Fitbit, I moved on to a Garmin Instinct II, a tracker my peers highly recommended.

Unlike my past Fitbit devices, the Garmin watch does broadcast my heart rate via a standard Bluetooth interface. This makes replacing the workflow a much easier job. For that to work, I just needed to figure out how to get the data, and then supply that data to OBS via the WebSocket interface. The library in use is bleak to communicate with the watch via Bluetooth LE, and obsws-python to publish heart rate to OBS.

Previously, our chatbot project only dealt with asynchronous libraries. Yet, this time the library that communicates with OBS is synchronous. Therefore, we are taking this opportunity to discuss how to coordinate them to work with each other. Specifically, we are solving the problem where the producer (the heart rate monitor data) is asynchronous, and the consumer (the sending of heart rate to OBS) is synchronous.

Building the Async Producer: Connecting via Bluetooth Low Energy

My journey on tracking my fitness data started with reading about running. One of the books I read was titled Body, Mind and Sport by John Douillard. Some of the practices advocated may seem controversial, though the idea of being mindful about my physical state stuck in my mind. As a result, I started running with deep breaths, and soon noticed running could be made bearable or even enjoyable with some effort. At the same time, I went looking for a chest belt to pair with my fitness tracker installed on my phone.

That was before wearable tech went mainstream, so my choices were rather limited. Chest straps to measure heart rate were common among runners, but they weren’t built to work with Bluetooth. I had to search high and low to finally find one that would work with my smartphone application. It was slightly more expensive than an ordinary tracker, but it was worth it. I ran with it for a few years, and benefited from having my current heart rate displayed on my tracker in real time.

No more estimating by poking my fingers to my neck and doing multiplication while I was busy catching my breath.

Photo by Luiz Rogério Nunes on Unsplash

When wearable technology started to take off, Sony later released their smart band. The companion app also provided some limited insight on how my body reacts to activities in my day-to-day life. Conveniently, it also broadcasts my heart rate through Bluetooth and switching to it in my sports tracker took almost no effort. No more having to prepare my chest strap by wetting it before a run — that was a definite win.

Enough storytelling for now, let’s shift our focus back to the project. Like my old Sony smart band I bought second-hand, my Garmin Instinct 2 also broadcasts my heart rate on Bluetooth on demand. A quick search yielded blehrm, and it provides an abstraction to bleak. Unfortunately, it wasn’t the right library for the purpose, and I soon moved on to bleak directly after some failed attempts.

Firstly, we start by finding our device. Call the BleakScanner.discover method with a reasonable timeout, as shown below:

from bleak import BleakScanner

ble_devices = await BleakScanner.discover(timeout=10.0)
print(f"--- Discovered {len(ble_devices)} BLE Devices ---")
for i, device in enumerate(ble_devices):
    print(
        f"#{i}: Name: {device.name}, Address: {device.address}, Details: {device}"
    )

Running that will return (device addresses are replaced with the placeholder

--- Discovered 8 BLE Devices ---
#0: Name: Instinct 2, Address: <address>, Details: <address>: Instinct 2
#1: Name: None, Address: <address>, Details: <address>: None
#2: Name: None, Address: <address>, Details: <address>: None
#3: Name: U-WTD838C, Address: <address>, Details: <address>: U-WTD838C
#4: Name: None, Address: <address>, Details: <address>: None
#5: Name: None, Address: <address>, Details: <address>: None
#6: Name: None, Address: <address>, Details: <address>: None
#7: Name: None, Address: <address>, Details: <address>: None

Jot down the address, as we will need it for the next step: establishing a connection to the device.

from bleak import BleakClient

HR_MEASUREMENT_CHARACTERISTIC_UUID = "00002a37-0000-1000-8000-00805f9b34fb"
async with BleakClient(
    <address>,
    timeout=15.0
) as client:
    await client.start_notify(
        HR_MEASUREMENT_CHARACTERISTIC_UUID,
        heart_rate_handler,
    )

In the snippet, we are sending the address we collected earlier to BleakClient with a reasonable timeout. Then we subscribe to the heart rate measurement event, and delegate the handling to a function.

With the help of Gemini, we saved time digging through the documentation, as it returned a ready-made snippet for the BLE connection and data parsing, which is shown below.

def heart_rate_handler(sender, data):
    """Callback to handle heart rate measurement data."""
    # The heart rate data structure is defined in the BLE spec:
    # Byte 0: Flags (contains format of HR value and if other fields are present)
    # Byte 1: Heart Rate Value (8-bit or 16-bit)

    # Check the first bit of the flags to see if HR is 16-bit (0x01) or 8-bit (0x00)
    flags = data[0]
    is_16_bit = flags & 0x01
    if is_16_bit:
        # 16-bit HR value is at index 1 and 2
        hr_value = unpack("<H", data[1:3])[0]
    else:
        # 8-bit HR value is at index 1
        hr_value = data[1]
    logger.info(f"[{flags:04b}] Heart Rate: {hr_value} BPM")

There we have it, we have half the program done, and it is able to fetch heart rate data periodically from the device.

Bridging the Gap: The Producer-Consumer Pattern

Photo by Lenzil Gonsalves on Unsplash

Continuing the story after I bought my second-hand Sony smart band. Electronics products die inevitably, as Sony wasn’t making a viable successor for the product I had to look elsewhere. That was when I first looked at Fitbit, particularly a Fitbit Charge 3. It was more than doubled the price of my smart band, though it came with more features. One feature I particularly liked was the sleep tracker, as I have really bad sleep pattern. It was working fine, though with a major drawback — I lost the ability to read my heart rate through the sports tracker app.

Since the device came with a screen itself, it did mean there was no need to whip out my phone constantly to check my heart rate. Not being able to log my run with heart rate data attached created a minor annoyance. On the other hand, I moved on to other sports already, namely boxing and weightlifting. The shift in activity did change the need as I didn’t need the data as often. For instance, the short-burst nature of weightlifting did not require constant monitoring of my heart rate; the main focus was just on whether I could lift the dumbbell or otherwise.

Again, the tracker died after serving for several years. I moved on to a Versa 3 on a good deal, despite the mild annoyance. Other brands were considered, but I got confused with the numerous lines of products and decided to stay with Fitbit for another time. Being a smart watch, Fitbit did release an SDK for their Versa watches, but they discontinued application development support when they released the successor, Versa 4. Regrettably, I didn’t put in the time and effort to develop anything as I was adequately happy with it.

As mentioned earlier, I found a way to broadcast my heart rate with an app offered in the Fitbit app store, and that led to the setup of a heart rate widget on my OBS setup. Coincidentally, I acquired Tetris Effect on my workstation, so I was curious to see if my heart rate changed throughout the game play. Surprisingly, it really did, and the recording clearly showed the heart rate fluctuations in action.

https://medium.com/media/018de8b0f30246afd93165f500a77b62/href

Granted, as the tracker was worn on the wrist, the response was somewhat delayed. Despite that, the stream showed as the tetrimino dropped faster, my heart rate also increased correspondingly. More often than not, the height of dropped blocks waiting to be eliminated also played a role in influencing the heart rate.

Now that we have the stream of heart rate data, how do we recreate the widget in OBS through our script?

Open Broadcasting Studio, or OBS in short, offers a way to interact with the sources on screen through the websocket interface. In our case, we are interested in sending our heart rate data to a text source. By using the obsws-python library, we can send the data as soon as we capture it from the device. We can establish this connection to OBS using the following simple code:

from obsws_python import ReqClient

obs = ReqClient(
 host=<host>,
 port=<port>,
 password=<password>,
)

All of these settings can be found in the OBS websocket setup page, as shown in the screenshot below:

Sending heart rate to the source is simple, through the obs.set_input_settings method

obs.set_input_settings(
 <identifier of the target source>,
 {“text”: f”{item:3d}”},
 True,
)

By assembling all these together we would have a complete program for the job. The only problem with putting everything together is that we are mixing asynchronous code calling bleak with a strictly synchronous and likely non thread-safe code by obsws-python. Fortunately, the data only flows one-way; otherwise it complicates the whole setup significantly more. Ideally the library that dealt with OBS websocket could be written in a more asynchronous manner but obsws-python was the only reliable library that I could find in a weekend.

The project started as a weekend project, and I stopped right after achieving a running proof-of-concept. Only later, when I finally had the time, did I start refactoring this into a more reliable program that properly handles errors and graceful shutdown. Considering we discussed rather extensively in the past on those topics, let’s skip to how we turn this into a producer-consumer design.

with ThreadPoolExecutor(max_workers=5) as executor:
   executor.submit(producer)
   executor.submit(consumer)

We want both the consumer and producer to remain isolated from each other, so we separate them into individual threads. Usually it is better to split them into different processes, but since we are only running one event loop for the producer part it is okay to stick with threads. In order to synchronize both of them, we need to pass a queue and an exit event to them as well. This can be achieved with functools.partial, or by writing a class with the dunder __call__ method, a topic we discussed previously. I will leave the implementation detail out to focus on the core concept itself.

Now, back to our producer, where we collect the heart rate, instead of printing the heart rate, we dump it to our queue:

logger.info(f”[{flags:04b}] Heart Rate: {hr_value} BPM”)
asyncio.create_task(asyncio.to_thread(queue.put, hr_value))

In case you missed our series of articles on AsyncIO (go read them, they are awesome!), queue.put is a blocking operation, so we are delegating it to asyncio.to_thread. For our purpose, we don’t exactly care about the outcome, but we want to ensure it is scheduled safely. Therefore, we pass the returned coroutine object to asyncio.create_task.

On the other hand, as the code dealing with OBS is synchronous, we need to plan carefully.

def consumer(queue: Queue, exit_event: Event) -> None:
    obs = ReqClient(
        host=<host>,
        port=<port>,
        password=<password>,
    )

    logger.info("Starting consumption loop")
    while True:
        # task 1: handle the exit event
        if exit_event.is_set():
            logger.info("Exiting consumption loop")
            break

        # task 2: publish to obs
        with suppress(Empty):
            if item := queue.get(timeout=5.0):
                obs.set_input_settings(
                    <identifier of the target source>,
                    {"text": f"{item:3d}"},
                    True,
                )

This simplified snippet sets up an infinite loop that does two things, firstly it checks if the exit_event is triggered, and secondly, it consumes the heart rate data from the queue and sends it to OBS. We would need to set a timeout and suppress the potential Empty exception when getting a value from the queue. The reason for this is otherwise the code will block until it receives a data, and we would never be able to capture exit_event that was emitted elsewhere.

Asyncio’s Role in Real-World, Non-Web I/O

Essentially, this is a fairly common Consumer-Producer application between two different components. While the consumer is depending on the producer to publish to OBS, the producer should not have to wait for it to complete. By separating them into different threads and using a queue for synchronization, we allowed both components to work independently. Unlike the telegram chatbot we built previously, this shows how we can adapt synchronous code into an asynchronous environment.

I appreciate the fact that Garmin allows the heart rate data to be made available through an open interface. If I were to get a new Versa watch, this project would not have been possible as the data is locked away from a proprietary API. As mentioned earlier, Fitbit also removed support for application development for the newer models.

That would mean no more Tetris Effect streams with heart rate readings for me.

https://medium.com/media/6fd1166a20e17dcd450a99014354e282/href

Not like anyone cared anyway.

So, is this project a good example as a response to the question posed early in the article? Is this a good demonstration of what asynchronous programming is capable of? I would argue yes, as this shows even though not all components are asynchronous, it is still possible to workaround it. When we have multiple components that need to talk to each other, asyncio is definitely a way to go, albeit requiring more thought and work into it.

That’s all I have for today! The code snippets are purposefully simplified for brevity, but if you are interested in the project, feel free to check it out on GitHub. Thanks for reading, I shall see you in writing again soon.

For the sake of transparency, this article was a collaborative effort with my editor, Gemini, who assisted me throughout the writing process. Beyond providing editorial suggestions to tighten the narrative and structure, the model was instrumental in accelerating the technical side of the project, specifically by generating and verifying the initial code snippet required to correctly extract and parse the raw heart rate data from the BLE characteristic — a small but critical step that saved me hours of digging through specification sheets. The final code and the story are mine.

From Blog Post to PyCon: My One-Week Crash Course in Public Speaking

Choon-Siang Lai — Wed, 12 Nov 2025 14:50:58 +0000

If you were to deliver a talk in a conference for Ruby engineers, what would the topic be? That was a question I asked my supervisor during PyCon MY 2025. He eventually answered that he did not have a topic. Somehow it didn’t surprise me: a lot of incredible engineers I have met in life would likely respond similarly.

Including myself.

My name tag as a speaker

Code vs. Conversation: The Socialization Problem

My decision of submitting a talk likely raised a couple of eyebrows. Even my partner, Mr. H joked that I had brought the stress entirely upon myself whenever I complained about the preparation work. My former colleague even asked point-blank why I would make such a decision as an introvert. He added that he would not only avoid giving a talk, but he wouldn’t even consider attending the event.

Perhaps our lives take on a different trajectory, but in some ways, I see the ability to effectively communicate ideas being a part of the adulting or socialization process. For instance, I was once tasked with manning a booth in two different trade shows previously in another job. We were exposed to vastly different audiences when we were rotating between the two shows. As a result, the ability to explain the same concept in different contexts is crucial for people to relate to the company’s product.

To a certain extent, that experience alone piqued my interest to improve on my ability to communicate ideas. KitFu Coda, this very blog can be seen as a conscious effort for the purpose.

Just a few days ago, I realized this writing project started slightly over a year ago. Looking back, it expanded to different side ventures throughout the period, including a video, and just recently a talk. For the talk, the story started in late June, where a friend pinged in the community chatroom about the call for paper for the upcoming PyCon MY. With nudges from my mentor and fellow friends, I submitted my entries.

A few days after the submission, I coincidentally received a job offer. Between the new job and a series of family matters, I quickly forgot about the submission. That changed about two months ago when the approval for one of my submissions on AsyncIO hit my inbox. As I was still juggling between work and personal matters, I was somewhat relieved that my lightning talk on creating a DSL didn’t make it.

From 20-Person Meetups to the Main Stage

The title screen of my slide (Background is generated by Gemini)

Speaking of public speaking, I previously co-organized Rust Malaysia meetups, where I had the chance to share my learning journey and host workshop sessions. Most of the time I was dealing with a maximum of just 20 attendees. It is also worth noting that my last talk happened pre-COVID.

I can’t believe that was 5 years ago.

PyCon MY 2025 took place in Sunway University on the 1st and 2nd November. Compared to the meetups I ran, the difference in scale was massive. For instance, in my session I estimated there were at least a hundred attendees; that alone was enough to make me uncomfortable.

Preparation for the talk only started one week before the actual event. Due to the change of domain and tech stack at work and a series of family matters, my life only started to settle down fairly recently. Fortunately, we are living in an era where AI tools are getting increasingly helpful and easily accessible. This is a luxury my younger self wouldn’t have imagined. Naturally, given the tight timeframe due to my endless delay, I started by getting NotebookLM to generate an overview based on the source material.

The generated presentation was almost perfect that I jokingly told my peers that I felt useless.

After that, I turned to Gemini to start drafting the actual script for the presentation. Just as in my usual workflow, I planned the script as a storytelling project. While not perfect, the overview and initial draft did give me an idea of how to proceed.

Moving on, I needed to start preparing the slides. Considering the workshop-like nature, I decided to go with a simpler slide design and chose Google Slides. It still does the job fine; however, some of the controls were either moved to a new position or were completely replaced by the Gemini integration.

The final script was only done about two days before the presentation. I handwrote it on my tablet, as this method is proven to help with memorization. Later, during the dinner hosted by the organizer for speakers, one of the speakers mentioned how he only managed to complete his slides 10 minutes before his session. If I were in his shoes, I likely would have lost sleep in the days leading up to PyCon.

Autopilot Engaged: The Crisis on Stage

Flustered me on stage (photo courtesy of PyCon MY)

Thinking back, my setup was likely overly complicated and hard to execute. My session was scheduled to be in the bigger auditorium hall. Unfortunately, it clashed with a talk by my project partner from SinarProject. Despite that, many showed up for the talk, and that amplified my uneasiness.

Ideally, I was supposed to have my timer on my phone, so as I went through my script on the tablet I would know how much time was left. Even with that seemingly perfect plan, juggling three devices at a time was almost impossible. I ended up forgetting to start the timer, and forgot about referring to the handwritten script, and just autopiloted through the presentation.

While I didn’t really keep track, I likely missed out a rough guesstimate of 40% of the content. On the other hand, I did put in effort to introduce only one idea in each slide so hopefully the core ideas are presented in a manageable pace.

If you are reading this and you attended the talk, feel free to reach out by commenting here for clarification.

The Return: Why Just Showing Up Was the Answer

If I can, you can too (photo courtesy of PyCon MY)

Attending PyCon as a speaker is indeed an interesting experience. While this is considered a smaller conference (my previous ones were PyCon APAC in KL and EuroPython 2022 in Dublin, Ireland), I was considerably more stressed this time having to deliver a talk.

Siang attending EuroPython 2022 - LaLoka Labs

Until I am done with the talk, that is.

One of the reasons I like attending a conference or meetup is getting to know what people are working on. Apparently, quite a number of speakers are already adopting AsyncIO in their work, which made me wonder if my talk really mattered. This revelation does make me wonder about the people who showed up at my session. Were they just curious about the topic, or wanted to discover new use cases, or simply just there to validate their adoption to the feature?

Perhaps the choice of topic is a major consideration; however, the willingness to put oneself out there matters more. I can now appreciate the support my mentor offered when I was complaining about Rust adoption at work: he pushed me into organizing meetups. Sometimes, the act of standing out and yelling is what it takes to attract more feedback. In this case, speaking in a conference calls back to the problem I had when I started writing about AsyncIO — the constant need for shared knowledge on the topic.

Thanks again for joining me in this wonderful journey. I would like to take the opportunity to express my gratitude towards the organizers for hosting me, and putting in effort to maintain the friendly atmosphere. I shall write again, soon.

This article was developed with the editorial assistance of a large language model, Gemini. The model served as a linguistic and structural advisor, helping the author refine the grammar, correct flow issues, and polish the final prose. The core content, personal voice, and experiences remain the exclusive work of the author, Kitfucoda.

The && Fallacy: How --command Fixed VS Code Extensions in a Declarative Nix Shell

Choon-Siang Lai — Sun, 19 Oct 2025 10:22:50 +0000

In the last article, we discussed the evolution of my development setup, focusing on how it became reproducible over time. It was originally written to document the process for my own amusement. Surprisingly, the stats tell a different story — apparently it resonated with enough people sharing the same pain. Over time, revisions were made alongside new discoveries, and I figured the journey was worth a retelling. Read on to find out about an unexpected milestone related to this very blog.

Cute illustration on the topic, by Microsoft Copilot

The Editor’s Blind Spot: Nix vs. VS Code Extensions

Up until the last article, we managed to get a rather consistent setup with home-manager to manage development tools, and subsequently retired tools like pyenv and rbenv with the use of nix-develop. Little did I know, it wasn’t the end of the story. This time, we are tackling a new issue — the lack of nix-develop support of the Visual Studio Code extensions for development work in Ruby.

Forcing the ruby-lsp extension to source an external Gemfile as detailed was a hackish solution. Although it worked for a bit, most of the time the extension failed to initialize the language server. Observing how inconsistent the behavior is, something must be wrong. On the other hand, setup seems impeccable, tests were passing, and projects were building fine too.

Unfortunately, not much luck with the team’s desired formatter.

It wouldn’t work in the editor.

A type checker extension called sorbet is also common, and that too wouldn’t work at all. Considering the team doesn’t type annotate the code, I quickly dismissed and deactivated the extension. Combining this with the previous problem where ruby-lsp failed to work, the itch grew as time passes on.

Photo by Abu Saeid on Unsplash

Worse still, a formatter that doesn’t work at all times yields errors in the automated checks on every push.

Something was definitely wrong, was it because of VSCode failing to pick up the environment in a nix-develop session? My initial attempts to fix the flake file to include a command to activate the language server properly as shared. Still, none of them seemed to work reliably.

How To: Use vscode-ruby-lsp with flake.nix based dev enviroment · Issue #1851 · Shopify/ruby-lsp

After hitting walls enough times, I decided to post my setup to the discussion board to seek for help.

The Log Line That Clicked Differently

A problem can only be solved if we can clearly define it. We know the setup is failing, but what caused it? Interestingly enough, achieving the problem definition is a series of unlikely coincidences. It almost felt like solving a jigsaw puzzle, where each seemingly unrelated piece ultimately leads to the solution. So what are the missing pieces stalling our progress in understanding what went wrong?

Let’s start with the discovery of previously unused --command argument for nix-develop.

Initially, the gemini-generated flake file for the nix-develop sessions contained some commands to start my favorite fish shell. The snippet on starting a new shell was taken out from the previous discussion as it was somewhat unrelated. It was rather complicated and bound to fail because I was expecting it to work differently in different scenarios. Naturally, I wanted to simplify it somewhat, and eventually I learned about the --command argument.

Instead of relying on a clever (or dumb, depending on how you perceive it) detection logic, I could start a new fish shell session with

nix develop /path/to/flake/dir --command fish

Besides a shell, it could also run any command in the environment defined by the flake file. Discovering this was a crucial step toward fixing the launch of the ruby-lsp language server in the editor.

Remember I posted an SOS message on the discussion board? Luckily, it caught the attention of Vinicius Stock, the project owner. Helpfully enough, he pointed out that an external Gemfile is not necessary, and asked me to check if the language server is launched properly. With that, I checked the log and found this line

2025-07-27 16:56:30.669 [info] (ship_it) Running command: `nix develop /Users/jeffrey04/Projects/home-manager/devenvs/ruby-3.3 && ruby -EUTF-8:UTF-8 '/Users/jeffrey04/.vscode-oss/extensions/shopify.ruby-lsp-0.9.31/activation.rb'` in /Users/jeffrey04/Projects/ship_it using shell: /Users/jeffrey04/.nix-profile/bin/bash

Despite seeing the log multiple times before, now that I know about the --command argument, the same line clicked differently. Does the ruby command after the && operator run in the nix-develop session? Validating in this case is rather easy, just run

nix develop /path/to/flake/dir && ruby --version

Oops, why am I being dropped to a new shell session? Let me exit the session, and sure enough, the ruby command refers to the version offered by the operating system.

❯ nix develop ~/Projects/home-manager/devenvs/ruby-3.3 && ruby --version
warning: updating lock file "/Users/jeffrey04/Projects/home-manager/devenvs/ruby-3.3/flake.lock":
• Added input 'nixpkgs':
    'github:NixOS/nixpkgs/544961dfcce86422ba200ed9a0b00dd4b1486ec5?narHash=sha256-EVAqOteLBFmd7pKkb0%2BFIUyzTF61VKi7YmvP1tw4nEw%3D' (2025-10-15)
(nix:ruby-3.3-env) wukong:ib-vpn jeffrey04$
exit
ruby 2.6.10p210 (2022-04-12 revision 67958) [universal.x86_64-darwin24]

Photo by Matias Malka on Unsplash

Clearly, the assumption that the command following the && operator is flawed from the very beginning. With this revelation, we can finally define what the problem is — the language server is not properly launched. In this case, all we need to do is to replace the boolean AND operator into --command.

Easy to say, but implementation is a challenge for someone who has no experience with vscode extension development like myself.

And this is exactly when vibe-coding is useful, with a clear problem definition and goal, the solution was just a prompt or two away with the help of gemini-cli. As expected, the patch worked flawlessly.

Firstly, add a new enum entry to ManagerIdentifier in vscode/src/ruby.ts

export enum ManagerIdentifier {
  Asdf = "asdf",
  Auto = "auto",
  Chruby = "chruby",
  Rbenv = "rbenv",
  Rvm = "rvm",
  Shadowenv = "shadowenv",
  Mise = "mise",
  RubyInstaller = "rubyInstaller",
  NixDevelop = "nix-develop", # add this line
  None = "none",
  Custom = "custom",
}

Then a new case in the runManagerActivation method

case ManagerIdentifier.NixDevelop:
  await this.runActivation(
    new NixDevelop(this.workspaceFolder, this.outputChannel, this.context, this.manuallySelectRuby.bind(this)),
  );
  break;

Followed by a new module at vscode/src/ruby/nixDevelop.ts that contains the activation logic:

import * as vscode from "vscode";

import { VersionManager, ActivationResult } from "./versionManager";

export class NixDevelop extends VersionManager {
  async activate(): Promise<ActivationResult> {
    const customCommand = this.customCommand();
    const command = `nix develop ${customCommand} --command ruby`;
    const parsedResult = await this.runEnvActivationScript(command);

    return {
      env: { ...process.env, ...parsedResult.env },
      yjit: parsedResult.yjit,
      version: parsedResult.version,
      gemPath: parsedResult.gemPath,
    };
  }

  private customCommand() {
    const configuration = vscode.workspace.getConfiguration("rubyLsp");
    const customCommand: string | undefined =
      configuration.get("customRubyCommand");

    return customCommand || "";
  }
}

Lastly, enable the option forrubyVersionManager in vscode/package.json

        "rubyLsp.rubyVersionManager": {
          "type": "object",
          "properties": {
            "identifier": {
              "description": "The Ruby version manager to use",
              "type": "string",
              "enum": [
                "asdf",
                "auto",
                "chruby",
                "none",
                "rbenv",
                "rvm",
                "shadowenv",
                "mise",
                "nix-develop", // add this line
                "custom"
              ],

Don’t forget about the actual config (omit the rubyLsp.customRubyCommand if flake.nix is in the workspace)

"rubyLsp.rubyVersionManager": {
    "identifier": "nix-develop"
},
"rubyLsp.customRubyCommand": "/Users/jeffrey04/Projects/home-manager/devenvs/ruby-3.3",

The patch is definitely not perfect, and it is reusing the optional customRubyCommand configuration item to define where the directory containing the flake file is. Yet, considering this likely affects only a small group of users, I don’t see a need to introduce a new configuration field for the purpose. Contributions are definitely welcome at the submitted pull request, as it has been stalling for months.

For the sorbet part, I managed to get it working one day out of boredom. Most of the Ruby code I have access for now is not type-annotated, so the effort is purely for fun. A quick and dirty way to do so, is building the language server application directly, and configure vscode as follows

"sorbet.enabled": true,
"sorbet.selectedLspConfigId": "without-bundler",
"sorbet.userLspConfigs": [
    {
        "id": "without-bundler",
        "name": "sorbet",
        "description": "Launch locally compiled sorbet",
        "command": ["/path/to/sorbet", "typecheck", "--lsp", "--dir=/path/to/project"]
    }
]

For some reason I needed to --disable-watchman flag in the command too, but as there’s no real need for now I am not going to dig further.

There you have it, with simplified flake files (removing the detection logic to start fish shell) and invoking fish shell explicitly with --command, and patches and hacks to enable the ruby toolings, we finally have a functional setup for development work in Ruby. Seeing the pull request being stalled for months (and their bot closed my PR recently due to inactivity) does make me upset for a little bit. Moving forward I may need to consider maintaining a fork for it as it does affect my work.

Stalled PR, Sudden Payoff: The Two Timelines

Photo by engin akyurt on Unsplash

Patches to the ruby-lsp were submitted in the late July, and I was hoping to start drafting this article when it received a review. Unfortunately, that didn’t happen. I was caught up with my new job and several major family issues while waiting on the PR. Things finally settled down for a little bit, and I should be back to a relatively more predictable routine. Hopefully this translates to a more consistent publishing schedule for KitFu Coda.

As for the unexpected milestone teased in the beginning, it is about a talk I am about to deliver during the upcoming PyCon Malaysia. My mentor encouraged me to submit for the CFP and it somehow went through. If you are interested in applying AsyncIO in projects, do feel free to drop by and say Hi. For reference, it is based on the articles I published earlier, as linked below.

I also discussed briefly with my supervisor in one of our 1 on 1 sessions, mainly to check on any necessary approvals or conflict of interest. He was incredibly supportive about it and offered some tips on presenting ideas to a wider audience. As I am new to Ruby, he also briefly explained how the language approaches asynchronous programming. On a lighter note, he’s even been checking on my preparation progress (zero so far), especially in recent sessions!

With this article checked off in my TODO, I can finally allocate some time on the preparation work. I sometimes feel like I’m running on an energy tank whose maximum capacity depletes as we age. These days, even finding time to game is a challenge. I feel bad for my character in Xenoblade X and I wonder if I’ll ever have time for the new Pokémon Legends Z-A game I’m about to purchase.

Finally, I would like to thank everyone for the support, especially to everyone who reads my work here. Though I have not been publishing much, since starting this technical writing project opens up fascinating new opportunities. Besides scoring a new job, I also started exploring scripted video making, and now a talk. I am thankful and hope to inspire more people through this trying time.

Thanks again for reading, I shall write again soon.

Throughout this journey, Gemini served as a valuable editorial companion, helping refine my prose. However, the voice and code remain entirely my own. I welcome your feedback and recommendations in the comments below, and invite you to subscribe to my Medium for more content on my development adventures!

Beyond rbenv and pyenv: Achieving Reproducible Dev Setups with Nix

Choon-Siang Lai — Sun, 20 Jul 2025 07:53:10 +0000

Setting up a development environment is a painful process, as I discussed previously when detailing how mine evolved to the combination of home-manager and Ubuntu Make. Little did I know, the article gained significant traction and likely resonated with many fellow developers. Coincidentally, I got into a new job earlier in the month, and this necessitated a revision to the setup on a MacOS environment. How did it go? Turbulent would be an appropriate adjective to describe the whole experience, as you may expect.

From Dotfile Hacks to Open Source: My Development Environment Evolution

A cute illustration from Copilot

The macOS Catalyst: A New Job, A New Challenge

Replicating a development setup over multiple machines can be a hard problem, and ensuring it works on different operating systems significantly increases that complexity. As we delve deeper into the topic, you will see how I shoot myself in the foot repeatedly, and the corresponding steps to recover.

Why a Mac?

Photo by Claudio Schwarz on Unsplash

Before starting my new role, I was asked if I would prefer a Linux machine or a Mac for work. Though I am relatively comfortable with a Linux machine, I picked Mac as I reckoned it is easier to get enterprise apps whenever necessary. I also reiterated this same reason when my coworker and I were pairing up for a task.

Despite working mostly with Ubuntu, I do have a Intel iMac for video calls and functioning as a second screen for documentation and web project previews. A minimal home-manager setup worked mostly fine, and I assumed expansion of the setup for development would be as painless.

Oh, how wrong I was.

As the assigned MacBook Pro became my primary device for work, all the minor problems surfaced. First, my shells were not initialized properly in the terminal emulator. After getting that fixed, I got adventurous and decided to retire rbenv to manage Ruby versions. Coupled with the struggle to set up my code editor, I was amazed how my feet survived all the gunshots.

Diving into the Rabbit Holes: macOS and Interpreter Woes

How hard can it be?

Photo by Nachristos on Unsplash

If one is looking for a statement to best represent underestimation, that would be it. Looking back, the problems look very trivial, but each was a great learning opportunity. Hang on tight, as we are going into a deep dive into all the rabbit holes I fell into with the revised setup, and to learn how to overcome them.

Compared to Windows, macOS offers a more familiar environment for developers with experience in Linux. We can interact with the system through a similar command-line interface. With a package manager, we can even add different shells not shipped with the operating system, like my favourite fish shell. Trying to configure an iTerm2 profile for fish is exactly the first hole I fell into.

Why would macOS not source .profile in a graphical session?

Unlike Ubuntu, macOS does not source .profile in a graphical session. Most of the time, the script updates the PATH variable, making locally installed applications discoverable. Subsequently, shell initialization scripts (like .bashrc for bash and config.fish for fish) then set up individual application based on the environment defined in .profile.

Multiple variations was attempted, until I stumbled upon a Q&A on superuser. Thinking it would be useful, I sent the proposed answer to Gemini and asked if it would be applicable for my case. Turns out I only needed a little tweak.

/bin/sh -lc "exec -l /path/to/fish"

This is a breakdown of the command:

/bin/sh: This specifies that the command should be executed by the Bourne shell, launching a new instance of this basic shell.
/bin/sh -l: this tells the shell to launch as a login shell, ensuring it sources standard login files like .profile.
/bin/sh -c: This tells the shell to execute the command provided in the string that follows.
exec: This crucial command tells the shell to replace the current /bin/sh process with a new process (in this specific case, the fish shell).
exec -l: This tells the new shell (e.g., fish) to launch as a login shell itself.
/path/to/fish: This is the full path to your fish shell executable. With the -l flag in the exec command, this fish shell will then source its own login initialization configuration file (e.g., config.fish).

Similarly, the command can be adapted to work with byobu, my preferred terminal multiplexer. The -l flag is omitted for the exec command as byobu is not a shell.

/bin/sh -lc "exec /path/to/byobu new-session"

For completeness’ sake, the setup for a bash profile is a lot simpler, as it sources .profile and .bashrc properly when launched as a login shell -l itself.

/path/to/bash -l

What a ride.

Shifting from Python to Ruby as my main development language presented an opportunity to review the toolings involved. Previously I relied on pyenv to provide isolated Python interpreters for individual projects. Eventually, the functionality was replaced by uv, which also manages pre-built CPython distributions as well. Unfortunately, I wasn’t aware of an alternative for Ruby, and had to continue relying on rbenv for the purpose.

One of the main goals of using home-manager, as discussed in the previous article, was to achieve idempotency and to isolate dev packages from the system package manager (such as apt in Ubuntu). Thus, we need to first define the packages in the home-manager flake file, as shown below:

packages = with pkgs; [
    ...
    # dev libraries
    openssl
    zlib-ng
    bzip2
    readline
    sqlite
    ncurses
    xz
    tcl
    tk
    xml2
    xmlsec
    libffi
    libtool
    libyaml
    libxml2
    libxslt
  ];

Essentially, this yields an isolated filesystem namespace that is unrecognized by the operating system. In order to get the compiler to utilize the pulled development package, we need to tell rbenv where to look for them, as shown in the shell script below:

#!/usr/bin/env sh

export CPPFLAGS="-I$HOME/.nix-profile/include $CPPFLAGS"
export CFLAGS="-I$HOME/.nix-profile/include $CFLAGS"
export LDFLAGS="-L$HOME/.nix-profile/lib $LDFLAGS"
export PKG_CONFIG_PATH=$HOME/.nix-profile/lib/pkgconfig:$HOME/.nix-profile/share/pkgconfig:$PKG_CONFIG_PATH
export C_INCLUDE_PATH=$HOME/.nix-profile/include:$C_INCLUDE_PATH
export INCLUDE_PATH=$HOME/.nix-profile/include:$INCLUDE_PATH
export LIBRARY_PATH=$HOME/.nix-profile/lib:$LIBRARY_PATH
export LD_LIBRARY_PATH=$HOME/.nix-profile/lib:$LD_LIBRARY_PATH

$HOME/.nix-profile/bin/rbenv install -vf 3.3.5

It worked fine, though I always found this a dirty hack, as it required explicitly defining the flags and include paths. There has to be a better way.

The Nix Pivot: Discovering a Declarative Path

If not rbenv, then how do we approach Ruby version management? Turns out we can use another feature provided by Nix to build a development environment through a declarative manner. So how is this relevant to the problem we are looking at though?

Photo by Jonathon B. Carreño on Unsplash

A friend of mine had been hinting for a while that I could completely replace rbenv with Nix but I didn’t explore deeper. That changed when I saw the announcement of the gemini-cli app. In the installation guide, it called for npx, but the tool was not available as a Nix package. After some exchanges with Gemini, it suggested I look into nix shell, and the conversation quickly evolved into an introductory session on nix develop.

Just a side note, gemini-cli was already packaged not long after the announcement, no need for npx. If I were to follow the installation guide, npx could be made available through a nix shell -p nodejs session. In case you missed how this could be related to our original problem, the equivalent for Ruby would be nix shell -p ruby.

What if I need to install gems that require some special dev libraries?

You just unlocked the reason the conversation turned into an introduction to nix develop. Like home-manager, the input for the command is a flake file, written in the Nix programming language, declaring the development environment setup. Naturally, this added complexity made me hesitant to commit to this strategy. I was still trying to get more comfortable with home-manager and wanted to take more conservative baby steps in adopting additional features offered by Nix.

Essentially, there are two important components to be declared in the flake: the list of packages and an initialization shell script. For example, we can define a development environment sporting Ruby 3.3 with the list of packages like so:

buildInputs = with pkgs; [
  # fish is kept so that fish-specific completions from
  # packages can be made available.
  fish
  # Git is needed for fetching gems from git repositorie
  git
  # The Ruby interpreter (3.3.8 as the time of writing)
  ruby
  # Bundler for managing Ruby gems.
  bundler
  # Common dependencies for building native extensions.
  cmake
  gcc
  gnumake
  # Libraries often required by gems.
  openssl
  pkg-config
  libmysqlclient
  shared-mime-info
  re2
  readline
  zlib
] ++ lib.optionals stdenv.isDarwin [
  # Add macOS-specific dependencies.
  libiconv
];

Secondly, the initialization script definition

shellHook = ''
  # Ruby-specific setup
  export FREEDESKTOP_MIME_TYPES_PATH="${pkgs.shared-mime-info}/share/mime/packages/freedesktop.org.xml"

  echo "Ruby dev environment activated for ${system}!"
'';

Incorporating these into the flake file makes nix develop a superior method to set up a development environment. Compared to just listing packages in nix shell, the shellHook component allows more control and further manipulation, enabling us to set up proper environment variables and run initialization commands. Additionally, the declarative nature of a flake file also means it is highly reproducible and can be easily shared.

A Harmonious Setup: Validation and Reflection

Remember I needed to redefine library path and compiler flags to compile Ruby interpreters? That problem eventually came back while I was installing gems for work projects. Long story short, I gave in and adopted nix develop. Does it work out of the box? Does the adoption of nix develop stop me from shooting myself in the foot?

In order to be cautious, I replicated and adapted the flake for my bigmeow side project, which is still managed by Poetry, and hence relied on pyenv. Fortunately, a satisfactory outcome was achieved after a few iterations. On the other hand, direnv also offers some integration with nix develop. Automated activation of a development environment can be achieved by adding one line to the respective .envrc:

use flake /path/to/folder # the folder that contains the flake.nix declaration

Photo by Sigmund on Unsplash

Applying the Ruby counterpart also went rather smoothly. There were some problems in the earlier iterations, but they were mostly trivial, and all it took was a regeneration by Gemini. Compared to the relatively smooth sailing in environment setup, taming my code editor to work with it was a lot more frustrating.

In recent years, most extensions bridging the code and development tools would often bundle the tool together. For instance, if I install an extension to format Python code with ruff formatter in Visual Studio Code, it would bundle ruff itself in the package. This saves the developer from needing to declare ruff as a dependency of the project.

Likely due to the highly modular design, the experience was a lot more complicated in the Ruby world. One of my work projects defined an ancient version of the Rubocop formatter, and none of the editor extensions offered in the marketplace would work with it. Forcing the extension to use the bundled gem would fail, because it wouldn’t work with the equally outdated rubocop extension gems (rubocop-rails, rubocop-minitest) installed in the project.

Updating the lock file is not a practical solution and I was stuck there for a while. Thankfully a coworker shared that I could explore the ruby-lsp extension. Apparently, this extension makes it possible to refer to an alternative Gemfile, i.e. I could have a Gemfile for all the development tools saved outside of the project. In this particular case, I could have a Gemfile like this

source "https://rubygems.org"

gem "ruby-lsp"
gem "rubocop", "1.26.1"
gem "rubocop-rails"
gem "rubocop-minitest"

Solving that yields an functional setup that I was happy enough for work. Looking back, everything seemed rather trivial, but in reality I was stuck for a couple of days. Despite the added complexity, getting all these parts working together felt like putting together puzzle pieces. Getting them configured correctly is giving the same satisfaction as solving a complex jigsaw puzzle.

Is this increased complexity worth the effort? Would there be a better alternative strategy out there? I am curious to know about them and am open to recommendations and corrections. Meanwhile, I do find the revision to retire version managers a fruitful learning experience. Another part I liked about the current implementation is the elimination of explicit declaration of dev packages to build the interpreters. If you ask me, I would say replacing the version managers and removing dev packages for interpreter compilation with nix develop flake files is totally worth the effort.

The Pygame Framework I Didn’t Plan: Building Tic-Tac-Toe with Asyncio and Events

Choon-Siang Lai — Sat, 05 Jul 2025 13:11:26 +0000

Over the last weekend, I spent some time mucking around with Pygame. Despite facing some challenges along the way, I ended up with a working prototype. There was neither design nor specification drafted for the project. Nonetheless, mid-development, a reusable design emerged, and we discussed that in last week’s article. Following that discovery, we will explore the implementation this week and answer the question ‚ “Can we make a framework out of it?”.

Cute illustration from Copilot

The Unexpected Framework: From Pygame Hacks to a Reusable Design

Understanding Awaitables: Coroutines, Tasks, and Futures in Python

As a brief recap to the last article, we know how Pygame is designed to be primitive, and lacks inherent structure. Building a Pygame application would therefore require a developer to begin by writing the main application loop. On the other hand, this also presents an opportunity to design a bespoke structure for graphical application development. By refactoring our work from last week, we will witness how a framework emerges, while adhering the original design goal to stay immutable and reliable.

Adapting the synchronous main application loop to AsyncIO took quite some effort. Despite the complexity involved, it was worth it because it broke the tight coupling of display update and event dispatching loop. Previously, events were only fetched once per display update cycle. If the application was written in a pure event driven manner, delegating work into multiple events would introduce latency.

Now that both event dispatching and display update run independently, ensuring synchronization of application states between the two components is also crucial. Immutability was therefore introduced such that application data can only be changed through publishing changes to relevant queues.

With all the work laid out, this opens up an interesting potential, allowing the orchestration code for event dispatching and display updates to be split from the game logic. Application setup work, such as event handler registrations and the initialization of application-specific data, is the primary area that differs from one application to another.

Sounds like a plan, let’s split it out, and complete the tic-tac-toe game we started.

Photo by Daniel McCullough on Unsplash

Building the Core: Immutable State and Asynchronous Orchestration

It shouldn’t be difficult, right? Given the design goal mentioned earlier, the splitting work was indeed not too difficult. On the other hand, keeping track of the declared dataclasses is another story. Regardless, we will uncover an architectural pattern that forms the framework core through some refactoring work. Some minor revisions will be made to the state management, as well as the main application loop. All this work yields a reusable framework for simple graphical applications.

Previously we were creating multiple queues for different fields in our Application dataclass. Having multiple queues serving similar purpose to update values is fine, though I personally find it wasteful. Why not just consolidate them?

And I did exactly that.

Photo by USGS on Unsplash

Firstly, we start by introducing an Enum to represent the field we intend to update

class ApplicationDataField(Enum):
    STATE = auto()
    ELEMENTS = auto()
    EVENTS = auto()

Secondly, we update the DeltaOperation to always require the field we intend to update

@dataclass
class DeltaOperation(ABC):
    field: ApplicationDataField
    item: Any

Lastly, we need to update the application_refresh function, which is the only place where the Application object is updated throughout the framework.

async def application_refresh(application: Application) -> Application:
    result = application

    with suppress(asyncio.queues.QueueEmpty):
        while delta := application.delta_data.get_nowait():
            field = application_get_field(application, delta.field)

            match delta:
                case DeltaAdd():
                    result = replace(
                        result,
                        **{field: getattr(result, field) + (delta.item,)},
                    )

                case DeltaUpdate():
                    result = replace(
                        result,
                        **{
                            field: tuple(
                                delta.new if item == delta.item else item
                                for item in getattr(result, field)
                            )
                        },
                    )

                case DeltaDelete():
                    result = replace(
                        result,
                        **{
                            field: tuple(
                                item
                                for item in getattr(result, field)
                                if not item == delta.item
                            )
                        },
                    )

    return result

Actual implementation is a little bit longer, but I truncated for brevity. Logically, the current version is rather similar to the previous version, though it only consumes from one single queue (application.delta_data) to update the states in an Application object.

After the minor refactoring to the state management, we shift our attention to the main application loop.

async def main_loop(
    application_setup: Awaitable[Application],
) -> None:
    pygame.init()

    application = await application_setup

    pygame.event.post(pygame.event.Event(SystemEvent.INIT.value))

    tasks = []

    tasks.append(
        asyncio.create_task(
            display_update(
                application.delta_screen,
                application.clock,
            )
        )
    )
    tasks.append(
        asyncio.create_task(events_process(application))
    )

    await application.exit_event.wait()

    pygame.quit()

It is almost similar to the original run function in our original code, with just 2 small changes. Remember I mentioned about wanting a window.onload like system events? Here we are emitting an INIT event before we start the display update and event dispatching loops. Additionally, setup code is passed in as an Awaitable object, that is only scheduled to run after pygame.init().

Understanding Awaitables: Coroutines, Tasks, and Futures in Python

That’s practically it, though extracting them out to another module, is a pain due to how these functions reference numerous dataclasses and other helpers. Alongside these changes, I also added some helpers to deal with in-game state management. Spiritually, the design of these helper functions loosely follows Redux, though the implementation is a lot simpler in my implementation. We shall see them in action when we go through the game implementation next.

Framework in Action: Bringing Tic-Tac-Toe to Life

Framework is now done, what’s next?

What’s a better way to test the framework, than building a game for real? Now that we delegate the orchestration work to another module, we can now focus purely on the game logic. Throughout the discussion, we can observe how the framework design enables building of a simple game with just dispatching events and implementing the handlers. Game state management as mentioned earlier will be discussed as well.

Firstly, let’s set-up the game:

async def setup() -> Application:
    application = Application(screen=pygame.display.set_mode((300, 300)))
    application = await add_event_listeners(
        application,
        (
            (SystemEvent.INIT.value, handle_init),
            (CustomEvent.RESET.value, handle_reset),
            (CustomEvent.MAP_UPDATE.value, handle_update),
            (CustomEvent.JUDGE.value, handle_judge),
        ),
    )

    return application

A new helper add_event_listeners was written for adding multiple event listeners, though they work similarly as the add_event_listener we discussed last week. In case you are wondering at this point, yes, this is all we needed to implement our tic-tac-toe game. Since we split out the orchestration code to another module, our run function now becomes:

from core import main_loop

async def run() -> None:
    await main_loop(setup())

Remember main_loop() takes an Awaitable as argument? This is the reason why setup() was written as a coroutine function, and passed into main_loop function without the await keyword.

Before we start the display update and event dispatching cycles, let’s initialize the game by drawing relevant elements to the application screen object:

async def handle_init(
    _event: pygame.event.Event, target: Application, logger: BoundLogger, **detail: Any
):
    for row, col in product(range(3), range(3)):
        element = await box_draw(target, row, col, 100)
        element = await add_event_listener(
            element,
            pygame.MOUSEBUTTONDOWN,
            handle_box_click,
        )
        await target.delta_data.put(DeltaAdd(ApplicationDataField.ELEMENTS, element))
        await screen_update(target, element) # type: ignore

    pygame.event.post(pygame.event.Event(CustomEvent.RESET.value))

During the initialization phase, the handler draws the 3 x 3 tiles to the screen via box_draw, and then registers a mouse click event to each of them. Each tile is then sent to 2 queues, one to populate the application.elements tuple (via target.delta_data.put), and another to update the display (via screen_update). Once done, the handler invokes the RESET event to start a new game.

async def handle_reset(
    _event: pygame.event.Event, target: Application, logger: BoundLogger, **detail: Any
):
    await state_merge(target, "board", winner=None, state=GameState.RING, board=None)

    for element in target.elements.values():
        delta = await box_redraw(target, (255, 255, 255), element, Symbol.EMPTY) # type: ignore
        await target.delta_data.put(
            DeltaUpdate(ApplicationDataField.ELEMENTS, element, delta)
        )
        await screen_update(target, delta)

As the game is resetting, it first reinitializes the game state via state_merge.The helper sends a DeltaOperation to update application.state to

{
    "board": {
        "winner": None,
        "state": GameState.END,
        "board": None
    }
}

Elements in the game board also get redrawn (via box_redraw) to ensure they do not contain a ring or cross symbol. As we did earlier, all of them are then sent to the queues for application and display update.

Our game is now accepting input. And once a click is registered, the registered event handler is called.

async def handle_box_click(
    _event: pygame.event.Event,
    target: Box,
    application: Application,
    logger: BoundLogger,
    **detail: Any,
) -> None:
    element = None

    match state_get(application, "board").get("state"):
        case GameState.END:
            await logger.ainfo("END")
            pygame.event.post(pygame.event.Event(CustomEvent.RESET.value))

        case GameState.TIE:
            await logger.ainfo("TIE")
            pygame.event.post(pygame.event.Event(CustomEvent.RESET.value))

        case GameState.RING if target.value == Symbol.EMPTY:
            element = await box_update(
                application,
                target,
                Symbol.RING,
            )
            await state_merge(application, "board", state=GameState.CROSS)

        case GameState.CROSS if target.value == Symbol.EMPTY:
            element = await box_update(
                application,
                target,
                Symbol.CROSS,
            )
            await state_merge(application, "board", state=GameState.RING)

        case _:
            await logger.aerror("Invalid click")

    if element:
        pygame.event.post(pygame.event.Event(CustomEvent.MAP_UPDATE.value))

Click events are handled differently, depending on the state of the board. If there is no valid moves available, in the events of an ended or tie game, then we log a message, and emit a RESET event. Otherwise, we update the clicked element to the symbol corresponding to the current state of the board, followed by a board state change for the next symbol. Lastly, we trigger a MAP_UPDATE event to update the current mapping of the board for judging.

@dataclass
class BoardMap:
    mapping: dict[tuple[int, int], Symbol]

async def handle_update(
    _event: pygame.event.Event, target: Application, logger: BoundLogger, **detail: Any
):
    await state_merge(
        target,
        "board",
        map=BoardMap(
            {
                (item.column, item.row): item.value
                for item in target.elements.values()
                if isinstance(item, Box)
            }
        ),
    )

    pygame.event.post(pygame.event.Event(CustomEvent.JUDGE.value))

BoardMap is just a dataclass keeping the current state of the board, what symbol is placed at each tile. As soon as it is updated, a JUDGE event is triggered to evaluate the game.

WIN_LINES = [
    # Rows
    {(0, 0), (0, 1), (0, 2)},
    {(1, 0), (1, 1), (1, 2)},
    {(2, 0), (2, 1), (2, 2)},
    # Columns
    {(0, 0), (1, 0), (2, 0)},
    {(0, 1), (1, 1), (2, 1)},
    {(0, 2), (1, 2), (2, 2)},
    # Diagonals
    {(0, 0), (1, 1), (2, 2)},
    {(0, 2), (1, 1), (2, 0)},
]

async def handle_judge(
    _event: pygame.event.Event, target: Application, logger: BoundLogger, **detail: Any
):
    if not (bmap := state_get(target, "board").get("map")):
        return

    tiles, winner = (), None

    for symbol in (Symbol.RING, Symbol.CROSS):
        places = {coor for coor, item in bmap.mapping.items() if symbol == item}

        for line in WIN_LINES:
            if line.issubset(places):
                winner = symbol
                tiles = line
                break

        if winner:
            break

    if winner:
        await logger.ainfo("Winner is found", winner=winner)
        await state_merge(target, "board", winner=winner, state=GameState.END)

        for element in target.elements.values():
            if (element.column, element.row) in tiles: # type: ignore
                delta = await box_redraw(target, (255, 255, 0), element, winner) # type: ignore
                await target.delta_data.put(
                    DeltaUpdate(ApplicationDataField.ELEMENTS, element, delta)
                )
                await screen_update(
                    target,
                    delta, # type: ignore
                )
    elif (
        len([symbol for _, symbol in bmap.mapping.items() if symbol == Symbol.EMPTY])
        == 0
    ):
        await logger.ainfo("End with a tie")
        await state_merge(target, "board", state=GameState.TIE)

Not the most robust code I would proudly print on a t-shirt and parade with it. Essentially, it judges the game to find the winning line and symbol. Redrawing the winning tiles will take place whenever applicable. Besides that, the relevant state of the board is also queued for update (either an END or TIE) whenever needed.

By utilizing events and state_merge strategically, the simple game of tic-tac-toe is implemented. All the event dispatching work and display update are completely handled by the framework itself. All we needed to do, when changes occur, is to submit the change to the relevant queue. While there are certainly potential improvements to be made, as a weekend project I deem it good enough.

Beyond Tic-Tac-Toe: Lessons Learned and What’s Next

Can we build more things with it?

Short answer to the question is yes, but I don’t see much point in doing so for most people. Instead, we should reflect on what we learned so far. What is the purpose of the project? What have we achieved so far? What do we do with this piece of code?

Extracting the orchestration code into a framework is a totally unexpected outcome. Yet, I am happy with the turns of events leading to what we have now. With some polish, the resulting framework should be able to be applied for other simple graphical applications. Personally, I am quite curious on seeing how the framework scales with more elements on screen.

Photo by Dan Freeman on Unsplash

As an experimental project, it certainly is quite rough on the edges. Much thought and polish are still needed in the framework API design for consistency. There are also some experimental code snippets that are not covered in the article as they are irrelevant to the game featured this week. Some of the most interesting helpers I can’t wait to try out are the functions inspired by setTimeout and element.dispatchEvent().

Perhaps a re-implementation of the code rain animation a-la the Matrix?

Code Rain’s Revelation: Embracing Existence Before Perfection

Now, back to the original question asked earlier, at the beginning of this section. The framework is proven capable of building a simple game, as shown in this article. Despite that, do go for a proper game engine like Godot for serious projects. Treat the work presented here as a personal passion project, though I am open for feedback and collaboration. Seriously, this exploration makes me appreciate how game engines do all the heavy-lifting work.

Finally, thanks for reading this far. If you are interested in checking out the project, it is still hosted on GitHub at the same repository. Though I cannot commit to a fixed publication schedule due to my new job, I shall try writing again soon.

A quick note on how this article came together. While Gemini helped with the drafting and refining of the language, all the core ideas, the technical insights, and every line of code you’ve seen are entirely mine. This project truly reflects my personal journey and passion for game development. If you enjoyed this deep dive and want to follow more of my explorations into coding and game development, consider subscribing or following me on Medium. Your support means a lot!

My Pygame Evolution: Embracing Asyncio and Immutability for Scalable Design

Choon-Siang Lai — Tue, 24 Jun 2025 17:13:34 +0000

Maintaining a blog is a lot of work, especially when it comes to finding new ideas. Over the weekend, I explored Pygame by building a simple game. I was very humbled by the experience throughout the build. Aha, this exploration turned out to be a worthy idea for a new article! Read on to relive the struggles involved while building a graphical game via Pygame together.

A cute illustration on the topic, by Microsoft Copilot

The Tic-Tac-Toe Catalyst: Unveiling Pygame’s Low-Level Nature

Over the weekend, in the EngineersMY chatroom, a friend shared his work in a recent game jam held in his workplace. He built the project with p5.js, piqued my interest. Eventually I decided to proceed with Pygame, and that decision led me to hours of exploration and perhaps the expense of a few hair follicles. Despite that, the lower-level library nudged me to build a custom architectural design that can potentially evolve into a proper framework.

Photo by Ryan Snaadt on Unsplash

Discussions about my friend’s project reminded me of a less than ideal job application that happened around late last year. The offered package was unattractive, and I was expected to work on graphical animation alongside full-stack development. My submitted video processing pipeline work received satisfactory feedback, though the hiring manager came back with another assignment. He requested a new task: to build a graphical application with p5.js, and it was also when I formally ended the application. That mention of p5.js was the first time in a formal setting.

Interestingly, I was still given a job offer despite all that, which I ultimately rejected.

Fast forward to the weekend, I consulted with my AI assistant and started my research about Processing, a project where p5.js was based on. Apparently there are ports for Python, though the official processing.py only works with Jython. All other alternatives either relied on the Java package or required special libraries. Consequently, I stumbled upon Pygame after a series of aimless searches.

Unlike Processing, Pygame felt like a lower level library for building graphical applications. Partially inspired by the discussion on the game jam, but mostly out of boredom, I decided to give it a try. The realization of how low-level Pygame is came when I was figuring out the specification and scope of my over-ambitious initial plan. At the time, my main loop was just barely running.

As a project to explore graphical application building, I eventually decided to limit the scope further to start small. Tic-tac-toe seemed like a good idea, and I should be able to build a reasonably functional version over a weekend. How wrong I was, as I am still doing some last minute changes as of writing this article…

Building a Reactive Core: Event-Driven Design in Pygame

Pygame is synchronous, and most of the objects and operations are not thread-safe. Due to the interactive nature of graphical applications, they often require a carefully planned strategy to achieve concurrency. AsyncIO is an obvious candidate for the task, though getting it to work with Pygame, requires some thought. Much of the implementation takes a cue from Javascript, especially in the event dispatching department. Immutability of certain components is also crucial, though mainly to keep me sane.

Let’s start with the main loop, essentially we want something like this

def run() -> None:
    pygame.init()

    continue_to_run = True
    clock = pygame.time.Clock()

    pygame.display.set_mode((300, 300))

    while continue_to_run:
        clock.tick(60)

        continue_to_run = event_process(pygame.event.get())

        pygame.display.flip()

    pygame.quit()

A potential problem I see with this arrangement is that, both handle_events and pygame.display.flip() are inevitably blocking. If any of them stalled for a prolonged period, we will miss the target frame rate. Ideally, we want something like this:

async def run():
    pygame.init()

    exit_event = asyncio.Event()

    pygame.display.set_mode((300, 300))

    asyncio.create_task(display_update())
    asyncio.create_task(event_process(exit_event))

    await exit_event.wait()

    pygame.quit()

Photo by Acton Crawford on Unsplash

A window popped up, and my event handling work seemed functional. Yet, sometimes when the application exits I get a SIGSEGV segmentation fault when exiting the program. After some research on the error, I found out that Pygame is not thread-safe, hence I should really avoid asyncio.to_thread. This means the following code, delegating the blocking display.flip() call to another thread, is invalid:

async def display_update():
    clock = pygame.time.Clock()

    while True:
        clock.tick(60)

        await asyncio.to_thread(pygame.display.flip)

With that discovery, I settled with the original main loop convention for a while, and only moved the event dispatching work to a coroutine.

async def run() -> None:
    pygame.init()

    clock = pygame.time.Clock()
    exit_event = asyncio.Event()

    pygame.display.set_mode((300, 300))

    while not exit_event.is_set():
        clock.tick(60)

        asyncio.create_task(event_process(pygame.event.get(), exit_event))

        pygame.display.flip()

    pygame.quit()

Unfortunately, due to the synchronous blocking nature of Pygame, the event_process tasks do not have a chance to run, though scheduled successfully. As my coding companion, the chatbot suggested adding asyncio.sleep(0) into the loop. According to the documentation:

Setting the delay to 0 provides an optimized path to allow other tasks to run. This can be used by long-running functions to avoid blocking the event loop for the full duration of the function call.

Adding the statement to the end of the loop allows the application to finally respond to events.

Previously, I mentioned I performed a last-minute update while writing this article. With this knowledge, we can apply the asyncio.sleep(0) fix to both display_update and event_process, enabling AsyncIO to run the scheduled tasks. In turn, this allows the refactoring of the huge main loop shown in the first snippet into the second snippet.

Photo by Scott Rodgerson on Unsplash

Before diving into the event dispatching part of the design, let’s talk about how I manage shared data. Certain components need to be shared across different parts of the application. Graphical applications are extremely interactive, hence I would need to respond to them concurrently. In this case, interactivity poses a new challenge — keeping track of changes of these shared objects.

Much of the design came from my experience with development work in JavaScript. Within a JavaScript application, it is possible to dispatch and subscribe to events almost everywhere, and changes can also be made to a page whenever we see fit. Keeping a reference while staying aware of these changes is a nightmare, and it is extremely unfriendly to my hair follicle.

Grouping them together into one dataclass, which serves as a registry is usually my first step. In this case, I prefer them to stay immutable, as shown below:

@dataclass(frozen=True)
class Application:
    screen: pygame.Surface = pygame.Surface((0, 0))
    elements: tuple[Element, ...] = tuple()
    state: GameState = GameState.INIT

The three objects consist of a pygame.Surface object for the main application window, an element tuple holding all the interactive widgets in the surface, and an Enum holding the current game state. Events are dispatched to be executed concurrently, so each execution is completely isolated from one another. Therefore, we need to synchronize the changes made to these shared objects to avoid race conditions.

Aiming to maintain the simplicity inspired by functional programs, I started by marking the dataclass as frozen, as shown in the snippet. Changes to the elements and state objects are deferred to two queues I introduced to the registry dataclass.

@dataclass(frozen=True)
class Application:
    ...

    state_update: asyncio.Queue[GameState] = asyncio.Queue()
    element_delta: asyncio.Queue[DeltaOperation] = asyncio.Queue()

Making the dataclass immutable would require a regeneration of application registry for every batch of events.

async def events_process(application: Application):
    while True:
        application = await application_refresh(application)

        await events_dispatch(
            application,
            pygame.event.get(),
        )

        await asyncio.sleep(0)

Whenever an event handler makes a change to the game state or manipulates the widgets on screen, the changes are pushed to the queue instead. Then the queue is consumed, to update the corresponding object:

from dataclasses import replace

async def application_refresh(application: Application) -> Application:
    with suppress(asyncio.queues.QueueEmpty):
        elements = application.elements

        while delta := application.element_delta.get_nowait():
            match delta:
                case ElementAdd():
                    elements += (delta.item,)

                case ElementUpdate():
                    elements = tuple(
                        delta.new if element == delta.item else element
                        for element in elements
                    )

                case ElementDelete():
                    elements = tuple(
                        element for element in elements if not element == delta.item
                    )

    with suppress(asyncio.queues.QueueEmpty):
        state = application.state

        while state_new := application.state_update.get_nowait():
            state = state_new

    return replace(
        application,
        elements=elements,
        state=state
    )

Wrapping the widget element with an ElementAdd, an ElementUpdate and an ElementDelete is just a way to provide metadata to indicate the manipulation type. On the other hand, the replace function, provided by the dataclasses module, is just a helpful shorthand in this case. Alternatively, we can recreate the Application object as follows:

    return Application(
        application.screen,
        elements,
        state,
        application.state_update,
        application.element_delta
    )

Moving on to the event dispatching part.

Due to my inexperience, I didn’t spent too much time drafting a design or a plan when I worked on the game. Still, as I went deeper I realized I really needed to start structuring the event handling code. Keeping that in mind, I continued with the build until I managed to get a bare minimum version running.

Drawing inspiration from JavaScript, I started my refactoring by revising the event handler registration. Directly inspired by the document.addEventListener method, my interpretation of it in this setting took this form:

async def add_event_listener(target: Eventable, kind: int, handler: Callable[..., None]) -> Eventable:
    event = Event(kind, handler)
    result = target

    match target:
        case Application():
            await target.event_delta.put(DeltaAdd(event))

        case Element():
            result = replace(target, events=target.events + (event,))

        case _:
            raise Exception("Unhandled event registration")

    return result

Besides the widgets in the application window, the application itself can also subscribe to an event. For that, I added a tuple of events as well as an event_delta queue to the Application dataclass. This function always returns either the application registry, or the recreated element. Publishing the recreated element to the element_delta queue is handled by the caller.

Unlike the display_update function, there is no clock.tick() function to regulate the timing of execution in the event_process function shown previously. After finishing scheduling the current batch, we want to immediately start fetching and processing a new batch of events as soon as possible. The current dispatching function is rather straightforward, as it just schedules the registered handler by cycling through the events tuple in the application registry as well as each of the elements.

async def events_dispatch(
    application: Application,
    events: Sequence[pygame.event.Event]
) -> None:
    for event in events:
        match event.type:
            case pygame.QUIT:
                application.exit_event.set()

            case pygame.MOUSEBUTTONDOWN:
                for element in application.elements:
                    for ev in element.events:
                        if not ev.kind == event.type:
                            continue

                        if element.position.collidepoint(mouse_x, mouse_y):
                            asyncio.create_task(
                                ev.handler(ev, element, application)
                            )

Basically these is all the code mainly for handling the orchestration work; it contains no game logic. Like how I was working on BigMeow, a structure emerges when I work more on the project. When I work on a personal project, I also tend to avoid writing object-oriented code as much as possible, and I am glad how well the current revision turns out.

Photo by BoliviaInteligente on Unsplash

Being a complete newbie to Pygame, Gemini proved helpful yet again. When I talked about the project with a friend, he jokingly asked if I vibe-coded it. However, the whole experience was not unlike how I ported the LLM word game to AsyncSSH, where Gemini assisted by helping me to navigate the documentation. Granted, the code example is usually good enough, but it helps to know how and why things are written in a certain way.

Lessons Learned and Horizons Gained: My Pygame Journey Continues

Refactoring work started right after I was more or less done with handling mouse click and triggered a change to game state. As the code gets cleaned up, it occurs to me that extracting the game logic into a separate module becomes trivial. To my surprise, after getting Gemini to review the code, I was told the separation would yield a mini framework for similar projects.

Not knowing where I could get feedback, I turned to Gemini to review my code, and we chatted about how this project could be extended. That follow-up discussion was a very enlightening session, apart from the revelation of an emerging framework. Presumably, the Application dataclass resembles an Entity-Component-System. This was definitely a concept worth adding to my learning backlog.

Photo by Glenn Carstens-Peters on Unsplash

If time permits, I would like to work on the separation and complete the game. Curiosity is definitely the biggest motivation behind the effort, rather than publishing the potentially bad framework. New system events like window.onload and window.onclose equivalents may also be implemented to better simulate my web development workflow. Follow the project on GitHub (inline link to project) if you are interested.

Unfortunately, I will not have as much free time for a while. On the flip-side, I no longer have to entertain needy and exploitative hiring managers too. With my incoming new job offer, I will adjust the publishing frequency for the coming weeks. I would also like to take this opportunity to thank everyone offering encouragement and help throughout this trying period. Spring does come, after winter.

To ensure this post is as clear and readable as possible, I leveraged Gemini as my editorial assistant. Gemini also provided complementary information to Pygame’s official documentation, helping to clarify complex concepts. However, please note that the voice and all code presented within this article are entirely my own. If you found this article helpful or insightful, consider following me on Medium and subscribing for more content like this.

From Dotfile Hacks to Open Source: My Development Environment Evolution

Choon-Siang Lai — Wed, 18 Jun 2025 12:32:20 +0000

Let’s talk about setting up a development machine for work today. If there is one thing techies love talking about, it’s our setup. Some take enormous pride in how they fine-tuned the process to perfection, often involving fancy or unusual components. I was like that once, and totally understood the motivation behind it. Over time, my priority shifted towards consistency over personality. Buckle up and read on about how I ended up with a combination of Ubuntu Make and home-manager.

A cute illustration by copilot

The Early Setup Struggles

Why the shift to value consistency? Computers, like many other consumer electronic devices, are bound to stop functioning after a while. Even if they last forever, upgrading is still inevitable as software becomes increasingly more capable. On the other hand, there are also people who would require multiple machines for work. Thus, devising a plan for setup may sound like a one-off effort, but it is important to ensure consistency and maximize productivity when executed repeatedly. The need poses a rather unique challenge that we are exploring in this week’s article.

My journey to achieve consistency began with setting up Vim. Being my preferred text editor at the time, I frequently shopped for new plugins at the project website. Eventually, I stumbled upon Pathogen by Tim Pope. Before this, all Vim plugins were installed into one place. Pathogen, however, made it possible to separate all plugins into their own folder. Eventually, this led me to writing a set of scripts to automate the work.

Photo by Elena Mozhvilo on Unsplash

Meet Vim-manager.

PHP was still the language I found myself most comfortable with at the time, and it was written when I was most obsessed with functional programming. Reading the code now would definitely give me a headache, but it addressed my need for consistency at the time. The setup process started by preparing a configuration file, as shown below,

{
    "pathogen": "git://github.com/tpope/vim-pathogen.git",
    "bundle": {
        "filetype-clojure": "git://github.com/guns/vim-clojure-static.git",
        "filetype-javascript": "git://github.com/pangloss/vim-javascript.git",
        "filetype-json": "git://github.com/rogerz/vim-json.git",
        "filetype-latex": "git://vim-latex.git.sourceforge.net/gitroot/vim-latex/vim-latex",
        "filetype-lisp": "https://bitbucket.org/kovisoft/slimv",
        "filetype-qml": "git://github.com/peterhoeg/vim-qml.git",
        "indent-php": "git://github.com/2072/PHP-Indenting-for-VIm.git",
        "omnicomplete-php": "git://github.com/shawncplus/phpcomplete.vim.git",
        "plugin-airline": "git://github.com/bling/vim-airline.git",
        "plugin-clojure-highlight": "git://github.com/guns/vim-clojure-highlight.git",
        "plugin-delimitmate": "git://github.com/Raimondi/delimitMate.git",
        "plugin-ctrlp": "git://github.com/kien/ctrlp.vim",
        "plugin-fencview": "git://github.com/mbbill/fencview.git",
        "plugin-fireplace": "git://github.com/tpope/vim-fireplace.git",
        "plugin-golden-ratio": "git://github.com/roman/golden-ratio",
        "plugin-l9": "https://bitbucket.org/ns9tks/vim-l9",
        "plugin-neocomplete": "git://github.com/Shougo/neocomplete.vim.git",
        "plugin-niji": "git://github.com/amdt/vim-niji.git",
        "plugin-php-namespace": "git://github.com/arnaud-lb/vim-php-namespace.git",
        "plugin-rainbow-csv": "git://github.com/vim-scripts/rainbow_csv.vim.git",
        "plugin-vawa": "https://bitbucket.org/sras/vawa",
        "plugin-vimcompletesme": "git://github.com/ajh17/VimCompletesMe.git",
        "plugin-vimwiki": "git://github.com/vimwiki/vimwiki.git",
        "plugin-vinegar": "git://github.com/tpope/vim-vinegar.git",
        "syntax-php": "git://github.com/StanAngeloff/php.vim.git",
        "theme-solarized": "git://github.com/altercation/vim-colors-solarized.git"
    }
}

Alongside the plugin configuration, the script also managed .vimrc, the application configuration file. Then, I would run a script to generate a Makefile, that detailed the steps to download, install, update, and clear the plugins and configurations. My eventual move to Neovim required just a minor revision to the script, and it continued to work well. Granted, there are plugins made for the purpose, but I still liked it for the simplicity, and most importantly, the idempotency.

In this context, idempotency means the script always yields a consistent result, no matter how many times you run it.

Eventually, I moved on to another editor for better user experience, but I kept Vim as my secondary editor for quick work, so the script lived on. Knowing how well it worked, that led me to think about expanding the project to cover other essential configuration files and setups for work. That thought eventually turned into a new project after I stumbled upon an introductory article on stow.

Enter the dotfiles era. At the time, I was spreading my work to two machines, my Surface Book 2 (RIP) and my desktop workstation. My laptop was constantly dying, and due to the way it is designed, it is practically unrepairable. At a time, I requested multiple replacements within a week. Needless to say, setting up a machine all over again repeatedly was a hassle.

My dying surface book

In order to achieve consistency, I tidied up my crucial system profile configuration .profile across different systems. Consequently, whenever I set up a new machine, I copy-paste the content over. Getting configuration done, however, was just half of the work. I still needed to install my tools like the interpreter, compilers, project, and version managers for work. These had to be done in the right sequence, as they could be dependent on each other. For instance, I couldn’t install Vim, until I properly set up Python, Node.js and Ruby.

Due to the complexity, I commenced work on the dotfile project rather unambitiously. As detailed in my recent article, I needed to address the “does it exist?” part first. Thus, the first iteration of the project, was just a bag of scripts to be executed in a specific order. Configuration files were placed with the stow tool, and the script would automate the installation of tools according to the configuration. Much attention was put into ensuring idempotency and cross-platform compatibility.

The Snap Saga

One thing I like a lot about modern Linux distributions is how a variety of software can be easily installed through their official software repositories. Within them, we can often find everything ranging from web browsers, game clients, streaming media players, and video editors. For most typical computer users, it is entirely sufficient. Unfortunately, that comes with a cost: more installed software means increased complexity during major operating system upgrades. Canonical introduced Snap as a solution to distribute user software we use daily, that is completely isolated from the system software packages. Let’s dive into it and see how if this works with our setup and what challenges it brings.

Photo by Alexander Shatov on Unsplash

Upgrading a system to the next major release is often rather painful by itself. There are usually thousands of files to be updated at once. Theoretically, including user software in the upgrade would be a good idea. Yet, more often than not, the added complexity causes random problems to occur post upgrade. Certain files may have gone mysteriously during the upgrade, or something corrupted during the hour-long upgrade process, and all these could have caused software to fail. Snap was designed as a solution to this problem, by separating user software updates from the system.

Snap was designed as a solution to this problem, by separating user software updates from the system. Instead of relying on the packaging system to manage software updates we use on daily basis, the work is delegated to snapd instead. In addition to that, Snap also offers other benefits like containerization and dependency management, but that’s out of the scope for our discussion.

If I have graphical access to the system, Snap is often my choice to get software installed. I am aware of the competing flatpak standard, but I prefer Snap as it is already built into every Ubuntu installation, my usual distribution of choice.

However, it is far from being perfect, especially when it comes to the support of east asian character (namely the Chinese, Japanese and Korean, or CJK) input. Additional effort is often needed by the software packager to properly integrate IBus, the input method for CJK characters into the Snap package. The issue raised on the matter for my code editor is still open as of the writing of this article.

So no, Snap is great, but doesn’t fit my ultimate goal of setting up a new machine quickly and easily.

Unifying with home-manager

I liked the idea of splitting user software from the system, and the dotfiles project did exactly that. All the configuration files and installed tools lived within my home directory. The setup shielded them from operating system updates and the system managed far fewer packages. The only problem with it at the time was I needed to be careful about the installation sequence. This was when home-manager came into the picture, as it was an almost perfect replacement to my bag of scripts hack.

https://medium.com/media/26572a87106c080c157db267b0f957c2/href

Like how I learned about stow, the discovery of home-manager was purely coincidental as well. The video above just appeared in my feed when I was browsing YouTube out-of-boredom. Perhaps something overheard my conversation with my friends about switching to NixOS? Anyway, after spending some time watching the video and learning the syntax, I started to experiment with it. Disregarding the hiccups I had with setting up version managers like pyenv, rbenv and phpenv, the whole migration was largely smooth sailing. As it also supports MacOS, I uninstalled Homebrew and replaced it with home-manager too.

Due to the design of Nix packaging system that powers home-manager, idempotency is always ensured. Previously I was doing it on a best-effort basis. With home-manager, all the managed configuration files are made read-only, and can be modified only through its configuration. The consistency brought by idempotency also allows all applications installed through home-manager to continue to work after major operating system upgrades.

Installation of graphical applications is supported, so I also attempted to install all applications I could find in Nix’s package repository. That mostly worked, but I was still having some random problems here and there with some of them. Overall, migrating to home-manager was a good call, though again, it is not perfect.

Discovering Ubuntu Make

To be frank, I don’t remember much about how I found Ubuntu Make. It likely happened when Firefox shut down the aurora release channel, and closed the personal package archive (PPA) hosting the package. Mozilla, the maker of the browser, replaced aurora with the developer release but there was no easy way to install it on Ubuntu. I suppose that eventually led me to Ubuntu Make after a series of web searches. The discovery was interesting, and I later found out it was really helpful for software developers like myself.

Photo by Hendrik Morkel on Unsplash

Ubuntu uses apt as the package manager, where it manages the packages installed to the system. As mentioned earlier, most of the user software can be found via the official package repository. In addition to free and open source software, Canonical, the company behind Ubuntu, also offers some non-free software packages through their partner repository. Alternatively, anyone can also host their software packages through the use of personal package archive (PPA).

Earlier when we discussed Snap, we knew how installing more packages to the system increases the complexity. This is the reason why PPAs are deactivated on every major operating system upgrade. All the approaches we discussed, are about separating user software management from the operating system. Ubuntu Make largely follow the principle, though it only installs and does not manage afterward.

But it is okay, as my installed Firefox Developer Edition updates itself, and my code editor would prompt me for a reinstall on every update anyway.

Essentially, Ubuntu Make is just a very well thought-out script to automate the installation work. It begins by grabbing a compressed archive from the website, and then deflating the content into a folder. Then it sets up the PATH, logos, and desktop file accordingly. In case of update, it would just replace the existing installation, and repeat the setup process.

The only problem with this is how the code is tightly coupled to the changes of the websites. In case of rebranding or whatever occasion leading to the change of website, a patch is needed to make sure the files are downloaded correctly. Fortunately, the code is written in Python, a language I am relatively more comfortable with. And yes, I submitted a handful of patches for software packages I rely on for work.

As a complement to my home-manager configuration, I am quite happy with it.

Reflections and Beyond

I like Ubuntu Make a lot. It saddens me that it is not getting enough recognition, perhaps because it looks rather boring, and isn’t built with a language that everyone celebrates. In a way, this article is written to express my appreciation to all the hard work put into all these projects, to make our lives easier. Additionally, I want to express my deep gratitude to the developers of Ubuntu Make for their invaluable guidance and help in merging my patches. If you work as a developer and happens to daily-drive Ubuntu, give it a try!

As a bonus, the developers are rather friendly when they respond to pull requests too.

Photo by Mimi Thian on Unsplash

The process of setting up a machine is often one-off, and most people wouldn’t really give much thought about it. However, for people who need to frequently work with multiple computers, having something that delivers consistent outcomes repeatedly is crucial. Better yet, it could be scripted and automated. Unfortunately, software installation is still a rather complicated matter, and every strategy comes with its unique pros and cons. I suppose this is why techies have strong opinions on how things should be done? In the end, the best strategy could just be a combination of everything.

This article was refined with assistance from an AI editorial assistant. While the content and voice remain entirely mine, this collaboration helped in the drafting process. For project collaboration or job opportunities, feel free to connect with me on LinkedIn or here on Medium.

Beyond Hardcoding: My Breakthrough in Testable Parallel Python

Choon-Siang Lai — Sun, 08 Jun 2025 16:43:57 +0000

Last week, we talked about removing hardcoded synchronization primitives. The refactoring was prompted by my revision to a take-home assignment I submitted for a job application. This week, let’s get into testing, while building a smaller-scale replica, building on what we learned. Read on to join me in a self-reflecting journey regarding take-home assignment and why I believe they often fail to serve their intended purpose.

A cute illustration by Copilot on the topic

Beyond the Code: The True Cost of Interview Assignments

One may ask, what’s the deal with all these take-home assignments that software engineers are required to take when we apply for a job? Why do engineers seem to not like them, but still debate endlessly on the topic? How does it work?

I considered myself lucky to only start encountering them in my current round of job applications. Though not uncommon in the past; they weren’t yet a widespread practice. For a backend developer like myself, the assignments typically require implementing a set of CRUD endpoints, sometimes involving some state management. Under normal working conditions, the whole process usually takes about 2–3 days. The time spent on writing code is usually rather short, compared to the time spent on setup and testing. The orchestration of various components like the database, cache, backend application, and server to ensure seamless integration is quite time-consuming.

If one is applying for a full-stack position, add frontend compilation and server setup to the list of tasks too.

Photo by Jefferson Santos on Unsplash

Most of the time, the hiring companies don’t offer any compensation upon submission. Spending multiple days on an assignment, then, is actually economically impractical. Regardless of how trivial the requirement reads, it still mimics real-world work to a certain extent. On one occasion, a company even gave me a requirement list that covered my future work with them, if I were hired. Hiring companies sometimes specify protocols and tech stacks, requiring extra learning time.

Some may wonder, won’t LLM be useful in this?

To a lot of engineers, LLM-empowered editors are already an integral part of their daily work. Though they could be useful, these editors usually require a subscription. Given the reality where the hiring company is already not paying for the work, subscribing to these services just for these assignments makes no economic sense. On the other hand, LLMs are not that useful in the setup and integration work.

Vibe-Coding, Storytelling, and LLMs: A Collaborative Approach

The Perils of Unstated Requirements

Regardless, take-home assignment can still be a good way to gauge a candidate’s ability. How are they executed so poorly that engineers detest them so passionately over social media? Seriously, I once posted a random rant over an extensive assignment requirement at LinkedIn, and that post made hundreds of thousands impressions and sparked a heated discussion on whether I should proceed.

Let’s look into the assignment we are discussing today, it was done for a job application referred by a contact. The scope of the project seemed manageable, and looked doable within a day. Background processing was not listed as a requirement, but I opted to include that in my submission anyway. Though the offer was unsatisfactory, I attempted anyway mainly to demonstrate my respect and appreciation to the referrer.

Little did I know, my oversight regarding the hardcoded queue caused much complication for the tests.

Life went on as usual after the submission, until the company replied with the same undesirable offer. Interestingly, the examiner noted that JWT was not implemented, an indication of my inability to adhere to requirements. Conversely, the extra effort on moving processing to background, auto container image building was barely acknowledged.

Photo by Wes Hicks on Unsplash

Naturally, that feedback triggered dissatisfaction and prompted me to revisit the original requirement document.

And no, there was no mention of JWT.

On my LLM-assistant’s advice, I politely declined the offer and did not attempt to elaborate on the omission of JWT in the initial requirement. Still, this highlights a problem — bad communication between the examiner and the applicant. Unlike in a daily work setting, where one has better access to coworkers or clients for clarifications. It is hard enough to get prompt responses from the hiring company these days, and this makes the take-home assignment work more like a one-off engagement. Requirement is sent, followed by a submission, with little room for revision and corrections.

If life is a role playing game (RPG), I probably need to consider dumping more points into my Intelligence to improve my psychic ability.

On the other hand, why can’t the hiring company communicate better?

Why penalize the candidate for not meeting implicit criteria? Funnily enough (OK, I was absolutely frustrated at the time), I was once rejected for trivial, random refactoring decisions despite the project worked as specified. The feedback was almost laughable as if they were joking, as it covered flaws such as certain classes should be stored in another module, incorrect pluralization handling of classes and table names, the omission of explicit module imports, and they even gave me a fail when they were unable to locate my frontend implementation. The feedback almost gave me an impression that the company doesn’t do peer review, suggesting I would be left to work on projects independently.

And yes, the contact person disappeared even after I pointed the actual line where the supposedly missing line is.

From Hardcoding Headaches to Testable Triumph

Photo by AbsolutVision on Unsplash

Why revisit the project then? Despite some minor problems with the tests, the project worked. Why undertake such thorough refactoring to remove the hardcoding, for a one-off assignment?

To a certain extent, it is the revelation that matters. The refactoring (which culminated in this revelation) was discussed last week, and this week we are going to see how passing synchronization primitives (like a queue) explicitly in a parallel programming setting improves testability.

Practically speaking, despite the experimental nature, BigMeow is my first formal attempt at asynchronous programming in a parallel setting. After spending so much time and effort in learning and re-learning AsyncIO with it, it is now becoming my go-to reference and template for projects requiring asynchronous and parallel setup. Moreover, I wrote a series of posts on asynchronous programming (here, here and here) based on the code written for BigMeow.

The revelation came when I was adding a scheduler to the chatbot: What if I remove the hardcoding on the synchronization primitives? Fixing it in BigMeow right away would have been a huge undertaking, so I went to the assignment with simpler scope. Structurally, both are similar, as the assignment practically shared much of its core structure with BigMeow. That worked right away, and as a bonus, it fixed all the unexplained problems I had with tests.

Oh, the hours I wasted in stress trying to fix the tests.

Refactoring on BigMeow was equally successful, though a lot more involved given the complexity. It is interesting to see how a sudden flash of insight impacts the thought process and changes how code is written.

Hands-On: Crafting Decoupled Background Processes

Today we are revisiting the assignment together, which is yet another unimaginative FastAPI prototype accompanied by a test suite. The technique detailed in the previous article is used to remove the hardcoded synchronization primitives. That change makes testing much more intuitive. In order to keep this concise, we are not implementing the implicit JWT requirement.

As shown in the article last week, we start from the main process, beginning with the setup of a ProcessPoolExecutor.

from project import web, background
from concurrent.futures import ProcessPoolExecutor
import multiprocessing as mp

# create the synchronization primitives
manager = mp.Manager()
exit_event = manager.Event()
task_queue = manager.Queue()

# start the processes in parallel
with ProcessPoolExecutor() as executor:
    executor.submit(web.run, exit_event, task_queue)
    executor.submit(background.run, exit_event, task_queue)

All the synchronization primitives are sent individually to the run() function this time, given the simplified scope of this article. Additionally we are omitting graceful shutdown for brevity, feel free to refer to our previous discussion on the topic.

Similarly, the setup for the FastAPI web module would look like this

@asynccontextmanager
async def lifespan(app: FastAPI):
    if not hasattr(app.state, "task_queue") and isinstance(
        app.state.task_queue, Queue
    ):
        raise RuntimeError("Task queue is missing")

        yield

app = FastAPI(lifespan=lifespan)
async def run(exit_event: Event, task_queue: Queue) -> None:
    app.state.task_queue = task_queue
    server = uvicorn.Server(...)
    asyncio.create_task(server.serve())
    await asyncio.to_thread(exit_event.wait)
    await server.shutdown()

Nothing new here, it was almost the exact code we just saw last week. Next comes the crucial part, where we are actually receiving a batch of data:

@dataclass
class SomeData:
    id: str
    num: int

@app.post("/submit/batch")
async def claim_submit(
    request: Request,
    : list[SomeData],
) -> None:
    await asyncio.to_thread(
        request.app.state.batch_queue.put, submitted
    )

Just assume we are returning a 200 OK every time without a body. After that, we can start by setting up a fixture for our TestClient

import pytest
from fastapi.testclient import TestClient

from project.web import app

@pytest.fixture(name="client")
def client_fixture():
    app.state.task_queue = multiprocessing.Manager().Queue()
    with TestClient(app) as client:
        yield client

With the task_queue set, we can write a test to see if the queue is receiving submitted data

@pytest.mark.asyncio
async def test_create_job(client: TestClient):
    submitted = [
        {"id": "RECORD01", "num": 1},
        {"id": "RECORD01", "num": 3},
        {"id": "RECORD01", "num": 5},
        {"id": "RECORD01", "num": 7},
        {"id": "RECORD01", "num": 15},
    ]

    response = client.post("/submit/batch", json=submitted)

    assert response.status_code == 200

    # check if the job is sent to the queue
    payload = client.app.state.batch_queue.get()

    for idx, record in enumerate(submitted):
        assert record['id'] == payload[idx].id
        assert record['num'] == payload[idx].num

The background processing module I submitted was very basic, it was mostly just a proof-of-concept based on the provided spec. For this simplified example, let’s just do a Fizz Buzz check on the submitted num. Similar to the web module, let’s start with the setup:

def run(exit_event: Event, task_queue: Queue):
    database = create_a_fictional_database_connection()

    asyncio.create_task(loop_queue(task_consume, task_queue, database))

    await asyncio.to_thread(exit_event.wait)

Firstly, we have the actual function processing the batch data, and the result is saved through a fictional database.save method,

async def process_payload(database, payload: list[ChargeIncoming]) -> None:
    result = []

    for record in payload:
        fizz = "fizz" if record.num % 3 == 0 else ""
        buzz = "buzz" if record.num % 5 == 0 else ""
        result.append(f"{fizz}{buzz}" or str(record.num))
    await database.save(result)

And the corresponding test would look something like, assume we have a fictional database fixture defined somewhere:

@pytest.mark.asyncio
async def test_process_payload(database):
    payload = [
        SomeData("RECORD01", 1),
        SomeData("RECORD01", 3),
        SomeData("RECORD01", 5),
        SomeData("RECORD01", 7),
        SomeData("RECORD01", 15),
    ]

    expected = [
        "1",
        "fizz",
        "buzz",
        "7",
        "fizzbuzz"
    ]

    await process_payload(database, payload)

    # test if the processing logic is done properly
    assert database.exists(expected)

Lastly, the queue consumption would look like

async def loop_queue(func: Callable[..., Awaitable[None]], *args: Any):
    while True:
        func(*args)

async def task_consume(task_queue: Queue, database):
    payload = await asyncio.create_task(
        asyncio.to_thread(task_queue.get, timeout=5.0)
    )

    await process_payload(database, payload)

Now the corresponding test

@pytest.mark.asyncio
async def test_task_consume(database):
    payload = [
        SomeData("RECORD01", 1),
        SomeData("RECORD01", 3),
        SomeData("RECORD01", 5),
        SomeData("RECORD01", 7),
        SomeData("RECORD01", 15),
    ]

    queue = multiprocessing.Manager().Queue()

    queue.put(payload)

    with patch('background.process_payload') as process_payload:
        await task_consume(task_queue, database)
        process_payload.assert_called_once_with(database, payload)

Comparing the tests for the FastAPI application and the background module, the benefit is more apparent in the latter case. With the explicit passing of the queue object as an argument, there’s no more guesswork on which queue to patch in test. Plus, as we are ensuring both the test and application are referring to the same queue now, we can pass real data into test, and ensure they are being received properly.

Impact & Insights: From Personal Challenge to Shared Knowledge

So my submitted solution showcased a way to process data in batch, in a separate module running in parallel. Mainly due to the time limit, and the scale of the exercise, I chose to propose a very basic implementation. The replacement of the queue can be done fairly easily with a more robust solution. In my humble opinion, the point of the exercise should be on the comprehension of requirement, and deliver a solution within a timeframe both parties find reasonable.

Photo by Akson on Unsplash

Documenting the insight into blog posts can be seen as my attempt to think out loud. Who knows if this could spark further discussion that yields a better design for similar setup? Granted, the length of this article exceeded my initial expectation, but understanding the context is also crucial to know how it leads to the revelation. Perhaps some people may appreciate the thought process too?

Earlier, I talked about adding a scheduler to my chatbot BigMeow. It took me quite a while to get it working in this asynchronous parallel setup. Perhaps we can go through the process of adapting it to our setup in a more generalized form in a future article. Stay tuned if you are interested in the topic.

At last, thank you again for reading this far, and I shall write again, next week.

For this article, I received invaluable editorial assistance from Gemini, my AI assistant. While Gemini helped refine the language and structure, please know that the ideas, personal voice, and all code snippets are entirely my own. If you’re interested in collaborating on a project or exploring job opportunities, feel free to connect with me here on Medium or reach out on LinkedIn.

Telegram Chatbot’s Evolution: Decoupling Parallel Python’s Shared State for Clarity

Choon-Siang Lai — Mon, 02 Jun 2025 15:47:11 +0000

After building the intro animation last week,,) I finally took the time to make a video based on the article on jQuery as promised. In the future, articles that are accompanied by a video will all have a play button emoji ▶️ added to the title. While video is still made irregularly, we are going to meet here more regularly. This week, let’s revisit our chatbot project discussed not too long ago. It mostly works fine, but there’s one thing I want to improve on — removing the hardcoded global queue.

How to write an AsyncIO Telegram bot in Python

Illustration from copilot for the topic

The Global Headache: When a Simple Assignment Becomes a Core Problem

Spreading workload to run in parallel is a hard problem, especially when it comes to keeping them synchronized. Previously in our chatbot, we split the bot and webhook to run in 2 different processes, managed by the ProcessPoolExecutor. The whole architecture mimics my experimental bot named BigMeow ..) The project is definitely flawed, as it is a culmination of my learning on AsyncIO without a formal introduction to the topic.

GitHub - coolsilon/bigmeow_bot

Not too long ago I was going through a job interview. As part of the process, I was asked to work on a take-home assignment. It was unimaginative as ever, though I took the opportunity to implement an experimental background batch processing module. The whole design, again, is similar to the chatbot, so it works as expected.

Photo by Pascal van de Vendel on Unsplash

When I finally started to write tests to validate my work, that was when my nightmare began.

Though I hacked through the tests, there were multiple minor issues left unresolved after I submitted my work. The hardcoded queue in the settings module severely limits the flexibility and hacks like patches needed to be applied in tests. Despite rejecting the offer, the frustration lingered on my mind for quite a while. How can I do better to avoid spending endless hours fixing tests? Most of my code is rather testable, but there must be something I am missing. During the weekend, I finally took some time going through the code to prepare a new article, and a revelation came and it clicked.

What if I stop hard-coding the queue and pass it into each process explicitly, like the exit event?

With some help from my LLM chatbot, I managed to remove the hardcoded queue. Then I proceeded to pass it into the child processes explicitly, for both the background processing module, and the FastAPI web application. We would talk more about it, including test preparation, in the coming week. For now, let’s shift the focus back to our chatbot.

Exposing the Entanglement: Diving into the Original Architecture

Photo by Josh Redd on Unsplash

Let’s briefly review the current setup, as I try to recall why it was written this way. By understanding the original design decision, we can then start working out a solution, addressing the inflexibility that came with the hard-coded problem.

The reason for having a settings module was basically I needed a place to store some configuration globals. Over time, the list of configurable options piles up. Seeing how convenient it is, I eventually set up a queue to allow the web application process to pass incoming messages from the webhook endpoint to the process hosting the Telegram bot. At the time, it was just a simple queue I created from multiprocessing.Queue. Nothing fancy, just a boring process-safe queue that can be safely shared by multiple processes.

On the other hand, in the main process, we created a multiprocessing.Manager, and created an Event object with it. The created Event object is also process-safe, i.e. can be shared by multiple processes. That event object is then passed explicitly to each child process, to be listened to for a cue to exit.

The multiprocessing.Manager here acts almost like a coordinator, making sure processes can communicate effectively with each other whenever needed. Unfortunately, this means messages would be passing through an additional layer before hitting the target. Still the overhead is worth it, when it comes to queues. There is a risk of non-managed queue getting stuck in a shut down, especially when it is not empty. The Queue object created with the multiprocessing.Manager would handle it gracefully.

Delving deeper into multiprocessing.Manager isn’t the point of the article, and I am clearly not capable of doing it as of now. Just understand it is beneficial in orchestrating synchronization between processes, and we are going to utilize it in our Telegram bot.

From Global to Explicit: Refactoring with Centralized State

Knowing the limitation is one thing; next we are going to fix it. As we tackle the hard-coding problem, we will see how it becomes more flexible, while not breaking the inter-process communication.

Photo by U. Storsberg on Unsplash

Since we created a multiprocessing.Manager already to create an Event object, let’s extend it to also create the queue. Move the telegram_queue to our main module

import signal
import multiprocessing
from functools import partial

def shutdown_handler(
    _signum, _frame, exit_event: threading.Event
) -> None:
    exit_event.set()

manager = multiprocessing.Manager()
exit_event = manager.Event()
telegram_queue = manager.Queue() # MOVED telegram_queue here

for s in (signal.SIGHUP, signal.SIGTERM, signal.SIGINT):
    signal.signal(
        s, partial(shutdown_handler, exit_event=exit_event)
    )

with ProcessPoolExecutor() as executor:
    executor.submit(process_run, tg_run, exit_event)
    executor.submit(process_run, web_run, exit_event)

They can be passed into the child processes as-is, but I prefer to group them into an object (for reference, BigMeow now has 11 synchronization primitives, namely queues, locks, and events). Let’s create a simple dataclass for this:

from dataclasses import dataclass
from threading import Event
from queue import Queue

@dataclass
class SyncStore:
    exit_event: Event
    telegram_queue: Queue

Notice multiprocessing.Manager creates proxies of the synchronization primitives. They often have the same programming interface as existing objects, hence the type annotation specifies threading.Event and queue.Queue. Next we instantiate an object by

import multiprocessing

manager = multiprocessing.Manager()
sync_store = SyncStore(
    manager.Event(),
    manager.Queue()
)

After that, we adapt the changes in our main module, remember to also pass the new shiny sync_store into our child processes in the executor.submit call

for s in (signal.SIGHUP, signal.SIGTERM, signal.SIGINT):
    signal.signal(
        s, partial(shutdown_handler, exit_event=sync_store.exit_event)
    )

with ProcessPoolExecutor() as executor:
    executor.submit(process_run, tg_run, sync_store)
    executor.submit(process_run, web_run, sync_store)

There is more work for the FastAPI web application module, so we are doing it last, For now, we first modify the function signature of run() in our Telegram bot, and update the exit_event.wait() call accordingly

async def run(sync_store: SyncStore) -> None:
    async with application:
        ...

        await asyncio.to_thread(sync_store.exit_event.wait)

        ...

Previously, the coroutine function message_consume implicitly read from settings.telegram_queue. As it is now made available to run(), we can proceed to pass it as an argument. Also there’s no reason why the bot’s application object to be global, let’s move everything into the function:

# remove this line
# application = ApplicationBuilder().token("Your telegram token").build()

async def run(sync_store: SyncStore) -> None:
    application = ApplicationBuilder().token("Your telegram token").build()
    async with application:
        await setup(application) # pass the application object explicitly
        # Starting the bot
        await application.start()
        # Receive messages (pass the object explicitly)
        asyncio.create_task(loop_queue(message_consume, application, sync_store.telegram_queue))
        # Wait for exit signal
        await asyncio.to_thread(sync_store.exit_event.wait)
        # Stopping the bot
        await application.stop()

Both loop_queue and message_consume would need to adapt to the change

from typing import Any
from telegram.ext import Application
from queue import Queue

# Take additional args to be passed to the coroutine function
async def loop_queue(coro_func: Callable[[], Awaitable[None]], *args: Any) -> None:
    while True:
        await coro_func(*args)

async def message_consume(application: Application, queue: Queue) -> None:
    with suppress(queue.Empty):
        await application.update_queue.put(
            Update.de_json(
                # no more consuming from settings.telegram_queue
                await asyncio.to_thread(queue.get, timeout=10),
                application.bot,
            )
        )

Finally, applying the changes to the FastAPI web application module. Like the Telegram bot module, we first revise the run() function signature

async def run(sync_store: SyncStore) -> None:
    ...

Then, we will have to register our store to the app.state ..) Also update the uvicorn config accordingly as app is modified in our function.

async def run(sync_store: SyncStore) -> None:
    app.state.sync_store = sync_store

    server = uvicorn.Server(
        uvicorn.Config(
            app, # we made changes to global app, so specify explicitly
            host="0.0.0.0",
            port=80,
            log_level="info",
        )
    )

    ...

Usually this is done in the lifespan, where we yield an object, that can be later accessed through request.state in the handler functions, as shown below:

from fastapi import FastAPI, Request

@asynccontextmanager
async def lifespan(app: FastAPI):
    yield {'foo', 'bar'} # yield the object here

app = FastAPI(lifespan=lifespan)

@app.get('/')
async def index(request: Request):
    print(request.state.foo) # 'bar'

Unfortunately, in our case app is instantiated outside of run() and it has no knowledge of sync_store at the time. Considering now we just store it at app.state directly, we could still define a lifespan handler, to check for the presence of the store, at startup:

@asynccontextmanager
async def lifespan(app: FastAPI):
    # do a check when the application starts up to ensure sync_store is available
    if not (hasattr("sync_store", app.state) and isinstance(app.state.sync_store, SyncStore)):
        raise RuntimeError("SyncStore object is not found")

    yield

app = FastAPI(lifespan=lifespan)

Lastly, update the webhook, to use the queue from sync_store, via request.app.state.sync_store.telegram_queue

@app.post("/webhook/telegram", include_in_schema=False)
async def telegram_webhook(
    request: Request,
    x_telegram_bot_api_secret_token: Annotated[
        str, Header(pattern=settings.TELEGRAM_SECRET_TOKEN, strict=True)
    ],
) -> None:
    asyncio.create_task(
        asyncio.to_thread(
            # change how the queue is accessed
            request.app.state.sync_store.telegram_queue.put,
            await request.json()
        )
    )

Although the refactoring looks rather significant, the core logic underneath is unaffected. Apart from changing how the queue is accessed in both the Telegram and the FastAPI web application, there’s no change on how it is done. Data is still being produced by the web application via .put(), and consumed by the .get() in the bot.

Beyond the Fix: Testability, Scalability, and What’s Next

So back to the original objective of this refactoring — remove the hard-coded telegram_queue. Fortunately, we have done so successfully, though the process was somewhat involved. The biggest advantage of passing them explicitly; is how it improves testability. No more patching and replacing globals are needed, just pass the test queue to the function, or properly populate the app.state while creating FastAPI’s TestClient fixture.

Testing would be covered next week.

We also explored grouping all the synchronization primitives into one object. When the number of child processes increases, the number of such objects would accumulate inevitably. Even if we cherry-pick, ensuring only the relevant ones go into each child process, some process would end up taking too many arguments. Thus, creating an object containing them can be a good workaround to the problem.

Other than the rather involved gymnastics of moving things around, there’s no change to the underlying workflow. Still it brings consistency compared to the original implementation. Now all synchronization primitives are all defined in the same place, all managed by the multiprocessing.Manager, and are introduced to each child process explicitly as arguments. Additionally, the usage of multiprocessing.Manager makes it easy to scale the application as the manager can run as a server.

Photo by Sharon Co Images on Unsplash

This is it, the journey started from a very unassuming take-home assignment and in the end became a valuable lesson. On a high level it isn’t quite significant, as it doesn’t bring much optimization (if not slower due to the change to the queue managed by multiprocessing.Manager). Even so, by applying what I learned to my chatbot, it immediately became relevant as the bot is still running out there serving its purpose. Turning the experience into this article; is also how I reciprocate to the community.

Hmm, that was the third time my LLM Editor suggested I should use a semicolon ..)

Anyway, that’s all I have for this week, thanks for reading, and I shall write again, next week.

Just so you know, while an AI assistant helped me refine this article’s clarity and structure, the voice, all the technical content, and every line of code presented here are entirely mine. This piece reflects my personal experiences and insights. If you’re interested in job opportunities or collaborating on projects, please feel free to reach out via LinkedIn or connect with me right here on Medium.

Code Rain’s Revelation: Embracing Existence Before Perfection

Choon-Siang Lai — Mon, 26 May 2025 19:51:02 +0000

I would like to start this article by thanking the editorial team of JavaScript in Plain English for featuring my previous article on setting up a LLM chatbot as my editor. Within the article I mentioned turning the final draft into a script for video production, and I am in the process of finally taking the plunge. Now that I have the script done, what else is needed? Let’s start by building a short intro animation.

From Node to Bun, Open-Source Python Projects & AI Storytelling

A cute illustration from Copilot on the topic

My Unexpected Journey: Why a Video Intro?

Despite hearing the suggestion numerous times, I never really considered going the video production route. The main reason was stated in the featured article — I am not ready to commit to a periodic release. Fast-forward to last weekend, I decided to publish my study notes on the book titled Dao of Functional Programming. Due to the complexity, I could only do it on an irregular basis. After publishing the first chapter, the implication became a big “Why not?” hovering in my mind.

Dao of Functional Programming (Chapter 1: Clean State)

Most of my recent articles are structured as a story writing project, including the one published last week. After publishing the article, I had the LLM chatbot assemble a video script out of the final draft. The proposed script, just as my previous attempts, appeared very actionable. After giving it a thought, I decided to bite the bullet.

The Unchaining: My Personal Journey Graduating from jQuery to Modern JavaScript

As a sidenote, I definitely can see the appeal of the LLM especially for people working solo.

Though I may not be able to finish the video in time, I read through the script to figure out the needed components for a video. First thing I noticed is a short intro sequence. Shouldn’t be too difficult, I figured, there should be a generator somewhere over the Internet. Let’s quickly make one and be done with it. However, after some failed attempts with the LLM video generator, I decided to just make a simple animated sequence myself.

A code raining effect inspired by The Matrix, how hard can it be, right?

After checking through my weekly TODO, I only had one take-home assignment for a job application. Considering I did not have much idea on how to do it, I allocated a day for that. The point was getting a usable version out anyway. Fortunately, it was not that difficult, I managed to whip out a version with jQuery, lodash and Chance.js within a day.

Yes, it is a simple web application.

On the other hand, I find the project more interesting than the coding take-home assignment I did the next day. Most of these assignments are about creating a prototype website involving a variation of CRUD (Create, Retrieve, Update, Delete) operations from scratch. Usually that would take 2–3 days end-to-end but I often allocate at most a day and a half for them.

On the other end of the spectrum, we have overly extensive assignments that even the hiring company would expect more than a day of work. Job applicants are often not being paid for the work (hence my 1 day allocation), but in this climate many would still submit their work without compensation. Alongside other problems shared by many fellow engineers when applying jobs, it prompted me to rethink my strategy.

This blog was started as my part of a longer-term plan, exploring options outside software engineering. Recently it turned into an instrument I use to improve my visibility in the job market, showcasing my ability to hopefully avoid take-home assignment in future applications. Being featured in a well-established article is a very unexpected confidence boost, and I can’t express my gratitude enough for that.

Beyond the Code: A Lunar New Year Reflection on Career, Recovery, and New Beginnings

Though it is equally important to acknowledge oneself internally.

While I still send out applications and meet folks for the possibility of collaboration, publishing content regularly is becoming a complement to my job hunting strategy. Producing video, in a way can be seen as an extension to all the technical writing I am doing here in this blog. A friend calls this “signalling”, but to me it is more an attempt to adapt.

The “Good Enough” Code Rain: Starting the Build

One common problem I often observe in life is that people often ask the wrong question “is it good?”, when there is nothing to improve on. This could be an oversimplification of life problems, but in my point of view, most things in life often boils down to either “does it exist” or “is it good?”. Both these questions are distinct from each other, but apparently enough people have a problem differentiating the two.

If there is one thing I learn, after meeting all the people approaching for project collaboration, that would be that ideas are worthless unless executed. There is never a shortage of brilliant ideas, the thing that is lacking is the will to bring them to reality. A conceived idea will inevitably grow over time and gets so ambitious such that it is close to impossible to begin working on it.

We do have a different name for this phenomenon — daydreaming.

The right question to start asking, in my humble opinion, is always “does it exist?”. In the case we are discussing today, it would be about the components of a video. There is none at the moment, so let’s start the collection and building. If the generator is not giving me the intro animation I wanted, then just build one myself, answer the existential question first.

And I did.

The crude initial version

It was a fairly simple animation featuring the code rain, ending with a display of the name of the channel. As mentioned earlier, it worked right away, which was good enough as a quick and dirty hack. Next the clip was uploaded t̶o̶ ̶s̶h̶o̶w̶ ̶o̶f̶f̶ ̶m̶y̶ ̶1̶3̶3̶7̶ ̶h̶a̶c̶k̶i̶n̶g̶ ̶s̶k̶i̶l̶l̶ to collect feedback from peers.

“Is it good enough?” should usually be asked, after answering the first “does it exist?” question. I fully expected to revisit the animation, addressing this question much later. In spite of that, Meta threw me a curveball after it ate my clips, returning an error saying I violate some rules. The linked help topic was a generic document on how to post videos.

Typical Meta, surprise surprise.

That prompted an immediate revision to the animation. If Meta doesn’t like the short clip, it means any uploaded videos containing the clip would be banned as well. To a certain extent, this is a blessing in disguise, I did receive feedback, though in the most unexpected way. Imagine how devastating this could be if I spent my whole life planning for it, only to find the work banned right away.

There are indeed rare occasions where these two should be considered in parallel. Yet, for most cases, where time and resources are limited, the two questions should be addressed separately, in the right order. We can spend our lifetime pondering how things can be perfected, but it changes nothing if no work is being laid out.

Existence != Perfection

Crafting the Rain: Modules, Events, and Iteration

This is the meta-approved (for now) version.

Meta-approved revision

Still based on the same code rain animation, but I am turning the channel name display into an ASCII art, and including the display of a video title. An outro is also added so it doesn’t end abruptly. Let’s discuss the implementation a little bit.

Coders writing for Node.js should be familiar with the module syntax ,,) shown below:

import defaultExport from './someModule';
import { otherExport } from './someModule';

const bar = 'bar';

export function foo() {
    ...
}

export default bar

Frontend development gets increasingly more complex over time. Gone are the days where all the frontend logic is expressed within one script alone. As the complexity grows, the amount of code also increases. Separating them into smaller parts eventually becomes a necessity.

So the module syntax is a welcome change, though I only knew the support was ported to the browser while I was porting grid-maker last week. All it needs is to change the tag’s type attribute to module.<br> </p> <div class="highlight"><pre class="highlight plaintext"><code><script type="module" src="./someModule.js"></script> </code></pre></div> <p></p> <p>Alternatively, you can just put the code inline<br> </p> <div class="highlight"><pre class="highlight plaintext"><code><script type="module"> import defaultExport from './someModule`; import { otherExport } from './someModule'; const bar = 'bar'; export function foo() { ... } export default bar </script> </code></pre></div> <p></p> <p>No more multiple <script> tags just to import different libaries, we now do it with the import statements, as it also supports importing from a URL. For example, to import functions from lodash, we write:<br> </p> <div class="highlight"><pre class="highlight plaintext"><code>import { range, reverse, } from "https://cdn.jsdelivr.net/npm/lodash-es@4.17.21/lodash.min.js"; </code></pre></div> <p></p> <p>The intro animation script was largely <a href="https://kitfucoda.medium.com/the-unchaining-my-personal-journey-graduating-from-jquery-to-modern-javascript-ca74a93a7b33">inspired by the</a><a href="https://kitfucoda.medium.com/the-unchaining-my-personal-journey-graduating-from-jquery-to-modern-javascript-ca74a93a7b33">grid-maker remake</a>. Any changes to the DOM would be initiated by event triggerings. So we are making a rather simple code rain animation in a 2D grid of monospaced characters. A “raindrop” effect is just a column of characters in the grid “activated” (fading in and out) one after another. We could chain all these activations together in a sequence of nested recursive function calls, risking blowing the stack if the number of rows increases.<br> </p> <div class="highlight"><pre class="highlight plaintext"><code>function activate_rain(char_in_grid) { const downstair_char = find_downstair_char(char_in_grid) do_the_animation(char_in_grid) if (downstair_char) { activate(downstair_char) } } </code></pre></div> <p></p> <p>Alternatively, we could let each activation trigger another activation below it by dispatching an event.<br> </p> <div class="highlight"><pre class="highlight plaintext"><code>function activate_rain(event) { const elem = $(event.target); do_the_animation(elem); find_downstair_char(elem).trigger("block:activate:rain"); } </code></pre></div> <p></p> <p>I did this in jQuery despite last week’s article, mainly because this is not meant to be deployed in a production environment. The function activate_rain was registered as a listener to the block:activate:rain, therefore firing the same event (after a delay) for the neighbour. The function exits after the dispatching of event, removing the risk of stack overflow. In the actual implementation, I also added a delay through setTimeout to the event dispatching to adjust the downward speed of a raindrop.<br> </p> <div class="highlight"><pre class="highlight plaintext"><code>setTimeout(() => find_downstair_char(elem).trigger("block:activate:rain"), 100); </code></pre></div> <p></p> <p>The full implementation can be <a href="https://github.com/Jeffrey04/kitfucoda-intro">reviewed at my GitHub repository</a><a href="https://github.com/Jeffrey04/kitfucoda-intro">.</a>.) Feel free to adapt and play with it. You may want to find a way to start a web server to serve the page to test it in your browser.</p> <h3> <a name="the-code-rains-enduring-lesson-execute-first-then-refine" href="#the-code-rains-enduring-lesson-execute-first-then-refine" class="anchor"> </a> The Code Rain’s Enduring Lesson: Execute First, Then Refine </h3> <p><img src="https://dev-to-uploads.s3.amazonaws.com/uploads/articles/9xau70vzs3ya6htns4za.gif"/><br> <em>My editor insisted this is needed again here.</em></p> <p>Initially I intended this to be a quick article on the making of the animation for the video production side-project. However the scope of the article grew when I was organizing my thoughts. For some reason it reminded me of a quick exchange on the two questions “does it exist?” and “is it good?” with my mentor not too long ago.</p> <p>We live in a very short-tempered world, where we expect things out fast and good. Sometimes it gets really easy to forget the difference between the two, especially when deadlines are approaching. Mixing the two up in such cases would inevitably generate a huge amount of stress. Even I find myself caught in situations like this before.</p> <p>The unexpected connection between my simplistic problem-solving view and the animation-making process was a surprising discovery. If the banning accident did not happen, I wouldn’t be able to make the connection, and we would read a very different article this week. Despite a round of revision, there are still numerous improvements I hope to achieve with it. Still, having it done checks an item of my TODO for a final video.</p> <p>I will pick an older article, and produce a video out of it with the help of LLM drafting the script. As it is going to be irregular, the review will come a little bit later, perhaps until I am more comfortable with the workflow. For now stay tuned, and thanks for reading this far. I shall meet you in my writing again, next week.</p> <p>While an AI editorial assistant helped refine this article’s language and structure, please know that my voice, ideas, and all code are entirely my own work. If <a href="https://kitfucoda.medium.com/">KitFu Coda</a> sparks your interest, feel free to reach out for job opportunities or project collaborations directly on <a href="https://kitfucoda.medium.com/">Medium</a> or via <a href="https://www.linkedin.com/in/jeffrey04/">LinkedIn</a>.</p> <hr>

The Unchaining: My Personal Journey Graduating from jQuery to Modern JavaScript

Choon-Siang Lai — Tue, 20 May 2025 06:08:43 +0000

When I was building a quick frontend to the LLM game, I used jQuery to quickly whip out a prototype. Only after I was happy with it, I ported the code to the modern DOM API. As a result, I totally removed the dependency on jQuery. This whole experience makes me wonder, do people still use jQuery, in this age of frontend engineering? I took some time over the weekend to port one of my old jQuery plugins. This is not intended as a how-to (we have enough of them everywhere), but rather a reflection on how things has changed over time.

A cute illustration generated by Microsoft Copilot

When jQuery Ruled the Web: My Journey Begins

I joined web development in an era where people finally realized JavaScript can do more than just adding fun animations to a page. With the ability to send HTTP request, this opens up a whole new world of interactivity. This newly discovered magic extended the imagination boundary and inspired new dynamic designs.

Previously, these kinds of interactivity could only be achieved with Macromedia/Adobe Flash/Shockwave, requiring a separate runtime installation.

New design patterns emerged, and eventually led to the creation of new JavaScript framework and libraries. In my first job, I was introduced to Yahoo UI (YUI) library at work. jQuery came out slightly later, and it eventually became the most popular library at the time, likely due to the simplistic syntax and efficiency. Sizzle, the query engine powering the library, was touted the best and fastest at the time.

Compared to the Java-like verbosity of YUI, jQuery was designed to get the same tasks done with less code. One thing I really like about the latter is the ability to chain function calls together.

$(element)
  .trigger("board:block:init:pre", [context, options])
  .addClass("block")
  .width(size)
  .height(size)
  .data("column", col_idx)
  .css({
    float: "left",
    "margin-right": options.margin_px + "px",
    "margin-bottom": options.margin_px + "px",
    "border-width": options.border_px + "px",
    "border-style": "solid",
    "border-color": options.border_color,
  })
  .trigger("board:block:init:post", [context, options]);

In the snippet above, the chain does:

Trigger a custom event with type board:block:init:pre with additional arguments to be sent to the handler
Add a class block to the selected element
Set the width and height of the element
Set a value to the data property named column
Set the inline CSS style declaration
Trigger another custom event with type board:block:init:post with additional arguments to be sent to the handler.

The compactness jQuery brings is still unrivalled, even with the much improved DOM API. We will briefly go through what jQuery does later, using my old jQuery plugin — grid-maker as an example.

At one time, I was building a lot of mini web apps, and they all have one single common element — a grid. You might be wondering, why not Flexbox? It was new at the time, and it seemed to work well, but it also brought more complexity. Even now, I still don’t fully get it, though I completed this cute gamified tutorial.

Anyway, after a few attempts, I revised the code, and decided to extract and reimplement the logic of building a responsive grid as a jQuery plugin. And yes, this library is very well deprecated, so go use a proper CSS grid. You may read this article as a tribute to jQuery’s greatness.

The Allure of Chaining: How jQuery Simplified My Development

Previously, we saw a snippet adapted from the grid-maker plugin. So let’s dive deeper into how do we start coding with jQuery. For DOM manipulation, we usually start by writing a query, following CSS selector rule. Say you want to look for the second div of class board in a div with class hello, you would need the selector div.hello div.board:nth-child(2). Pass the selector as string to the jQuery function (by default aliased to a dollar sign $) to get started, as shown below:

$("div.hello div.board:nth-child(2)")

Now we can do DOM manipulation to the returned jQuery object wrapping the matched element. It can now move around the tree of DOM elements or be populated with other elements. We also saw how the HTML attributes (data-column, class) and inline style could be modified. The one change per function call is likely following the builder pattern. Here comes my favourite feature — the chaining of function calls (many considered this a Fluent Interface). Let’s say we want to adapt the creation of 10 rows from the grid-maker

$(this)
  .append(
    $.map(_.range(options.row), function (row_idx) {
      return $(document.createElement("div"))
    }),
  )
  .append(
    $(document.createElement("br")
      .css("clear", "both")
  )

The last append() could be rewritten as follows to be more concise

.append($.parseHTML('<br style="clear: both;" />'))

Just a side note, $.map functions similar to the new .map() function for arrays. In modern JavaScript the rough equivalent would be

_.range(10) // this underscore method returns a series of 10 numbers in array
  .map((row_idx) => document.createElement("div"));

Yes, the jQuery plugin was written in the pre-ES6 era.

The grid-maker was made to be event-driven. Web applications were starting to get more complex (and hence the birth of more capable frameworks like React, Angular, Vue etc) at the time. As the complexity grew, keeping track of things became more difficult. And through the building of the mini web apps, I figured the best strategy would be delegating all the updates to the event listeners. Registering an event listener is easy, just tag a on() call to the chain. Let’s say we want to delegate the further initialization work of each created row to a function called row_init, we can do this

.append(
    $.map(_.range(options.row), function (row_idx) {
      return $(document.createElement("div"))
        .on("board:row:init", row_init)
    }),
  )

Then we could trigger the actual initialization with trigger(), additional data can be passed to the event handler in the second argument as a list:

.append(
    $.map(_.range(options.row), function (row_idx) {
      return $(document.createElement("div"))
        .on("board:row:init", row_init)
        .trigger("board:row:init", [
          some,
          other,
          arguments
        ]);
    }),
  )

Though not present in the grid-maker, jQuery was also known for simplifying HTTP requests. It eventually introduced the Promise interface in later versions, similar to the one built into the modern JavaScript now. Yes, it is similar to the new FetchAPI.

$.ajax("https://example.com")
  .then(
    () => { console.log("success" },
    () => { console.error("failure" },
  )

Next we will look at the porting process.

Side-by-Side: Unpacking Modern JS vs. jQuery’s Approach

For simpler web applications, I still prefer to keep it simple. Possibly due to my familiarity with jQuery, I still build my first iteration of work with it. Why is that the case? I hear you ask, but hopefully the reason becomes clear as we move along.

We started by selecting an element, let’s emulate jQuery’s behavior, so the solution can be applied to not just 1, but possibly multiple matches. Let’s start with finding the list of elements using document.querySelectorAll, as shown below:

const elems = document.querySelectorAll("div.hello div.board:nth-child(2)");

Then we need to loop through them, this is easy, just chuck everything into a loop:

elems.forEach((board) => { ... })

For each matched board, we populate the rows

_.range(options.row).forEach((row_idx) => {
  const elem = document.createElement("div");
  const br = document.createElement("br");
  br.style.clear = "both";

  board.appendChild(elem);
  board.appendChild(br);
});

A reference of the row’s div element is kept for event registration purpose, to be shown later. The first major difference would be the losing of (subjectively) elegant chaining syntactic sugar. I also know I could use innerHTML property for the br tag, but I don’t like the idea of injecting raw HTML string into it.

Now, let’s talk about custom events. I decided the library to be event-driven because I wanted to avoid tracing a long list of function calls when a change happens. This is particularly useful when you have multiple elements to be updated whenever something happens. Instead of chaining them together in the event handler, let each individual element subscribe to changes. As an additional benefit, you get a more testable code as the scope gets broken down into smaller units.

Let’s start with event listeners, the equivalent of jQuery’s .on() method in JavaScript is .addEventListener().

elem.addEventListener("board:row:init", row_init);

In order to trigger the event manually, the equivalent of .trigger() would be the .dispatchEvent() method. However, this time, we would need to create a CustomEvent for the purpose. This is where you find the abstraction offered by jQuery valuable.

elem.dispatchEvent(new CustomEvent("board:row:init"));

We needed to pass additional argument just now, but unfortunately, it is done a little bit differently with vanilla JavaScript. Pass the additional data through event.detail by adding a second argument to the CustomEvent instantiation.

elem.dispatchEvent(new CustomEvent(
    "board:row:init",
    { detail: {
      some,
      other,
      arguments
    }}));

And event handler function would have to be declared differently, as shown below

// jQuery version
function row_init(event, some, other, arguments) { ... }

// vanilla JavaScript version
function row_init(event) {
  const {some, other, arguments} = event.detail;
  ...
}

The new syntactic sugar on object destructuring is used here to somewhat replicate the behaviour.

The FetchAPI equivalent to $.ajax would be somewhat similar for people who are already familiar with jQuery’s Promise object.

fetch("https://example.com")
  .then(
    () => { console.log("success" },
    () => { console.error("failure" },
  )

Just for a visual comparison, the screenshot shows the difference of the two implementations.

Code comparison

Different formatters were used so the line count isn’t the best way to determine which is better. However, even with all the new API and syntactic sugar, the vanilla JavaScript version still feels a lot verbose. It is especially evident when it comes to event handling, with the explicit CustomEvent object instantiation. Also losing DOM manipulation methods like appendTo(), prepend(), prependTo() etc. will require developers to rethink the population sequence. Throughout the process of porting legacy code to modern JavaScript, I find this reference titled “You might not need jQuery” useful, especially for people who are looking to “graduate” from jQuery and embrace the new APIs.

There are certainly a lot more to cover when it comes to migrating away from jQuery, as shown in the linked website. The point of this exercise is more about the question we ask in the beginning — do people still code using jQuery now, with all these new toys brought by the recent updates to the API and language itself?

A “Coming of Age” in Code: What My jQuery Journey Taught Me

As mentioned earlier, the idea of this article came from a realization of building prototype with jQuery by default. If I had kept the original revision done purely with jQuery, I wouldn’t have to port grid-maker to prepare for this article. Having the original and ported code, and comparing them side-by-side is definitely a very interesting experience, considering the original code was written close to a decade ago.

I am still in awe to see the original code still works fine in action today.

Would this exercise change my habit of not starting with jQuery in the future? Likely not. Grabbing jQuery and start working almost feels like a muscle memory deeply ingrained in my body. Or, it could be just because it worked so well and rarely failed me. Heck the prototype application I built still works to this day, what more can I ask?

On the other hand, I certainly see some footprints left by jQuery in the new APIs. They could all be purely coincidence, but it was indeed almost revolutionary at its prime.

If I could make an analogy, I would see this article as a coming of age film, showing the process of us growing out of jQuery in the context of web development. It is still great, but it is likely time to move on. If you are new into the field, may this article serve as a decent introductory to the library.

This article was shaped with the editorial assistance of Gemini. Rest assured, though, that the technical code and the voice telling this story are entirely my own. If my work resonates with you, I’d love to connect for job opportunities or project collaborations. You can reach me here on Medium or find me on LinkedIn.