DEV Community: Shri

Only One Per Customer

Shri — Mon, 25 May 2026 13:44:52 +0000

henge pushes config to edge devices. Semantically, each device should connect once - except:

What if a device is misconfigured with the details for another device
What happens if a device disconnected, but we haven’t picked it up yet
How about if a device has a zombie connection and decides to reconnect
Any other cases I’d not thought about.

So, we need to figure out how to handle a second connection from the same device.

Allow multiple connections

For each device, we track multiple connections. Instead of a map to eachchannel, we’d have one to an array of chans ([]chan)

Every operation on channel then becomes a loop.

For the case where each correct case is most likely to be one device to one connection, this option has a lot of additional complexity, work and defect surface area.

Reject additional connections

This option is simpler. If a device is already connected, we reject any additional connections.

This option is certainly simpler than allowing multiple connections. However, we are dealing with a network and there are many reasons why a connection does not close properly.

To be able to mitigate some of the issues, we’d have to allow the user to manually clear an open connection so that a device can re-connect.

The endpoint is easy enough to write, but requiring user intervention is not something I like having as part of a product.

We could add a heartbeat, and close the connection down if it’s doesn’t receive a heartbeat for a configured amount of time. This reduces the reliance on user intervention by replacing it with wait time. Of course, it also now adds complexity.

Close existing connection on reconnect

How about on reconnect, we close any existing connections and reconnect to the new request?

Instead of one chan per connection, we now have a Subscription

type Subscription struct {
    stream   chan henge.ConfigChangedEvent // actual stream of events
    closeCmd chan struct{}                 // on msg, unsubscribe
}

And the connection code involves a little more of a dance:

func (c *ConfigEventBroker) Subscribe(deviceId string) *Subscription {
    c.mu.Lock()
    defer c.mu.Unlock()
    sub, found := c.channels[deviceId]
    if !found {
        sub = &Subscription{
            stream:   make(chan henge.ConfigChangedEvent, 1),
            closeCmd: make(chan struct{}, 1),
        }
        c.channels[deviceId] = sub
    } else {
        // already connected
        // close previous connection
        sub.closeCmd <- struct{}{}
    }

    return sub
}

and the connection itself respects the quit command:

    for {
        select {
        case snap := <-sub.stream:
            err = sendSSEEvent(w, f, Snapshot{
                Version: snap.Version,
                Values:  snap.Config,
            })
            if err != nil {
                slog.Warn("error while emitting sse event", "err", err)
                return
            }
        case <-sub.closeCmd:
            // client reconnected. We can close this one
            return
        case <-req.Context().Done():
            r.broker.Unsubscribe(deviceId)
            err = r.devices.SetConnected(deviceId, false)
            if err != nil {
                slog.Warn("unable to set connect status to disconnected", "err", err)
            }
            return
        }

We don't have to:

Track multiple connections
Have Heartbeats
Require user intervention

In the event of rapid reconnect flurry coinciding with a slow handler, it should
still behave correctly, albeit slower. This rare edge case is accepted. If a
device is reconnecting that quickly to a slow handler, there are probably much
bigger problems at play.

Always On

Shri — Tue, 14 Apr 2026 18:51:09 +0000

I knew as soon as my phone rang what it was about.

It was the same every time. I would drag myself up to answer the phone - my body, my mind screamed at me, but I had gotten good at overriding every instinct through sheer willpower. I could hear the apologetic tone on the other side, and I could recognise some of the voices after a while. I mustered up all of my strength to be and sound as awake as possible. I needed to be professional even if I was still in my underwear.

I had a glass of water on my bedside table. I’d pick that up and head to my office in the spare room. The computer was always on and always ready to go — like me I guess. I’d log on to the servers, and check the logs. If I can identify which one fell out of the group, I can restart just that one. If I was too late or if the issue had escalated, I’d have to restart the whole cluster — shut them all down, give them a few seconds, then bring each one up, while keeping an eye on them. I could do it half asleep after a while.

Falling back asleep wasn’t a breeze either - I was tired - exhausted - but I was now also wired. Waking up in the morning was harder - the alarm would go off and my body would be limp. I still remember the sheer power of will to drag myself into the shower, then carry on with the rest of the day.

Of 266 incidents over about two years, I answered 156.

I remember one particular night, though I do not remember how many times I’d woken up beforehand. I was already tired.

megabus.com had gone offline. I got an alert. “But someone else is on call tonight,” I told them. “We already tried them twice,” came the reply. I had to deal with this. I had to deal with this.

I remember sitting at my desk at home working on fixing it. At some point, something was different, though I don’t remember what. While I was working on fixing it, I remember being overcome with an overwhelming impulse to get up from the chair and walk away — I almost imagined myself walking away. I resisted and shut down that impulse. I fixed megabus as I had always done. In fixing megabus though, something broke inside me, somewhere deep, in the very core of my being. I was never the same again.

I analysed the system top to bottom, inside and out. I even waded through JVM internals.

It got incrementally better, more stable. I think I rewrote every component that wasn’t the core ticketing system. In the end, what pushed it over the line were two unexpected changes. Automated nightly restarts of each node in the cluster and a rate limiter.

On the 10th December 2012, the system had a sale event. We had a bank of screens on a wall with all the key stats for the system. It looked cool, and we felt a bit like we were on a TV show. At peak, nearly 15,000 concurrent sessions — six or seven times the average. Over 30,000 bookings in a single day, three times more than the normal amounts across all systems.

We watched it closely, all day. Nothing broke. Nothing screamed. Everyone smiled, but there was no celebration.

Did They Have a Problem That Year?

Shri — Tue, 14 Apr 2026 09:24:26 +0000

2008 was a heck of a year for kraya, and for me. We were already operating megabus.com in the UK, USA, and Canada, along with Oxford Tube, the sales website for coach usa - all for Stagecoach.

We were also working on the fringe website. We integrated the website with the brand spanking new ticketing system - which cost nearly £900k.

We were also hosting websites for Boots, Kellogg’s Food Service and dozens of other clients.

All of this was held together by three or four developers, two systems administrators and me.

On the 13 June (incidentally, I got married on the same date years later), as I was just getting ready for a wild night on the town, a call comes through - which John answers.

I still remember them laughing and then doing a double take “oh, you’re serious? let me get Shri”

It was the fringe. We’d already known that they were having trouble with their ticketing system. I’d even pitched in, made suggestions - looked at their code to try and help, but none of that was enough. I expected an update.

They wanted to know if we could put together an interim booking system for them over the weekend. I wasn’t sure. I told them I’d speak to my team and get back to them.

I wasn’t involved with the work on the fringe up until this point. I knew very little about it. I was focused on megabus.com. The US version of the site had a big marketing campaign happening in a few days and that was what I was meant to be focused on.

By the time I put the phone down, Chris, who had been the lead on the fringe already had a answer. “We can do it!”

“But how - it’s got to take more than a weekend - right?”

The fringe website was already built well and had a clean layer interfacing with the new ticketing system. In fact, that was the bulk of the work that year.

So.. Chris told me - all we would have to do is to implement the functionality within that thin layer, fattening it up.

He believed we could do it. I believed him.

I hopped in a cab, headed over to the fringe to talk it through. I didn’t promise them we’d be able to get something ready by Monday, but I promised we’d do our best.

We were in the office on the weekend, writing code. I remember working on the basket, sending diffs over email and generally having a good time.

I even had some megabus US fun to keep me entertained in the form of issues with loading sheets - I was already in the office, so it was one step easier to fix.

One of the bits of functionality which took a surprising amount of time was the seat allocation. None of us had to worry about that before - it was just capacity management on megabus. For the fringe though, we had to allocate actual seats with seat numbers and everything.

With the fringe we had multiple tables which all joined together (thanks hibernate) to encode a tremendous amount of detail about the seating plans - including their physical location on a map.

It was too much detail for us, so we had to simplify it all down to get it working in the timeframe. We kept most of the rest of the structures intact to keep the data migration easier once the ticketing system was fixed.

By Monday, we had each managed at the most 6 hours of sleep each of the previous three nights. I still have vivid memories of a suit of armour that we put together using packing material while we were waiting for bits of data or details of logic.

I broke the MySQL replication at 01:24, fixed by 01:32

I remember the delirium setting in. Email sent to the client with “fun fun fun fun fun fun fun” as the subject

There were random emails to my brother “I’m still here!”

I also remember making makeshift beds with bubblewrap to get a wee nap here and there. We were all so exhausted - pumped up on coffee and nicotine.

Finally at 03:35 on the Tue email to client: “DONE DONE DONE DONE DONE NODE NODE NODE NODE”

Then at 05:33, requesting a PostgreSQL server rebuild for megabus US for their marketing campaign.

At 10am on Tuesday, the fringe is finally able to sell tickets. The website promptly fell over from the load, but we nurse it back and it sells 65k+ tickets in the first week.

It would be at least two more weeks before the ticketing system is fixed and brought back in.

For the work we did for them that year, and the previous one, we effectively only charged about 30% - because that’s all they could afford. This year, we asked if they could put our name on the website.

Over the next two weeks(while megabus US was on their marketing campaign), we fought many battles. There were 750 duplicate bookings. Numerous customer complaints (thanks to our name being on the website) - almost all of them blaming us for the failure of the ticketing system. People did not understand that we put in the interim one, not the one that failed.

Press releases went out from the fringe - only two credited us. Both misspelt the company name. Both called us a web design company — which, we were not, had never been, and had no interest in becoming.

In truth, I wanted to be a hero - I think we all did. What we really wanted was an acknowledgement of what we had done - which was nowhere to be found. We got paid though - at least for a part of our effort.

For many years after that, I would tell people with pride - “did you know - I saved the fringe, back in 2008,” which was inevitably met with something like “oh, did they have a problem that year?”

What we Carried

Shri — Mon, 13 Apr 2026 19:02:04 +0000

I started my company in 2000. I was 17. I built megabus.com in 2003. I was 21.

It started off small, and little by little, I carried more and more. I became we, and we carried more and more. Before we realised, we were carrying a great deal. Ultimately, though, if something went seriously wrong, it would be on my shoulders.

The chart does not capture the scaling of the organisation, or other departments like tech support or hosting, which had dozens of clients.

Data mined by Claude from my emails, issue trackers and code repos

The section at the top is the number of active committers for that quarter. You can see my on my todd at the start and the rise and the fall of the dev team.

The very peak of it was in 2008. We built a booking system over the weekend for the Edinburgh festival fringe because their brand new £800k+ system could only handle one person at a time.

There were a handful of us building it while I was simultaneously prepping and managing the megabus US systems for a sales campaign. At the same time we were operating megabus across the UK, USA, and Canada, Oxford Tube, Coach USA, the Fringe website itself and numerous other smaller hosting clients like Boots, Kelloggs Food Service, and so on.

We were a total of ~4 developers and two systems administrators holding all of these together.

I was recently reminded of a story a friend of mine told me. Before he was my friend, he worked with me, and one of the things we did together was one the big megabus deployments when we migrated to a Java EE ticketing system.

One part of the migration was the data. I had developed a tool to migrate the data and on the evening - everything was prepared, we were off peak, and we had taken the site offline. He ran the script, which went on for a wee while and it failed. It was not meant to do that.

The way he tells the story, he told me about the failure. I come over, look at the errors, say “hmmm, that’s interesting,” and head off to have a cigarette. A few minutes later I go over to my desk type away furiously, then asked him to run it again.

It worked, and completed.

I remember that night. I don’t remember what the problem was or how I fixed it, but I do remember that moment when I went over to see how it had failed. In the short walk from my desk to his, I reiterated in my mind, all the possible backup plans - with the worst case scenario being to call off the migration on that day. We would do it another day. It would cost money, but it would be do-able. I was ok with that.

I was curious as to where my limits were, so I kept pushing, until I would meet with a wall.

There were no rails and there were no railings - just a cliff edge, unmarked… I didn’t know that - I expected a brick wall.

The worst of it would be only a few years later, in 2011. We built a new Java EE ticketing system for a fraction of what it should have cost in about 30% of the time it needed.

I personally responded to over 250 out of hours emergency tickets over an 18 month period. That was hard!

I had run off a cliff edge, and like the roadrunner in the cartoons, it took a while before I realised there was no ground beneath me.

A few years after I ran off the cliff, the company shut down. A few years later, I would start my active recovery journey through therapy. A few years later still, when I felt ready for a leadership role, I was asked to lead a problematic team - a role they struggled to fill for a while.

The team worked hard and delivered but struggled with the perception of poor delivery. Trust was thin, stress was high and morale was low. The situation was so bad that the week before I was supposed to start, the scrum master who was supposed to be my guide through it all quit.

I was warned by multiple people that this job was loaded with problems. I took on the job anyway, without a real guide, straight into multiple serious issues.

I loved it and managed to turn the whole thing around in my first week. Delivered key items, laid the foundations of trust and improved morale. It took a bit longer to bed everything down. Within weeks, I was asked if I would take on leading the entire digital team.

It was here, many years later, I got a sense of how unusual it was for such a tiny team to do so much.

It was here, many years later, I got a sense of how it is to have guardrails, to have support, peers to lean on.

It was here, for the first time I realised that the job didn’t have to be a lonely one.

Whatcha Thinking?

Shri — Mon, 13 Apr 2026 11:05:48 +0000

I loved working on megabus. I was in love with it. My girlfriend at the time had a habit of asking what I was thinking about when I looked deep in thought. The answer - every single time, was inevitably megabus. She eventually stopped asking.

I was 22 years old.

When I built the original prototype for megabus.com, I built it using PHP + PostgreSQL. I put together a document detailing my reasoning for these choices. I quoted 33 days for it, built it over six weeks and charged £13,200.

The support contract was £300/month - for one day a month. On the first day, megabus.com sold 200 orders.

When megabus had its first expansion, I was up overnight bringing new servers online and scaling it live. I loved it - my code was finally being tested.

Over a week, I’d probably burned through many days of effort. I remember the project manager specifically asking me to invoice for the extra work I put into it. I even said that I would - except I didn’t.

I’ve had a long time to think about this - why did I not send that invoice? I even had approval.

The answer, as with most things of this nature is complicated. I loved the work and I didn’t want it to end. I didn’t want a potential conflict trying to figure out what a reasonable amount was to charge. I felt that I should have done a better job in the first place - I felt responsible that I had not told them that scaling of this nature would not have worked without prep work.

I had not scaled anything before.

I was 22 years old.

I was super grateful that someone believed in me. I naively assumed that they saw all the extra effort I was putting in and that they would reward me for it - that they would have my back.

I remember adding a bunch of different bits of functionality because I wanted it there. I didn’t want to go through the process of quoting for it, and it getting potentially rejected, not to mention the waiting for decisions. One key bit of functionality I remember is adding in a percentage load column for the loading sheets. I built it, showed it - they loved it! It went live. I did not charge for it.

At this point, the vast majority of my time was spent on megabus - very little of it actually paid for.

At a glance, based on the emails sent, I probably spent a minimum of 10 days each month supporting megabus when I was charging for one day.

In Jan 2004 - I proposed doubling the contract to two days for £4,800/year. It probably kicked in in Feb 2004. By March 2004, the site exceeded that revenue each day.

In the following months, I probably spent, on average a minimum of at least double the time I was paid for. I should have charged for it.

I grew the team, and the support contract based on the minimum I needed to maintain the product - not based on the amount of time I was spending.

For my 28th birthday, my girlfriend at the time organised a cake which was a image representing kraya - which was basically megabus. I felt bad that she thought that kraya was the most important thing in my life - she was right - but it still felt bad. kraya had other clients at the time, but my time wasn’t monopolised by other clients, or indeed by kraya - my heart still belonged to megabus.

And it would all have been all fine too, except for a grave miscalculation I made.

In 2010, after trying to rebuild the ticketing system for £500k, and making some mistakes with people I trusted, kraya ended up in £150k in the hole. We needed some money urgently.

I was desperate and naively, I reached out to stagecoach for help. I thought they were my friend - that they would have my back.

They understandably lost a great deal of trust in my ability to manage and lead my company. I trusted the wrong person - but that was still my mistake. They were right.

I thought that I’d built up enough goodwill that they would help me through this. I’d felt I would have way more than that “in the bank” in terms of goodwill. I learned that professional relationships do not work that way that dark afternoon, standing outside my office on the phone, in the rain.

They didn’t make my life easier. Instead, I’d ended up rattling the cage - they were now panicked - realising their over-reliance on an organisation that could disappear at any point.

Instead of support, I had further actions, renegotiating the contract and what felt like punitive, and definitely invasive reporting obligations.

I was hurt and angry. I had poured my heart, my soul - hey, my very life into this product that I loved.

Suffice it to say - I got no help - no loan, no offer of investment - though they did suggest buying us outright - which I rejected.

I signed a contract under circumstances I would not wish on anyone.

The best I got from them was a challenge - if we were really spending more time than we were charging for - prove it. I did! We documented every minute we were spending - I wasted my time on spreadsheets, pointless meetings and work to try and rebuild the broken trust.

We went from £300k in the hole to £200k profit within a year. We charged for a whole year in support around 20% of what the system made in a day.

I was 28 years old.

Around the same time, I was also dealing with the operational aftermath of trying to build a java EE ticketing system over six months for £500k. I thought it would take a year and cost £1m. In hindsight, it needed two years and probably three million pounds.

Over 18 months, I personally answered over 250 out of hours emergency calls. We had a rota and others on call too - but I took the vast majority of these calls. I felt bad putting others through what I knew was gruelling.

All of this led me down a narrower and narrower path to a serious breakdown - though I didn’t know enough to name it until many years later. All I knew - all I felt was that something broke in me.

We managed to resolve all of the issues, but the deployment of that version kept getting pushed.

Stagecoach cancelled the contract in 2012. They had started building a ticketing system in-house two years prior - the cost of my grave mistake. I wasn’t able to make the meeting - I was in India, and at the same time as the meeting, I was meeting for the first time the one who is now my wife.

I was 28 years old. I spent the next 15 years putting myself back together.

How much did it cost them to build it inhouse? If I had charged for my time from the start, would we all have been better off?

I still feel something deep inside me every time I see a megabus - a sense of pride mixed in with a deep sense of sadness - not for what I lost - but for what could have been.

I am 44 years old, and I am starting again.

Publishing from hugo to dev.to

Shri — Tue, 23 Sep 2025 08:30:54 +0000

I have been pondering federating parts of my blog to dev.to for a bit more visibility.

However, I had a couple of issues:

With html (instead of markdown), it would not pick up the code blocks correctly
With markdown, it would render the relative links incorrectly

What I really wanted was a way to render the hugo markdown into Jekyll style (which is what forem wants) but with the links rendered.

This was a little more complicated than I would have liked.

Goals

At a minimum, I wanted two main things:

Render code blocks correctly
Relative URLs should be rendered as absolute (because they won’t work on dev.to)

Pre-existing Solutions

I found hugodevto which looked promising except:

Not a fan of having to manually update hundreds of posts with the devto id
Had a couple of fiddly bits to get it working (my plain text outputs had some troubles for unknown reasons)
It rendered image urls, but not regular urls.

Ultimately though, it felt a bit bulkier than what I was looking for

Using Hugo

I had done enough work with hugo and output formats to have a vague idea of how to make it work.

Limitations

There are a few limitations to doing it this way though. Hugo does not provide render hooks for everything . You will end up with html in the output. However, since Forem (and Jekyll) will just render them, it fits my use case. It won’t work as well if you try and use this to generate like for like markdown usable in Jekyll.

Define a new content type

We want a new content type which will output markdown

```toml [outputFormats.jekyll] mediaType = "text/markdown" baseName = "index" isPlainText = true isHTML = false notAlternative = true path = 'jekyll' # put the output in `public/jekyll` so it's easier to find [outputs] page = ['html', 'jekyll'] # Output all pages in our jekyll format as well ```

You also need a basic template before hugo will output our markdown files.

layouts/_default/single.jekyll.md

---
title: {{ .Title }}
published: true
date: {{ .Date }}
{{- with .Params.tags }}
tags: [{{ delimit . ", " }}]
{{- end }}
canonical_url: {{ .Permalink }}
---

{{ .Content }}

With this, if you hugo build, it’ll render the .md files, but the content will be html.

Render code blocks as markdown

We can use the code block render hook to “convert them” back to markdown.

layouts/_default/_markup/render-codeblock.jekyll.md

Render absolute urls

I already have a pretty extensive render-link so updating it was just a case of making a copy of it and replacing relative url references with absolute ones.

It’s fine for it to be in html because Forem will still render it correctly. They could be rendered as markdown and it should work just as well.

layouts/_default/_markup/render-link.jekyll.md

{{ /* other content */ }}
  <a href="{{ printf "%s#%s" .PageInner.Permalink $u.Fragment | safeURL }}" {{ with .Title }}title="{{ . }}"{{ end }}>{{ $text }}</a>
{{ /* other content */ }}

Output

With these relatively minor changes, I was able to render markdown files I could then pop into dev.to and it works for the handful I set up.

Next Steps

Images

One big glaring omission is images - it’s not as relevant for me because I rarely use images.

I expect it to be easy enough to use the image render hook to achieve this.

Automation

Once the above two are done, it would be good to automate it. I could

get all the posts
get all the local posts
map them based on the canonical url (which is returned by the endpoint)
upload updated ones.

Not all my posts are technical, so I’d also want to add a field to the frontmatter (devto_published) and figure out a way to push updates only if there are changes.

Conclusion

I have a working solution for now - and if dev.to brings enough traffic / value, then I’ll consider spending a bit more time adding polish.

For the time being though, seems to work!

Feel free to use any of this code (such that it is) as you wish.

Using `locateFile` to have js and wasm in different locations with emscripten

Shri — Thu, 18 Sep 2025 09:15:42 +0000

Originally published at drone-ah.com

As part of building shine, I am using
lume and webassembly with zig.

zig, through emscripten generates both a js and wasm file, which, by default are
expected to be in the same directory.

I wanted to put them in different places, and struggled to get that working with
lume for a bit. I did eventually solve it though:

Include emscripten js file

Firstly, the js file output from emscripten should be included in you page
manually. I had used site.add which meant that it was loaded before we could
put the override for locateFile in window.Module.

I am using the
simple-blog theme, so I add the
following line to the top of src/index.vto

<script defer src="/js/shine.js"></script>

Override `Module`

You can override how emscripten finds the wasm file in the
Module Object.

This can be done in the html file in a script block, or even better, in a js
file that is included automatically.

js/main.js felt like a good place.

window.Module = {
  locateFile: function (path, scriptDirectory) {
    if (path === "shine.wasm") {
      return "/static/shine/shine.wasm";
    } else {
      return scriptDirectory + path;
    }
  },
};

Closing

With this setup, I can not only have the js and wasm files in different
locations, it's also easy to modify / augment the Module object.

Calling Javascript from Zig through WebAssembly

Shri — Wed, 17 Sep 2025 14:52:20 +0000

The next step for shine is to build a bridge between zig and javascript .

I am currently planning to using supabase for storage. Unsurprisingly, it does not have a zig sdk. It does, however, have a javascript sdk.

If I can write basic CRUD operations in javascript and call that from zig through webassembly, that could make that integration a lot easier.

Goals

There are a few ideal restrictions for me - mainly because writing javascript is not fun for me.

Use the Supabase js/ts library through zig
Use TypeScript as much as possible. (I don’t love TypeScript, but at least it’s not javascript)
Keep as much of the supabase related code in the web part so that deno and lume can handle any heavy lifting.

Options

I can’t use FFI(Foreign Function Interface):

extern "env" fn jsLog(ptr: [*]const u8, len: usize) void;

extern "env" fn jsLog(ptr: [*]const u8, len: usize) void;

const wasm = await WebAssembly.instantiateStreaming(fetch("prog.wasm"), {
env: {
jsLog: (ptr, len) => {
/* read from memory and console.log */
},
},
});

because imgui pulls in emscripten, which means we don’t have the ability to call instantiateStreaming.

With emscripten, declaring an external function is easy enough.

extern fn jsLog(ptr: [*]const u8, len: usize) void;

There are a couple of options to wire them up to the javascript:

`library.js` / `mergeInto`

This option requires javascript files on the zig side. If you want to start with typescript, you’ll need to integrate a transpiler into the build chain as well.

First, you want a javascript file - let’s call it libshine.js, and pop it into a js dir.

// js/libshine.js
// Emscripten will provide these globals at link/runtime
declare var mergeInto: (lib: any, funcs: Record<string, Function>) => void;
declare var LibraryManager: { library: any };
declare function UTF8ToString(ptr: number, len?: number): string;
mergeInto(LibraryManager.library, {
jsLog: (ptr: number) => {
const msg = UTF8ToString(ptr);
console.log("🟢 Zig says:", msg);
},
});

We then need to pass this js file into the build step

as part of my sokol build step, I pass it in as .extra_args.

// create a build step which invokes the Emscripten linker
const link_step = try sokol.emLinkStep(b, .{
.lib_main = shine,
.target = opts.mod_main.resolved_target.?,
.optimize = opts.mod_main.optimize.?,
.emsdk = dep_emsdk,
.use_webgl2 = true,
.use_emmalloc = true,
.use_filesystem = false,
.shell_file_path = opts.dep_sokol.path("src/sokol/web/shell.html"),
// set the js file here
.extra_args = &.{
"--js-library", "js/libshine.js",
},
});

We can then call it from zig, with something like:

pub fn main() void {
jsLog("hello from zig");
}
extern fn jsLog(ptr: [*]const u8) void;

From my firefox console:

Lume live reloading is ready. Listening for changes...     localhost:3000:102:15
🟢 Zig says: hello from zig                                shine.js:3168:11

`EM_JS` / `EM_ASM`

The other option is to use EM_JS which involves writing a wee bit of C, which can embed the javascript.

In theory, it’s as simple as:

#include 
<emscripten.h>
EM_JS_DEPS(bla, "$UTF8ToString");
EM_JS(void, jsLog, (const char* s), {
console.log(UTF8ToString(s));
});

and adding it into the build file:

// build the main file into a library, this is because the WASM 'exe'
// needs to be linked in a separate build step with the Emscripten linker
const shine = b.addLibrary(.{
.name = "shine",
.root_module = opts.mod_main,
});
// get the Emscripten SDK dependency from the sokol dependency
const dep_emsdk = opts.dep_sokol.builder.dependency("emsdk", .{});
// need to inject the Emscripten system header include path into
// the cimgui C library otherwise the C/C++ code won't find
// C stdlib headers
const emsdk_incl_path = dep_emsdk.path("upstream/emscripten/cache/sysroot/include");
shine.root_module.addCSourceFile(.{
.file = b.path("src/libjs.c"),
.flags = &.{}, // optional extra emcc flags
});
shine.addSystemIncludePath(emsdk_incl_path);

The calling code in main.zig remains the same:

pub fn main() void {
jsLog("hello from zig");
}
extern fn jsLog(ptr: [*]const u8) void;

However, this didn’t work, and failed with:

error: undefined symbol: jsLog (referenced by root reference (e.g. compiled C/C++ code))
warning: To disable errors for undefined symbols use `-sERROR_ON_UNDEFINED_SYMBOLS=0`
warning: _jsLog may need to be added to EXPORTED_FUNCTIONS if it arrives from a system library
Error: Aborting compilation due to previous errors

Thanks to some help from flooh (who btw put together the sokol and sokol-zig packages as well the sokol-imgui-sample template which I used to kick start this project.), I was able to get it working.

Turns out the c file needs to have a function in it that is used in the zig file - it doesn’t need to do anything.

So, based on the suggestion, libjs.c changes to:

#include 
<emscripten.h>
EM_JS_DEPS(bla, "$UTF8ToString");
EM_JS(void, jsLog, (const char* s), {
console.log(UTF8ToString(s));
});
void dummy(void) {};

and in main.zig:

pub fn main() void {
dummy();
jsLog("hello from zig");
}
extern fn jsLog(ptr: [*]const u8) void;
extern fn dummy() void;

From my firefox console:

Lume live reloading is ready. Listening for changes...     localhost:3000:102:15
🟢 Zig says: hello from zig                                shine.js:3168:11

You can see a working example in [my forked repo](

`EM_JS` directly through `zig` [unsuccessful]

Looking at the macro for EM_JS and with my good friend ChatGPT, I attempted translating it to zig and made some progress, but ultimately failed to get it working. I’ll leave the work here in the hopes it might be helpful.

#define _EM_JS(ret, c_name, js_name, params, code)                             \
_EM_BEGIN_CDECL                                                              \
ret c_name params EM_IMPORT(js_name);                                        \
__attribute__((visibility("hidden")))                                        \
void* __em_js_ref_##c_name = (void*)&c_name;                                 \
EMSCRIPTEN_KEEPALIVE                                                         \
__attribute__((section("em_js"), aligned(1))) char __em_js__##js_name[] =    \
#params "
<::>
" code;                                                       \
_EM_END_CDECL

The above macro translates to zig roughly (with help from ChatGPT) as:

extern fn jsLog(ptr: [*]const u8) void;
/// 2. Keep a reference to avoid the linker removing the function.
///    Same role as __em_js_ref_* in the C macro.
pub export const __em_js_ref_jsLog = &jsLog;
/// 3. Embed the JS implementation in a special section called "em_js".
///    Emscripten will scan this and inject the code into the output JS.
export const __em_js__jsLog align(1) linksection("em_js") =
"(const char* s)
<::>
{ console.log(UTF8ToString(s)); }\x00";
pub export fn dummy() void {}

I added a pub fn and called it from main:

pub fn log(ptr: [*]const u8) void {
jsLog(ptr);
}

Which gave me the familiar error about not being able to find jsLog.

comparing the linker sections gave some clues:

❯ wasm-objdump --section=linking -x <path/to/libjs.o>
libjs.o:        file format wasm 0x1
Section Details:
Custom:
- name: "linking"
- symbol table [count=9]
- 0: F 
<dummy>
func=1 [ binding=global vis=hidden ]
- 1: D 
<__em_js_ref_jsLog>
segment=0 offset=0 size=4 [ binding=global vis=hidden ]
- 2: F 
<jsLog>
func=0 [ undefined explicit_name binding=global vis=default ]
- 3: D 
<__em_js__jsLog>
segment=1 offset=0 size=53 [ exported no_strip binding=global vis=hidden ]
- 4: S <.debug_abbrev> section=7 [ binding=local vis=default ]
- 5: G 
<env.__stack_pointer>
global=0 [ undefined binding=global vis=default ]
- 6: S <.debug_str> section=9 [ binding=local vis=default ]
- 7: T 
<env.__indirect_function_table>
table=0 [ undefined exported no_strip binding=global vis=default ]
- 8: S <.debug_line> section=10 [ binding=local vis=default ]
- segment info [count=2]
- 0: .data.__em_js_ref_jsLog p2align=2 [ ]
- 1: em_js p2align=0 [ RETAIN ]

and the zig object:

❯ wasm-objdump --section=linking -x js.o
js.o:   file format wasm 0x1
Section Details:
Custom:
- name: "linking"
- symbol table [count=6]
- 0: F 
<dummy>
func=1 [ binding=global vis=default ]
- 1: D 
<__em_js_ref_jsLog>
segment=0 offset=0 size=4 [ binding=global vis=default ]
- 2: F 
<jsLog>
func=0 [ undefined explicit_name binding=global vis=default ]
- 3: D 
<__em_js__jsLog>
segment=1 offset=0 size=4 [ binding=global vis=default ]
- 4: D 
<__anon_946>
segment=2 offset=0 size=54 [ binding=local vis=default ]
- 5: T 
<env.__indirect_function_table>
table=0 [ undefined exported no_strip binding=global vis=default ]
- segment info [count=3]
- 0: .rodata.__em_js_ref_jsLog p2align=2 [ ]
- 1: em_js p2align=0 [ ]
- 2: .rodata.__anon_946 p2align=0 [ ]`

From what I could understand (which is little), it looks like __em_js__jsLog in the zig obj is a pointer while from C, it’s the full string.

hardcoding it as a static array helped:

export const __em_js__jsLog align(1) linksection("em_js") = [_]u8{
'(', 'c','o','n','s','t',' ','c','h','a','r','*',' ','s',')',
'<',':',':','>','{',' ',
'c','o','n','s','o','l','e','.','l','o','g','(',
'U','T','F','8','T','o','S','t','r','i','n','g','(',
's',')',')',';',' ','}','\x00',
};

The output from this is a little more promising

❯ wasm-objdump --section=linking -x js.o
js.o:   file format wasm 0x1
Section Details:
Custom:
- name: "linking"
- symbol table [count=5]
- 0: F 
<dummy>
func=1 [ binding=global vis=default ]
- 1: D 
<__em_js_ref_jsLog>
segment=0 offset=0 size=4 [ binding=global vis=default ]
- 2: F 
<jsLog>
func=0 [ undefined explicit_name binding=global vis=default ]
- 3: D 
<__em_js__jsLog>
segment=1 offset=0 size=53 [ binding=global vis=default ]
- 4: T 
<env.__indirect_function_table>
table=0 [ undefined exported no_strip binding=global vis=default ]
- segment info [count=2]
- 0: .rodata.__em_js_ref_jsLog p2align=2 [ ]
- 1: em_js p2align=0 [ ]

Let’s look at the two side by side

# From C
- 3: D 
<__em_js__jsLog>
segment=1 offset=0 size=53 [ exported no_strip binding=global vis=hidden ]
# From zig
- 3: D 
<__em_js__jsLog>
segment=1 offset=0 size=53 [ binding=global vis=default ]

There are some clear differences in how the two are output and I am already beyond my knowledge level here - so I’ll leave it to someone who knows this stuff better (or wait until I do)

You can check out the code in the branch of my forked repo

Next steps

My plan is to use EM_JS through C to implement glue JavaScript functions - something like:

EM_JS(void, jsLog, (const char* s), {
Module.jsLog(UTF8ToString(s));
});

By doing this, I can have one-line js code in the .c file and all the implementation can go into the web side (and can easily be TypeScript too).

window.Module = {
jsLog: function (msg) {
console.log("🟢 Zig says:", msg);
},
};

Auto reload WASM with zig+lume

Shri — Wed, 17 Sep 2025 11:20:51 +0000

I’ve been taking some time off to rest and recover from health issues that made
it hard to focus. To ease back in, I’ve started a small project:
shine

The project would do well to be multiplatform - mobile and web. The obvious
choice was flutter and I have enjoyed working with it before.

However, as I'm currently in love with zig, I wanted to work with
that instead.

Libraries

Graphics

I've been playing with raylib and that was my initial instinct.
However, raylib
does not support iOs and
has issues with wasm

I considered a few options, including

sdl, which looked great but was perhaps a little too low level for me.
jok - does not support mobile and possibly has a little more than I needed.

In the end, I decided to go with sokol and
sokol-zig.

While a little more lower level than raylib, it has:

a modern clean api
first class mobile support
first class wasm support

UI

I've been working with dvui a lot recently. Unfortunately, it
doesn't support sokol. imgui is a better
option.

There is even a
template project that I could start from.

WASM first

To keep things straightforward, I decided to start with wasm. If I make the site
mobile friendly, I could see how it goes and see if it needs a mobile version.

I will need some shared data storage and have been considering
supabase, which has javascript
libs. By using wasm, I can effectively shim in js functions to
handle that instead of having to write bare rest calls from zig.

Structuring the project

Is this a zig project with a web component, vice versa or indeed two independent
parts that work together.

I did a fair amount of web searching to see if there was some guidance I could I
find for a good way to structure a relatively straightforward zig+js project.

I could not find one. In the end, I decided to keep it fairly straightforward.

- shine/
  - src/ # zig code
  - web/ # all the web stuff

Frontend

After a bit of research, and realising that I will probably need a little bit of
supporting content around shine, I decided to go
with lume

I used the simple-blog theme as
a template to start from. I could have just pulled the template in but I wanted
a custom homepage.

When I tried to add an index.md, it complained about two files wanting to
write index.html. From what I could find, the easiest way to override the
homepage was to just pick up the theme and edit it - which was easy enough.

WASM => frontend

I didn't want to copy over the wasm and the js file every time, so I added a
couple of steps to build.zig right after the link_step (also included below)

// build.zig
// create a build step which invokes the Emscripten linker
const link_step = try sokol.emLinkStep(b, .{
    .lib_main = shine,
    .target = opts.mod_main.resolved_target.?,
    .optimize = opts.mod_main.optimize.?,
    .emsdk = dep_emsdk,
    .use_webgl2 = true,
    .use_emmalloc = true,
    .use_filesystem = false,
    .shell_file_path = opts.dep_sokol.path("src/sokol/web/shell.html"),
});
// attach to default target
b.getInstallStep().dependOn(&link_step.step);

// Copy shine.js from default emscripten output
const js_install = b.addInstallFileWithDir(
    b.path("zig-out/web/shine.js"),
    .{ .custom = "../web/src/static/shine" },
    "shine.js",
);
js_install.step.dependOn(&link_step.step);
b.getInstallStep().dependOn(&js_install.step);

// Copy shine.wasm from default emscripten output
const wasm_install = b.addInstallFileWithDir(
    b.path("zig-out/web/shine.wasm"),
    .{ .custom = "../web/src/static/shine" },
    "shine.wasm",
);
wasm_install.step.dependOn(&link_step.step);

These steps will copy across the wasm and the js file across to static/shine.
I wanted to put the js in src/js and the wasm in the static dir. However, the
js file expects the wasm in the same dir. I tried overriding locateFile but it
didn't work.

<!-- index.vto -->
<script>
  window.Module = {
    locateFile: (path, prefix) => {
      if (path.endsWith(".wasm")) {
        return "/static/shine/shine.wasm";
      }
      return prefix + path;
    },
  };
</script>
<script src="/js/shine.js"></script>

I was able to get lume to process the javascript file by adding it.

// _config.ts
site.add("static/shine/shine.js");

Conclusion

With all of these set up, I was able to run:

zig build -Dtarget=wasm32-emscripten --watch

in one window. This command will rebuild wasm and provide it to lume whenever
the zig code changes.

deno task serve

Running this in another window will mean that lume will rebuild on any changes,
including a new wasm file and redeploy. The redeploy will trigger an auto-reload
of the page as well if I have it in a browser.

I now effectively have automated reload with changes if I make changes in either
zig or the frontend.

I don't have hot reload - but this is pretty good for now.

Building Pong in Zig with Raylib – Part 1: Setup, Paddles, and Ball

Shri — Sat, 28 Jun 2025 19:38:29 +0000

Before continuing development on Triangle - my larger, arcade ARPG, factory game, I wanted to take a step back and build something small and familiar. Pong felt like the perfect choice: quick to prototype, easy to understand, and a good warm-up before diving deeper into raylib again.

This post goes alongside my video on YouTube , and walks through the early steps of the project: setting up the game, getting something on screen, and implementing the paddles and the ball.

Why Pong?

Sometimes, before getting deeper into a project, it helps to do something simple just to get your hands moving again. I hadn’t written Zig in a couple of weeks, and wanted a fast feedback loop to:

Get back into using raylib-zig
Have a bit of fun
Remind myself why I made certain decisions in the first place

Project Setup

I’m using raylib-zig for this. You can scaffold a new project with:

./project_setup.sh <project-name>

You could also use zig init and then add the dependency in manually if you prefer.

Drawing the Playground

The Pong “arena” is simple:

Two paddles
A ball (currently static)
A center dividing line
Placeholder for scores

Implementing Paddles

The paddles have:

A position (top-left corner)
A fixed size
A simple render function

const std = @import("std");
const rl = @import("raylib");

const Paddle = @This();

pos: rl.Vector2,

pub fn init(x: f32) Paddle {
    return .{ .pos = .{
        .x = x,
        .y = 200,
    } };
}

pub const size = rl.Vector2{ .x = 25, .y = 100 };

pub fn render(self: Paddle) void {
    rl.drawRectangleV(self.pos, size, .white);
}

It might have been nice to be able to have the paddle calculate more of its own values, but the more I think about it, the more it makes sense to keep the logic in paddle simpler.

Ball Placeholder

To wrap things up for this session, I added a static ball in the center of the screen. It has a radius, position, and velocity fields ready to go. Rendering is straightforward:

const std = @import("std");
const rl = @import("raylib");

const Ball = @This();

pos: rl.Vector2,
r: f32 = 16,
vel: rl.Vector2 = .{ .x = 0, .y = 0 },

pub fn render(self: Ball) void {
    rl.drawCircleV(self.pos, self.r, .white);
}

What’s Next

This was mostly about warming up, but the next episode will tackle:

Adding velocity to the ball
Basic collision detection with paddles

I’ll try and think about simplifying paddle logic, especially the awkward symmetry between left and right.

You can find the full source code on GitHub .

See you in part 2!

Directed Acyclic Graphs and Executing Tasks in Order (and in Parallel) Based on Dependencies

Shri — Mon, 07 Nov 2011 23:44:36 +0000

A little while ago, there was a requirement to write a tool that could take a number of tasks each with a set of dependencies and execute them in parallel while taking the dependencies into account.

The tasks themselves were meant for data migration but that is not particularly relevant. We were writing a number of tasks which all had a set of dependencies (some of the tasks did not have any dependencies or the process could of course never start).

It was assumed that there were no cyclic dependencies (which would be error in this particular case anyway)

Bearing in mind that this was a quick and dirty tool for use three times, some of the bits in here could do with tidying up.

Each task was defined to implement the following interface

    public interface Task extends Runnable {

        public String getName();

        public Set getDependencies();

    }

It should all be self explanatory. Extending the Runnable interface ensure that we can pass it into threads and other relevant bits of code. The getDependencies is expected to return the name of the tasks that it depends on.

The basic task runner which I describe below does not check if the task described in any list of dependencies actually exist. If an non-existing dependency is defined, it will likely just throw a Null Pointer Exception. I wrote this a long time ago, so don’t actually remember.

The BasicTaskRunner which we used to run the tasks implemented the TaskRunner Interface

    public interface TaskRunner {

        public boolean addTask(Task task);

        public boolean prepare();

        public boolean start();

        public void waitToComplete();
    }

The addTask method simply added it to a map from String -> Task and threw an exception in the event of a duplicate task being added in.

       @Override
        public synchronized boolean addTask(Task task) {
            LOG.info("Adding task: " + task.getName());

            if (tasks.put(task.getName(), task) != null) {
                throw new RuntimeException("Task with same name already exists: " + task.getName());
            }

            return true;
        }

the prepare method just calls a method to buildGraph. This uses the jGrapht library to build a Directed Acyclic Graph

       private boolean buildGraph() {

            LOG.info("Building DAG of tasks");
            graph = new SimpleDirectedGraph(DefaultEdge.class);

            LOG.info("Adding tasks");

            for (Task task: tasks.values()) {
                graph.addVertex(task);
            }

            LOG.info("Adding Relationships");

            for (Task task: tasks.values()) {

                if (task.getDependencies() != null) {
                    for (String depend: task.getDependencies()) {

                        Task dependOnTask = tasks.get(depend);

                        LOG.info("Adding relationship between " + task.getName() + " and " + dependOnTask.getName());
                        graph.addEdge(dependOnTask, task);

                    }
                }
            }

            return true;
        }

So we create a simple directed graph, loop through the tasks, then each of its dependencies to create an edge, which we then add to the graph. Simple stuff really.

the start method, which actually executes the task is as follows:

       public boolean start() {

            int cpus = Runtime.getRuntime().availableProcessors();

            executor = new ThreadExecutor(cpus, 60, new LinkedBlockingQueue());

            numTasks = graph.vertexSet().size();
            LOG.info("Starting... Num Tasks: " + numTasks);
            startTime = System.currentTimeMillis();

            scheduleTasks();

            return true;

        }

As a basic algorithm, we pick up the number of available processors and use that many threads. scheduleTasks is a pseudo-recursive function whose role is to add the currently executable list of tasks into the executor to execute.

       private void scheduleTasks() {
            if (graph.vertexSet().size() == 0) {
                executor.shutdown();
            }

            synchronized (graph ) {
                Iterator iter = new TopologicalOrderIterator(graph);
                Set executing = new HashSet();

                while(iter.hasNext()) {

                    Task task = iter.next();
                    //System.out.println(task.getName());
                    if (graph.incomingEdgesOf(task).size() == 0 && !executing.contains(task)) {
                        executor.execute(task);
                        executing.add(task);
                    }

                }
            }

        }

If there are no tasks left to execute, we shut the executor down. All being well, we add every single task in the graph that has no dependencies to be executed. The threadpool ensures that any tasks that cannot currently be executed are queued.

We use a custom version of the threadpool as follows:

       private class ThreadExecutor extends ThreadPoolExecutor {

            public ThreadExecutor(int corePoolSize, long keepAliveSeconds, BlockingQueue workQueue) {
                super(corePoolSize, corePoolSize, keepAliveSeconds, TimeUnit.SECONDS, workQueue);
            }

            @Override
            protected void afterExecute(Runnable runTask, Throwable e) {
                super.afterExecute(runTask, e);

                if (e == null) {
                    completed((Task) runTask);
                } else {
                    failed((Task) runTask, e);
                }
            }

        }

The main purpose of this is to use the completed and failed callbacks to ensure that on complete, dependent tasks can be executed. On fail, we ensure that dependent tasks are not executed. The code currently does not allow for tasks that are left behind and will hang indefinitely after executing all tasks it can.

       public void completed(Task t) {
            LOG.info("Completed Task: " + t.getName());

            synchronized (graph) {
                graph.removeVertex(t);
            }

            long timeTaken = (System.currentTimeMillis() - startTime);
            int tasksComplete = numTasks - graph.vertexSet().size();

            long timePerTask = timeTaken/tasksComplete;

            long totalTime = timePerTask * numTasks;
            long timeToComplete = timePerTask * graph.vertexSet().size();

            LOG.info(" ## Tasks left: " + graph.vertexSet().size()
                   + " ## Elapsed: " + timeTaken/1000
                   + " ## Est. Total " + totalTime/1000
                   + " ## E.T.A : " + timeToComplete/1000);

            scheduleTasks();
        }

        public void failed(Task t, Throwable e) {
            LOG.fatal("Failed Task: " + t.getName(), e);
            scheduleTasks();
        }

On completion of a task, we simply remove the task from the graph. The frees up all its dependencies to be executed. We add these tasks into the list by calling scheduleTasks again. There is nothing more for us to do when a task fails except to schedule any other tasks that can be executed. In theory, this call is redundant since any tasks that could be executed before the failure are already in the queue. Any tasks that can be completed on the completion of another item will be initiated on the completion of that task.

I hope the above makes sense and has been helpful. The code for the full class including further logging statements follows. Please bear in mind that this was hacked together over a couple of hours for something that was to be executed a grand total of three times.

    public class BasicTaskRunner implements TaskRunner {

        private static final Logger LOG = Logger.getLogger(BasicTaskRunner.class);

        private Map tasks = new HashMap();

        private DirectedGraph graph;

        private ThreadExecutor executor;

        @Override
        public synchronized boolean addTask(Task task) {
            LOG.info("Adding task: " + task.getName());

            if (tasks.put(task.getName(), task) != null) {
                throw new RuntimeException("Task with same name already exists: " + task.getName());

            }

            return true;
        }

        @Override
        public boolean prepare() {

            LOG.info("Preparing task runner. Num Tasks: " + tasks.size());

            buildGraph();

            return false;
        }

        private boolean buildGraph() {

            LOG.info("Building DAG of tasks");
            graph = new SimpleDirectedGraph(DefaultEdge.class);

            LOG.info("Adding tasks");

            for (Task task: tasks.values()) {
                graph.addVertex(task);
            }

            LOG.info("Adding Relationships");

            for (Task task: tasks.values()) {

                if (task.getDependencies() != null) {
                    for (String depend: task.getDependencies()) {

                        Task dependOnTask = tasks.get(depend);

                        LOG.info("Adding relationship between " + task.getName() + " and " + dependOnTask.getName());
                        graph.addEdge(dependOnTask, task);

                    }
                }
            }

            return true;
        }

        public void waitToComplete() {
            try {
                executor.awaitTermination(3, TimeUnit.DAYS);
            } catch (InterruptedException e) {
                // TODO Auto-generated catch block
                e.printStackTrace();
            }
        }

        private long startTime;
        private int numTasks;

        @Override
        public boolean start() {

            int cpus = Runtime.getRuntime().availableProcessors();

            executor = new ThreadExecutor(cpus, 60, new LinkedBlockingQueue());

            numTasks = graph.vertexSet().size();
            LOG.info("Starting... Num Tasks: " + numTasks);
            startTime = System.currentTimeMillis();

            scheduleTasks();

            return true;

        }

        private void scheduleTasks() {
            if (graph.vertexSet().size() == 0) {
                executor.shutdown();
            }

            synchronized (graph ) {
                Iterator iter = new TopologicalOrderIterator(graph);
                Set executing = new HashSet();

                while(iter.hasNext()) {

                    Task task = iter.next();
                    //System.out.println(task.getName());
                    if (graph.incomingEdgesOf(task).size() == 0 && !executing.contains(task)) {
                        executor.execute(task);
                        executing.add(task);
                    }

                }
            }

        }

        public void completed(Task t) {
            LOG.info("Completed Task: " + t.getName());

            synchronized (graph) {
                graph.removeVertex(t);
            }

            long timeTaken = (System.currentTimeMillis() - startTime);
            int tasksComplete = numTasks - graph.vertexSet().size();

            long timePerTask = timeTaken/tasksComplete;

            long totalTime = timePerTask * numTasks;
            long timeToComplete = timePerTask * graph.vertexSet().size();

            LOG.info(" ## Tasks left: " + graph.vertexSet().size()
                   + " ## Elapsed: " + timeTaken/1000
                   + " ## Est. Total " + totalTime/1000
                   + " ## E.T.A : " + timeToComplete/1000);

            scheduleTasks();
        }

        public void failed(Task t, Throwable e) {
            LOG.fatal("Failed Task: " + t.getName(), e);
            scheduleTasks();
        }

        private class ThreadExecutor extends ThreadPoolExecutor {

            public ThreadExecutor(int corePoolSize, long keepAliveSeconds, BlockingQueue workQueue) {
                super(corePoolSize, corePoolSize, keepAliveSeconds, TimeUnit.SECONDS, workQueue);
            }

            @Override
            protected void beforeExecute(Thread thread, Runnable runTask) {
                super.beforeExecute(thread, runTask);

                Task task = (Task) runTask;

                LOG.info("Starting task: " + task.getName());
            }

            @Override
            protected void afterExecute(Runnable runTask, Throwable e) {
                super.afterExecute(runTask, e);

                if (e == null) {
                    completed((Task) runTask);
                } else {
                    failed((Task) runTask, e);
                }
            }

        }

    }

PostgreSQL performing huge updates

Shri — Sun, 06 Nov 2011 12:45:41 +0000

PostgreSQL is a pretty powerful database server and will work with almost any settings thrown at it. It is really good at making do with what it has and performing as it is asked.

We recently found this as we were trying to update every row in a table that had over eight million entries. We found in the first few tries that the update was taking over 24 hours to complete which was far too long for an update script.

Our investigation of this led us to the pgsql_tmp folder and the work_mem configuration parameter.

When the query was being executed, we checked the pgsql_tmp folder to see how was space being utilised in there. We already knew about the pgsql table from past experience. We had a server running out of disk space and rapidly. We had narrowed it down into this folder. In cancelling the query referenced by the tmp files in here, we were able to free up literally gigabytes of disk space...

We had found roughly half a gig of temporary files in here. This led us to investigate the configuration file.

The one parameter that stuck out was work_mem which was set to a default of 1mb which I guess might make sense under most circumstances but not in this one. According to the postgresql documentation

work_mem (integer)

Specifies the amount of memory to be used by internal sort operations and hash tables before switching to temporary disk files. The value is defaults to one megabyte (1MB). Note that for a complex query, several sort or hash operations might be running in parallel; each one will be allowed to use as much memory as this value specifies before it starts to put data into temporary files. Also, several running sessions could be doing such operations concurrently. So the total memory used could be many times the value of work_mem; it is necessary to keep this fact in mind when choosing the value. Sort operations are used for ORDER BY, DISTINCT, and merge joins. Hash tables are used in hash joins, hash-based aggregation, and hash-based processing of IN subqueries.

This would tell us that the total memory usage with work_mem could be several times the value set here and setting it to half a gig would probably be a terrible idea for a heavily utilised production server. However, for the migration process when we need to update over 8,000,000 rows, it might be a good temporary fix.

After updating the work_mem to 512mb, we found that no more tmp files were created and the whole thing was done in memory.

When updating so many rows, there area a few other things to consider.

Firstly, autovacuum will likely kick in several times to vacuum the table. You'll probably want to disable this for the duration of the update statement and run a vacuum afterwards.

sql --disable auto vacuum ALTER TABLE sometable SET ( autovacuum_enabled = false, toast.autovacuum_enabled = false );

You can switch autovacuum back on after the update statement has completed

sql --enable auto vacuum ALTER TABLE sometable SET ( autovacuum_enabled = true, toast.autovacuum_enabled = true );

A few other things you want to take a look at are the

fsync parameter (I usually have this set to off anyway since the servers are pratically fully redundant)
checkpoint_segments: I changed this to roughly 5 times the original value (check the log to see if it says that its checkpointing too often)
checkpoint_completion_target: I changed this to 0.9

With all of these updates, we were able to bring the total time of the update down to a few hours.

DEV Community: Shri

Only One Per Customer

Allow multiple connections

Reject additional connections

Close existing connection on reconnect

Always On

Did They Have a Problem That Year?

What we Carried

Whatcha Thinking?

Publishing from hugo to dev.to

Goals

Pre-existing Solutions

Using Hugo

Limitations

Define a new content type

Render code blocks as markdown

Render absolute urls

Output

Next Steps

Images

Tags

Automation

Conclusion

Using `locateFile` to have js and wasm in different locations with emscripten

Include emscripten js file

Override Module

Closing

Calling Javascript from Zig through WebAssembly

Goals

Options

library.js / mergeInto

EM_JS / EM_ASM

EM_JS directly through zig [unsuccessful]

Next steps

Auto reload WASM with zig+lume

Libraries

Graphics

UI

WASM first

Structuring the project

Frontend

WASM => frontend

Conclusion

Building Pong in Zig with Raylib – Part 1: Setup, Paddles, and Ball

Why Pong?

Project Setup

Drawing the Playground

Implementing Paddles

Ball Placeholder

What’s Next

Links

Directed Acyclic Graphs and Executing Tasks in Order (and in Parallel) Based on Dependencies

PostgreSQL performing huge updates

Override `Module`

`library.js` / `mergeInto`

`EM_JS` / `EM_ASM`

`EM_JS` directly through `zig` [unsuccessful]