DEV Community: Ersin Buckley

What do I want from your new API?

Ersin Buckley — Sat, 21 Sep 2024 21:10:00 +0000

My hands are sweaty as they race across the keyboard. My eyes dart through the stream of log messages going past. There it is! The reason I woke up at 2:30am on a Sunday. A broken API integration caused by an update I didn’t notice was supposed to happen 12 months ago.

So, you are writing a new API product, and I am going to be leading an effort to integrate with it. Read on for my list of what you should be caring about and the must-have features.

TLDR? The most important parts of your api design are:

Never break backwards compatibility
Be respectful of performance and stability
Give me tools for security and access management
Break the APi with care
Give me a great onboarding and documentation experience
Make it gorgeous and easy to understand

Stability and Backwards compatibility

The number one thing that I want from your API is that it never breaks, and my integration just works forever! Be gentle if you are going to break it. Once I have written the code, don’t make me go back and look at it 6 months, 1 year, 5 years later.

10% of effort is spent on implementing successful software and 90% is spent on operation and support. I call this the 10-90 rule. Most blogs, products and new developers focus on making the initial implementation easy. The reality is that business will spend 90% of the effort on operating and supporting a successful integration. An API really explodes this factor because so many external developers are also involved for each partner or customer that uses your product. A significant break in the integration could mean 100’s of developers need to stop and fix something. This level of distraction is churn just waiting to happen.

A surprising example of not breaking the API is Windows. A decades old binary will still run, because of the stellar focus on backwards compatibility. Keeping the ‘classic’ software running is a huge factor in the ongoing enterprise success that windows has. Many small and medium size businesses depend on these niche databases and applications. A huge sector of IT professionals are employed to operate and support them.

How do you avoid breaking backwards changes?

Use versioning to make sure that deprecated functions are never removed but newer implementations are available.
Never remove a field from the returned data type.
Do not rename a field.
Never re-type a field.

Sticking to these basic rules are the most important thing you can do to make me happy with your API product.

Performance and Scalability

Your API should never cause me a butt-clenching text message or phone call at 1:30am in the morning. To do this, you need to shift my focus to a better implementation which can handle an unexpected increase in the scale or performance of your API.

If a request is slow it should timeout. If it is always slow, it should be asynchronous. Timeouts are super simple to implement, and you can usually just add some configuration to your load balancer for it. To really handle long running requests though you need to have a ‘start the work’ endpoint and a ‘get the work’ endpoint. This is an idempotent approach that allows really easy caching of requests. It will result in less work done by the servers, as holding multiple long running requests open is more likely to result in timeouts and re-running the request.

Rate limiting is another must have for your API. It is important because I will be forced to think about retry logic during implementation, and this will yet again save me from an unplanned late evening wake-up call.

When you are developing the webhook feature, please be kind and provide me options for queuing or load shedding behaviour. I have never implemented a push integration which hasn’t eventually gone haywire at some point. This could mean your product has gone viral (good), or the vendor has gone wild (bad). In either situation I have been extremely grateful for systems put in place ahead of time to handle this load.

How to version and break changes

Never remove or break an API that is still being used. You need to wait at least a year, really! I have actually worked with customers and partners that do a once a year API call. When the yearly call breaks, you can probably figure out what that person thinks about your product when they come to fix it.

Instead of changing the meaning of an existing API, you need to introduce new versions. One easy way to do this is to introduce a new name for the API method. Or if you have a whole suite of breaking changes to endpoints you might mint a major version upgrade in the URL path.

But before you go blasting ahead on your API, I want you to ask yourself.

Do I really need to remove/break that api?

The best way to break it is to get on a call and help every customer using the API to upgrade their integration. If you did this you will be 1000% better than every other company that has an API product. This is something that I think is a very cool way to do something which doesn’t scale, and can delight your customer.

Security and access management

Give me tools to audit, manage and control how API keys are used.

Adding new keys or revoking old keys
Setting expiry times for temporary keys
Setting granular permission for the key
Seeing the usage of the key
Setting billing limits

You should care about these features because they will reduce your customer support workload. They help customers when their integration goes wild. If something bad happens, all the important audit data is available through the usage history for the API.

Clear Documentation and onboarding

Every commit that changes the API should come with a changelog update and a new documentation site. Tools like openapi help with the generation of docs and you should consider something similar for your platform. Include all the things, like generated types and field by field data model comments. Offering client libraries I can npm install or go get are nice to have, but not necessary in my opinion.

When I get round to integration of your API I’m going to start with the documentation. This is your opportunity to set me on the right path and make something that is high quality and won’t break on xmas day when I’m enjoying time with my family.

Another amazing thing you could consider here is ‘white glove’ level of service, give me access to a developer who knows your API and can help me integrate. Another opportunity for standing out amongst a field of mediocre at best API products.

Style doesn’t matter but you asked!

Least of all and last in the list is on the API style. HTTP, REST-ish and json formatted is just fine thanks!

I like plain old HTTP because I can call it from just about anything from curl through to any old other programming language.

Using roughly REST (no need to be zealots) style to me means using clear URL paths, and GET/POST/DELETE methods. It is incredibly nice to have cache headers in the response so that I can check if the rest of the body needs to be read or if I can simply wait and query again later for new data.

There are a lot of options for API style, graphQL, protoubf, xml, soap. Style is much less important than any of the other points here. Reason for being chill on API style is that it is more of an implementation factor than a support factor. One style from another doesn’t represent too much difference in the ongoing support of the integration.

Too long, did read

Thank you for taking the time to implement an API in your product, I really appreciate it and I love the ability to integrate and automate on top of the platform. It is important, and will be ever more important to the decision makers and buyers of software to have API functionality.

Most important in your implementation is to respect the 90% of time spent on supporting the API. You should be thinking about this with every increment on the product. We underestimate the pain and time that goes into the operation of software. If you follow my suggestions here your support people will be happy and your customers unburdened.

My favourite type of API integration is the one I do once and think about very little for the next few years.

If you are putting the effort into making an API product you should care about all this stuff like I do! If you don’t care about it but you're reading this anyway maybe we should chat and I can figure out how to help you on your journey.

Reflection is slow in Golang

Ersin Buckley — Sat, 25 May 2024 21:10:00 +0000

Recently I’ve been processing huge amounts of AIS (Automatic Identification System) data at my day job. By improving performance of the data processing, we enable prompt alerts and facilitate fast feedback loops, empowering platform users with real-time insight. Removing reflection from the hot loop resulted in a 29x performance improvement. Read on if you want to dive head first into what reflection is and why it’s not a good idea to call it thousands of times per second.

Generally speaking, the Go language is considered a performant and efficient language compared to its peers, but there are certain foot guns when it comes to using the reflection package. If you are using the package, it’s definitely worthwhile building out some simple benchmark tests to ensure you aren’t introducing a performance issue.

What is reflection anyway?

Reflection is a standard library package in go that lets you introspect the type system to dynamically figure out what the types and values of an any type are at runtime. This is the perfect solution if you want to write some code that iterates through the fields of a struct, or you want to write some code that reads the struct tag comment and does something smart with it.

In short, it’s the way to ‘meta program’ and write code that works on your own programs state at runtime. Let’s check out an example.

package main

import (
    "fmt"
    "reflect"
)

// A generic function to print struct details
func PrintStructDetails(data any) {
    val := reflect.ValueOf(data)
    typ := reflect.TypeOf(data)

    // Check if we're dealing with a struct
    if val.Kind() != reflect.Struct {
        fmt.Println("Not a struct!")
        return
    }

    fmt.Println("Type:", typ.Name())
    fmt.Println("Fields:")

    // Iterate over struct fields
    for i := 0; i < val.NumField(); i++ {
        field := typ.Field(i)
        value := val.Field(i)

        fmt.Printf("\tName: %s Type: %s Value: %v Tag: %s\n", field.Name, field.Type, value, field.Tag)
    }
}

Try it: https://play.golang.com/p/TVx-vokhiyY

In the above example, we have created a function PrintStructDetail, that will print out the names and fields of a struct programmatically. It takes an ‘any' type as the parameter, and then if it is a struct type it iterates through the fields and prints out the name and value of the field.

Why use reflection?

You probably have already used reflection in golang without realising it. For example, json encoding and decoding in the standard library is reflection based, it does something like the above to map parsed json values into a struct type based on the values of the field names.

Albeit you are unlikely to need to use reflection in your day to day coding, it can be used to make some magical things happen in Go, it’s a great way to extend the language. Advanced Go programmers should definitely add reflection to their tool belt.

Today however, we won’t be going in depth on how and why to use reflection, we’re going to show you how we can make parsing massively more performant by not using it.

Why is reflection slow?

To keep it dense and too the point, reflection is slow because of missed compiler optimization and extra memory allocation. When we call reflection on a type, it means the go runtime needs to allocate more memory to store the types, fields, names and tagblocks. All of this extra runtime stuff provides indirection in the generated assembly code. Processors are extremely efficient at doing the same operation many times in a row, because they can ‘pipeline’ multiple bits of data into the same operation. When reflection is used the pipeline optimization is not possible, furthermore – you need to do slow allocations and lookups on the heap to figure out what the type is before you can do the actual operation or work on the data

You don’t need to know exactly the theory on why this is slow to get the benefits of not using reflection though. To prove that it makes sense, just benchmark the code in practice, and use production telemetry to show how pulling out reflection can result in a performance improvement.

TLDR: if your types are massive or you are frequently invoking reflection, then you should look at pulling out the need for reflection from the code.

A deep dive on AIS parsing

AIS (Automatic Identification System) is a radio based protocol for helping ships and anything on the water avoid collision. My team is concerned with ingesting a global dataset of the stuff. Speed of ingesting the data is paramount. The first step of all our work involves decoding []bytesinto a struct so that we can perform analytics and build datasets for training machine learning models.

There are many AIS message types (27 of them), most commonly they are a bag of fields related to a ship’s position, heading, course, callsign and mmsi (a kind of unique vessel identifier).

type PositionReport struct {
    Header                    `aisWidth:"38"`
    Valid                     bool            `aisEncodeMaxLen:"168"`
    NavigationalStatus        uint8           `aisWidth:"4"`
    RateOfTurn                int16           `aisWidth:"8"`
    Sog                       Field10         `aisWidth:"10"`
    PositionAccuracy          bool            `aisWidth:"1"`
    Longitude                 FieldLatLonFine `aisWidth:"28"`
    Latitude                  FieldLatLonFine `aisWidth:"27"`
    Cog                       Field10         `aisWidth:"12"`
    TrueHeading               uint16          `aisWidth:"9"`
    Timestamp                 uint8           `aisWidth:"6"`
    SpecialManoeuvreIndicator uint8           `aisWidth:"2"`
    Spare                     uint8           `aisWidth:"3" aisEncodeAs:"0"`
    Raim                      bool            `aisWidth:"1"`
    CommunicationStateNoItdma `aisWidth:"19"`
}

https://github.com/BertoldVdb/go-ais/blob/master/messages.go#L32-L50

In the AIS parsing library, it’s represented by the above struct. The struct tag contains a aisWidth parameter that defines the number of bytes to ingest for that part of the message. A message is simply a sequence of bytes, where certain chunks of bytes represent different fields of data.

You can probably imagine how parsing works already. The parser will iterate through the fields of the struct, reading the ‘width’ number of bytes, reflecting on the type of the field, then calling the write conversion method to turn that data into something like a uint8, or a bool, or a string.

OK, so how does this all relate to the slowness of reflection? This approach works perfectly fine for lower volumes of data, but for larger datasets, you have repeated calls to reflect.TypeOf reflect.ValueOf. All this really adds up to the amount of time it takes for parsing a message.

How did I speed it up?

One day, while getting to the bottom of another issue in the codebase, I had just run the heap profiler on the decoder component. What I saw was a huge number of allocations in the reflect package. Specifically, the thing that iterates over struct fields, and figures out the type of these fields. Because of what I know about reflection being a ‘slower’ operation than usual, I had the confidence to dive in and perform some tests .

Ultimately, this led me to creating a small custom code generator that removes reflection from the parser, in favour of a more simple parser. I was able to use the existing metadata about the types provided in the struct tags, but generate a parser that didn’t call reflect at runtime at all.

Removing reflection resulted in a 29x speed improvement

Fortunately I work for an organisation that values open source, and I was able to contribute this change back to the source of the parser..

https://github.com/BertoldVdb/go-ais/tree/master/parser_generator

It’s a dense 500 lines, but it gets the job done. The code works by knowing which struct represents a message that can be parsed, and the code integer that maps to it. It was important to reuse as much of the existing parser as possible, because it is known to be ‘complete’ and correct. After running both implementations side by side over massive datasets of AIS, I was able to confirm this, and with confidence move the change into production.

Would I recommend this to others?

The way parsing works in this ais library is an interesting one. It works well as an expressive way to show how the AIS format works. However, I think it’s not a very idiomatic approach for a go codebase. I’m not sure I would recommend the code generation approach for building your own new parser either. Parsers are a really important advanced topic to learn. For anyone who hasn’t dabbled with parsers I would recommend a deep dive on the topic.

A good small hand rolled parser is worthwhile maintaining and developing, and I would avoid formal grammars and intermediate languages where possible. The problem with these parser generator approaches is that you often end up with a custom DSL that is difficult for others to pick up and learn.

There is a good amount of public knowledge that backs up this approach to building parsers, and you can see that most languages do not rely on formal or custom grammars like BNF, instead deciding to focus on a hand rolled approach

One recommendation I will make before wrapping this up, is that you should check out this really great book. In the parsers and compilers book, you go through a step by step test first approach to building a tokeniser, lexer and interpreter for a tiny language called ‘monkey’. I found it to be a very awesome intermediate -> advanced level book, and it uses the go language (even better!).

My Blog now supports RSS

Ersin Buckley — Thu, 01 Feb 2024 22:10:00 +0000

Behold, the latest and greatest feature on my tiny blog platform. RSS feeds! Yes reader, we now support the uber trending web 2.0 tech from 2005.

Now, is it relevant to have rss in 2024? I'm not sure the answer to that, and I'm not going to spend time on it today. Stay tuned if you want to get straight in to the nitty gritty details of how to add a rss feed to any site built using golang.

What is RSS?

RSS is a way to subscribe to a website, and get updates when new content is published. It's a simple format, and a simple protocol. The protocol is based on http, and exposes itself via a url with an XML format.

How to tell the web about your RSS feed

The first step is to tell the world about your RSS feed. This is done by adding a link tag to the head of your html page. This is a simple html tag that looks like this:

<link rel="alternate" type="application/rss+xml" title="RSS Feed" href="/rss.xml" />

How to generate the RSS feed

Here's what we are trying to generate, a simple XML document that looks like the following:

<rss version="2.0">
    <channel>
        <title>Hacker News</title>
        <link>https://news.ycombinator.com/</link>
        <description>Links for the intellectually curious, ranked by readers.</description>
        <item>
            <title>Making a PDF that&#x27;s larger than Germany</title>
            <link>https://alexwlchan.net/2024/big-pdf/</link>
            <pubDate>Wed, 31 Jan 2024 22:47:47 +0000</pubDate>
            <comments>https://news.ycombinator.com/item?id=39210507</comments>
            <description><![CDATA[<a href="https://news.ycombinator.com/item?id=39210507">Comments</a>]]></description>
        </item>

The key parts that should be standing out are title, link, description and pubDate. These are what we will use to build up our own feed.

Building all that XML by hand is a pain, so we are going to do this the easy way. Behold xml -> golang generation. This online tool is
the easy way to build up your type structs, simply grab that whole document and paste it in to the tool. It will generate the golang structs for you automagically.

The generator will split out a big nested type. The code will contain all the necessary annotations and incantations to make your rss feed, we simply need to fill in the blanks with our own feeds parameters.

rs := Rss {
 //... a whole bunch of fields here
}
err := xml.NewEncoder(w).Encode(rs)

Actual encoding can then be done using the familiar and easy to use Encoder interface.

A gotcha with dates

There is one catch with the generated type, it expects the pubDate to be a string, we need to serialize a date string for this field. This is all well and fine, you probably assume that a good all ISO8601 date will work.
You would be wrong to think this. It's closer to RFC822. So you might think the built in RFC822 formatter from Go's great standard library. Well, almost, but not quite! The RSS feed is particularly janky, and requires us to use a 4 digit year.

To save you a bunch of time just use "02 Jan 2006 15:04 MST". Oh, and make sure to include the timezone it was posted in. This is a requirement of the RSS spec. Check out the below example

    gmtTimeLoc := time.FixedZone("GMT", 0)

    for _, param := range fp.Posts {
        itm = append(itm, Item{
            Title:       param.Title,
            Link:        "https://" + param.URL,
            Description: param.Description,
            PubDate:     param.PubDate.In(gmtTimeLoc).Format("02 Jan 2006 15:04 MST"),
            GUID: struct {
                Text        string `xml:",chardata"`
                IsPermaLink string `xml:"isPermaLink,attr"`
            }{
                Text:        "https://" + param.URL,
                IsPermaLink: "true",
            },
        })
    }

The above is an example of how I mush my regular Post type into the rss type. As you can see I'm making the date field be in the gmtTimeLocation, and using the expected time format string.

How to make the rss endpoint

The final step is to make the endpoint that will serve the rss feed. This is a simple http.Handler that will serve the rss feed.

func rssHandler(w http.ResponseWriter, r *http.Request) {
  //... generate the rss feed, posts, fill out the Rss type.
  w.Header().Set("Content-Type", "application/rss+xml") // always include this
  w.WriteHeader(200) // always include this
  //... write the rss feed to the response
  err := xml.NewEncoder(w).Encode(rs)
  if err != nil {
      http.Error(w, err.Error(), 500)
  }
}

As you can see, it's all straight forward and familiar. That's one of the gorgeous things about RSS feeds and the go language. Simple, easy to understand and easy to implement. A great evening project to make your blog just a tiny bit more awesome.
The important part in the above example is to have a Content-Type header. After all this

Wrapping up

The RSS feed is a cool little evening project I implemented to make this tiny blog platform more awesome. Golang as a language makes it easy to do all this with just the standard library. There's a few gotcha's but nothing too crazy. I hope you enjoyed this little write up, and I hope you enjoy the RSS feed. If you're reading this from a RSS feed, cool! Didn't know people actually still used these things!

Screens First, Data Model Second

Ersin Buckley — Wed, 24 Jan 2024 04:59:41 +0000

Read a great little post from a dev reflecting on what they have learned over the last 30 years building systems. They say.

Doing the screens first and persistence last is a common top-down approach and it is a very bad mistake

I respectfully disagree. But we also shouldn't just do screens first and not consider the persistence (modelling) of the data. Let's talk about the problem and a better way to build high quality software that lasts.

The big problem

Like many things in life, the sure thing is change and death. Software is no different. Projects start off at a great pace, with enthusiastic engineers and users, but over time they accrue cruft. Before you realize it, the project is a nightmare to change. Velocity crawls to a halt. People leave, new engineers take forever to learn the software. It's a 'giant ball of mud'.

Is 'screen first' or 'data model first' going to get you to your inevitable demise faster? I think this is the wrong question! Taking an absolute view is absolutely the wrong way, and will lead you to many WTFs on the project.

A 'data model first' approach wil harden and become brittle to new and unexpected ways of querying the data. It will accrue cruft related to features that are never needed. In my experience I've seen teams sink big amounts of time in to work that is never needed, which just clutters the way forward and makes the software harder to work with.

'Screens first' as a mandate is no better. Projects that do this end up in what I call 'firebase hell'. There is often no data model, confusing denormalisation. Sometimes, if you are in a special hellish place, you might end up with a nosql model when a relational model is desperately needed. This is an avoidable nightmare if you consider the data model from day one.

Why do we take these hard positions on one side or the other? Is it a skill issue? I say no, I think it's natural -- it happens, don't beat yourself up reader, we all make mistakes. How do we fix it?

The perfect team

We need the three musketeers of software development. Product, backend and design. We need to make sure the solution actually solves the problem by (product). A designer to make the screens to solve the problem, and a data modeller to give us a model that scales beyond the life of this one application. Including this diverse set of mindsets is critical. It allows a team to succeed in the short term, while laying a foundation that can be built on in to the future.

If you're working on your own, it's important to wear all the hats. Don't worry about being balanced or perfect at all of these, it's ok to be specialized in one area. Just respect that these points of view are of equal importance!

The designer is going to make screens which you can put in front of customers, validating that they solve the problem. Do not worry about making pixel perfect figma designs. My favorite prototype tool is a screenshot or simple powerpoint. Use what works best and fastest, some developers I know like to do this with plain html/css if that is there jam. There are interesting AI tools that might even work in this space such as v0.dev. The important part about this is to ask the right questions when you are talking to the customer. Ask "What is your biggest problem with the existing solution?" "Show me how you solve the problem now?". Listen very hard to their answers, build many iterations of the screens.

A good data modeller is going to be able to sit in on these interviews with customers, and their mind will spin with the ways you could persist and model this data. A graph, a document store or a relational database? The modeller will ask questions to tease out the best data model, validating or invalidating the mental model.

In an ideal world, this happy trio of people (or alternate personalities), is going to be able to quickly iterate and test ideas and models of the system. I can't stress more how important it is to have everyone along for the design journey.

Wrapping this all up

So if you've made it this far, I hope you are convinced that the absolute position of 'screens first' or 'data model first' comes from a place of pain, and will end in pain. It's a natural reaction to a bad experience. Instead, try and think about the problem you are solving, and the people you are solving it for. Try and get a good team together (or adopt multiple personalities), and iterate on the design and data model. You will be rewarded with a better product, and a better experience building the software.

Creating the Local First Stack

Ersin Buckley — Mon, 18 Dec 2023 19:18:59 +0000

This week I've been busy leading up to the holiday season working on my CRDT (conflict free replicated data type) project. My goal is to test out some of the assumptions I made about building a really effective stack for offline and local first software. Stay tuned to read more about how it went on my journey to implement a sync-server for CRSQLite using Go, and a frontend using React + wasm SQLite. If you want to know a bit more about why I'm doing this, check out last weeks article Local First Software.

The foundational technology I want to build on for this project is CR-Sqlite. You can think of it as a Git layer on top of SQLite. What it allows you to do is create a normal relational schema, then mark tables as replicated, you will then able to track the changes to these tables through a simple query. On the other side of the synchronization picture, you can handle merges by inserting changes in to the database and the CR-Sqlite plugin will handle updating the right values in the database all in the background.

Keeping two databases in sync

The greatest thing about CRSQLite is that it just works on top of an existing database. We start with a schema like this:

create table foo (a primary key not null, b);
create table baz (a primary key not null, b, c, d);
select crsql_as_crr('foo');
select crsql_as_crr('baz');

Then we can get a stream of update events made to the database with a query like this!


result, err := conn.Query("SELECT * FROM crsql_changes WHERE db_version > ? ", currentVersion)

The special relation crsql_changes contains the column, row and database version for that column and row. Using this data we can 'materialize' a view of the table at a certain version. Check out the implementation, you can simply query the database to get the list of changes needed to synchronize two databases

To merge changes, simply INSERT into crsql_changes and your rows/columns will seamlessly be updated. This is really cool because it's just plain old SQL, and it's dead simple to interact with from Go, using the database/sql package. You can check out the implementation at the time of writing here.

See this all working together in the "dumb version" implemented in this test. The test creates two databases in memory, inserts values into one and merges changes between each other to validate that the two databases end up in the same state. After implementing this I'm feeling confident that the method will work, and I have everything needed for the back-end server to implement merging of database states.

Lets move on.

Implementing the server

How shall we merge the []Change values in a meaningful way over the network? My test just passes them through memory and that's not really realistic for our desired application. The requirement I have is to support many web clients and a simple server in the background for caching and orchestrating the web clients. The server is just like a client, except it's always online and available for peers to use as a source for synchronizing data.

We can solve this with a service! Now there are many ways I could have started, but I decided to test out gRPC along the way. This was a mistake. I hoped for the best, but gRPC ended up not being a good choice for the web client. Why? you ask. The gRPC protocol works with all the bells and whistles of http when used server to server, but web clients are not as great. The Javascript client is dependent on http 2.0, and it requires a proxy like Envoy to work with a browser. What's more, I didn't love the structure of the generated web client. So through the process of working on this 'local first stack' I actually got sucked in to a big rabbit hole in making the rpc system work. I ended up going with Connect which is a tool that can create a service from a protobuf service definition, that also talks a simple http 1.1 protocol. What ultimately sold me on this solution as the best is that it also came with a very nice to use web client generation, and even plugs in to my favorite react http helper useQuery.

So with the backend all done and implemented, I have a simplified service definition here, an implementation of the server in go and a client implemented in typescript.

Implementing the frontend

The coolest thing in this whole stack is that we have the same exact database on the frontend as the backend. A build of the whole SQLite project, including the CR-Sqlite plugin. This is cutting edge stuff, let's go dive in and see how it works.

The database is this really high performance low level piece of systems programming, how the heck is it running on the web?

Great question, dear reader! Despite the database being written in C and Rust, we can use this from javascript using a simple database oriented API, in the exact same way as we interacted with the db in our backend.

The technology that unlocks this is Web Assembly (WASM). A very low level language that can run in the browser, and can expose API's to your plain old javascript. This is something that comes in the browser, and more and more it is becoming widely used on the web. Compiling the C and rust code with LLVM enables the CR-SQLite project to run on the web. I can use all this really advanced and deeply complicated stuff through simple npm imports. Got to love the 2023 way of plugging in thousands of lines of advanced database with a few lines of bun install :D

Wrapping it all up

Now the frontend is far from perfect so far, there are in-fact critical bugs with the synchronization work, but I have the bones of the project setup -- and over the next weeks I plan to continue refining and improving on it so that I can test my theory about Local First software being a simpler, faster and better way to build software. It's amazing how much work I can leverage already in the CRDT space, thanks to the way the web as a platform has evolved and the focus and efforts of people to develop great open source tools for RPC, user interface and databses.

And one last note, if you've made it this far. Thanks for reading! It would be awesome to hear from you, so feel free to reach out on the twitter, or the comments, help me push this project (writing) a bit further. My goal has been to write once a week, and I'm hoping that the process is one that will help me hone my craft.

Offline eventually consistent synchronization using CRDTS

Ersin Buckley — Sun, 10 Dec 2023 03:02:01 +0000

The 2023 way is to lock your data in a walled garden, keeping it safe and secure in someone else's computer (a server). You put trust in the service and hope for the best. Hope for no leaks, hacks, security breaches or rogue employees. When the service ends hopefully there are backups available and a way to extract your own data. This pattern is one most of you are familiar with. It's the way web 2.0 works, and by 2023 it's old hat!

Today; we talk about a niche suite of algorithms that enables a better way, Conflict Free Replicated Data Types (CRDTs) enable an eventual data sync data between peers. This can be, servers/browsers/phones or anything in between. It's a cool alternative to locking data behind a server, and it makes some hard features easy

Users always having a copy of their own data
Easy support for offline mode
Simplified architecture

The local-first software movement aims to put that control back in users' hands. Groups like Ink & Switch Research have been doing some fascinating work in this space. Their insights into eventually consistent, yet resilient systems inspired me to embark on my own learning journey with CRDTs.

In this series, I plan to document my progress building a small prototype application using conflict-free replicated data types. CRDTs provide a neat way to replicate application state across devices without worrying about sync conflicts. The goal is to see these patterns in action by implementing database synchronization between server, desktop, mobile, and browser-based peers of the app.

What are CRDTs?

So what exactly are conflict-free replicated data types? They are specially designed data structures that allow for eventual consistency of application state. What I mean by eventual consistency is that nodes can update their local copy of state independently and immediately then propagate changes to other nodes asynchronously, yet all nodes eventually converge to the same global state due to a mathematically-guaranteed merge procedure.

This magic lies in the CRDTs' commutative, associative, and idempotent merge function.

Basically, no matter what order changes or updates arrive across nodes, applying them results in the same output state.

The most simple example of this is a Set. If multiple nodes perform an 'Add' operation, the same eventual state is reached. When you need to 'merge' state you simply do a 'Union' operation of each state and a final shared state is reached. The fantastic part is it doesn't matter what order of events happen on each of the peers, it can seamlesly be merged together at some point of time in the future. Pretty nifty! This is the key property that eliminates update conflicts even across disconnected and independent replicas.

By removing the need for coordinating events, you get this awesome property of local first editing. Immediate feedback can be given to the user, and whenever a sync is possible it can happen in the background

The prototype

Theory is great, but how can we apply this in practice? Instead of starting from 0, and writing a CRDT, let's try and leverage an existing project to do the heavy lifting. My choice is crSQLITE, an extension for SQLite to support CRDT merging of databases. Under the hood, the extension creates tables to track changes and allow inserting into an event log for merging states of separated peers.

What's great about choosing SQLite as a platform, is it is a ubiquitous development environment. I can get this built for every platform, and all my favorite libraries are setup to work with SQL already.

Now, I wish I could share with you the skeleton of requirements for a big amazing project, but I'm just not quite there yet. Ideas I've been thinking about include:

An extension to my markdown editor markdown editor in tailwind react and wails
Simple diagram tool
Time tracker
????

No matter what, my goal with this is to test out this CRSqlite thing, and build something desktop based using wails, which can be run on my desktop, laptop(s), phone and anything with a web browser. So without further ado, I'm introducing the newfangled cr-sqlite-go, a simple library that wraps the CRSQLite extension and provides a simple API for interacting with the database.

My vision is to eventually build an application that seamlessly synchronizes across peers, with offline support and including as many platforms as possible. Hopefully along the way we can learn some things about CRDT, and give my stack choice a good workout.

Should I be hacking on the weekend?

Ersin Buckley — Fri, 01 Dec 2023 02:10:19 +0000

Balancing professional growth and personal time is a constant challenge in our industry. Inspired by the question in episode 210 of the soft skills engineering podcast,I am reflecting on my own approach to hiring and interviewing for jobs and how we should think about side projects and the github profile.

Before I got my job as a web developer, I was studying 2 hours per night. Now that I've started working (been here for two years) I want to come home and relax! How much time do you spend reading about new technology, or working on side projects? I plan on getting a new job in the future, and I feel like I need to start studying again. My github is empty since I started this job, should I be worried about that?

When you first start in the industry, you will be going from this time where you probably spent every waking hour looking to make yourself ready for work by doing every leetcode/sideproject that will make you look like a good candidate. This changes almost immediately as you sink in to a 40 hour week of actually doing the work. It's completely natural to not want to code at home. This is healthy and good. Some of the best engineers I have worked with haven't written a single line of code at home. You learn on the job, and you get to grow with every new problem you solve.

An empty Github Profile might feel confronting when it comes to moving from work mode to searching mode, but reality is, it doesn't matter! A delicious github graph filled with green squares might be a great dopamine hit for you, but it's not going to be the deciding factor for hiring someone who already has industry experience.

Now, for those of you who know me, you know I'm a huge fan of side projects. That is like 50% of the purpose of this blog is to share from my own experiences from coding for fun. But even I haven't always felt the need to hardcore code all day and all night. When I've had really engaging or demanding professional work, I'm more likely to go outside and touch grass, or spend time with my family. I think that is a healthy balance. There are times when it's the opposite though, if my day job isn't involving much code, and more in the analysis space I will find time to pique curiosity with side-code.

So on to the next part; you're looking for that second job? Well, the github profile isn't going to be a factor (sorry). A full profile may be pivotal in your first role, but in my experience it plays a much smaller part in each subsequent role. What's more important is that you can talk about the work you have done professionally. Be very precise in telling the story of your work; Describe the problem, explain the context and then dive in to how you use your tools to solve it. This is the kind of thing that will get you hired.

When does it make sense to pick up side projects ahead of a job search?

This might make sense if you are looking at picking up a new technology. If you can find a way to branch out within your current role, do it. Experience with real world projects is 1000x more valuable for lateral shifts. Finally, if you can't shift your current role to the new stack/skill, then totally put your time into a nice real world project simulation.

Focus on distilling a project to just the parts you might need in interviewing in the new role, and aggressively cut scope on anything you already have experience in within your current role. This is the kind of thing that will optimize your time, and get the most pay-off for shifting to a new skillset.

In my own experience, side projects have infrequently come up as a topic of discussion. What does come up more frequently is the knowledge gained from working on the project. Being able to compare my experience on a project to the role I'm interviewing for has been helpful in giving context about who I am as a technical contributor, and how I might approach the trade-offs from technology decisions.

Navigating this balance between professional growth and personal growth is something that will always be a tension. There is no obvious one-size-fits-all approach. Whether you hack on the weekends or not, prioritize a balanced approach. Try not to get caught up on expectations for your side projects to mean anything in the interview. Be real with yourself about what you are working on achieving here. Don't Fixate on the Github contributions graph. Finding a healthy work-life balance is the most important thing of all, and the best way to do that is to listen to your gut, and don't take things too seriously.

Thanks for reading! I'm eager to hear your own experience with side projects and job seeking :)

Should I learn golang in 2023

Ersin Buckley — Sun, 26 Nov 2023 01:16:44 +0000

The year is 2013, you've just landed your first real engineering job and you get to churn out angular.js and PHP. What is even better, you are paid a handsome junior salary to do it. In other words you are living the dream.

If I could ask one thing of my future self at the time it would be. Should I learn the Go language?

The Go language was really fresh at the time. It had come up in one or two conversations with my mentor. The hype online was starting to build, interesting tools like docker were being built on the language, and it was popping up all the time on the front page of Hacker News. What's more, it was supported by google.

New Zealand had very few opportunities to use Go professionally. We tend to be microsoft centric down under. On the face of it, Golang doesn't look like the best tool to pick up next. If job optionality was most important, one would be better off learning C# or Java.

Despite all this, you should learn Go first young Ersin. Start building side projects, don't fret that there are no job opportunities yet, even if it doesn't end up in paid work, you will learn along the way. You will learn a new type system, you will be exposed to the kind of problems the language is good at solving (concurrency, distributed services, systems programming). The types of blog posts and tutorials coming out centered around go will stretch your knowledge and make your more well rounded. This is the kind of thing that compounds and makes you better years down the track.

The side projects you build might seem inconsequential, but each new project you build hones that muscle. Reading code and trying new things are how you grow. When the day job becomes routine, a side project will keep you invested in your craft. The effort will pay off eventually, I promise.

If you take one point from this, always take a bet on niche and emerging technologies.

The prevailing wisdom is that you should stick with learning the most popular tools, to give you the best chance of landing jobs. I think doing just the safe bet is a mistake. We should adopt a barbell strategy in technology. Build a solid base with safe choices to earn your way in to the industry -- then put time into many niche, emerging and wildly interesting technologies. This gives you the security of stable work, while broadening your horizons in a direction that brings you joy. Eventually, an opportunity will come up in one of these niches.

It is an amazing thing to be ahead of the curve on an emerging niche technology. Projects are more likely to be green-field. The people who are drawn to organizations that use them are typically more engaged in the work, and you will find a higher degree of alignment in the team. People aren't just there for the money or because they want to get a job at all costs. Not to mention if that tool becomes wildly successful, you will be an outlier expert of which no one can catch up

What about in 2023?

So a decade later, If I was asked the same question "Should I learn go?"; I would say Yea! but the reasons are different now. Although it's still a niche technology in New Zealand, there is always at least a couple jobs on the board. You could bet your career on Go as the 'safe' option.

After that first job, you might be better off focusing your efforts on a more radical technology. Instead of Go, branch out and pick up another emerging niche technology. You never know where it will take you, so find what piques your interest and dive in head first! It's hard to picture where you will be in ten years time, so best not get overly focused on the outcome but instead focus on the journey.

How to build a desktop markdown editor

Ersin Buckley — Thu, 16 Nov 2023 20:22:44 +0000

Building a Markdown Editor with Wails, React and Tailwind

Are you tired of dealing with the complexities and massive builds associated with Electron for desktop applications? Do traditional frameworks like QT, GTK, or Win32 make you want to cry?
Today we are going to build a markdown editor using Wails, React and Tailwind.

Simplifying Desktop App Development

Building a desktop application has been an ugly duck since the dawn of web 2.0. The situation has been even more dire for golang enjoyers.
You are forced into a few options, mostly wrapping existing UI libraries, and non of which really bring the productivity and joy of writing backend
services using golang. The most popular option, Electron, is a huge stack of javascript, and while it's easy to get started, it's bloated and you are forced
to write so much javascript that it's going to make you pull your hair out.

Let's delve into a better way, by creating a markdown editor using Wails, React and Tailwind.

Our basic requirements for this app will be:

    preview markdown text
    edit markdown text
    create a file
    save a file
    export a file (as pdf)

Step 1: Bootstrap!

Wails follows a Rails-like philosophy, providing a batteries are included and opinions come thick and fast. The opinion it does not have, is what html framework you use for the view layer. To make things really easy to get started, it makes it straight forward to kick start from templates.

Let's get started from this awesome pre-made vite+react+ts+tailwind template.

wails init -n write -t https://github.com/hotafrika/wails-vite-react-ts-tailwind-template
cd write
wails dev

Queue waiting forever to let npm install + build for the first time. This is a one time thing, and it's worth the wait, I promise!

Take a second now to get oriented, the structure is like so, and every wails app will look the same:

- build (contains built binaries, installers, etc)
- frontend (contains the frontend code)
- wailsjs (contains the go code)
- main.go (the entrypoint for the go code)
- wails.json (the wails configuration file)
- app.go (the main wails application file)

Let's start with the design: here is what I used, thanks to v0.dev for saving me all the time! This is pretty much a copy/paste job of the html into app.tsx. I chose not to import the v0 library -- I'm just not convinced it's worth it for such a simple design.

Step 2: Render and update markdown

Wails provides a server/client model, seamlessly wiring the Golang Backend to the Javascript frontend. In our markdown editor component you will see some simple snippets like this:

// at the start of the file
import {Save, SendMarkdownToRenderer, Open, Export} from "../wailsjs/go/main/App";


// inside the Component

 const updateValue = async (event: React.ChangeEvent<HTMLTextAreaElement>) => {

   const value = event.target.value

   const html = await SendMarkdownToRenderer(value)

   setMd(html);

 }

// many lines skipped ...

<textarea

           onChange={updateValue}

           ref={ref}

           className="flex min-h-[80px] rounded-md border border-input px-3 py-2 text-sm ring-offset-background placeholder:text-muted-foreground focus-visible:outline-none focus-visible:ring-2 focus-visible:ring-ring focus-visible:ring-offset-2 disabled:cursor-not-allowed disabled:opacity-50 w-full h-full bg-white dark:bg-gray-700"

           id="markdown-input" placeholder="Type your markdown here..."></textarea></div>

As you can see, we have a really simple textarea with an onChange function. When the textarea changes, an event is fired, the function gets called and we execute SendMarkdownToRenderer. The function is a code generated client that maps to Go code. Let's check that out.

https://github.com/ebuckley/write/blob/335bc096973f3b5a368e0257c1ffcd053a6c513c/write/app.go#L55C1-L60C1


func (a *App) SendMarkdownToRenderer(content string) string {

   a.markdown = content

   a.renderedHTML = lib.RenderMD(content)

   return a.renderedHTML

}

This fantastically simply little function is how we persist state to the 'server' and go context, and then pass back a string value to the client for rendering via the setMD state setter.

Important: there is a lot of cool stuff going on here. First of all, we have this really awesome code generation that just automatically happens as you save your files. A go function gets seamlessly and magically turned into something callable from a react application. This kind of magic RPC is really effective for a UI application where the server is close to the client, and it's just quite delightful for working with.

A fun digression: If you squinted at this you could look at the recent tech twitter memes about 'use server' and call this 'use golang'.

Step 3: New file, Save File, Open file

To implement this trio of features, we are going to use the same exact pattern using onClick handlers for buttons. Again, we leverage the codegen that wails provides to automagically wire up. Let's dive into the Save function in app.go:


func (a *App) Save() (string, error) {

   hd, err := homedir.Dir()

   if err != nil {

      return "", err

   }


   chosenPath, err := runtime.SaveFileDialog(a.ctx, runtime.SaveDialogOptions{

      DefaultDirectory:          path.Join(hd, "Documents"),

      DefaultFilename:           a.filename,

      Title:                     "Save file",

      ShowHiddenFiles:           false,

      CanCreateDirectories:      true,

      TreatPackagesAsDirectories: true,

   })

   if err != nil {

      return "", err

   }

   err = os.WriteFile(chosenPath, []byte(a.markdown), 0644)

   if err != nil {

      return "", err

   }

   return chosenPath, nil

}

The framework provides a runtime package which allows calling a few functions which provide common cross-platform utility. Here, we call SaveFileDialog to present the user with a native and familiar way of choosing a filename and location to put the markdown. The function returns a path chosenPath and we use that in a straight up WriteFile call.

Our state a.markdown is kept in sync with the frontend via the previous onChange handler from step 1.

Travel warning: The SaveFileDialog takes a context, it's really important that this is a runtime context, the runtime context is stored on the app struct, and you need to always remember to pass this so that all the magic works! This planned to change in wails v3, but that is still in development at the time of writing this.

challenge

Take a look and see if you can find out how the New File and Open File work. Start out in app.tsx and trace your way back to app.go!

Step 4: Yak shaving markdown export

Ok this turned out to be a H U G E timewaste, but it was a lot of fun. As you might be aware, there are a lot of techniques for exporting PDF files, I decided to go for the native and non browser based pure pdf library fpdf. Thankfully, our markdown -> html compiler also supports extending via a renderer function and walking the parsed AST. I had a great time adapting a scuffed and minimalist little pdf renderer. You can check it out here.

For the wails part, it's again very simply an onclick handler for a button alongside an exported method on App.go We can simply take that markdown content and put it through our renderer, passing a path to save as a result from the SavefileDialog

Warning: don't use this pdf renderer code, it crashes on codeblocks! It's just an example approach which I think could be extended quite nicely.

Step 5: CI/CD

Eat your vegetables, make your code hygienic and never leave a bad build on origin main.

I went to the effort of setting up CI/CD for this project to show how you can cross compile for windows and linux. To do this I setup a github action which tests/builds on push and releases when you tag the repo with something matching v*.

Yes. You are seeing this correctly, it took me 16 builds to get a release. Anyway... So that you don't need to waste as much time, check out my deploy action here.

The interesting parts are:

You need to install a few dependencies sudo apt install libgtk-3-dev libwebkit2gtk-4.0-dev nsis
Don't forget to get the paths right
Don't forget to use the permissions block in your action to allow it to cut new releases
Copy the binary out of the build directory between builds, because build/bin will be overwritten on builds

As a result, I think we have a solid base for a project which can be extended, bug smashed and it all gets automagically released through github actions. A nice place to stop for this little exercise.

Wrapping it all up with a disruptive and transformative conclusion

Wails is a gamechanger for GUI applications written in Golang -- simple, productive and downright magical.
You can think of it as a 'web style' MVC for your frontend, or 'use golang' for desktop applications. What sets it apart? Bring any frontend view layer! HTML to Svelte, it all works. Binaries are tiny 10Mb instead of electrons 100+Mb. As far as Golang desktop frameworks go, I think Wails is undisputed as our best option.

You can check out the code I wrote for this project here, and get started with wails here.

a rusty code cracker

Ersin Buckley — Fri, 10 Nov 2023 18:50:27 +0000

Are you looking for just the challenge with no spoilers? Stop now, go back to last weeks post!

Still here? Great! Let's dive in to the solution.
Why did I do this in rust? Well.. it's not because I had the need for extreme memory safety. I didn't need the zero cost abstractions, but i guess they are nice.
The thick and well supplied third party ecosystem wasn't needed. Didn't need the safe concurrency.

I chose rust because I'm interested in the language, and I wanted to pick it up again.

Recently, I went along to the local rust meetup in Christchurch, and I enjoyed the vibe. People are interested in talking allocations, safe handling of errors, and
working with systems that require some high performance.

Writing systems that need to go fast is really fun. You get a sweet dopamine hit from making memory go down and cpu flame graphs chill out.

My hope is that rust is a tool that helps me stretch that urge, and at some point in the future I would very much like to apply it to a professional problem.

So let's dive in to the code. I'm going to skip over the boring bits, and focus on the interesting parts.

Part one, finding possible words:

Let's start by building our dictionary of possible words. I'm depending on the built in dictionary file in linux, on my machine I chose /usr/share/dict/american-english.

fn get_dictionary(dictionary_path: &str) -> Vec<String> {
    let mut words = Vec::new();
    for line in read_to_string(dictionary_path).unwrap().lines() {
        // if it contains ' then strip it out and save it...
        let mut thisline = line.to_string();
        if line.contains("'") {
            thisline = thisline.replace("'", "").to_string();
        }
        words.push(thisline.to_uppercase());
    }
    words
}

As you can see, I'm reading the file in to a string, and then iterating over the lines. I'm also stripping out the ' character, because it is not a valid character in our code cracker alphabet.

Next we need to represent our hint and substitute alphabet as a type. As what you might find after years and years of programming, most problems can be solved with simple vectors and hashmaps. So let's represent our substitution alphabet with an array, and our hint with a vec.


// a hint is a Vec<i32> where each item in the hint represents an index into our substituion alphabet

struct Problem {
    code: [Option<char>; 26],
    hints: Vec<Vec<i32>>
}


fn find_options(hint: &Vec<i32>, code:[Option<char>; 26], words: &Vec<String>) -> Vec<String> {
    let word_length = hint.len();
    let mut options = Vec::new();
    'outer: for word in words.iter() {
        if word.len() != word_length {
            continue
        }
        // this might be it!
        options.push(word.clone())
    }
    options
}

In this super simple first cut, we just iterate through all the possible words, and check if the length of the word matchces the hint, if it does, push it on to the options vector.

Part two, optimizing options:

We are going to extend find_options so that we can select just the words that make sense. To do this we need a function to fill in the known parts of our hint word with the actual char from the substitution alphabet.

fn to_letter_map(hint: &Vec<i32>, code: [Option<char>; 26]) -> HashMap<usize, char> {
    let mut out = HashMap::new();
    for n in 0..hint.len() {
        let elem = hint[n];
        let codeindex = (elem - 1) as usize;
        let opt = code[codeindex];
        match opt {
            Some(v) => {
                out.insert(n, v);
            }
            None => {
                continue
            }
        }
    }
    out
}

This one builds up a HashMap that maps the substitution alphabet value to it's actual character, so we can easily lookup what letters to search for.

Now we have what we need to breakdown words with letters that are already known, lets attempt to filter the word list even more by checking that they match the 'pattern' laid out by the hint.
So for our hint 1 7 7 R 7 21 we would expect the word to have the same letter for the 2nd, 3rd and 5th letters. We implement a check_word function to do so.

fn check_word(hint: &Vec<i32>, code:[Option<char>; 26], word: &String) -> bool {
    assert_eq!(hint.len() , word.len(), "should be the same length");

    let mut code = code.clone();

    for (i, c) in hint.iter().enumerate() {
        let current_word_char = word.chars().nth(i).unwrap();
        let id = (c-1)as usize;
        let matches_code = code[id];
        match matches_code {
            Some(code_value) => {
                // if the current value exists as a code but doesn't match the current word
                if current_word_char != code_value {
                    return false
                }

            },
            None => {
                code[id] = Some(current_word_char);
            }
        }
    }
    return true
}

Of note in this function, we use a 'clone()' on our substitution alphabet (code). This is important because we want to not mutate the original one, and this function will be called many times as we check each candidate word.

Now we can extend the find_options function, putting together our word_map and check_word functions into the loop.


fn find_options(hint: &Vec<i32>, code:[Option<char>; 26], words: &Vec<String>) -> Vec<String> {
    let word_length = hint.len();
    // word_vec is a map of index of hint -> char, built from referencing the code (substitution alphabet)
    let word_vec = to_letter_map(hint, code);
    let mut options = Vec::new();
    'outer: for word in words.iter() {
        if word.len() != word_length {
            continue
        }
        // NEW: in this loop we check that the candidate word matches the letters already known from the hint.
        for (index, word_char) in word_vec.iter() {
            if word.chars().nth(*index).is_some_and(|c| c != *word_char){
                continue 'outer // if it's not then bail to the outer loop `outer
            }
        }

        let matches_pattern = check_word(hint, code, word);
        if !matches_pattern {
            continue 'outer
        }
        // this might be it!
        options.push(word.clone())
    }
    options
}

This is how far I got with the challenge! you can check out the code on github.

I'm really new to rust, and this was a fun one to flex my Rustacean skills. Looking forward to putting new skills to use in real projects some time in the near future :)

Code Cracker challenge

Ersin Buckley — Thu, 02 Nov 2023 09:23:45 +0000

This code cracker challenge is a great exercise to flex your programming muscles, especially in something new (like I did with rust). Let's break it down:

Challenge 1: Finding Possible Words
Create a function that takes input text and outputs the substitution alphabet and each clue. For example:

example input

14 R
1 7 7 14 7 21
14 7 3 7 14 1 7

example output

Alphabet:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26
_ _ _ _ _ _ _ _ _  _  _  _  _  R  _  _  _  _  _  _  _  _  _  _  _  _

Clue 1:
1 7 7 R 7 21

Now, search through a dictionary (e.g., /usr/dict/words) to find words matching the clue's length.

Challenge 2: Optimizing Options
Print out all possible words that match the pattern. For 1 7 7 R 7 21, the output would be:

1 7 7 R 7 21

0: DEERES
1: JEERED
2: LEERED
3: PEERED
4: VEERED

Challenge 3: Play the Game
Allow the player to fill in new letters in the substitution alphabet to play the next word. This could be a handy hint buddy for cracker enthusiasts.

Challenge 4: Solving a Full Game
Create the ability to input an entire game, letting the program figure out a solution.

This challenge is not only a great exercise but also a fantastic way to learn a new programming language. Happy coding!

Looking for some more samples, check out the original authors work! codecracker.co.nz

As for my own solution? I'll post it soon, promise!

The 10 year blog platform

Ersin Buckley — Sun, 29 Oct 2023 19:29:58 +0000

TLDR: I migrated all my posts to a shiny new, handrolled personal blog at www.ersin.nz

Ok I'm kidding. I may have been writing code for cash for ten years now, but this tiny platform has been the product of two weeks casually plugging away in my editor, in between looking after my 5 month old daughter and working full time on starboard.nz.

This blog exists for three reasons. I wanted to learn something new, liberate my data from the centralized blogging platforms, and experiment with SEO.

My goal is to build a platform that can last for 100 years.

So Buckle up, we're going deep.

Yea I know it's not going to be here for even a half century. I'm setting my sites long term, and I'll be happy if it lasts a quarter of that. Shoot for the moon, hit the outer atmosphere. Big goals are important, no?

My rationale is that making my blog tech so simple, so boring will let me finish and move on to the next thing. The biggest impediment to building a 100 year blog, is gaining the inertia to keep writing, and build an audience that can pull you towards continuing in your writing.

You are probably asking; Why not post on a major platform Ersin? Major platforms suck, they don't help me with distribution or gaining an audience, and they don't let me mess around with some code and learn something new! At least through this process I am getting to build and I'm learning something new at the same time.

Don't get me wrong, if you're already famous you probably should use a platform for posting your content. But I'm not there yet. I'm still building.

If you're not interested in running a $5 vm, you definitely shouldn't be doing this. If your life isn't spent in git, then you should run now. Stop reading! Go away!

Great, now the noobs have been filtered out, let's continue.

This blog has been back-filled with some things I have written over on dev.to. Not only is it important to bring my old stuff in to the future, it's important to keep my content rolling. When you are working on software, having real data is also amazing and gives you the best possible experience for design. Let's not spend our days looking at lorem ipsum boilerplate eh?

My old blogging platform helpfully provided a tool to export your data (thanks gdpr), and this comes in the form of a simple JSON structure, this let me build a little tool which converts old blogs to new blogs.

So what does a new blog look like? Glad you asked!

Blog posts are in hashicorp config language
The content is markdown
They can sit in a file under a folder and will automagically be loaded and embedded in to the binary.

page "render-massive-lists-on-the-web-2h3c-temp-slug-7179880" {
    title = "Render massive lists on the web"
    description = "Rendering a huge list is almost never the right solution, but this post is here to help you out if it..."
    keywords = ["javascript"]
    content = <<EOT
Rendering a huge list is almost never the right solution, but this post is here to help you out if it is, and you want to do it as fast as possible.
EOT
  posted = "2023/04/05"
}

I am aiming to avoid needing a database for this application. All the content will be in the git repo, and the fact that it uses HCL is more of a quirk in that I wanted to pique my interest of the language. In 2023 I have picked up terraform for the first time, and it's just so much better than writing YAML. I want to spend more time writing code than HCL, and hopefully the least amount of time writing YAML.

Let's be real, no one really likes the look of tailwind when they first pick it up. But after 4 years of skepticism, I'm all in on utility first CSS. Now I'm not the best at slicing and dicing boxes on the web, but I've been centering divs since float: was the best technology we had. Utility first CSS is something that has been sitting in the background for longer than tailwind has. Tailwind is having it's moment because the old way was too rigid. We can copy a design system and easily extend it to new patterns when using tailwind, that's much much harder with the old way.

HTMX is my other pick here. Right now I'm only using it in two places. We are boosting, which should result in a marginally faster click through for you. And we are building the subscribe/contact form using our dear htmx. A tool that is gaining huge popularity now, based on a technique that has been happening for about 33 years? I'm all about it htmx is old but gold.

If you've made it this far. Thanks for hanging! What can you expect in the future? More technical content, my goal with this blog is to help make you a better programmer, and give you some ingisht in to the mind of a decade experienced software engineer. If I'm smashing goals out of the park, I will post weekly, but at a minimum I will post something once a month. For now you can expect blog posts, source code releases and the infrequent real world meetup talk.

What's next? Well, after a really energizing session at the local rust meetup, I'm hoping to pickup my learning journey there. On another note, I have countless dead projects that might be worth a deep dive. Or we could dive in to the real world of software development, and maybe I could answer a reader question.

Thanks for hanging!