DEV Community: Alex Day

Shrinking a materials database

Alex Day — Sun, 09 Aug 2026 14:08:23 +0000

Shrinking datasets is quite contextual. It depends on how the dataset is going to be used. Its the same concept as optimizing databases, based on the read/write patterns.
PACLEC and Traffic Patterns are the key determining factors.

Hosted on fly.io : https://materials-db.fly.dev/

The need for this was not to integrate with another external API for retrieving information on different materials and not think about rate-limiting. This was made as a part of a bigger system: Research Agent for Researchers

The materials trajectory database was around 12Gigs and is mostly used for training models like CHGNet.
But I needed it only for responding to query on finding materials and it didn't need all the variations.

The aim was to be able to serve it using my most loved database - sqlite, in under reasonable timeframe: <500ms.

A detailed write up is provided in my blog: https://ikouchiha47.github.io/2026/08/05/materialsdb-11gb-to-266mb.html

Scheduling concurrency

Alex Day — Sun, 09 Aug 2026 13:36:13 +0000

Have been trying to look at 3 different languages: Go, Kotlin, Elixir/Erlang mostly to understand their concurrency models.

This article concerns mostly about Go's preemptive scheduling. Looking at the dense article on go: https://go.googlesource.com/proposal/+/master/design/24543-non-cooperative-preemption.md

Here is what I understand.

Before Go 1.14: Cooperative preemption at function prologues

Go used compiler-inserted cooperative preemption points in function prologues up to and including Go 1.10. This means Go could only switch between concurrently-executing goroutines at specific points - and the compiler ensured that all local GC roots were known at those safe-points, enabling precise garbage collection.

The problems this caused were real and serious. In really extreme cases, it could cause a program to halt entirely.

For example, when a goroutine spinning on an atomic load starved out the goroutine responsible for setting that atomic.

They tried to fix this by inserting preemption checks at loop back-edges (the obvious next step), but even their most efficient approach - called "fault-based preemption" - added a geomean slowdown of 7.8% on a large benchmark suite.

It also had implementation downsides: it couldn't target specific threads or goroutines, was "sticky" in that they couldn't resume any loops until all loops were resumed, and interfered with debuggers.

Go 1.14+: Signal-based non-cooperative preemption

Non-cooperative preemption switches between concurrent execution contexts without explicit preemption checks or assistance from those contexts - the same way modern operating systems switch between threads. Without this, a single poorly-behaved goroutine can wedge a Go application, much like how a single poorly-behaved application could wedge an entire OS.

The mechanism: Go implements this by sending a POSIX signal to stop a running goroutine and capture its CPU state. If a goroutine is interrupted at a point that must be GC-atomic, the runtime simply resumes the goroutine and tries again later.

Why SIGURG specifically? It meets all the criteria: it's passed through by debuggers by default, isn't used internally by libc in mixed Go/C binaries, can happen spuriously without consequences, and is extremely unlikely to be used by an application for its real meaning - since out-of-band data is basically unused, and because SIGURG doesn't report which socket has the condition, making it pretty useless for its original purpose.

SIGURG is a different beast altogether, it sits at the intersection of:

OS signals
Compiler internals
Runtime/GC Designs

It needs more reading. Will come back to that later.

Rebuilding Sofle-choc on android

Alex Day — Sun, 09 Aug 2026 13:34:28 +0000

I built a custom Android keyboard. It got weird.

A full IME - written in Kotlin, handling every keypress, suggestion, and cursor movement myself. The idea was to replicate the Sofle Choc split. Because.

Here's what actually surprised me building it:

Backspace into a committed word is a whole thing.

Once a word is committed, the suggestion engine loses track of it. Backspacing into it looks like a new character to the IME. Getting autocomplete to pick up the fragment again meant wrapping three API calls in beginBatchEdit/endBatchEdit to stop the editor firing callbacks mid-operation. AnySoftKeyboard does the same thing. Took me way too long to figure out why.

Next-word prediction is more interesting than I expected:

After every committed word, surface what probably comes next - T9 style, but without the hardware. Two layers: a static English seed (Norvig's bigram corpus, MIT licensed) and a personal layer that learns from your actual typing. The scoring uses temporal decay from librime - recent use contributes a near-full boost, something you typed 1000 commits ago contributes almost nothing. Old habits fade. New ones take over within a few hundred keystrokes.

Learned about bigrams, infigrams, suffix-array with binary search and a new shit called #beam_search from github.com/rime/librime

Mixed language (Banglish, Hinglish) works without any language detection, kinda - the personal layer is just a string map. It doesn't care what language you're in. Again, ASK showed the way for MVP.

The suggestion interface mirrors librime's Grammar pattern, context passes through the function call, not stored as shared mutable state. Took one refactor to get there but it cleaned up everything downstream.

A seven part series on the same:
https://ikouchiha47.github.io/keyboard/