DEV Community: Seiya Izumi

Curing a Transcription Worker That Kept Dying on Cloud Run with Streaming

Seiya Izumi — Tue, 16 Jun 2026 15:08:10 +0000

An audio transcription worker kept getting killed by OOM on Cloud Run.

Bumping the memory allocation calmed it down for a while, but that only inflated the bill without fixing anything. Crash, raise the limit, crash again — that was the steady state.

In the end, changing how the processing was structured improved memory, cost, transcription accuracy, and testability all at once. This post is a record of the path I took from treating symptoms to curing the root cause.

The punchline up front: what actually worked wasn't a clever algorithm. It was simply dropping "load everything into memory, then process" in favor of a streaming design — "read and forward as you go." That's it. But getting there required clearly articulating why it was dying, and identifying where the existing setup was carrying a structural impossibility.

What the System Did

It transcribed the audio from user-uploaded mp4 files by sending it to a third-party transcription Web API. The slightly awkward part: that API only accepts wav. Uploads come in as mp4, the API wants wav. The conversion had to happen somewhere.

The runtime was GCP Cloud Run, the language was Go. The architecture was an asynchronous job: an upload enqueued a task to Cloud Tasks, which dispatched a worker that ran the transcription.

The Setup I Inherited, and Its Limits

By the time I took it over, the inside of the worker worked like this:

Launch ffmpeg from the application process
Split the mp4 into 15-second wav files
Load each wav into memory and send it to the transcription API

It worked. But the worker was dying to OOM constantly.

Why? Breaking it down calmly, two memory-hungry factors were stacked on top of each other:

ffmpeg ran as a separate process, carrying its own resident memory
wav is uncompressed, so it sat in memory far heavier than the equivalent mp4, and the design held that data in memory

And Cloud Run handles multiple tasks concurrently within a single container. So when several transcription tasks piled into the same container, this heavy processing multiplied accordingly. The moment single-task peak memory × concurrent tasks exceeded the container's memory ceiling, the whole container went down to OOM. A single large mp4 could push it over on its own, and overlapping tasks made it worse. In short, it was a design guaranteed to fall over eventually.

What the Number "15" Was Telling Me

What snagged my attention while reading the code was why the split unit was 15 seconds.

Digging in, it turned out to be a compromise the previous engineer had found within the memory constraints. Make the chunks longer and each wav gets heavier, pressuring memory and increasing OOMs. Make them shorter and memory eases up, but now utterances get chopped at chunk boundaries, hurting transcription accuracy — and the request count goes up too.

15 seconds was the equilibrium point of that tug-of-war.

And here's what struck me: the number 15 wasn't a solution to the problem — it was a constraint born out of being shackled to the memory limit. Memory and accuracy, two genuinely separate concerns, were pushing against each other on a single parameter: the split length. Raise one and the other falls. The previous engineer had been carefully balancing on top of that, but if the tradeoff didn't exist in the first place, nobody would have to agonize over it.

Another Parameter, Set Carelessly

The same structure showed up in the concurrency setting. The worker's concurrency had no sign of being deliberately tuned — it was set roughly. My guess is it came from a cost/ops instinct: "I don't want to spin up a lot of instances."

But this meshed in the worst possible way. To save on instances, you set concurrency high. Multiple tasks then pile into one container, the heavy memory processing multiplies, and OOM kills the container.

The setting meant to reduce instances was, instead, killing the containers. Raise memory and the cost climbs; concurrency tugs against instance count; split length wavers between memory and accuracy. Small tradeoffs tangled everywhere, and the whole thing was stuck at an equilibrium nobody was happy with.

Don't Let the Diagnosis Stay a Guess

Before touching anything, I first confirmed that memory really was the cause. Deciding the reason by hunch wastes time on misdirected fixes.

I used Cloud Trace to repeatedly follow which segments of a request inflated resources, and how that correlated with container OOM terminations. That narrowed it down to the wav memory loading and the ffmpeg process. Once the source is pinned down, the direction of the fix follows naturally — reduce what gets loaded into memory.

The Shift in Thinking: Stream While Reading, Not Load Then Forward

The essential waste in the old setup was right here: expanding the entire file (or the entire split wav) into memory before sending it to the API. But looking closely, the transcription API actually supported streaming input.

So this became possible: read the mp4 from storage a little at a time, convert it into audio chunks on the fly, and pipe it straight to the API. There's no need to pool the whole thing in memory anywhere. Only the small chunk currently being processed sits in memory, and what has finished streaming is released.

Concretely: read the mp4 from storage incrementally with a section reader, parse the container with a stable third-party demuxer to extract audio chunks, decode those with fdk-aac, and pipe the result to the transcription API as a stream.

The crux of this design is that peak memory stops depending on file size. Ten minutes of audio or an hour, what sits in memory at once is only the chunk being processed. The peak becomes roughly constant.

What to Do with ffmpeg — Isolation, Not Elimination

Here I had a call to make. The old setup used ffmpeg for the mp4 → wav conversion. What to do with it in the new one?

I wanted to avoid launching ffmpeg as a separate process on every transcription. The reasons stacked up:

Spawning a process per task means processes proliferate with concurrency, and resident memory piles up
Process startup/teardown has latency, and handing data over stdin/stdout copies it across the process boundary
Above all, an external process's memory usage can't be controlled or observed from the application side

Even if streaming keeps the application's peak memory down, if I can't hold the reins on what an external process consumes, I can't guarantee memory by design. On Cloud Run, where the memory allocation maps directly onto the bill, being able to guarantee the peak ties directly to severing the cost-vs-stability tradeoff.

That said, I couldn't eliminate ffmpeg entirely. The mp4 files users upload vary in codec and container layout. The demuxer and fdk-aac can cleanly handle only known, standard formats. Push arbitrary input straight into the hot path and it breaks easily on edge cases.

So I split it like this: run ffmpeg exactly once at upload time, re-encoding into a form fdk-aac can handle while normalizing the container and metadata. From there on, the pipeline assumes only normalized mp4 flows through it.

In other words, ffmpeg wasn't eliminated. The heavy, potentially-unstable re-encoding work was pushed out of the repeatedly-executed transcription path and into a one-time pre-processing step at upload. No matter how many times transcription runs, no matter how many retries fire, the re-encode never runs again. The hot path only ever deals with clean, standard input, free from edge cases.

Looking back, this was the same idea as severing the memory multiplication: push heavy and unstable work out of the repeatedly-executed path into a one-time pre-process, and keep the hot path light and predictable. The same principle ran through both the memory problem and the input-handling problem.

What Paid Off

Memory and Cost

Peak memory stopped depending on file size, and memory spikes vanished in the vast majority of cases. It wasn't just that OOM went away — because the peak became constant, the container stopped dying even with a lower memory allocation, which translated straight into cost savings. The "raise memory ⇄ crash" tug-of-war itself disappeared.

Transcription Accuracy

This was a side effect I hadn't aimed for, and it was a big one. By dropping fixed-length splitting and streaming the audio as a continuous stream, the problem of utterances getting chopped at chunk boundaries disappeared by construction. The API can process with context intact, so dropped or misrecognized words straddling boundaries went away. The accuracy problem the previous engineer had been desperately balancing with the number 15 simply resolved itself as a byproduct of the design change.

The Parameters Disappeared

Both split length and concurrency had been parameters demanding tightrope tuning. With the ceiling on peak memory lowered, the three-way tradeoff of "instances I want to run ⇄ concurrency ⇄ memory safety" loosened all at once, and neither needed nervous adjustment anymore. Rather than tweaking values as a symptomatic fix, I changed the structure of the problem — so the very things needing adjustment ceased to exist.

Tests Became Writable

Modest but meaningful: testability. The old setup depended on ffmpeg as an external process, so verifying it meant actually launching that process — effectively only possible via E2E. Slow, fragile, environment-dependent.

In the new setup, both demux and fdk-aac decoding became in-process library calls, so I could verify units like "feed this input, get this chunk, get this decode result" with unit tests. In practice it was still E2E-only at the time, and expanding unit test coverage remained on the to-do list — but at least the structure now allowed it.

Lined up side by side — control over memory, ease of mid-stream resumption, testability — these were all consequences of one thing: pulling state that had been held by an external process back into my own code. A single decision brought several good things along with it.

Design Calls and the Tradeoffs I Accepted

It wasn't all a tidy success story. Some things I consciously settled for.

fdk-aac as the decoder. It was what I landed on after searching for an AAC decoder usable from cgo at the time. Pure-Go implementations exist, but many cap out at the AAC-LC profile, and I had no confidence they'd cover the variety of input. cgo complicates the build, so I prepared the native dependency as a Docker image and pulled only the build artifact in via multi-stage build at Cloud Run deploy time, eliminating the need to recompile fdk-aac on every build.
The normalization phase has its own cost. Re-encoding involves audio codec conversion, so if it's lossy the audio quality degrades once (whether that's within a range that doesn't affect transcription accuracy is worth examining). Storing the normalized file means holding it alongside the original, and it inserts an extra step between upload and transcription-ready. I judged the payoff of a stable hot path to be worth more.
No support for formats other than mp4 — by design from the start. Input was a domain restricted to mp4, so I avoided over-abstraction (YAGNI).
Not writing the demuxer from scratch was deliberate too. Container parsing is a minefield of edge cases; rolling my own would shoulder all the breakage risk. Riding on a mature third-party library sidestepped that minefield. Working at a low level wasn't the goal — running stably was.

Closing

The lesson from this work is simple: if a parameter has to be tuned endlessly to stay stable, it may not be a value to adjust but a structure to remove.

The 15-second split length, the carelessly-set concurrency, the up-and-down of the memory allocation — all of it was tug-of-war happening on top of one foundation: "load everything into memory, then process." The instant I changed the foundation to streaming, the tug-of-war itself vanished.

And the idea of "pushing heavy, unstable work out of the repeatedly-executed path into a one-time pre-process" applies well beyond this one incident. Keep the hot path light and predictable. Reclaim, into your own code, the state an external process was holding. Do that, and memory, cost, and testability tend to follow.

Kyrage: A TypeScript-First Database Migration Tool for Modern Development

Seiya Izumi — Sun, 14 Sep 2025 08:48:29 +0000

When working on personal projects with CockroachDB, popular tools like Drizzle and Prisma presented unexpected challenges. Drizzle's CockroachDB support was still in planning stages, and while PostgreSQL compatibility seemed promising, migration attempts resulted in data type-related errors. Prisma had its own set of CockroachDB-specific issues.

Most traditional Node.js migration tools require manually writing SQL migration files. What was missing was a tool that could automatically generate migrations based on schema differences while keeping everything in the TypeScript ecosystem - something that could provide the declarative approach similar to Ruby's ridgepole or Atlas, but with better integration for Node.js projects.

Creating Kyrage

This gap led to the creation of Kyrage (kirāju) - a personal project designed to fill this specific need.

IzumiSy / kyrage

A minimal, database-agnostic, schema-based declarative migration tool for Node.js ecosystem

kyrage

A minimal, schema-based declarative migration tool for Node.js ecosystem

kyrage (kirāju) automatically generates and applies database migrations by comparing your TypeScript schema definitions with your actual database state. No more writing migration files by hand!

Why kyrage?

Traditional database migrations require manually writing up/down migration files every time you change your schema. This is error-prone and time-consuming.

kyrage takes a different approach:

✍️ Define your desired schema in TypeScript
🔍 kyrage compares it with your actual database
🚀 Automatically generates the necessary migrations
✅ Apply migrations with a single command

This is a style of managing database schema that is called as Versioned Migration Authoring by Atlas.

Supported Databases

Dialect	`dialect` value	Dev Database
PostgreSQL	`postgres`	✅ Docker container
CockroachDB	`cockroachdb`	✅ Docker container
SQLite	`sqlite`	✅ File-based (no Docker required)

Note: The Dev Database container reuse feature (kyrage dev start) is only available for PostgreSQL and…

View on GitHub

Since most development work happens in Next.js + TypeScript environments, the goal was to unify everything within a TypeScript codebase while providing a minimal, schema-based declarative migration tool.

How Kyrage Works

Kyrage automatically generates and applies database migrations by comparing your TypeScript schema definitions with your actual database state.

The tool follows what Atlas calls the "Versioned Migration Authoring" approach:

Connect to your database and introspect the current schema
Compare your TypeScript schema definition with the current state
Generate a migration file (in JSON format) based on the differences
Apply the migration automatically

Under the hood, kyrage leverages Kysely's migration functionality for SQL execution and version management, while providing its own CockroachDB dialect support.

Key Features

Schema Definition

Define your database schema using intuitive TypeScript syntax. The schema definition benefits from Kysely's type system, providing excellent IDE autocompletion and type safety.

import { column as c, defineTable as t } from "@izumisy/kyrage";

export const members = t("members", {
  id: c("uuid", { primaryKey: true }),
  email: c("text", { unique: true }),
  name: c("text", { unique: true }),
  age: c("integer", { nullable: true }),
  createdAt: c("timestamptz"),
});

export const posts = t("posts", {
  id: c("uuid", { primaryKey: true }),
  author_id: c("uuid"),
  title: c("text"),
  content: c("text"),
  published: c("boolean", { default: false }),
  published_at: c("timestamptz", { nullable: true }),
});

Kyrage also supports indexes, foreign key constraints, composite primary keys, and composite unique constraints:

export const posts = t(
  "posts",
  {
    id: c("uuid"),
    author_id: c("uuid"),
    slug: c("text", { notNull: true }),
    title: c("text"),
    content: c("text", { notNull: true }),
  },
  (t) => [
    // Composite primary key
    t.primaryKey(["id", "author_id"]),

    // Unique constraint with custom name
    t.unique(["author_id", "slug"], {
      name: "unique_author_slug",
    }),

    // Foreign key with cascade delete
    t.reference("author_id", members, "id", {
      onDelete: "cascade",
      name: "posts_author_fk"
    }),

    // Unique index
    t.index(["slug", "title"], { unique: true })
  ]
);

Dev Database Support

Kyrage includes built-in mechanism to spin up ephemeral database for development as Docker container. This addresses common challenges in team development environments.

To use this feature, add the dev configuration to your kyrage config:

export default defineConfig({
  database: {
    dialect: "postgres",
    connectionString: "postgres://postgres:password@localhost:5432/mydb",
  },
  dev: {
    container: {
      image: "postgres:17"
    }
  },
  tables: [members, posts],
});

Then generate migrations using the dev database:

# Generate migration using a clean, ephemeral database
$ kyrage generate --dev
🚀 Starting dev database for migration generation...
✔ Dev database started: postgres
-- create_table: users
✔ Migration file generated: migrations/1755525514175.json
✔ Dev database stopped

This feature eliminates environment dependencies by removing the need to share production database credentials with developers. Each developer works with a clean slate, preventing conflicts that arise when multiple team members make schema changes simultaneously. The approach also reduces the risk of accidental production data modifications while ensuring consistency across the team since everyone uses the same Docker image.

The implementation uses Testcontainers internally, making it seamless to run in CI/CD environments like GitHub Actions.

Getting Started

Installation and Setup

# Install kyrage
npm install @izumisy/kyrage

Create configuration file

kyrage.config.ts:

import { defineConfig } from "@izumisy/kyrage";
import { members, posts } from "./schema";

export default defineConfig({
  database: {
    dialect: "postgres",
    connectionString: "postgres://postgres:password@localhost:5432/mydb",
  },
  tables: [members, posts],
});

Basic Workflow

# Generate migration
$ npx @izumisy/kyrage generate
-- create_table: members (id, email, name, age, createdAt) 
-- create_table: posts (id, author_id, title, content, published, published_at)
✔ Migration file generated: migrations/1754372124127.json

# Preview SQL before applying
$ npx @izumisy/kyrage apply --plan
create table "members" ("id" uuid not null primary key, "email" text not null unique, ...)
create table "posts" ("id" uuid not null primary key, "author_id" uuid not null, ...)

# Apply migration
$ npx @izumisy/kyrage apply
✔ Migration applied: 1754372124127

Advanced Features

Persistent Dev Databases

Kyrage provides the ability to persist dev database containers across kyrage sessions, enabling applications to continuously connect to the same dev database container managed by kyrage

# Start persistent dev database
$ kyrage dev start
✔ Applied 2 migrations
✨ Dev database ready: postgresql://postgres:password@localhost:32768/test

# Get connection URL for your app
$ DATABASE_URL=$(kyrage dev get-url) npm run dev

# Check running containers
$ kyrage dev status
Running: abc123def456 (postgres:17)

# Clean up when done
$ kyrage dev clean

Migration Squashing

For iterative development, kyrage supports squashing multiple migrations to clean up the migration history before applying to production. This feature treats the Dev Database like a feature branch - you can generate multiple migrations during development iterations, then squash them into a clean, consolidated migration when ready to merge to the main database.

Here's how the workflow looks:

# During feature development - generate multiple migrations
$ kyrage generate --dev
✔ Migration file generated: migrations/001_add_users_table.json

$ kyrage generate --dev  
✔ Migration file generated: migrations/002_add_email_column.json

$ kyrage generate --dev
✔ Migration file generated: migrations/003_add_email_index.json

# Before merging to main - squash into a single clean migration
$ kyrage generate --squash
✔ Squashed 3 migrations into: migrations/004_user_management_feature.json

# Apply the clean, squashed migration to production
$ kyrage apply
✔ Migration applied: 004_user_management_feature

This helps organize changes before applying them to production.

Environment-Specific Configuration

Thanks to the unjs/c12 configuration system, kyrage supports environment-specific settings:

export default defineConfig({
  $development: {
    database: {
      dialect: "postgres",
      connectionString: "postgres://dev:dev@localhost:5432/myapp_dev"
    },
    dev: {
      container: { image: "postgres:17" }
    }
  },

  $production: {
    database: {
      dialect: "postgres",
      connectionString: process.env.DATABASE_URL!
    }
  },

  tables: []
});

Why Choose Kyrage?

Kyrage offers several compelling advantages over traditional migration tools. Its TypeScript-first design keeps everything in your familiar development ecosystem with excellent IDE support and type safety. The declarative approach means you define your desired database state and kyrage figures out how to get there, eliminating the tedious process of writing manual migration files.

The Dev Database support removes common team development friction by providing isolated environments for each developer. Rather than reinventing database migration fundamentals, kyrage builds upon proven tools like Kysely while adding modern conveniences that align with contemporary development practices.

And let's be honest - if you're building applications with CockroachDB in the TypeScript ecosystem, kyrage is pretty much your only option that actually works! While other popular tools struggle with CockroachDB compatibility, kyrage was specifically designed with this use case in mind, thanks to Kysely's excellent dialect extensibility!

TypeScript All the Way

Kyrage is designed as a minimal tool focused solely on schema migrations - it doesn't include query building capabilities. This intentional design pairs perfectly with Kysely and kysely-codegen to create the ultimate TypeScript database development experience.

The recommended workflow combines kyrage for schema management with kysely-codegen for type-safe query building:

# Spin up Dev Database to get url
$ kyrage dev start

# Update your schema and generate migration (automatically apply the changes)
$ kyrage generate --dev

# Generate type-safe query builder types from your updated schema
$ DATABASE_URL="$(kyrage dev get-url)" kysely-codegen

This combination gives you declarative schema management through kyrage while leveraging Kysely's powerful type-safe query building capabilities. Other similar libraries that generate type-safe query builders from database schemas work equally well in this setup, allowing you to choose the tools that best fit your project's needs.

Future Roadmap

The kyrage project has several exciting features planned that will significantly enhance the developer experience:

Export API: Reverse-engineer existing databases into TypeScript schemas - perfect for migrating legacy projects to kyrage
Multi-database support: Expanding to DuckDB, SQLite, and MySQL for broader ecosystem compatibility
Hooks API: Execute custom logic during migrations (automated backups, notifications, CI/CD integration)
Seed data pre-population: Automatically populate development databases with test data for streamlined development

These features aim to make kyrage not just a migration tool, but a complete database development workflow solution.

Getting Involved

Kyrage is open source and ready for you to try! Whether you're tired of writing migration files by hand, struggling with CockroachDB compatibility, or just want a cleaner TypeScript-first database workflow, kyrage might be exactly what you've been looking for.

The project welcomes contributors, feedback, and real-world usage stories as it continues to evolve.

Loading state design revisited in Elm

Seiya Izumi — Sat, 15 Aug 2020 14:58:30 +0000

If you are a frontend dev, I suppose that you have been frequently through the scene that you implement loading view in the frontend application.

Elm is a domain-specific language built for frontend applications, so it is pretty usual to design loading view. The great thing is that, Elm has Maybe that says if the value is available or not in a type level. Using Maybe in designing loading state in Elm is really naive pattern, but this is legit.

type alias Model =
    { user : Maybe User
    , bookmarks : Maybe Bookmarks
    }

Now, we can implement init function as follows. Nothing is a variant that says no value available, so we can understant that this function returns empty Model anyway.

init : Model
init =
    { user : Nothing
    , bookmarks : Nothing
    }

As I said this is a naive pattern, this Model design is failure in an Elm way.

First of all, Maybe does not carry any of context information. Maybe is just a type says it can be Nothing or Just, so it does not tell you why it is Maybe. This is meta, but the bigger your codebase gets, the more critical how to convey the context of it by code is.

Maybe is way better than Boolean, but using it too many times is almost equal to Boolean Identity Crisis.

Unleash Custom Type

Let the code tell us its context. This is exactly what Elm as a typed language should be, and where Custom Type comes into.

type Model
    = None
    | UserLoaded User
    | BookmarksLoaded Bookmarks
    | AllLoaded User Bookmarks

Our Model is now way more descriptive than before. We can understand that bookmarks and an user are needed to be waited until fetched.

In addition to the descriptivity, this also has advantage that we don't have to explicitly unwrap them out to non-Maybe types anymore! If every single data waited to be fetched is all wrapped with Maybe, it bothers us that we always have to unwrap it every single time even at where we are sure that it would never be Nothing. We can now extract all data just by a single pattern matching of Model.

I have learnt this pattern from the video, "Make Data Structure" by Richard Feldman at Elm Europe 2018.

Make Data Structure by Richard Feldman

Pitfall: Combinatorial Explosion

However, we would easily come across the case that designing Model simply with Custom Type does not actually work out.

Imagine that some more additional data sources are needed to be fetched in initialization. Working with two is still duable by designing your Model with a simple Custom Type like the previous, but how about more of it? The bigger the number gets like three, four, five, the harder it is to cover all their patterns up.

This is a kind of combinatorial explosion. If you design waiting state on your Model which waits 3 resources at the intialization.Maybe has cardinaliy of 2 which is the number of exponent of waiting resources, so the result is 3^2=9. Now we can see that the number of variants on your model is 9. It exceeds 9 if additional error handling is needed.

The number of variants can easily be exploded. Using a simple Custom Type in desiging waiting state in your Model is totally unrealistic if it is more than two.

elm-multi-waitable

As one of the solution for the variant explosion in Model, I crafted a small package that encapsulate multiple Maybes from Model. This package is well tested on our product built with Elm.

IzumiSy / elm-multi-waitable

A small package like a traffic light

The great thing this package exactly improve is your Model design that needs waiting multiple resources. If your application has to wait 3 resources at the initialization, your Model will look like as follows.

type Model 
    = Loading (MultiWaitable.Wait3 Msg User Options Bookmarks)
    | Loaded User Options Bookmarks

Very neat! Even if we need more of resources waiting, the variants never be exploded anymore.

The rest that wires everything up is as follows.

init : Model
init =
    Loading <| MultiWaitable.init3 AllFinished


type Msg
    = AllFinished User Options Bookmarks
    | UserFetched User
    | OptionsFetched Options
    | BookmarksFetched Bookmarks


update : Model -> Msg -> ( Model, Cmd msg )
update model msg =
    case (model, msg) of
        ( Loading waitable, UserFetched user ) ->
            waitable
                |> MultiWaitable.wait3Update1 user
                |> Tuple.mapFirst Loading

        ( Loading waitable, UserOptions options ) ->
            waitable
                |> MultiWaitable.wait3Update2 options
                |> Tuple.mapFirst Loading

        ( Loading waitable, BookmarksFetched bookmarks ) ->
            waitable
                |> MultiWaitable.wait3Update3 bookmarks
                |> Tuple.mapFirst Loading

        ( Loading _, AllFinished user options bookmarks ) ->
            ( Loaded user options bookmarks, Cmd.none )

MultiWaitable module essentially provides statemachine-like functionality. Internal implementation of this package actually uses multiple Maybes to keep track of status of what has been done and what still not, but it is well encupsalated.

wait[N]Update[N] function MultiWaitable provides returns a tuple contains updated model and Cmd which publishes a completion Msg registered by init[N] function that initialize MultiWaitable at first. Once all waiting state internally kept is marked as completed, wait[N]Update[N] function emits a Cmd for the completion Msg.

What this package brings developers is exaclty a great concentration on Model designing itself. You would be able to design your Model in a descriptive way without any contextless, distractful multiple Maybes.

Cleaner model, greater maitainability

Model is exactly the place where has the bigger portion of clue to know how your application works at a glance, but it gets massive in no time when the application grows. This is no doubt.

Clean model does not let you misunderstand your application and reduce bugs in implementing new features. Designing loading state in a clean way is one of the easiest things we can tackle right now.

Always keep eyes on your Model and let it have enough context.

Designing Opaque Type for form fields in Elm: Part 2

Seiya Izumi — Sun, 12 Apr 2020 12:35:59 +0000

In the previous article, we implemented Username module which encapsulates validation logic of form fields.

That's really enough at that moment, but it still has more room to get improved in viewpoint of extensibility.

Say, if we would like to implement one more another field of Email that have almost the same logic of Username, but with different validation rule. At some point, we can follow rules of WET (Write Everything Twice). Abstract something out too early is always a way to ruin your application. So, it would be better to be dumb about it just by cloning functions of Username into Email module. Then, module relation will be as follows.

It looks fine for now, but getting stinky. Being WET is nice, but don't always be wet. As solution toward this kind of case, we can take advantage of module composition in order to make modules highly coherent and reusable without breaking border of responsibility each module has. This is really similar to class inheritance, but way better than that.

Explicitness and Implicitness

Elm does not have class-based syntax so that naturally we always will take step to use module composition to extend behavior of one another module. This is a nice thing that we don't have any chance to get into pitfalls of class inheritance.

Class inheritance is usually seemed to be implicit, because the all public behaviour of parent classes is carried over by a child class, sometimes intentionally, but sometimes not. This means class inhertance is not suitable to pick only some specfic behaviours of other classes.

Composition seems way explicit on the contrary. If we don't have to inherit some specific functionality, just ignore it. It even has chance to inherit behaviours of compositing classes just by delegation. Simple and concise.

Using module composition in Elm is a key to make your modules extensible and reusable all the way. Extracting business rules in your application into Opaque Type is nice to decouple interface and implementation. Module composition can empower it to be reusable in a good manner.

Overall design

Now, let's apply module composition into our application.

Right before using composition, each module had its own implementation intendedly duplicated. However, now, their internal implementations are delegated to Field module that has fundamental implementation abstracted out of Username and Email.

Field module

Field module now has three states initially Username had before.

validator field in Common record is important. That is defined as a function interface that gets injected in init function. The reason why validator interface requires implementation to return ( String, err ) tuple is that, even though validation fails, the current value is needed to be accessible to show it on input field. If it has only err, the value typed by users will be lost. This must not.

module Field exposing (Field, init)


type alias Common err =
    { value : String
    , validator : String -> Result ( String, err ) String
    }


type Field err
    = Partial (Common err)
    | Valid (Common err)
    | Invalid (Common err) err


init : String -> (String -> Result ( String, err ) String) -> Field err
init value validator =
    Partial
        { value = value
        , validator = validator
        }

Of course, Field module has input function and blur function as well, but they are more abstract.

module Field exposing (Field, init, input, blur, mapErrorToString)

import Html exposing (Html)
import Html.Attributes exposing (type_, value)
import Html.Events exposing (onBlur, onInput)


-- ...

input : (Field err -> msg) -> msg -> Field err -> Html msg
input onInputMsg onBlurMsg field =
    let
        onInputHandler =
            \value ->
                onInputMsg <|
                    case field of
                        Partial { validator } ->
                            Partial
                                { value = value
                                , validator = validator
                                }

                        Valid { validator } ->
                            value
                                |> validator
                                |> mapResult field

                        Invalid { validator } _ ->
                            value
                                |> validator
                                |> mapResult field
    in
    Html.input
        [ type_ "text"
        , value <| toString field
        , onBlur onBlurMsg
        , onInput onInputHandler
        ]
        []


blur : Field err -> Field err
blur field =
    case field of
        Partial { value, validator } ->
            value
                |> validator
                |> mapResult field

        _ ->
            field


mapErrorToString : (err -> String) -> Field err -> Maybe String
mapErrorToString translator field =
    case field of
        Invalid _ err ->
            Just <| translator err

        _ ->
            Nothing

Some more internal functions

-- internals


mapResult : Field err -> Result ( String, err ) String -> Field err
mapResult field result =
    let
        validator_ =
            case field of
                Partial { validator } ->
                    validator

                Valid { validator } ->
                    validator

                Invalid { validator } _ ->
                    validator
    in
    case result of
        Ok value ->
            Valid
                { value = value
                , validator = validator_
                }

        Err ( value, err ) ->
            Invalid
                { value = value
                , validator = validator_
                }
                err


toString : Field err -> String
toString field =
    case field of
        Partial { value } ->
            value

        Valid { value } ->
            value

        Invalid { value } _ ->
            value

Next, let's look into how to use this Field module under the way of module composition in order to create specialized modules.

Username module

As the way of module composition, almost all exposed function delegates its functionality to internal dependent module. Username module is just a wrapper of Field in this meaning that provides specialized behaviours extended from Field module.

One more great thing is that, interface of Username module is almost not changed since the last article even though its internal functionality is updated and being delegated! Actually, errorString function was newly introduced in place of error function that is described in the last article, but this is just an optional change.

Everything stays same overall. The stability of module interface is always important to avoid unintentional effect to other functionality.

module Username exposing (Error(..), Username)

import Field


type Username
    = Username (Field.Field Error)


type Error
    = LengthTooLong
    | LengthTooShort


empty : Username
empty =
    Username <|
        Field.init "" <|
            \value ->
                if String.length value > 20 then
                    Err ( value, LengthTooLong )

                else if String.length value < 5 then
                    Err ( value, LengthTooShort )

                else
                    Ok value

input function and blur function are, as they exactly show, just does delagation to functions provided by Field module.

errorString is a little different. It works like a translator of Error type in this Username module. Error patterns vary on every module extensing Field so that Username module must have responsibility of translating error type to String message as a specialized module.

module Username exposing (Error(..), Username, blur, empty, errorString, input)

import Field
import Html exposing (Html)


-- ...

input : (Username -> msg) -> msg -> Username -> Html msg
input onInputMsg onBlurMsg (Username field) =
    Field.input (onInputMsg << Username) onBlurMsg field


blur : Username -> Username
blur (Username field) =
    Username <| Field.blur field


errorString : Username -> Maybe Error
errorString (Username field) =
    Field.mapErrorToString
        (\error ->
            case error of
                LengthTooShort ->
                    "Length is too short"

                LengthTooLong ->
                    "Length is too long"
        )
        field

This article has shown only the example of Username type, but with this strategy, it would be way easier to implement some similar types like Email, Biography that need validation like this. That's because essential logics to help us implement it as fields are now reusable.

Complete example is on Github.

IzumiSy / elm-compositional-form-field

Typed form implementation in Elm

Designing Opaque Type for form fields in Elm: Part 1

Seiya Izumi — Wed, 08 Apr 2020 14:36:53 +0000

This is an article that shares my knowledge on designing Opaque Type for form fields.

Let's say, we have this kind of a simple form.

-- A simple String value form


type alias Model =
    { username : String }


init : Model 
init =
    { username = "" }


type Msg
    = UsernameInputted String


update : Msg -> Model -> Model
update msg =
    case msg of
        UsernameInputted value ->
            { username = value }

This is pretty much fine for now, but one day we feel like to add validation to username that it must be more than 5 characters and less than 20 characters.

Opaque Type for validations

Now, this is the time to use Opaque Type to encapsulate validation logic into the separated module. We introduce Username type which wraps String as its internal data.

type Username =
    Username String


new : String -> Maybe Username
new value =
    if String.length value > 20 then
        Nothing

    else if String.length value < 5 then
        Nothing

    else
        Just (Username value)

new function now here is a point that does validation of inputted String. If the inputted data is valid, it returns Just.

Okay, now it seems that we can start using Username as a field value like below.

type alias Model =
    { username : Username }

However, it is impossible actually.

Design failure

The main reason is time to run validation. In current design, we have validation mechanism in new function so that it always works when Username data is initialized with String value. The time to initialize it is that update function handling message attached to onInput event coming from views, so Username initialization is going to be triggered every single time when users type their keyboard to fill the field.

It means that, even though the users don't finish typing their value on the field, validation errors will pop up, because the value they are typing partially to fill the field is always invalid. Errors will appear from the beginning. This is so annoying!

So, in case of this, we need to change timing to trigger off validation. The thing to keep in mind is that, Opaque Type for input fields will have "partial" state which describes "users don't need to get validation at this moment". Don't start validation from the very beginning, but do it right once users finished their typing on the field.

Let's see how we can make it better!

Improvement

Now, Username type has three patterns. Every variant has String that describes the current value of it. When the data is Partial, it means that the data conveyed on it does not get validated.

module Username exposing (Username, Error(..), empty, error)


type Username
    = Partial String
    | Valid String
    | Invalid String Error


type Error
    = LengthTooLong
    | LengthTooShort


empty : Username
empty =
    Partial ""


error : Username -> Maybe Error
error username =
    case username of
        Invalid _ error ->
            Just error

        _ ->
            Nothing

The intitial state of Username is always Partial as empty functions shows. It will not be changed until users finished filling value of it.

Then, when is Partial changed into other two variants? That's exactly what view and blur function do.

The moment to start validation is when users removed focus from input fields, so this module provides blur function for it. blur function is expected to be used in update function handling onBlur event. It triggers off validation at once.

On the contrary, the handler for onInput event does not trigger validation during Partial is given to it. It always is just waiting that blur function gets called. Once validation has started by blur function, the handler for onInput triggers off validation every time!

module Username exposing (Username, Error(..), empty, error, input, blur)

import Browser
import Html exposing (Html)
import Html.Attributes exposing (type_, value)
import Html.Events exposing (onBlur, onInput)


-- ...


input : (Username -> msg) -> msg -> Username -> Html msg
input onInputMsg onBlurMsg username =
    let
        onInputHandler =
            \value ->
                onInputMsg <|
                    case username of
                        Partial _ ->
                            Partial value

                        _ ->
                            validate value
    in
    Html.input
        [ type_ "text"
        , value <| toString username
        , onBlur onBlurMsg
        , onInput onInputHandler
        ]
        []


blur : Username -> Username
blur username =
    case username of
        Partial value ->
            validate value

        _ ->
            username


-- internals


validate : String -> Username
validate value =
    if String.length value > 20 then
        Invalid value LengthTooLong

    else if String.length value < 5 then
        Invalid value LengthTooShort

    else
        Valid value


toString : Username -> String
toString username =
    case username of
        Partial value ->
            value

        Valid value ->
            value

        Invalid value _ ->
            value

validate function and toString function are pretty simple. They two probably have no need to be described. One is just to do validation, and the other is to extract String from Username type.

Wire it up!

Now, we don't have to care timing or anything in converting a primitive value of String into Username. It all is completely handled by Username module itself. Gee!

module App exposing (main)

import Html exposing (Html, div)
import Username


type alias Model =
    { username : Username.Username }


init : Model
init =
    { username = Username.empty }


type Msg
    = UsernameInputted Username.Username
    | UsernameBlurred


update : Msg -> Model -> Model
update msg model =
    case msg of
        UsernameInputted username ->
            { username = username }

        UsernameBlurred ->
            { username = Username.blur model.username }


view : Model -> Html Msg
view model =
    let
        error =
            model.username
                |> Username.error
                |> Maybe.withDefault ""
    in
    div
        []
        [ Username.input UsernameInputted UsernameBlurred model.username
        , div [] [ text error ]
        ]


main : Program () Model Msg
main =
    Browser.sandbox
        { init = init
        , view = view
        , update = update
        }

One-file demo on Ellie is here

Why I created validatable-record

Seiya Izumi — Sat, 18 Nov 2017 05:11:42 +0000

IzumiSy / validatable-record

Immutable.js Record powered with validate.js

validatable-record

Immutable.js Record powered with validate.js

Install

$ npm install --save validatable-record

Usage

ValidatableRecord returns Record in Immutable.js for extending your own class. Usage is almost the same as Record in Immutable.js, but it has the power of validate.js. With ValidatableRecord, you can define models with built-in validation logic.

const ManRecord = ValidatableRecord({
  name: null,
  age: null
}, {
  name: {
    presence: true
  },
  age: {
    presence: {
      message: "is invalid"
    }
  }
});
class Man extends ManRecord {
  ...
}

const man = new Man({
  name: "Justine";
  age: 25
});

man.validate() // == true

// Of course you can use `Immutable.Record` methods
man.size        // 2
man.get('name') // "Justine"
man

…

View on GitHub

A few month ago, I created a small module composed of Immutable.js and validate.js. This is a kind of implementation of my approach to the thought about where do you put your validation logic in a front-end application. The field of massive front-end application architecture is totally different from your hobby apps if you value things like scalability or maintainability.

Where to put your own validation logic in your application is not actually a small concern. However, at the beginning phase of front-end application development, the validation logic is often not seemed to be important. In this kind of situation, the view layer tends to have a role of validation, because it is pretty usual for lots of people to think that validation is only the job of view layer.

There is no correct answer to “Where should I put my own validation logic?”, because it really depends on cases, but the sure thing is that the validation logic in view layer should not be the one which relates to your business rule.

Let’s think about a common validation rule like “Your name must not contain special characters”. It is mostly a sort of requirement by infrastructure layer which usually includes things like database, external API server, and so on. However, the validations like “You cannot order this item more than one” or “Free shipping is not available in your membership” are different. These are unique to business rules of your own application. Uniqueness is that, whether the rule is required only because of your business rule or not.

Business rules as domain layer are never affected by other ones such as view layer and infrastructure layer, because view and infrastructure are just implementations which provide your own business as a software and implementations never change what your business rule is. In this concept of layered architecture, validation logic in almost all cases is better to be a part of domain layer. Validation is one of the business rules which compose your application as a domain, not as infrastructure or view.

validatable-record is a small module built with Immutable.js and validate.js in order to define immutable record model with validation rules. Unlike ActiveModel in RoR, there is no standard built-in model validation module in JavaScript, so almost all the time if you would like to put your validation logic in a part of domain model, you will have to write write your own implementation. In the situation, validatable-record is available as one approach to built-in model validation.

const ManRecord = ValidatableRecord({
  name: null,
  age: null
}, {
  name: {
    presence: true
  },
  age: {
    presence: {
      message: "is invalid"
    }
  }
});

// You can use ManRecord as Immutable.Record
class Man extends ManRecord {
  ...
}

const man = new Man({
  name: "Justine";
  age: 25
});

// You can check if the instance is valid or not
man.validate() // == true

// Of course you can use `Immutable.Record` methods
man.size        // 2
man.get('name') // "Justine"
man.get('age')  // 25

ValidatableRecord returns Immutable.Record enhanced with validate.js by taking constraint rules as a second argument. Of course, you can still use methods in Record. You can also get the given error message after validation. More about on https://www.npmjs.com/package/validatable-record.

What I would like to stress in this article with my approach of validatable-record is, scaling front-end application sometimes needs serious planning on responsibility of every layers. Developing front-end application is getting easier than before today, but still software engineers are required to think deeply about architecture from the view point of scalability and maintainability. I would be grad if this article could take you one step back and re-think about your own great application architecture on front-end.

DEV Community: Seiya Izumi

Curing a Transcription Worker That Kept Dying on Cloud Run with Streaming

What the System Did

The Setup I Inherited, and Its Limits

What the Number "15" Was Telling Me

Another Parameter, Set Carelessly

Don't Let the Diagnosis Stay a Guess

The Shift in Thinking: Stream While Reading, Not Load Then Forward

What to Do with ffmpeg — Isolation, Not Elimination

What Paid Off

Memory and Cost

Transcription Accuracy

The Parameters Disappeared

Tests Became Writable

Design Calls and the Tradeoffs I Accepted

Closing

Kyrage: A TypeScript-First Database Migration Tool for Modern Development

Creating Kyrage

IzumiSy / kyrage

A minimal, database-agnostic, schema-based declarative migration tool for Node.js ecosystem

kyrage

Why kyrage?

Supported Databases

How Kyrage Works

Key Features

Schema Definition

Dev Database Support

Getting Started

Installation and Setup

Create configuration file

Basic Workflow

Advanced Features

Persistent Dev Databases

Migration Squashing

Environment-Specific Configuration

Why Choose Kyrage?

TypeScript All the Way

Future Roadmap

Getting Involved

Loading state design revisited in Elm

Unleash Custom Type

Pitfall: Combinatorial Explosion

elm-multi-waitable

IzumiSy / elm-multi-waitable

A small package like a traffic light

Cleaner model, greater maitainability

Designing Opaque Type for form fields in Elm: Part 2

Explicitness and Implicitness

Overall design

Field module

Username module

IzumiSy / elm-compositional-form-field

Typed form implementation in Elm

Designing Opaque Type for form fields in Elm: Part 1

Opaque Type for validations

Design failure

Improvement

Wire it up!

Why I created validatable-record

IzumiSy / validatable-record

Immutable.js Record powered with validate.js

validatable-record

Table of Contents

Install

Usage