DEV Community: Bigi Lui

Karaoke Maker: From Music Video to KTV

Bigi Lui — Wed, 11 Jun 2025 22:53:18 +0000

A few years ago I first learned about Ultimate Vocal Remover, a desktop program created to work with several different ML models that split instrumentals and vocal tracks from a song. I learned about this in the context of learning about voice cloning back then. When I saw how well the vocal remover models worked, I immediately thought it would've been perfect for making karaoke videos.

The problem with many karaoke videos (those found in karaoke places) is multiple folds: (1) poor instrumental quality that sometimes sound nothing like the original song (2) pitch is sometimes off (3) the videos are sometimes completely unrelated to the original song or music video.

I finally got around to building something from this idea I had for a few years. This is now working in Python script format; perhaps one day I'll build a desktop app or web app with this tech.

Core Idea

The primary goal is to take a standard video file (e.g., MP4) as input and produce a new video file with two key features:

Karaoke Audio Mix: The audio is re-mixed into a stereo track where the left channel contains the instrumental-only version of the song, and the right channel contains the original, unprocessed audio. This allows for easy toggling between karaoke and original versions in a media player (common in karaoke places). I also added an option to use just the instrumental-only track, if that's what we want to output.
Synchronized Lyrics: The lyrics of the song are transcribed and embedded into the video as karaoke-style subtitles, highlighting each word as it is sung.

Technology Stack

The application is written in Python, orchestrating several key pieces of tech:

Audio/Video Processing: ffmpeg is used as the workhorse for all audio/video manipulation, including extraction, stream copying, and final re-assembly.
Vocal Separation: onnxruntime runs the UVR_MDXNET_Main.onnx model, a high-quality vocal separation model from the Ultimate Vocal Remover project.
Speech-to-Text: faster-whisper, an optimized implementation of OpenAI's Whisper model, is used for fast and accurate transcription of the isolated vocals to generate word-level timestamps.

Application Workflow

The conversion process is a four-step pipeline. The entire process is automated by a single Python script.

Step 1: Audio Extraction

The process begins by using ffmpeg to extract the full original audio track from the input video file. It is saved as an uncompressed WAV file. This original audio track is also preserved for use in the final video's right audio channel. (If using that option)

Step 2: Vocal Separation

The extracted WAV file is fed into a vocal separation module. This module uses the pre-trained UVR_MDXNET_Main.onnx model. This is an MDX-Net (Music Demixing Network) model that excels at separating music into its component stems. We configure it to produce two outputs:

Instrumental Track: A WAV file containing only the music.
Vocal Track: A WAV file containing only the vocals.

Step 3: Lyrics Generation

The isolated vocal track is then passed to the transcription module. We use the faster-whisper library with the small model for a good balance of speed and accuracy. The library is configured to perform transcription and generate word-level timestamps. These timestamps are then formatted into an Advanced SubStation Alpha (.ass) subtitle file. The .ass format is specifically chosen because it supports karaoke-style effects, allowing us to specify the duration of each word for the "bouncing ball" highlighting effect.

Step 4: Final Video Assembly

In the final step, ffmpeg is called again to construct the output karaoke video. It combines four distinct sources:

Original Video Stream: The video (without audio) is copied directly from the input file to prevent any loss of quality from re-encoding.
Instrumental Audio (Left Channel): The instrumental track from Step 2 is mapped to the left channel of the new stereo audio track.
Original Audio (Right Channel): The original, unprocessed audio from Step 1 is mapped to the right channel.
Karaoke Subtitles: The .ass file from Step 3 is burned directly onto the video frames.

The result is a self-contained karaoke video file, ready for playback.

Here is an example of what the program did:

Original music video: https://www.youtube.com/watch?v=Oextk-If8HQ

Karaoke output: https://www.youtube.com/watch?v=0dBf14_TLZc

Clean monolith code repo structure for a small JAM web app

Bigi Lui — Sun, 24 Jan 2021 23:21:44 +0000

About a year ago I set out to start a new web app project, that would combine the uses of some of my favorite technologies that I've picked up over the past few years.

One of my main beefs with existing standards for modern web apps (typically in a server-side-rendered React stack) is how difficult the code repo's structure is to work with everyday. One of my specific goals in this project is to figure out a way to structure a web app, with both its backend and frontend, in a mono-repo (monolithic code repository), in a way that is easily understood and navigated.

I should emphasize that I also specifically picked to go with a JAM stack for this project, which means that this is certainly entirely not applicable for a server-side-rendered codebase (e.g. Next.js with SSR, etc.). The frontend is a statically-generated site.

The other emphasis I would make is that I'm going with a full Javascript codebase (so a frontend JS framework, with a node.js backend, instead of something like python or ruby). This allows me to share some code between backend and frontend, which is another goal of mine.

Folder structure

I figured the easiest way to understand the folder structure I want to describe is to visualize it as a tree from the code editor.

Immediately you'll notice a few things: There's only a simple layout of a backend folder, a frontend folder, and a small handful of other root level files and folders. We'll do a quick run-down of the important elements.

Naming

You'll immediately notice that instead of going for more technically-focused terms like client or server, I opted to go with a very English-centric vocabulary for the folders, i.e. just frontend and backend.

I wanted there to be no thinking required to immediately figure out how the codebase is laid out, no matter how many years I've been away from the project and not remember anything.

Using terms like client or server always gives me a double-take. If I don't remember that this is a JAM stack app at all, server makes me question whether it has anything to do with a server-side-rendered code, the server of the frontend site, or if it would be the API backend. (or even some other server-side scripts or configuration) Using backend instead makes it clear that it is indeed the API backend, and not anything else.

`backend`

As this is a JAM stack app, this is where the API server is. For my project, I chose to go with a Fastify server. The main thing to point out here is what is contained within the backend folder. In short, imagine if you were building a backend-only project (Pick an example Fastify project if you'd like). Simply, the entire repo of the backend project would belong within here, except package.json.

(Later we'll look at what the root level package.json contains and see how it is shared with frontend.)

`frontend`

I'll just come out and say I'm not a fan of React. I chose Svelte for the project (and I believe I'll choose this for every personal web project going forward), but whether you choose Svelte, Vue or React, the same concept should apply here.

Similar to backend, imagine you were building a frontend-only project. The entire repo of the frontend project would belong within here, again except for package.json.

`lib`

This is where backend and frontend shared modules are. In my case, I am building a simple card game. Some of the logic of the game belongs here. Later on, we'll take a look to see what a module that has different behavior on frontend and backend might look like here.

`package.json`

This probably what ties it all up. The important things here to look at are the scripts section and the dependencies.

  "scripts": {
    "check-format": "prettier --list-different './lib/**/*.js' './test/**/*.js'",
    "format": "prettier --write '*.js' './**/*.js'",
    "lint": "node ./node_modules/eslint/bin/eslint './lib/**/*.js' './test/**/*.js'",
    "test": "tap ./test/**/*.test.js",
    "backend": "node backend/index.js | pino-pretty -m message --ignore level,pid,hostname,v",
    "backend:raw": "node backend/index.js",
    "start": "node backend/index.js",
    "frontend": "node frontend/export-env.js && rollup -c -w",
    "frontend:serve": "sirv frontend/public",
    "build": "node frontend/export-env.js && rollup -c"
  },

The first four, check-format, format, lint, test are mostly standard scripts for use with any js projects. They are applicable for both frontend and backend source files. My tools of choice are prettier, eslint and tap which are pretty standard. I have a strong preference for tap over any other testing frameworks like mocha or jest for the simplicity.

The following scripts are more interesting. The backend scripts make it clear which part of the stack they are starting. The default backend starter script is what I use for development, and it pipes logs through pino-pretty to make it much easier to read. The generically named start script is to work with Heroku. I deploy my projects to Heroku so having the default server start script named start makes it a breeze to set up with no additional config needed.

The frontend scripts are similar: the main frontend script is what I use during local development (supports hot reload with rollup). The build step is for Netlify to run on deploy. The export-env.js is a small script that runs in order to take environmental variables I define on Netlify and write them into a file so the frontend bundle can use them. Needless to say this shouldn't contain any secret tokens or anything; but we can use these for any client-side public tokens we use (e.g. Google Analytics, etc.).

Dependencies

One notable thing here is that, every frontend-only package dependency in the project can simply be put in devDependencies. Because the app is a JAM stack one, the frontend is entirely statically-generated. The generation process happens during build-time, hence only devDependencies is needed.

For shared or backend-only dependencies, we would include the package in dependencies. This actually helps keep the list of dependencies very small for the backend app.

`rollup.config.js` changes

A default rollup config file that comes with a sample Svelte/Sapper project is typically good to start, only having to update the path to the frontend app itself. In my case, the relevant lines in the rollup config look like this:

{
  input: 'frontend/src/main.js',
  output: {
    sourcemap: true,
    format: 'iife',
    name: 'app',
    file: 'frontend/public/build/bundle.js'
  },
  plugins: [
    svelte({
      // enable run-time checks when not in production
      dev: !production,
      // we'll extract any component CSS out into
      // a separate file - better for performance
      css: css => {
        css.write('frontend/public/build/bundle.css');
      }
    }),
  ]
}

`export-env.js` for frontend

As mentioned before, this file allows us to take Netlify-configured environmental variables and put them into the bundle. For safety reasons, we specify exactly which env vars we want to take (so we don't accidentally add secrets to the Netlify configuration and expose them to clients):

'use strict';

/**
 * This script is for Netlify builds.
 * Netlify allows you to set up env vars, but they only exist at build time.
 * We run this as part of the build to generate a json file, which will then get
 * loaded for use in frontend.
 */

require('dotenv').config(); // for local builds with .env file

const fs = require('fs');

fs.writeFileSync(
  './frontend-env.js',
  `'use strict';\n\nmodule.exports = ${JSON.stringify({
    // The following list of env vars should match what's set on Netlify
    ENV: process.env.ENV,
    WEBSOCKET_URL: process.env.WEBSOCKET_URL
  })};\n`
);

No `packages` folder, no separate npm packages, no lerna

This is really my big thing here. When I worked with a mono-repo web app at a job in the past, I hated the multiple packages setup with lerna. The lerna bootstrap process always takes way too long, and it was often error-prone, not knowing when you have to bootstrap again and such.

(And this is a small pet peeve of mine, but I hated how the root level of a frontend app using lerna would have a package.json and packages directory, which really messes with my tab-auto-complete on the command line.)

Here, all of my frontend code (Svelte components, or Vue/React components if you choose) is in frontend. Because I use Svelte and Rollup, I automatically get the benefit of tree-shakes, so even if my project grows large and I start having unused packages they won't pollute the bundle.

Because I'm not doing server-side-rendering, there is generally little "isomorphic" code. The only shared code I have, is explicitly and intentionally placed into the lib folder at root so I know they are shared; and there's no mistake. These are typically functional modules with logic only.

Example shared module with different behavior on frontend and backend: `logger.js`

One of the modules in my lib folder is the logger module logger.js. The reason I don't just simply have separate logger modules in frontend and backend is that I want to be able to use the logger in other lib shared code themselves as well. My logger.js module code looks like this:

'use strict';

const { ENV } = require('./constants');

if (typeof window === 'undefined') {
  // server: use pino
  const pino = require('pino');
  module.exports = pino({ timestamp: pino.stdTimeFunctions.isoTime });
} else {
  // client: logging only if dev
  const logFn = (level, obj) => {
    if (ENV === 'dev') {
      console.log(level, obj);
    }
  };
  module.exports = {
    fatal: obj => logFn('fatal', obj),
    error: obj => logFn('error', obj),
    warn: obj => logFn('warn', obj),
    info: obj => logFn('info', obj),
    debug: obj => logFn('debug', obj),
    trace: obj => logFn('trace', obj)
  };
}

There are several things going on here:

I use typeof window === 'undefined' to detect for the global window object to determine if this is backend or frontend.
On backend, I use the pino logger.
On frontend, only in a dev build, it uses console.log. Otherwise, it's a no-op.

Conclusion

I'm not expecting this blog post to start a big movement away from the current startup best practice with file structures in mono-repo codebases. I'm merely introducing how I'm doing it, and perhaps you'd find it useful for your projects, find flaws and make improvements to it.

If there's any interests, I'd be happy to put together a sample project you can clone and run immediately with this structure, so you can play around with it more easily.

My current new Windows PC setup essentials in 2020

Bigi Lui — Mon, 16 Nov 2020 07:37:44 +0000

I've bought and set up a lot of new desktop/laptop PCs (usually Windows, in some cases Mac or Linux) over the past two decades; and the set of essential software I installed on a new computer has also evolved over the years.

Most recently when I've set up a new Windows laptop for the family, I've found that the set of software I now install for a new Windows computer is now relatively meager and each are now much higher quality than the generations before.

This has become a sort of compilation of "best practices" setup for myself / my family, so I figured it's nice to document it.

Windows Updates

This has really replaced a lot of things over the years; most notably in terms of security. I make sure to open up Windows Update (multiple times, and multiple restarts throughout the process) to be certain that I am fully up-to-date.

Security

In the past I've had various anti-virus (e.g. F-Prot, AVG, Avast, etc.), firewall (e.g. ZoneAlarm, PrivateFirewall, etc.), anti-spyware (e.g. Spybot Search & Destroy), etc. but it seems that Windows Defender, Windows Firewall and Malicious Software Removal Tool has evolved enough over the years that they have been a good enough replacement for everything, along with up-to-date windows security updates.

Language

Language tools in Windows is another category where Windows has completely replaced the need for 3rd party. As a Chinese person, who types some Chinese text from time to time, I've been through software like RichWin, TwinBridge, NJStar, etc. over the years, but nowadays I simply make do with the Pinyin input that comes with Windows as well as CantoneseInput.com, a web-based input editor.

Some Windows settings

Cursor size

It's probably something that comes with old age, but I find that on almost every computer I've set up, the default cursor size (considering the screen resolution and screen sizes nowadays) is too small to see. I bump this up a bit as one of the first things to set.

Trackpad scroll direction

Maybe I'm old school and not of the "mobile generation", but I find that I can only use the mobile scroll direction on mobile. It just doesn't make sense on a laptop; but unfortunately both macOS and Windows decided nowadays that the mobile scroll direction (opposite scroll) should be the default. I have to flip this every time.

Blue Light Filter - Flux

It's probably something that comes with old age again, but I simply can't use a regular-colored screen in the evening hours anymore. It gives me a migraine pretty much within a minute of use. I have to get Flux.

There are a number of options now for blue light filters, and mobile OS like Android even comes with this feature by default now. On Windows I still use Flux, which I've used for many years. It has a lot of niceties around scheduling and easy customization of color temperature and such.

Browser - Edge, Chrome, Firefox

I'll be honest, I think we're well past the days of the Microsoft browser being the butt of the joke (as in, Internet Explorer). Edge is a very legitimate browser in its own right. Both as a dev, and for simply having more options as a user when needed (believe or not, there are still sites out there that don't work on certain browsers), I like having all 3 installed. Note that this also means manually updating Edge to the latest because the one that comes with Windows 10 as of now (Nov 2020) still isn't the latest.

Adblock plugin - ublock origin

Lucky for us, in 2020, ublock origin is available on all 3 browsers. This is the only adblock or privacy blocking plugin I'd use. The more popular AdBlock Plus is way too bloated.

Office - SoftMaker FreeOffice

This was a relatively recent discovery for me. I had stopped using Microsoft Office many years ago mainly to be cost-conscious; and for years it has been sufficient to use either Apache OpenOffice or Google Docs. More recently I found out about FreeOffice and in my brief experience so far it seems superior to OpenOffice, which is slow and often looks dated. Admittedly, I have not tried LibreOffice, but currently I'm happy with FreeOffice.

Graphics - Krita

Krita was another relatively recent discovery for me. For more than 20 years I had used Jasc Paint Shop Pro 6 for basic graphics editing needs (mostly things like resizing/scaling, canvas sizing, or cropping images). In the last few years it has finally begun to age and I've run into a lot of issues with more modern image formats and such (even transparent PNGs aren't handled well).

For this reason, the last few years I found myself doing a lot of work with the web-based Pixlr Editor, and more recently Photopea. However, web-based editors always felt lacking and slow. I've tried using Gimp (another free alternative) as well but have never been satisfied.

Having tried Krita briefly, I can say it appears to satisfy all my basic graphics needs quite well and I can see myself being a long term user.

Communications - Zoom

In these COVID times with social distancing, a lot of times hanging out with friends comes down to video conferencing. In most cases, Zoom is everyone's choice, so here it is. Interesting to note that over the years, the choice of communication software has ranged from ICQ, AIM, MSN messenger, Skype to combo ones like Trillian or Adium. Nowadays text-based messengers have mostly moved to options that are on mobile like WhatsApp and Facebook messenger. Video calls, though, are still common on the computer.

I hope this list helped you discover some useful free software that you previously did not know about as well!

Fav HN Comment - "Less Code Fallacy"

Bigi Lui — Wed, 14 Oct 2020 06:30:35 +0000

Hacker News is certainly a good source of aggregated news (tech news, but also general news) and information. But more than that, I find the comments section to be the best. Every now and then I come across absolute gems. Brilliant people are everywhere, but not every one of them writes blogs! For the most part, these have been buried in my "upvoted comments" section, and not many people ever got a chance to read these comments. This is a blog series I hope to highlight some of my favorite HN comments in the past.

On Software will eat software in a remote-first world (Original article), comment written by 0xB31B1B:

I think your fallacy is in the "less code" assumption when you say "The result is less code". I'd argue that empirically we've seen this to be false. The result isn't less code, at least in a global sense, its more productivity, more features, more customization, and more specificity at a cost of less code/feature. Software has really interesting economics where as the cost/feature decreases by a factor, say 1x, then the set of features that can be profitably worked on expands by like 10x, so paradoxically, as cost/feature decreases, it makes sense to hire more engineers and expand R&D budgets.

This is such counterintuitive insight that I admit I had not thought of until I read the comment, yet it's so true.

The topic of the thread was about no-code/low-code tools (e.g. site builders, etc.); but I think it applies just as well to basically any level of abstractions and frameworks.

What used to take a lot longer, newer frameworks and technologies allow you to build software much faster. Technologies across the spectrum including AWS (in general), NoSQL data stores (compared to SQL DBs), frontend frameworks, enable you to build entire web apps as a solo dev -- the exact same apps that used to take entire teams to build.

I worked on text-based Facebook games back in 2008. We ran LAMP stack with an in-house built MVP framework in PHP, with basically no frontend framework (raw native JS only), running on bare metal servers that our infrastructure team had to assemble in our data centers.

The same type of games could be built in 2020 with modern technologies (React/Svelte for frontend, Redis for caching and various data structures stores for performance, MongoDB/Couchbase for persistent storage) and infrastructure (AWS with EC2/ECS/EKS for auto-scaling, RDS for a managed data store, Elasticache for Redis) and scale to the similar volume of traffic (several millions of daily users) with a much smaller team, with even more stable infrastructure than what we used to have.

What happens then? As a company, in this specific example, it then becomes profitable to build a lot more features, or simply build more games. Because you can now build the same product you did 10 years ago with a 10x smaller team (as an example), that frees up 90% of your team cost -- which in turn means (all else being equal) you could theoretically invest the 90% of the team you freed up to build 9 more games/apps, for 10x the profit. Of course, this is just to demonstrate a point -- in the real world, things aren't quite as simple and straightforward; but the same concept is certainly applicable.

Big2 - Breaking Down a Project Into Phases

Bigi Lui — Sun, 27 Sep 2020 05:10:59 +0000

Finding time to work on a hobby project is among one of the hard things about doing one. What makes it harder is the context you lose between the time you work on it. And when it comes to a reasonably sized project -- anything bigger than something you could complete over a weekend, with some degree of complexity that requires some planning and system design -- losing the context that was once fresh in your head could also mean losing interests/passion and abandoning ship.

The Big2 game project was like that for me. I started it a few months ago, and life got in the way and I paused on it.

When I started it, I had big plans of online play that works both synchronously & asynchronously, as well as various UI features and ideas. I started coding it all in one big sprint. I did take the time to organize code files sensibly, with clean backend/frontend separation (more on the stack and system design here) while taking advantage of Svelte's component structures; and follow best practices with the actual code.

Nevertheless, having taken a hiatus for a few months, coming back to it was hard. A lot of the system design I had in mind of client-server communication (how the server handled client connections and distribute data to each) was lost in my head and what code I had written was hard to follow.

I decided to take a step back from all of that network play code mixed in with game client/UI code, and broke down the project to different phases that I could work on separately, with each phase being small enough for me to complete over a few weeknights.

It came down to the following 3 main areas:

Game Logic

This further breaks down to two sub-points:

Pure functions that handle the logic of the Big2 game rules

Before I even split the project up to the 3 main areas, this was actually the very first part of the codebase I worked on. It made sense at the time too; I foresaw having to have game logic like this on both client side and server side (client side could quickly determine if a move is valid before a server request, and server to validate and execute a move). The actual final design turned out a little different, but this shared lib was useful nevertheless, because having a local game mode.

These are pure functions (in functional programming terms) where you provide an input value, and it runs through some logic and gives an output. No database, no state, no request context, no client.

For example, here's what a function might look like, which validates a move when everybody passed and it's your turn to play any card:

/**
 * next is a card or array of cards the player wants to play
 */
module.exports = next => {
  const cards = Array.isArray(next) ? next : [next];
  if (cards.length === 1) {
    //
    // Singles
    //
    return true;
  } else if (cards.length === 2) {
    //
    // Pairs
    //
    if (cards[0].order !== cards[1].order) {
      return false; // not pair
    }
    return true;
  } else if (cards.length === 3) {
    //
    // Triples
    //
    if (cards[0].order !== cards[1].order || cards[0].order !== cards[2].order) {
      return false; // not triple
    }
    return true;
  } else if (cards.length === 5) {
    //
    // Five Card Set
    //
    const fiveCardType = getFiveCardType(cards);
    if (!fiveCardType.type) {
      return false; // invalid set
    }
    return true;
  } else {
    return false; // invalid move
  }
};

All of the game logic of a game of Big2 is written as a set of a dozen or so pure functions like the one above. This includes everything from checking whether a set of five cards is a straight / flush / full-house / etc., checking if the cards you're playing is greater in power than what was played by the previous player, and so on.

Pure functions that handle players starting games and taking turns

Once the above set of pure functions for game logic is done, this is really a small subset that takes care of the flow of the game.

This set of pure functions would take in the entire Game State object, and determine whose turn is next, what happens when they take a turn or do a pass, update the Game State for everything that needs to happen, and so on.

Local Game Mode

This phase basically allowed me to flesh out the UI and frontend.

Once all of the above pure functions that handle game logic and game flow are done, working on the local game mode is completely Svelte frontend code.

The concise nature of Svelte frontend components code as well as the structures of how a Svelte web app is laid out make this just about as easy and fun as frontend could be to work on. As someone who used to advise against React and for simply using jQuery frontends for hobby projects, I have to say Svelte takes the cake in ease of development. Svelte offers the best of both worlds -- ease of getting started and concise code like a jQuery ad-hoc frontend would give, while providing the code structure and component-based design like React does that a big frontend project needs, in order to not become a spaghetti mess that a jQuery frontend codebase often becomes.

Online Game Mode

Leaving this part to the end allowed me focus on the networking aspect of it, as well as other online features like setting your name and avatar.

At this point, all game logic is done, and I have a simplified game frontend mostly working in a local game mode. Sure, the online mode will have some extra features, but at this point the UI is now in a state where I can make iterative updates, instead of starting from scratch.

This let me dive back into the websockets network game code I had when I first started the project, made sense of what was there, made necessary revamps to better work with the now more independent components of the rest of the codebase.

At the end of the day, getting the online game mode working became a lot easier and shorter than I would've anticipated. I remember at one point coming to a mental revelation of something like, "Wait, this is it. I think I'm actually only a few steps away from a fully working online game. I basically just wire up an API request of what happens when a player clicks a button to take a turn, and the rest of the websockets code that takes a request, execute it, and broadcast the updated Game State to all clients, would just handle the rest." Indeed, once I did that, the client code would receive an updated Game State, render the entire game stage (game board) correctly; and then automatically all players in the game have a fully working game.

What a satisfying feeling! I hadn't even expected the online game to be fully working that soon, but there it is.

Svelte Reactivity and Stores

Bigi Lui — Thu, 27 Aug 2020 05:42:25 +0000

I don't code in Svelte for my day job, so I only get to work with it on my hobby projects. Like most people, I only occasionally have time to work on them. I will have an ongoing project that I just randomly take a hiatus from because life, and then a month/a few months would pass by (or maybe even a year later), I get back to work on it.

Svelte is an intuitive framework and once you learn the basics, it's easy to navigate a Svelte codebase and get things done. Occasionally there are quirks that catch me off-guard, though.

One of the things that have caught me multiple times now, after taking a break and coming back to a project, is how Svelte handles reactivity with its Stores.

Let's say you have some Writable Stores like this, set up in a stores.js file:

import { writable } from 'svelte/store';

export const status_message = writable('Howdy');
export const game_state = writable({});
export const cards = writable([]);

You use these stores normally in your other Svelte components. So you have things like this:

<script>
  import { status_message } from './stores';

  status_message.set('Hello!');
</script>

<div>{$status_message}</div>

All good so far. Let's say though in some cases you put an object or an array into a Writable Store, like the example above with game_state and cards. Because usually you intend to update only some subset of the fields within the object, you may write your code like this in a component that updates it:

<script>
  import { game_state } from './stores';

  $game_state.current_turn = 2;
</script>

<div>Current turn: {$game_state.current_turn}</div>

And you'll find that game_state does not seem to be updated, even though you're using it as an Auto-subscribed Store by virtue of using $.

You have to use `.set()`

The gotcha here is that reactivity is triggered by the .set() call, not simply the data in the Writable Store being updated. So the way to do something like the example above would be:

<script>
  import { game_state } from './stores';

  $game_state.current_turn = 2;
  game_state.set($game_state);

  // Alternatively, use .update() instead of .set()
</script>

<div>Current turn: {$game_state.current_turn}</div>

When you think about the internal workings of Svelte, it makes sense; the specific functions of .set() or .update() on a Writable Store is what kicks off re-evaluation and reactivity based on the value of that store.

When I'm just coding though, I tend to think of a Store as simply a variable that can be accessed globally from anywhere in the frontend app and that changes to it immediately reflect everywhere else. Because of that, I tend to forget what I really should do to kick off a reactive update, especially when what's in the Store is an object or array.

Svelte/Sapper Issue with Switching Components

Bigi Lui — Mon, 24 Aug 2020 05:48:34 +0000

I ran into what might be my first Svelte bug when working on the Big2 card game project. Very roughly, I have a page setup that looks like this (<script> and <style> sections omitted, and unrelated component properties omitted):

<main>
  {#if $state === 'game'}
    <GameStageLocal />
  {:else if $state === 'deal'}
    <DealAnimation onDone={dealingDone} />
  {:else}
    <div>Initializing...</div>
  {/if}
</main>

The <DealAnimation> component plays some game start animation and when it's done, it calls the onDone handler passed into it. What it will do is to update state (a Writable Store) to game so the interactive game UI will start.

As you can deduce from the code above, $state is an auto-subscribed store so when its value changes, it would change the DOM on the page such that <GameStageLocal> is now what's on the page, and <DealAnimation> would be removed from the DOM.

When testing it, the Firefox console gave the following error:

TypeError: t.parentNode is null

It was a bit baffling since I couldn't really find an error in my Svelte code and there wasn't any error in the Svelte dev build output. Upon searching online, I came across this StackOverflow post which has exactly my issue. Although the answer mentions that:

Ran into this error recently. In my case it was due to another library changing the DOM. (In my case fontawesome)

I was not using another library to change the DOM; only Svelte. Either way, though, the described fix in the answer worked for me, which was to wrap my components inside those conditions in another node, like below:

<main>
  {#if $state === 'game'}
    <div>
      <GameStageLocal />
    </div>
  {:else if $state === 'deal'}
    <div>
      <DealAnimation onDone={dealingDone} />
    </div>
  {:else}
    <div>Initializing...</div>
  {/if}
</main>

And this worked fine.

Parting Thoughts

I think this does go to show the relative immaturity of Svelte in general, compared to React or Vue. I remain a fan of Svelte, but nevertheless it's good to keep in mind that when you use a small framework, you're bound to run into actual bugs in the framework.

Miner - Blog Framework Based on Sapper/Svelte

Bigi Lui — Sat, 01 Aug 2020 22:12:16 +0000

I open sourced the framework that powers my blog, and called it Miner!

The README of the repo describes pretty well what it is and the story behind making it. It's based on Sapper, the web app framework based on the Svelte UI framework.

Sapper supports regular SPA mode (Single Page Application), as well as server-side-rendering options, or even all the way to static site generation. The static site generator mode is what I use out of it.

Now, Sapper is a great web app framework, but it's not specific for creating a blog. There are many basic features one would want for a blog, such as managing each post in Markdown files, defining meta-data for each post (e.g. as YAML Frontmatter), OpenGraph social tags for social network sharing optimizations, sharing icons, comments widget, and more.

The Miner framework provides all of this out of the box. Quite literally, you can grab the repo, start writing Markdown files in the posts directory and you'll have a blog running.

The Name

I'm not great at naming and I thought for a while what to name it. I ended up with Miner -- admittedly in this day and age, it's going to make folks think of crypto miners.

In reality, it's an extension of the name Sapper. Miners are a specific type of sappers in war that attack forts. I thought it's appropriate since Miner is a framework for building sites of a specific narrow category (a blog).

Doing my part to push Svelte

If you've been reading my blog, you know I'm a fan of Svelte. As a modern frontend framework, I think it's the one that makes the most sense among Angular, React, Vue, and now Svelte. Admittedly, out of these, I've only had extensive experience with React and Svelte; and only a cursory exploration with Angular and Vue in the past. From my experience though, a web app built on Svelte has the most concise code. Components have the cleanest code among different frameworks.

Svelte is still relatively small, but if you look at the community, there's already a lot of things built upon it. Case in point, searching on npm for a Svelte component for a specific thing usually yields some results. The other day, I was looking for a color picker and found one. If you're looking for a calendar component, there's one.

If you've really been reading my blog, you probably also realize it's not a secret that I dislike React. Right now, the tech industry as a whole basically develops frontend in React. I'd like to do my part to push for other frameworks to be viable, employable skillsets. Or, if you'd like to hear ir phrased more negatively, I'd like to do my part to dethrone React's monopoly status as the frontend framework of choice in the tech industry/among startups. Getting just one more developer comfortable and in favor of Svelte is another step toward that goal.

Other Svelte Blogging Frameworks

Because Sapper stops short of providing features of a blog, many people who've built a blog with Svelte wrote tutorials to build a blog with Svelte/Sapper, or in some cases even provided a sample repo, which is quite similar to Miner actually.

However, I believe that a batteries-included framework can contribute a lot to adoption of a technology. Learning of any new tech takes ramp up time and effort. As individual elements, Svelte and Sapper may be easy to learn; and then once you've learned them, building basic features like those for a blog may be easy tasks; and finally wrapping your head around static site generation and a Netlify deploy may be simple once you've taken the time to read about it.

But add all those together, now you've got a pretty high barrier of entry, if all you're trying to do is to start a blog. You could start one on Wordpress, or another static site blog generator that's more popular like Hugo (written in go) or Jekyll (written in ruby)... or you could take your time and learn all of above about Svelte, and then pick Sapper. I think most people would avoid this last route, given the options.

However, what if you look at Jekyll's "Get up and running in seconds." tagline, see the 4 commands it takes to start a blog:

gem install bundler jekyll
jekyll new my-awesome-site
cd my-awesome-site
bundle exec jekyll serve

And compare it to Miner, which also involves just 4 commands to start:

npx degit "bigicoin/miner#master" my-blog
cd my-blog
npm install
npm run dev

Now you start to have competition. Now you're talking! That, at the end of the day, is what I'm trying to achieve.

On being a tech lead

Bigi Lui — Fri, 10 Jul 2020 21:28:48 +0000

A friend and coworker of mine from Zynga, Ellen, maintains a Slack group of tech leaders from various companies that I'm a part of. Recently, Ellen shared the following article on engineering leadership with the group: The Engineer/Manager Pendulum.

I found it really insightful and echoing a lot of thoughts I've had, but hadn't been able to articulate. Here is a couple of excerpts from it that really rings true to me, and some points that I'd like to add to them.

What one can learn from the pendulum

The best frontline eng managers in the world are the ones that are never more than 2-3 years removed from hands-on work, full time down in the trenches. The best individual contributors are the ones who have done time in management.

And the best technical leaders in the world are often the ones who do both. Back and forth. Like a pendulum.

...

Conversely: the best tech leads in the world are always the ones who done time in management. This is not because they’re always the best programmers or debuggers; it’s because they know how to get shit done, which means they know how to communicate and manage other people.

Up until the last few years, I had been stubbornly avoiding getting into a people management role. My nearly-6-year tenure in Zynga was spent squarely on the individual contributor / architect track. I took something that had some truth to it -- that "I'm not a people manager type" -- and mentally went all-in on it and did not even consider the possibility of going into management. (Zynga, like most large tech companies, had a management track and an IC track, so this was not an issue. As an early employee and important IC, I leveled up quickly from Software Engineer through Principal to eventually Architect.)

Stumbling upon a hybrid team lead / manager role

After shutting down my startup CoinTent (where I spent 3 years hacking up products and running dev ops, doing nothing about team management or tech leadership), I was made an offer to join Massdrop as a Principal Software Engineer, first as an IC. Soon after I joined, a team lead role had opened up. Since I had expressed that I wasn't really into people management, I was given the opportunity to be a hybrid team lead -- giving technical guidance, direction and planning for the team, but not having to do the meaty people management stuff (like holding 1:1s, performance reviews, that sort of thing. I would be able to still do some hands on coding with the remaining time that I had.). The Senior Director of Engineering offered to run those 1:1s himself for my team members directly, so my team would all officially direct-report to him instead.

Even then, I learned a lot about team leadership from my brief time there (I was in that role for only about a year until I moved for personal reasons -- mostly to switch to a fully remote company, instead of working in San Francisco any longer.). I must admit that having been an IC for a decade+ before this, it truly had not occurred to me what all the things are that a team lead is responsible for and why team leads are apparently perpetually busy and hard to reach. Why are they never available even though they barely get any code done?

What does a team lead do?

Let's look at some scenarios. We could get started with the various leadership meetings you'd be regularly in. Those take up a fixed amount of your work days every day -- whether it's project planning, meeting with business operations teams about their needs, or with product managers on consumer product (I've had all of these regularly). But that's the typical stuff you already know about. Let's talk about the remaining hours of your day outside of those meetings instead. Say you run a team of 5, and 4 of them have one question for you that day. Each question takes you anywhere from 15-30 minutes to answer and/or starting a conversation; whether it's because there's some issues they need help figuring out, or because they need some context in feature development, or some direction in design approaches. There goes most of your day. You only had 2-3 hours of non-meeting time to begin with, so those are the same time slots your team will reach out to you because they can actually find you at your desk.

Gaining insights and a new appreciation of team leads and their actual work is just scratching the surface of it. Then there's the real challenging part of being a team lead, which is dealing with humans. This is not about people management; but about actual tech discussions and direction. Working in a company of any reasonable size, you'll inevitably work with many different personalities on the team. Being a tech lead is not about ruling with an iron fist, that this is the system architecture you've designed, and that everyone is doing it this way now, or having everyone develop systems you designed. On any team with intelligent engineers, from a team of 5 you'll get 5 different ideas and system designs to solve a business problem. Your job as a tech lead is not just to evaluate the ideas based on merit, business context, time-frame; but also to facilitate discussion and narrow down on a solution design, while keeping everyone happy and productive and making sure no one feels like they haven't been heard or respected; all the while making sure progress is made in a timely manner (i.e. quickly) and moving forward in general! Think about this again. This isn't people management work; this isn't holding 1:1 meetings or performance reviews. This is the work that a tech lead does.

On working with people

The experience I described above echoes the following excerpt from the article:

They still need the full manager toolset. They’ll need to know how to rally people and teams and motivate them, or how to triage and restart a stalled project that everybody dreads. They still need to connect the dots between business objectives and technical objectives, and break down big objectives into components. They need to be able to size up a junior engineer’s ability and craft a meaningful assignment, one that pushes their boundaries without crushing them … then do the same for another twenty contributors. This is management work, from the slightly shifted perspective of “Get Thing X Done” not “care for these people”.

So these tech leads usually spend more time in meetings than building things, and they will bitch about it but do it anyway, because writing code is not the best use of their time. Tech is the easy part, herding humans is the harder part.

I feel that one of the more important and more delicate aspect for a tech lead to "herd humans" (in order to "get shit done") is to both know the personalities of your team members very well, as well as utilizing their strengths correctly.

Some of your team members come from backgrounds that have taught them to build for the proper engineering solution, for the long term, not incur tech debt. Sometimes, with the business context you have as a tech lead, you know that speed is necessary and sacrifices are needed. You have to juggle this delicate balance, while keeping the team happy. In other cases, it might be the total opposite. You got a builder in your team that's about coding up features as quickly as possible; so you'll want to encourage proper unit testing, documentation, and thinking of edge cases.

Some of your team members will have strong opinions. They may be right a lot of times, and sometimes they may be right only because of the context and background they came from. Either way, they are imposing strong opinions on other team members. In cases like this, your job may be to keep an open mind and listen and take what is appropriate. In some cases you may have to hold your urge to defend your solution in a technical debate (that most engineers tend to do); and sometimes you might just be surprised how much merit the idea/design of someone with the opposite opinion of yours actually has. Your job is to keep things moving, and keep things moving forward; and not dwell on "who wins" the technical debate.

Sometimes less mature engineers like to debate for the sake of debate. Your/other team members' solutions are not wrong, but they like to point out what the issues are, what edge cases are not being covered, and so on. In those cases, know to pick your battle and let folks take the position they insist on but keep things moving forward. You'll notice that's something I'm repeating, because that's really the key here -- move things forward and get things done.

Of course, in yet another scenario, you could have a team member who has too weak of opinions but have brilliant thoughts. It's also your job to draw out ideas from these folks.

Ultimately, being an effective tech lead is about getting things done. It's both about getting things done yourself, and making sure that your team gets things done (and coherently). In some cases that aligns with proper engineering. Sometimes it doesn't. And even when it doesn't, it doesn't necessarily mean you go cowboy all the way. There's a balance to strive for. In certain other scenarios it could align with cowboying code. There's just some things and some time, in a business, where the thing has to ship tomorrow. You have to understand the context fully and the consequences fully and know what needs to be done. That's the job of a tech lead.

Microservices and Monorepo, React and jQuery/no frameworks

Bigi Lui — Sun, 24 May 2020 22:44:07 +0000

The debates of microservices vs monorepo never really end, with articles arguing over one side or another. I came across this article against microservices the other day, and while it has some very valid points and yes, if you're starting a new company or a new project and say "I know! I'll use microservices as backend for this", it's probably the wrong call; I'd probably argue that at a certain point in the growth of an organization, microservices begin to make sense.

While thinking about that, that got me thinking -- that kind of statement is really not unlike the argument for React vs not-react (the framework-less option usually being jQuery with some static HTML) in frontend.

If you're starting a hobby project and you just want to build an MVP, using React is probably overkill, unless you're working on a highly interactive web app (dashboards, charts, etc.). For most web sites with some minimal set of interactivities, static HTML and jQuery is probably sufficient.

The difference between the React vs jQuery question in frontend and the Microservices vs Monorepo question in backend is probably the point on the spectrum at which it makes sense to switch. For frontend, once you get a certain size of a frontend web app (that isn't even very big), it probably starts to make sense to start using a framework already (although for myself, I'd definitely prefer Svelte over React). For backend, you can go a looooong way on a monorepo setup before moving to microservices. Many legit tech startups still run fully monorepo and that's completely valid.

MS Edge's Maps behavior with scroll

Bigi Lui — Fri, 15 May 2020 06:59:45 +0000

I typically use a mix of Chrome and Firefox in my day to day computing, on both Mac and Windows. Once in a while when I'm on my personal Windows laptop, I click on one of those Win10 pre-login screen with pretty scenery and see where it is. It opens up a Bing result page on MS Edge. Then I click on the Bing Maps page to see where that place is.

It catches me off-guard every time I do this that when I scroll (mouse scroll, or trackpad scroll with two fingers), it scrolls the map instead of zooming in and out.

This is one of those cases where, yeah, maybe it makes more sense that when you "scroll" in a mouse or trackpad, the map scrolls, but the pre-established convention is different and it just doesn't feel good as a UX. It's like on the NES, once in a while you run into a game where A is shoot and B is jump. It's weird and it doesn't feel great.

By the way, this isn't a Google Maps or Bing Maps difference, it's a browser difference. Google Maps and Bing Maps behave the same on Chrome and Firefox -- in both cases, scrolling on trackpad will zoom. On MS Edge, both Google Maps and Bing Maps will cause the map to scroll when you scroll on trackpad.

Bootstrapping a Svelte project

Bigi Lui — Tue, 28 Apr 2020 04:32:10 +0000

In my last article I mentioned starting on a card game web app using Svelte as a frontend. (By the way, it may sound weird to use an app frontend framework for a game, but I think for a card game it works fine. It's not like I need a physics engine or anything ;)

As I learn more about Svelte, I want to write some articles on some key learnings I have along the way.

What's Svelte?

Most frontend devs are familiar with React and Vue. Svelte is another framework in the same category. Its focus is on reducing boilerplate code, using vanilla js, and speed (by virtue of not having a virtual DOM). Its syntax is heavily borrowed from Vue, with each component looking something like this:

<script>
  /**
   * Programmatic logic about the component, in vanilla JS
   */
  setTimeout(() => {
    alert('Hello world');
  }, 3000);
</script>

<style>
  /**
   * CSS of the component here
   */
  div.example {
    color: #f00;
  }
</style>

<div class="example">
  Hello! Once this component loads, you will see an alert box in 3 seconds.
</div>

How do I start a Svelte project?

Right from the Svelte homepage, you will see a few cli commands that let you start a project from the sveltejs/template. It's a great place to start for a first project. It starts you off with just one Svelte component on one page, with a default rollup config. Rollup is the bundler Svelte suggests, but you can use other bundles if you like.

But once you've started from the template and begin coding, you may quickly come to a question...

How do I do routing?

Because the template starts you off with one page and one component, you'll quickly wonder how best to do routing if you're trying to build a web app of any reasonable size (not just one page).

From what I've gathered, there are a number of options of popular libraries to do routing with, with different goals in mind.

Probably the most feasible one for a bigger web app is Sapper. As their doc mentions, it's heavily inspired by Next.js. Sapper is opinionated and basically gives you a src/routes directory where you put your pages, with each page being a component. (Should be familiar for those who come from React/Next.js) Sapper can easily be configured to be used as a static site generator (JAM stack), or a standard server-side-rendering setup with a nodejs server. When you go with Sapper, you will want to use the sapper-template instead of the sveltejs/template to get started, as their getting started guide mentions.

(Funnily enough, remember that in my first dev.to article I mentioned how one of the secret sauces of dev.to being so fast was because of its use of InstantClick, i.e. preloading links on mouse hover? Sapper comes with this functionality built in as well.)

As you can imagine, Sapper does add a considerable amount of complexity to your Svelte app. If you're a veteran frontend developer coming from React and Next.js, this won't be an issue for you.

What if you wanted to build a simpler app? In my case, I am building a simple card game, with a landing page and a game page. The game page is hash id based for different "game rooms"; and it contains a lot of dynamic interactivity on it.

For this use case, I've found the svelte-spa-router library to be an excellent option. It only supports client-based routing (with a # hash) and does not do server-side-rendering. However, you can easily run a JAM stack setup with the static site generator setup that comes with svelte template. I basically start from the sveltejs/template, add svelte-spa-router as a dependency into my package.json and start from there.

You're in a good place to start coding

With a Svelte project fully set up along with a router library of your choice, you should be ready to start coding away your new app frontend. Enjoy!