DEV Community: Jimmy Yeung

Journey of Systematically Cut Our Monorepo CI Time in Half

Jimmy Yeung — Thu, 26 Mar 2026 21:51:34 +0000

The engineering team maintains a Python/TypeScript monorepo with 22+ Django microservices, a React frontend, and shared libraries — all running through GitHub Actions CI. Over time, CI had become the productivity bottleneck. PRs routinely waited 10-15 minutes for green checks, and the slowest pipeline consistently exceeded 10 minutes. Engineers were context-switching while waiting for builds, or worse, stacking PRs on top of unverified ones.

I ran a focused initiative to systematically identify and eliminate CI bottlenecks. This post is a record of how I approached it.

1. Finding the Bottleneck

You can't optimize what you can't see. So my first instinct was to write a script to pull GitHub Actions workflow run data from the API and compute aggregate statistics — average, P50, P75, P90 — for both wall clock time and per-job durations across any date range.

This immediately told me two things:

The largest service's unit-test job was the critical path — at P50 of ~10 minutes, it single-handedly determined the CI wall clock time.
Other services stayed around 8-9 minutes — they were running multiple jobs in parallel so further investigation was needed.

Having this data let me prioritize ruthlessly. Instead of optimizing everything at once, I focused on the jobs that actually moved the wall clock needle.

And it's always satisfying to compare the before vs. after statistics.

2. Low-Hanging Fruit — Modernizing the Toolchain

My first lever was swapping slow tools for faster alternatives. The Rust-based ecosystem has matured to the point where several tools are genuine drop-in replacements, so this felt like a no-brainer:

Yarn to Bun (not Rust-written, but famous for its speed and maturity)
Webpack to Rspack
storybook-rsbuild
Adopt eslint-plugin-oxlint, disabling ESLint rules that oxlint now handles

Each swap reduced CI time per job by 30-40%.

3. Low-Hanging Fruit — Docker Build Optimization

Docker builds were another time sink. I attacked from multiple angles:

Enabled .dockerignore — builds weren't using ignore files because the build context was outside their scope
Optimized uv sync in Dockerfiles — eliminated unnecessary dependency installation loops; uv build --package uses build isolation and never touches the workspace venv
Cleaned up stale cross-service dependencies — after migrating to uv workspaces, some services had phantom dependencies on unrelated services, triggering unnecessary rebuilds

This reduced Docker build CI time by about 50%.

4. Test Duration Enforcement & Slow Test Fixes

After tackling the low-hanging fruit, I turned to the tests themselves. The repo already had a check-test-durations composite action that parses JUnit XML and reports the top 10 slowest tests — but it was only wired up in 2 of 22 services.

Rolling Out Visibility

I added the duration check to all workspace CI workflows with continue-on-error: true — visibility without blocking builds.

This immediately surfaced tests taking 10-30+ seconds across services that had never been profiled.

Fixing the Worst Offenders

With data in hand, I targeted the slowest individual tests. Some were making unnecessary network calls, others were creating excessive test data, and a few were simply doing too much in a single test. I fixed tests exceeding 10 seconds across the two slowest services, bringing meaningful improvements before touching any CI infrastructure.

(One fun find: somewhere the application code called time.sleep(10), and the test didn't mock it — so the test took at least 10 seconds for free. Sometimes reducing test duration is that simple.)

Enforcing the Threshold

After the initial round of fixes, I made the duration check a hard enforcement — any individual test exceeding 10 seconds fails the build. This prevents slow tests from creeping back in, turning a one-time fix into a sustainable practice.

5. Caching: Every Layer Counts

GitHub Actions provides a 10 GB cache quota per repository. In a monorepo with 22+ services, that's not much — so every cache needs to earn its space. I identified and enabled caching at four layers of the CI pipeline.

`.git` Cache

In a monorepo, git clone with fetch-depth: 0 (needed for change detection and visual regression diff baselines) is expensive — the repo has significant history. I set up a scheduled workflow that runs every 6 hours to pre-populate the .git directory cache on master.

`uv` Cache

I enabled setup-uv's built-in cache across all CI workflow files.

The cache is keyed on the uv.lock hash, so it invalidates exactly when dependencies change. Impact: ~5 seconds saved per job on the "Install uv" step (8s → 3s average). Small per-job, just a few hundred KB.

`node_modules` Cache

The frontend workflows cache node_modules after bun install. The cache key was previously overloaded — some callers passed a branch name (${{ github.head_ref }}), others a static date string. Branch-keyed caches created redundant entries for the same lockfile, wasting quota.

I simplified to a single lockfile-based key using bun.lock.

GitHub Actions already scopes cache access by branch (PRs can read from the base branch's cache but not other PRs'), so per-branch keys were pure overhead. This change eliminated redundant cache entries and improved hit rates.

ESLint Cache

ESLint is one of the slower steps in frontend validation. I enabled its persistent cache by restoring .eslintcache between runs.

On cache hit, ESLint only re-lints files that changed since the last run, skipping the majority of the codebase.

6. Test Parallelization (with Cost in Mind)

Parallelization is the most effective lever for reducing wall clock time — but in GitHub Actions, more parallel jobs means more billable minutes. Every matrix shard spins up a fresh runner, installs dependencies, and tears down. The setup overhead isn't free. I approached this deliberately, targeting parallelization where the payoff justified the cost.

pytest-xdist: Free Parallelism

The lowest-cost optimization is utilizing all CPUs within a single runner. I enabled pytest-xdist with -n auto in services that were running tests serially.

For one service, this cut pytest execution from ~4m30s to ~2m07s — roughly 2x faster with zero additional runner cost.

Matrix Sharding: Deliberate Trade-off

For the largest service (8,674 tests, ~10 minute P50), single-runner parallelism wasn't enough. I restructured its CI into 7 matrix shards to keep each under 5 minutes.

The cost trade-off: 7 shards means 7x the setup overhead (dependency installation, database creation). Total billable minutes actually increased. But the wall clock time — what engineers wait on — dropped from ~10 minutes to ~5 minutes. For a team making dozens of PRs daily, the productivity gain far outweighs the compute cost.

Takeaways

Measure first, always. A simple Python script using the gh CLI gave me reproducible P50/P75/P90 data across any date range. Way more reliable than eyeballing the Actions UI — and it makes before/after comparisons trivial.
Modernize your toolchain — it's free speed. Bun, Rspack, and oxlint are drop-in replacements that delivered 30-40% speedups per job with minimal migration effort. Highest ROI work I did.
Instrument test durations, then enforce them. A 30-second test hiding in a 6-minute suite is invisible until you look. These are often the easiest wins — and a hard time budget prevents regression.
In a monorepo, cache quota is a shared resource. GitHub Actions' 10 GB limit is shared across all workflows. A poorly-keyed cache doesn't just waste space — it actively evicts caches that matter.
Parallelism costs money — spend it wisely. pytest-xdist within a single runner is free performance. Matrix sharding trades billable minutes for wall clock time. Make that trade deliberately, and know the numbers before and after.

Journey migrating to uv workspace

Jimmy Yeung — Mon, 02 Feb 2026 03:21:33 +0000

My company has migrated from poly-repository to a single monorepo recently due to various reasons, like

we always want our internal packages up-to-date and in-sync of each other -> there's little gain in maintaining versions for our internal packages (not the main point of the blog today).

And for dependency management, we are still using the poly-repo way of

each repo (now dir in monorepo) has its own requirements.txt and its own virtual environment. We will spin each virtual environment up during local development

And of course...the pain points were real:

Slow setup: Full environment setup took 12+ minutes, with libraries being redundantly rebuilt for each service
Version drift: Service A might run Django 4.2 while Service B ran Django 5.1, leading to subtle bugs when code was shared

When uv introduced workspaces, we saw an opportunity to modernize. The goal: a single uv.lock file guaranteeing consistent dependencies across all services, with faster installation via shared caching.

This is the story of that migration.

Phase 1: Aligning Dependencies (The Hidden Prerequisite)

Before we could create a unified workspace, we faced an uncomfortable truth: UV workspaces enforce a single version per package across all services.

Our per-service approach had been hiding version conflicts for years. When we analyzed our codebase, we found:

Package	Service A	Service B	Service C
Django	4.2.x	5.1.x	4.2.x
factory_boy	3.2.x	3.3.x	3.2.x
pytest	7.4.x	8.0.x	7.4.x

With pip's per-venv approach, this "worked" because each service lived in isolation. With UV workspaces, we needed one truth.

The migration began before the migration: we spent time upgrading packages across all services to consistent versions, fixing breaking changes from major version bumps along the way.

Key learning: UV workspaces expose hidden version conflicts that pip's per-venv approach masks. Resolve these conflicts before migration, not during.

Phase 2: Migrating Libraries First

With dependencies aligned, we started with the foundation: our shared libraries. This was strategically important because:

Libraries have fewer dependencies than services
All services depend on libraries, so migrating them first tests the workspace machinery
Subsequent service migrations become simpler

The Workspace Structure

We created a virtual workspace root at your_workspace/pyproject.toml:

[project]
name = "your-workspace"
version = "0.0.0"
description = "Your Monorepo Workspace - Virtual root package"
requires-python = ">=3.12"

[tool.uv]
package = false  # Virtual workspace - don't install this package
index-strategy = "unsafe-best-match"

[tool.uv.workspace]
members = [
    "internal-package-a",
    "internal-package-b",
    "internal-package-c",
    # ... more packages
]
exclude = ["not-yet-migrated"]

[tool.uv.sources]
internal-package-a = { workspace = true }
internal-package-b = { workspace = true }
internal-package-c = { workspace = true }
# All workspace packages declared as sources

After the library migration, local setup time improved from 12 minutes to 9 minutes, since libraries were now built once and cached.

Learning: The Dependency Bloat Discovery

When we migrated our first service, we hit an unexpected problem. setting up in dependent services started failing with installation errors for packages.

The investigation revealed a fundamental difference:

Approach	How it installs internal-package-a	Dependencies installed
Pip (old)	`-e file://../internal-package-a#egg=internal-package-a`	None - only adds source to `sys.path`
UV workspace	`uv sync` with PEP 621	All - follows the spec correctly

The old pip approach was a hack. It only added the package source to the Python path without installing any of its dependencies. This "worked" because:

Dependent services only imported specific utility modules
Common deps like Django were redundantly declared in each service's requirements.txt

UV was doing the correct thing by installing all declared dependencies.

The Solution: Optional Dependencies

The problem was that internal-package-a conflated two concerns:

The running service: Needs a, b, c, x, y, z
The exported libraries: Only need a, b, c

We split them using optional dependencies:

[project]
dependencies = [
    # Minimal deps for exported libraries (what consumers actually need)
    "a",
    "b",
    "c",
]

[project.optional-dependencies]
service = [
    # Runtime deps only needed when running the actual service
    "x",
    "y",
    "z",
]

Now:

Consumers get minimal dependencies by default
Running the service locally: uv sync --extra service
Production Dockerfiles: uv export --extra service

Phase 3: Service-by-Service Migration

With the pattern established, we migrated services in dependency order:

First: Services that only depend on libraries
Then: Services with inter-service dependencies
Finally: Services with the most dependents

Each service migration followed the same checklist:

Move requirements.txt contents to [project].dependencies
Separate service-only deps into [project.optional-dependencies].service
Add to workspace members list
Update Makefile to use uv sync --package <name>
Update Dockerfile (this is where things got interesting...)

Learning: The Docker Editable Install Trap

This was our most painful discovery. After deploying a migrated service to QA, we saw:

ModuleNotFoundError: No module named 'configurations'

The mystery: django-configurations was definitely installed during the Docker build. Even stranger, the same image behaved differently depending on the pod:

DB job pods: Worked fine
Runner/worker pods: Failed with import errors

The Root Cause

When uv export generates requirements, local workspace packages appear as paths:

./internal-package-a
./internal-package-b
./internal-package-c

When uv pip install sees a local path (even without -e), it creates an editable-style install:

__editable__.internal_package_a-0.0.0.pth
__editable___internal_package_a_0_0_0_finder.py

These .pth files add path hooks that reference the source directories (/internal-package-a). In our multi-stage Docker build:

Build stage: Source directories exist (we COPY . .)
Final stage: Only /usr/local/lib is copied, source dirs don't exist
Python startup: .pth files try to hook non-existent paths, causing unpredictable import failures

Why `--no-editable` Doesn't Help

We tried uv export --no-editable, but it only affects the output format:

# With --no-editable:
./internal-package-a

# Without --no-editable:
-e ./internal-package-a

Both are still local paths. uv pip install treats them identically.

The Fix: Build wheels for local packages first

RUN uv export --package your-service --extra service \
        --frozen --no-dev --no-emit-package your-service > /requirements-raw.txt && \
    # Extract local packages (lines starting with ./)
    grep '^\./' /requirements-raw.txt > /local-packages.txt && \
    # Filter out local paths for external deps
    grep -v '^\./' /requirements-raw.txt | grep -v '^[[:space:]]*#' > /requirements.txt && \
    # Build wheels for each local package
    mkdir -p /wheels && \
    while read -r pkg; do \
        uv build --wheel --out-dir /wheels "$pkg"; \
    done < /local-packages.txt && \
    # Install external dependencies
    uv pip install --system -r /requirements.txt && \
    # Install local packages from wheels (NOT source paths)
    uv pip install --system /wheels/*.whl

Wheels are self-contained. When installed, they extract directly to site-packages with no references to source directories.

Verification

We added a check to catch regressions:

RUN ls -la /usr/local/lib/python3.12/site-packages/__editable__* 2>/dev/null && \
    echo "ERROR: Editable installs found - these will fail at runtime!" && exit 1 || \
    echo "OK: No editable installs"

Results

After migrating all libraries and services:

Metric	Before	After
`make setup` time	~12 min	~1 min (a lot faster!!!)
Virtual environments	One per service	Single workspace venv
Dependency consistency	Hoped for	Guaranteed by `uv.lock`
Config files	Multiple `requirements.txt`	Single `pyproject.toml`

Recommendations for Your Migration

Start with a spike: Migrate one service end-to-end to understand the full scope
Invest in dependency alignment upfront: This is the hidden work that makes everything else possible
Plan for Dockerfile changes: The wheel-building pattern should be templated across services
Communicate timing: Multi-branch repos need coordination to avoid merge hell

Conclusion

Migrating to UV workspaces was a significant undertaking, touching every service and library in our monorepo. The upfront investment in dependency alignment and the debugging of Docker editable install issues were the hardest parts.

But the result is worth it: a single source of truth for dependencies, faster local development.

If you're managing a Python monorepo and suffering from version drift or slow setup times, UV workspaces are worth the investment. Just watch out for those editable installs.

Django leaky connection pool

Jimmy Yeung — Thu, 07 Aug 2025 20:49:47 +0000

Disclaimer

In the case described below, we are NOT using traditional Django web server. We are just leverage django as a framework and start our own grpc server. That's why we hit this tricky scenario, that's used to be handled by django automatically

Background

Django web server will close old connection when web request starts / finishes using signals: https://github.com/django/django/blob/5.2.5/django/db/__init__.py#L62-L63

# Register an event to reset transaction state and close connections past
# their lifetime.
def close_old_connections(**kwargs):
    for conn in connections.all(initialized_only=True):
        conn.close_if_unusable_or_obsolete()


signals.request_started.connect(close_old_connections)
signals.request_finished.connect(close_old_connections)

Yet, request_started and request_finished will only be fired upon web requests. If we're not using django for web server, we need to bite the bullet and handle close connection on our own.

What we did

Interceptor on request

We have a grpc.ServerInterceptor, which closes the connection before request and after response.

class DatabaseInterceptor(grpc.ServerInterceptor):
    ...
    def before_request(...):
        django.db.close_old_connections()

    def after_response(...):
        django.db.close_old_connections()


server = grpc.server(
    ...
    interceptors=[
        DatabaseInterceptor(),
    ]
)

Define an internal ThreadPoolExecutor that helps us to close connection

We need this because the new threads spawned will get a connection if it runs a database query -> but will never return the connection back to the pool

class DatabaseThreadPoolExecutor(ThreadPoolExecutor):
    ...
    def submit(self, fn, /, *args, **kwargs):
        try:
            return fn(*args, **kwargs)
        finally:
            django.db.close_old_connections()

Other alternatives but not considered

We considered the on_commit callback like

transaction.on_commit(django.db.close_old_connections)

such that developers don't need to always remember to use the predefined DatabaseThreadPoolExecutor.

However Django doesn't provide rollback hook, that makes it not a good solution because we do want to release connection whenever there's an exception in an atomic block.

Thoughts

This is very tricky to discover but we cannot blame django for this. Django is built for handling web requests, it's on us using the framework to do what it's not supposed to do.

Yet, hope this helps anyone who face this situation and provide some insights. :D

Upgrade from psycopg2 to psycopg3 in Django

Jimmy Yeung — Sat, 10 May 2025 20:42:17 +0000

Why Psycopg3

1. No upcoming features for psycopg2

Psycopg2 is not expected to receive new features anymore
- From psycopg2 Pypi description:

The psycopg2 package is still widely used and actively maintained, but it is not expected to receive new features.

Psycopg 3 is the evolution of psycopg2 and is where new features are being developed: if you are starting a new project you should probably start from 3!

2. More modern features in psycopg3

Psycopg3, being the successor of psycopg3, presents a familiar interface for everyone who has used Psycopg 2 or any other DB-API 2.0 database adapter, but allows to use more modern PostgreSQL and Python features.(cite)

3. More community support psycopg3 like Django

Django starts to support psycopg3 in Django 4.2. And Django 5.1 also introduces connection pooling with psycopg3, that gives us the incentive to upgrade both Django & psycopg3, so as to unleash us from various features going to be introduced in psycopg3. First being the connection pooling.

How to upgrade

psycopg3 already provided the differences to psycopg2 (Ref). What I'm trying to illustrate below are the gotchas when I tried to update my repository using Django to it, which is not mentioned in the aforementioned link.

1. psycopg2 and psycopg3 cannot co-exist

I tried pip install psycopg and my unit tests immediately fail, possibly due to some mixed imports.

We could take reference to django as to how they handle the case by try ... except ImportError ...:

try:
    from psycopg import ...
except ImportError:
    from psycopg2 import ...

2. Test coverage

We have 2 imports but they can't co-exist, so how should our tests run?

We were using coverage run previously. And now as we're maintaining both packages usages, we will need to make use of coverage combine. I.e.

From:

coverage run -m ${TEST_COMMAND}
coverage report

To:

# temp workaround becoz psycopg2 and psycopg have mixed imports.
# So we need to temp. uninstall psycopg and run the original tests in psycopg2.
# Will fallback to install deps again and raise error so it doesn't disrupt the dev experience.
pip uninstall -y psycopg && coverage run -p --data-file=.coverage.psycopg2 -m ${TEST_COMMAND} || { \
    ${DEPENDENCY_INSTALLATION_COMMAND} \
    exit 1; \
};
# Install again for psycopg3 tests
${DEPENDENCY_INSTALLATION_COMMAND}
coverage run -p --data-file=.coverage.psycopg3 -m ${TEST_COMMAND}

# Finally, combine coverage & generate report
coverage combine --append --keep  --debug=pathmap
coverage report

3. DateTimeTZRange changed to TimestamptzRange

(Should be fairly easy and no need to explain)

From:

from psycopg2.extras import DateTimeTZRange

To:

from psycopg.types.range import TimestamptzRange

4. Set CONN_MAX_AGE = 0

CONN_MAX_AGE is a Django setting that controls the lifetime of connections managed directly by Django. It dictates how long Django keeps a connection open before closing and reopening it for subsequent requests.

CONN_MAX_AGE = 0 (default): Connection is closed at the end of each request.
CONN_MAX_AGE > 0: Connection persists for that many seconds.
CONN_MAX_AGE = None: Connection persists indefinitely (until unusable).

When using psycopg3's native pooling via OPTIONS['pool'], the psycopg_pool library takes over the lifecycle management of connections within its pool. Django requests connections from this pool. To avoid conflicting connection management behaviors, it is generally recommended to set CONN_MAX_AGE = 0 when psycopg_pool is active. This ensures Django doesn't prematurely close connections that the pool intends to keep alive and manage. The pool's max_lifetime and max_idle parameters should govern connection longevity.

5. Django adapter

Psycopg3 has the adapters documented here, but not in the aforementioned "difference page".

An example I faced is with JSON adaptation:

from psycopg2.extras import Json
conn.execute("INSERT INTO mytable VALUES (%s)", [Json(thing)])

Now in psycopg3, we need to do the following:

from psycopg.types.json import Jsonb
conn.execute("INSERT INTO mytable VALUES (%s)", [Jsonb(thing)])

How to enable connection pool

In Django documentation, it already stated that we could set "pool" in the OPTIONS part of the database configuration, just stating here for visibility.

# settings.py
# parameters like max_size could be found in https://www.psycopg.org/psycopg3/docs/api/pool.html

DATABASES["default"]["OPTIONS"]["pool"] = {"max_size": <your_desierd_number>}

Conclusion

Overall it's fairly easy to upgrade from psycopg2 to psycopg3 for my Django project, I guess most of the heavy liftings have been done by Django already. Hopefully that helps solve the problems others are facing too!

Confident feature release - use dry-run

Jimmy Yeung — Sun, 16 Jul 2023 07:21:20 +0000

Background

We have feature releases every day - and in every case, we would love to verify if the business logic of that feature is correct or not.

One normal approach is to

have a feature flag, all codes are hidden behind the feature flag and we only switch to the new flow when the feature flag is on.
Do unit tests, integration tests, stress tests for possible scenarios
Have the QA team to carry out end-to-end or even further exploratory tests before flipping the feature flag.

It looks promising. But sometimes the flow is so complicated that we may not cover all possible scenarios in our test suites. How could we be more confident?

Dry Run Mode

In one project (e.g. a web application using Django, React, Postgres), we were doing a large refactoring of an existing feature. We introduced the dry run mechanism so that we can simultaneously run the old endpoint and new endpoint; but the new endpoint is only in dry run mode. In this particular case, it could mean creating new db columns with prefix dry_run - and we are only writing to the dry_run_* columns in dry run.

Behaviour

With feature flag OFF

use old endpoint as source-of-truth
but call the new endpoint with a new boolean request parameter called dry_run. With dry_run is True, we just update the new dry_run_* columns.
We could report any discrepancy if it turns out the dry_run_* is NOT the same as the original ones.

With feature flag’s OFF, we are observing no error + data (dry_run_*) is correct + QA passes; we could toggle the feature flag with confidence.

Benefits

The dry run mechanism indeed helps us to detect the missing logics with real user behaviours. It’s useful since it’s always hard to mimic all user behaviours in QA environment.

Some Gotchas

Although it brings some benefits, we also need to be cautious to avoid affecting production results. Here are some thinking directions and examples:

1. Avoid saving stale data

Usually we make use of .save() in Django for updating. It will generate a query like UPDATE … SET <ALL_FIELDS> … WHERE id = … to update all the columns. For some critical queries, we have applied pessimistic lock with select_for_update so that the rows returned by SELECT query are locked until the entire transaction is committed.
However, not all updating queries apply pessimistic lock. Reasons could be

there’s only one endpoint updating the table - we make use of this nature, so that we don’t apply the locking to have some performance gain.
They rarely hit concurrency issues so it’s not worthwhile to introduce locking
etc.

And in the parallel run world, we’re writing to the same rows and thus saving stale data is a big concern. It also might not be worthwhile to introduce locking to the original flow because it slows down the performance.

How we’re trying to solve this problem

We made use of update_fields argument in .save() to specify which fields to update. This way we don’t need to apply locking. But we should remove them as part of the feature flag cleanup process to avoid confusion.

# Old endpoint: 

update_fields = [
    f.name
    for f in SomeTable._meta.get_fields()
    if f.name not in ("id", "dry_run_column_1", "dry_run_column_2")
    # Boolean flag that indicates if the field has a database column associated with it
    # Source:
    # https://docs.djangoproject.com/en/4.2/ref/models/fields/#django.db.models.Field.concrete
    and f.concrete
]
row.save(update_fields=update_fields)

# New endpoint

update_fields = ["date_modified", "dry_run_column_1", "dry_run_column_2"]
row.save(update_fields=update_fields)

2. Keep performance in mind

Since we’re simultaneously running 2 flows, extra overhead shall be introduced. We should always keep performance in mind and incur as less overhead as possible

How we’re trying to solve this problem

We make use of void instead of await in React frontend very often when we try to call the new endpoint in the frontend. In this way, the frontend doesn’t require waiting for the response before proceeding.

3. Avoid doing same behaviour as the original flow when it’s in dry run mode

It sounds easy, but we need to do it with extra care. E.g.
Make sure it’s a read-only endpoint if we don’t need to write anything in dry run mode. One way of doing this is using

SET SESSION CHARACTERISTICS AS TRANSACTION read only;

in postgres to ensure we're only doing read-only operations to db. Also we should avoid publishing any user-interaction events for analytics (if have) when it's in dry-run mode.

4. Avoid anything which could disrupt the original flow

E.g. We could throw an error from the new endpoint, but we definitely don’t want to show the error to users if it’s in dry run mode; because it has nothing to do with the current flow.

5. Deprecate the dry run mode as part of the clean-up

Always keep the codebase clean, we should deprecate the dry run mode afterwards by: Removing the dry_run_* columns and make sure to sync with other skateholders, make sure to clean up everything in downstream usages.
Removing the code branches if it’s unnecessary after feature flag is on
etc.

Conclusion

To me, having the new logic with dry-run mode in production is a double-edged sword. On one hand, it is helpful to ensure the business logic is correct; but on other hand, it also creates loopholes which may affect the original flow too. We just need to handle it with extra care.

Thus it may not worth to do dry-run for every feature release. But it's worth considered doing so if that feature is of paramount importance (e.g. directly tied to the company's income).

TL;DR, do at your own risk ;D Hope that's helpful.

Put Overlays on image and fit in different aspect ratios

Jimmy Yeung — Sat, 10 Sep 2022 05:54:17 +0000

Preface

I would not say I am a frontend guy. But most of the time in my company I need to do projects in a fullstack way, meaning that I need to touch the frontend side of work as well. And here comes an interesting problem for me to solve ;D

Problem

We need to highlight certain phrases of an image full of text. We're given sets of (x,y,height,width) of the image. The (x,y,height,width) are in percentage so that we can simply do the following to draw an overlay on the image (See example below for highlighting the text danger and science)

overlay {
    left: x%;
    top: y%;
    height: height%;
    width: width%;
}

Ok so how about next?

What if we need to fit it into a viewport which has different aspect ratio? While at the same time highlighting the same thing as the original one?

Fit into a different viewport

In order to do that, we need to know

which part of the original image should we show, and
how we scale the new positions

Assumptions

For simplicity, assume width should always be preserved; as we're fitting a 4629/10110 image to a 700/500 viewport

which part of the original image should we show

The css property background-position comes into handy. And big thanks to the article from https://dev.to/this-is-learning/all-you-need-to-know-about-background-position-3aac, which clearly explains how background-position works. If we know how it works, we could then calculate the new positions by simple maths.

TL;DR

we need to know the minimum y so that we can set that as the new background-position-y
then we need to calculate the new y' and height' after the scaling. (I'm skipping the maths here, but if you are interested, feel free to take a look at the example below)

And at last we could result in something like this ;D

Sum Up

Of course it's not perfect solution. I'm just trying to demo the css properties we can make use of when tackling a problem. There are also other handling/optimisation needed, e.g.

what if the highlight (partially) lies outside of the viewport?
what if some coordinates cannot fit into the viewport with x unchanged; then we need to scale x too.
others...

Love to hear if there're other approaches towards this too!

Check if a file is a subset of another file using bash script

Jimmy Yeung — Sat, 22 Jan 2022 01:48:05 +0000

Scenario

I need to check if a file is a subset of another file into the CI pipeline. Thus bash script is chosen since it's performant and we don't need to install extra dependencies into the CI pipeline.

diff
The first command comes to my mind is diff, which is a really powerful command telling the difference between two files.

However it's too powerful. diff "predicts" which line needs to be changed in order to make the two files identical; which is unnecessary for my use case.

E.g. (Example from GeeksToGeeks)
```
$ cat a.txt
Gujarat
Uttar Pradesh
Kolkata
Bihar
Jammu and Kashmir

$ cat b.txt
Tamil Nadu
Gujarat
Andhra Pradesh
Bihar
Uttar pradesh

$ diff a.txt b.txt
0a1
> Tamil Nadu
2,3c3
< Uttar Pradesh
 Andhra Pradesh
5c5
 Uttar pradesh
```
comm
Without further digging into diff, I found another command comm which is simple and just fit in my use case.

comm returns 3 columns:
- first column contains names only present in the 1st file
- second column contains names only present in 2nd file
- the third column contains names common to both the files
E.g. (Example from GeeksToGeeks)
```
// displaying contents of file1 //
$cat file1.txt
Apaar 
Ayush Rajput
Deepak
Hemant

// displaying contents of file2 //
$cat file2.txt
Apaar
Hemant
Lucky
Pranjal Thakral

$comm file1.txt file2.txt
                Apaar
Ayush Rajput
Deepak
                Hemant
        Lucky
        Pranjal Thakral
```
And to check if one file is a subset of another file, we just need the 1st column. We could just do -23 to neglect the 2nd and 3rd column. I.e.
```
comm -23 file1.txt file2.txt
```

Conclusion

At last, I just end up with this simple bash script to check the subset condition:

#!/bin/bash    
SUBSET="<subset_file_path>"
SUPERSET="<superset_file_path>"
CHECK=$(comm -23 <(sort $SUBSET | uniq ) <(sort $SUPERSET | uniq ) | head -1)

if [[ ! -z $CHECK ]]; then
  echo "Detected extra line in $SUBSET and not in $SUPERSET."
  echo $CHECK
  exit 1
fi

Added the extra sort and uniq commands there just to make sure we're comparing two sorted and deduplicated files.

Debounced Hover on Nested Components Using Event Delegation

Jimmy Yeung — Fri, 10 Dec 2021 08:20:42 +0000

Consider the following case:

const FirstLayeredElement = () => (
  <>
    <SecondLayeredElement/>
    ...
  </>
)
const SecondLayeredElement = () => (
  <>
     <DeepestElement1/>
     <DeepestElement2/>
     ...
  </>
)
const DeepestElement1 = () => (
  <span>...</span>
);
const DeepestElement2 = () => (
  <span>...</span>
);

where we want to fire a function logAnalytics()

when the cursor is hovered on a DeepestElement for some seconds (e.g. 1s)
and we want to know which DeepestElement is captured (Consider some of the info needs to come from the parent components, so we couldn't simply add a listener in DeepestElements)

One of the approach is

pass onMouseEnter handlers into nested div, with the use of debounce from lodash-es

const FirstLayeredElement = () => {
  return (
    <>
      <SecondLayeredElement
        onMouseEnter={(labelType) => logAnalytic(labelType, "Some other info")}
      />
      ...
    </>
  )
}
const SecondLayeredElement = ({onMouseEnter}) => {
  return (
     <DeepestElement1
       onMouseEnter={onMouseEnter}
     />
     <DeepestElement2
       onMouseEnter={onMouseEnter}
     />
     ...
  )
}
const DeepestElement1 = ({ onMouseEnter }) => {
  // Delay for one second
  const debouncedMouseEnter = onMouseEnter
    ? debounce(onMouseEnter, 1000)
    : undefined;
  return (
    <span
      onMouseEnter={() => debouncedMouseEnter("someLabelType1")}
    >
      ...
    </span>
  );
};
const DeepestElement2 = ({ onMouseEnter }) => {
  // Delay for one second
  const debouncedMouseEnter = onMouseEnter
    ? debounce(onMouseEnter, 1000)
    : undefined;
  return (
    <span
      onMouseEnter={() => debouncedMouseEnter("someLabelType2")}
    >
      ...
    </span>
  );
};

But seems lots of useless listeners are added...could we do it in a simpler way?

Event Delegation Approach

First we define a hook useDebounceHover, the input onHover will be called onMouseOut if the time difference between onMouseOver and onMouseOut > 1s (onMouseEnter cannot be used in event delegation, check here and here for more details)

import { DOMAttributes, MouseEvent, useRef } from "react";
const ComponentIdToTypeMapping = {
  some_data_id_1: "someLabelType1",
  some_data_id_2: "someLabelType2",
  ...
}
const useDebounceHover = <T = Element>(
  onHover?: (event: MouseEvent<T>) => void,
  duration = 1000,
): Pick<DOMAttributes<T>, "onMouseOver" | "onMouseOut"> => {
  const labelToHoverDurationMap = useRef({
    some_data_id_1: 0,
    some_data_id_2: 0,
    ...
  });

  const handleMouseOver = (event: MouseEvent<T>) => {
    const labelType = ComponentIdToTypeMapping[event.target.dataset.id];
    if (labelType) {
      labelToHoverDurationMap.current[labelType] = Date.now();
    }
  };

  const handleMouseOut = (event: MouseEvent<T>) => {
    const now = Date.now();
    const labelType = ComponentIdToTypeMapping[event.target.dataset.id];
    if (labelType) {
      if (
        onHover &&
        now - labelToHoverDurationMap.current[labelType] > duration
      ) {
        onHover(event);
      }
      labelToHoverDurationMap.current[labelType] = 0;
    }
  };

  return { onMouseOver: handleMouseOver, onMouseOut: handleMouseOut };
};

export default useDebounceHover;

And so you could:

const FirstLayeredElement = () => {
  const { onMouseOver, onMouseOut } = useDebounceHover(logAnalytic);
  return (
    <div 
      onMouseOver={onMouseOver}
      onMouseOut={onMouseOut}
    >
       <SecondLayeredElement/>
       ...
    </div>
  )
}
const SecondLayeredElement = () => (
  <>
     <DeepestElement1/>
     <DeepestElement2/>
     ...
  </>
)
const DeepestElement1 = () => (
  <span data-id="DeepestElement1">...</span>
);
const DeepestElement2 = () => (
  <span data-id="DeepestElement2">...</span>
);

The presentation layer should be simpler coz with the hook, we just need to

add a parent div for onMouseOver and onMouseOut
add data-id to the deepest components

Conclusion

Note that React has done some optimization so the performance w/o event delegation are similar. Event Delegation does not help in performance in React. But for simplicity, actually my team prefers to use Event Delegation.

But again, there's always trade-off and it depends on different cases ;D.

Performant script in updating a large table🚀

Jimmy Yeung — Mon, 06 Sep 2021 16:50:04 +0000

From time to time, it's inevitable to update some fields in the database via some manual scripts, no matter it's data patching or some random requirements from the business side.

I often have a hard time in writing a script that updates a large table. It just takes so much time in running once, not to mention if you need to debug/optimise after each run.

Use Case

In one of my sprint, I need to append a json object into a jsonb column, in which the json object is computed differently for each row. There are millions of records in the Postgres table and it takes hours to run the script for testing. In order to minimise the time taken for each test so that I can finish my work on time, I tried to compare different ways in order to acquire the fastest result:

(In the examples below, I'm using 10000 records for each case).

Naive Django ORM .save() way
```
for record in records:
    json_value = <computation on json_value>
    record.json_array_field.append(json_value)
    record.save()
```
with cProfile: 6951470 function calls (6891465 primitive calls) in 31.840 seconds

This is definitely the slowest way as it hits the database and executes the update query for every row. Just the first try, we could definitely make things faster.
Django Django ORM bulk_update way
```
for record in records:
    json_value = <computation on json_value>
    record.json_array_field.append(json_value)
MyModel.objects.bulk_update(records, fields=. ["json_array_field"])
```
with cProfile: 5072220 function calls (4612182 primitive calls) in 20.213 seconds

Slightly better. bulk_update is actually suggested by Django for optimization as bulk_update just executes one query. But it's still a lot of time, I tried to find if there's better way to improve.
Django bulk_update with batch_size
```
for record in records:
    json_value = <computation on json_value>
    record.json_array_field.append(json_value)
MyModel.objects.bulk_update(
    records, 
    fields=["json_array_field"], 
    batch_size=1000
)
```
with cProfile: 5077900 function calls (4617619 primitive calls) in 11.765 seconds

Django: If updating a large number of columns in a large number of rows, the SQL generated can be very large. The batch_size parameter controls how many objects are saved in a single query.

Performance are further improved by dividing the updates into 10 batches. Not bad, but again, we could make things faster.
SQL way
```
values_to_be_updated = [(json.dumps(<computation on json_value>), i.id) for i in records]
with connection.cursor() as cursor:
    execute_values(
        cursor,
        "UPDATE my_table "
        "SET json_array_field = json_array_field::jsonb || "
        "(v.new_json_value)::jsonb FROM (VALUES %s) as v (new_json_value, bid) "
        "WHERE id = v.bid::uuid;",
        values_to_be_updated,
    )
```
with cProfile: 802769 function calls (802756 primitive calls) in 3.603 seconds

That calls it a day since it reduces down to just ~3 seconds for updating! It's one tenth of using the .save() one. I just feel so dumbed on trying different ORM methods at first and spent hours waiting it finish running. I should definitely use the SQL way at first. (Imagine if we are updating millions or billions of rows, how huge the difference would be.)

Key Takeaway (for me)

Maybe it's well-known already, but just want to say it again: when writing a one-off bulk insert/update script in a table with large amount of records, do try consider running a SQL at the very first beginning. That could save you a lot of time on waiting and optimising.

But if you needs to consider readability or you do not care about performance, ORM could be a better choice. Just remember in terms of performance, SQL always wins ;D

DEV Community: Jimmy Yeung

Journey of Systematically Cut Our Monorepo CI Time in Half

1. Finding the Bottleneck

2. Low-Hanging Fruit — Modernizing the Toolchain

3. Low-Hanging Fruit — Docker Build Optimization

4. Test Duration Enforcement & Slow Test Fixes

Rolling Out Visibility

Fixing the Worst Offenders

Enforcing the Threshold

5. Caching: Every Layer Counts

.git Cache

uv Cache

node_modules Cache

ESLint Cache

6. Test Parallelization (with Cost in Mind)

pytest-xdist: Free Parallelism

Matrix Sharding: Deliberate Trade-off

Takeaways

Journey migrating to uv workspace

Phase 1: Aligning Dependencies (The Hidden Prerequisite)

Phase 2: Migrating Libraries First

The Workspace Structure

Learning: The Dependency Bloat Discovery

The Solution: Optional Dependencies

Phase 3: Service-by-Service Migration

Learning: The Docker Editable Install Trap

The Root Cause

Why --no-editable Doesn't Help

The Fix: Build wheels for local packages first

Verification

Results

Recommendations for Your Migration

Conclusion

Django leaky connection pool

Disclaimer

Background

What we did

Interceptor on request

Define an internal ThreadPoolExecutor that helps us to close connection

Other alternatives but not considered

Thoughts

Upgrade from psycopg2 to psycopg3 in Django

Why Psycopg3

1. No upcoming features for psycopg2

2. More modern features in psycopg3

3. More community support psycopg3 like Django

How to upgrade

1. psycopg2 and psycopg3 cannot co-exist

2. Test coverage

3. DateTimeTZRange changed to TimestamptzRange

4. Set CONN_MAX_AGE = 0

5. Django adapter

How to enable connection pool

Conclusion

Confident feature release - use dry-run

Background

Dry Run Mode

Behaviour

Benefits

Some Gotchas

1. Avoid saving stale data

2. Keep performance in mind

3. Avoid doing same behaviour as the original flow when it’s in dry run mode

4. Avoid anything which could disrupt the original flow

5. Deprecate the dry run mode as part of the clean-up

Conclusion

Put Overlays on image and fit in different aspect ratios

Preface

Problem

Fit into a different viewport

Assumptions

which part of the original image should we show

TL;DR

Sum Up

Check if a file is a subset of another file using bash script

Scenario

Conclusion

Debounced Hover on Nested Components Using Event Delegation

Consider the following case:

One of the approach is

`.git` Cache

`uv` Cache

`node_modules` Cache

Why `--no-editable` Doesn't Help