DEV Community: Libme

asyncio.gather Raised an Exception but the Other Tasks Kept Running: TaskGroup vs return_exceptions

Libme — Sat, 14 Nov 2026 22:58:58 +0000

If one coroutine in asyncio.gather() raises, the exception propagates to your await, but the sibling tasks are not cancelled. They keep running in the background, still holding connections and still writing. Since Python 3.11, asyncio.TaskGroup fixes this: the first failure cancels the siblings, and you handle the result as an ExceptionGroup. Use gather(..., return_exceptions=True) only when you genuinely want every result, failures included.

What actually happens when gather fails?

The symptom usually shows up as a "ghost" side effect: a request handler returns a 500, yet a database row or an outbound API call from a sibling coroutine still lands a second later. Here is a minimal reproduction:

import asyncio

async def worker(name: str, delay: float, fail: bool = False) -> None:
    try:
        await asyncio.sleep(delay)
        if fail:
            raise RuntimeError(f"{name} failed")
        print(f"{name} finished")
    except asyncio.CancelledError:
        print(f"{name} cancelled")
        raise

async def main() -> None:
    try:
        await asyncio.gather(
            worker("a", 0.1, fail=True),
            worker("b", 0.5),
        )
    except RuntimeError as exc:
        print("caught:", exc)
    await asyncio.sleep(1)  # keep the loop alive to observe "b"

asyncio.run(main())

Output:

caught: a failed
b finished

Task b was never cancelled. The caller already moved on to its error path while b completed its work. The asyncio documentation describes this behavior for the default return_exceptions=False: the first exception is propagated to the awaiter, and the other awaitables in the sequence are not cancelled.

Takeaway: gather reports the first failure but does not clean up after it.

What do people try first (and why it falls short)?

Three dead ends come up repeatedly:

Wrapping the gather in try/except and cancelling the gathered future. Cancelling the future returned by gather does cancel the children, but you only get that handle if you stored it, and by then the failure has already been raised and the siblings have had time to do damage.
Adding return_exceptions=True and forgetting to check the results. Exceptions are now returned as ordinary list items, so nothing raises at all. A failed task becomes a silently ignored object in a list.
Bumping timeouts. A timeout around gather cancels the whole thing, but it does not help when the failure is fast and the siblings are slow.

Takeaway: each workaround either needs manual bookkeeping or turns a loud failure into a quiet one.

How does TaskGroup change this?

asyncio.TaskGroup (Python 3.11 and later) gives structured concurrency: the async with block does not exit until every task in it is done, and if any task fails, the rest are cancelled.

import asyncio

async def worker(name: str, delay: float, fail: bool = False) -> None:
    try:
        await asyncio.sleep(delay)
        if fail:
            raise RuntimeError(f"{name} failed")
        print(f"{name} finished")
    except asyncio.CancelledError:
        print(f"{name} cancelled")
        raise

async def main() -> None:
    try:
        async with asyncio.TaskGroup() as tg:
            tg.create_task(worker("a", 0.1, fail=True))
            tg.create_task(worker("b", 0.5))
    except* RuntimeError as eg:
        for exc in eg.exceptions:
            print("caught:", exc)

asyncio.run(main())

Output:

b cancelled
caught: a failed

Two things trip people up here. First, errors always arrive wrapped in an ExceptionGroup, even if only one task failed, so a plain except RuntimeError: will not catch it; you need except* (also 3.11+). Second, the try/except* has to wrap the async with, not sit inside it, because the group raises when the block exits.

The cost is real: you cannot collect partial results from the survivors, because they were cancelled. If you want "run all, tell me which failed", TaskGroup is the wrong shape.

Takeaway: TaskGroup trades partial results for guaranteed cleanup, and wraps every error in an ExceptionGroup.

When is return_exceptions=True the right choice?

When the tasks are independent and a partial result is useful, such as fetching 50 URLs where 3 failing should not discard the other 47:

import asyncio

async def fetch(i: int) -> int:
    if i % 4 == 0:
        raise ValueError(f"item {i} is bad")
    await asyncio.sleep(0.01)
    return i * 10

async def main() -> None:
    results = await asyncio.gather(*(fetch(i) for i in range(1, 9)), return_exceptions=True)
    ok = [r for r in results if not isinstance(r, BaseException)]
    failed = [r for r in results if isinstance(r, BaseException)]
    print("ok:", ok)
    print("failed:", [str(e) for e in failed])

asyncio.run(main())

Always split the results explicitly, as above. One subtlety: with return_exceptions=True, a CancelledError from a child can also appear in the results list as a value, so check against BaseException, not just Exception.

Need	Use	Siblings on failure	Error shape
All-or-nothing, clean shutdown	`TaskGroup`	Cancelled	`ExceptionGroup`
Best effort, want every outcome	`gather(return_exceptions=True)`	Keep running to completion	Values in a list
Legacy code on Python 3.10 or older	`gather` plus manual cancel	Keep running	First exception only
Background work that must outlive the caller	`create_task` with a stored reference	Independent	Handled in the task

Takeaway: choose by whether a partial result is worth more than a guaranteed stop.

Why do fire-and-forget tasks disappear mid-run?

A related trap: asyncio.create_task(coro()) without keeping the returned task. The event loop holds only a weak reference to tasks, so an unreferenced task can be garbage-collected before it finishes, sometimes producing Task was destroyed but it is pending! in the logs. Keep a strong reference and drop it when the task is done:

import asyncio

background: set[asyncio.Task] = set()

def spawn(coro) -> asyncio.Task:
    task = asyncio.create_task(coro)
    background.add(task)
    task.add_done_callback(background.discard)
    return task

If an exception in such a task is never retrieved, you will see Task exception was never retrieved at garbage collection time. Add a done callback that logs task.exception() if you want failures visible immediately.

Takeaway: an unreferenced task is a task you have agreed to lose.

What if you are stuck below Python 3.11?

The taskgroup and exceptiongroup backport packages exist on PyPI, but check their maintenance status before depending on them. Alternatively, if you already use AnyIO or Trio, both provide task groups with the same cancel-siblings semantics. If you want that behavior across Python versions without writing your own plumbing, AnyIO is the library that gives you task groups on top of asyncio with a consistent API. The drawback is an extra dependency and a slightly different cancellation model than native asyncio, which can surprise you when mixing the two styles in one codebase.

FAQ

Does asyncio.gather cancel other tasks when one raises?
No. With the default return_exceptions=False, the first exception propagates to the awaiter, and the remaining tasks continue running. Use asyncio.TaskGroup if you want them cancelled.

How do I catch an exception from asyncio.TaskGroup?
Use except* around the async with block, for example except* ValueError as eg:. Errors are always delivered inside an ExceptionGroup, so a plain except ValueError: will not match.

Should I use gather or TaskGroup in new code?
Prefer TaskGroup on Python 3.11 and later when tasks belong together and one failure should stop the rest. Keep gather(return_exceptions=True) for independent, best-effort fan-out.

Bottom line

For new code on Python 3.11+, default to TaskGroup so a failure cannot leave orphan work behind. Reach for gather(return_exceptions=True) when you want every outcome and will inspect each one. Hold strong references to any task you spawn and do not await. On older Python, either upgrade or use AnyIO or Trio to get the same structured behavior.

LLM Provider Returns 529/503 Overloaded: LiteLLM vs OpenRouter vs a Hand-Rolled Fallback

Libme — Fri, 13 Nov 2026 21:20:50 +0000

If one provider's outage takes your feature down, you need a fallback path, and you have three realistic options. A hosted router such as OpenRouter is the least code but adds a third party to your request path. A self-hosted layer such as LiteLLM keeps keys and logs in your hands but is something you now operate. A 40-line wrapper around two SDKs is enough when you only ever call two models.

The symptom that sends people searching for this looks like 529 overloaded_error, 503 Service Unavailable, or a stream that opens and then dies halfway. I'll walk through what to fix first, then compare the three approaches. Model names below are placeholders; substitute whatever you run as of November 2026.

Why does retrying the same provider not fix a 529 or 503?

Retries handle blips. Overload and regional incidents last minutes to hours, and a retry loop with backoff just queues your users behind the problem. Worse, if every worker retries, you add load to a provider that is already shedding it.

The dead end I see most often: bumping max_retries in the SDK from 2 to 10. That turns a fast failure into a 90-second hang, and your own request timeouts then fire first. Retries are for 429s with a Retry-After header and for transient network errors. A different provider is the answer to "this one is down."

Takeaway: retry for blips, fail over for outages, and never use the same mechanism for both.

What should trigger a failover, and what should not?

Before choosing a tool, decide the policy, because every option below is only as good as the rules you feed it.

Error class	Fail over?	Why
5xx, 529, connection reset, timeout	Yes	Provider-side or network problem
429 rate limit	Retry first, then fail over	Often your own quota; the backup may have headroom
400 invalid request / context too long	No	The backup will likely reject it too, or silently truncate
401/403	No	Your key is wrong; failing over hides a config bug
Content-policy refusal	No	Not an outage; behavior differs by model

The row people get wrong is context length. A fallback model with a smaller context window will fail your longest requests, which are usually your most important ones. Check the window of every model in the chain before you ship it.

Takeaway: a fallback chain that ignores 4xx semantics will mask your own bugs as provider flakiness.

How does LiteLLM handle fallbacks?

LiteLLM is an open-source Python library plus an optional proxy server that exposes an OpenAI-compatible API in front of many providers. Its Router takes a list of deployments and a fallback map.

import os
from litellm import Router

router = Router(
    model_list=[
        {
            "model_name": "chat",
            "litellm_params": {
                "model": "anthropic/claude-sonnet-4-5",  # placeholder
                "api_key": os.environ["ANTHROPIC_API_KEY"],
            },
        },
        {
            "model_name": "chat-backup",
            "litellm_params": {
                "model": "openai/gpt-4o",  # placeholder
                "api_key": os.environ["OPENAI_API_KEY"],
            },
        },
    ],
    fallbacks=[{"chat": ["chat-backup"]}],
    num_retries=1,
    timeout=20,
)

resp = router.completion(
    model="chat",
    messages=[{"role": "user", "content": "Summarize this ticket."}],
)
print(resp.choices[0].message.content)

What it does well: keys never leave your infrastructure, you can set per-model timeouts and cooldowns, and the same code path covers dozens of providers. What tripped me up: the fallback fires per request, so during a long outage every request pays the primary's timeout first unless you configure cooldowns for failing deployments. Also, running the proxy server means you own its uptime, its database if you enable spend tracking, and its upgrades. If you want a self-hosted gateway with fallbacks, budgets, and one OpenAI-compatible endpoint for every team, LiteLLM is the one that gives you that without sending traffic through a third party.

Takeaway: LiteLLM trades a service you rent for a service you run, and the timeout-before-fallback latency is the part to tune.

How does OpenRouter handle fallbacks?

OpenRouter is a hosted API that fronts many providers with one key and one billing relationship. You send an ordered list of models and it tries them in turn.

import os
from openai import OpenAI

client = OpenAI(
    base_url="https://openrouter.ai/api/v1",
    api_key=os.environ["OPENROUTER_API_KEY"],
)

resp = client.chat.completions.create(
    model="anthropic/claude-sonnet-4.5",  # placeholder
    messages=[{"role": "user", "content": "Summarize this ticket."}],
    extra_body={
        "models": [
            "anthropic/claude-sonnet-4.5",
            "openai/gpt-4o",
        ]
    },
)
print(resp.choices[0].message.content)

This is the least code, and one account replaces several provider invoices. The drawbacks are real, though. Your prompts and your users' data pass through one more company, which matters if your contracts or compliance posture name specific subprocessors. The router itself becomes a single point of failure, so you have traded N providers' outages for N providers' outages plus one. And pricing includes a platform layer; check the current fee model on their site rather than trusting my memory. If you want one key and one invoice and don't want to run anything, OpenRouter is the lowest-effort way to get multi-provider fallback.

Takeaway: OpenRouter removes operational work and adds a dependency you cannot debug from the inside.

When is a hand-rolled fallback enough?

If you call exactly two models and the request shapes are simple, a wrapper is shorter than the config for either tool. The key is classifying errors, not looping blindly.

import os
import anthropic
import openai

primary = anthropic.Anthropic(max_retries=1, timeout=20)
backup = openai.OpenAI(max_retries=1, timeout=20)

def ask(prompt: str) -> str:
    try:
        msg = primary.messages.create(
            model="claude-sonnet-4-5",  # placeholder
            max_tokens=1024,
            messages=[{"role": "user", "content": prompt}],
        )
        return msg.content[0].text
    except (anthropic.APIConnectionError, anthropic.APITimeoutError,
            anthropic.InternalServerError):
        pass  # 5xx, 529, network, timeout: fail over
    # 4xx (BadRequestError, AuthenticationError) propagate on purpose
    r = backup.chat.completions.create(
        model="gpt-4o",  # placeholder
        messages=[{"role": "user", "content": prompt}],
    )
    return r.choices[0].message.content

The limitation: it does not scale. The third provider, tool calling with different schemas, and streaming mid-failure each add code, and that is the point where one of the two tools above starts paying for itself. Also note that a stream that dies halfway cannot be transparently resumed on another model; decide up front whether you discard the partial output and restart.

Takeaway: two models and no tool calls means write the wrapper; anything more and you are rebuilding a gateway.

Comparison

	LiteLLM (self-hosted)	OpenRouter	Hand-rolled
Keys and prompts stay with you	Yes	No	Yes
Ops burden	Medium (proxy, DB optional)	None	Low, grows with providers
Added failure point	Your own proxy	Their service	None
Providers supported	Many	Many	Whatever you code
Best for	Platform teams, budgets	Prototypes, small teams	Two-model apps

FAQ

Should I retry or fail over when my LLM API returns 529 overloaded?

Retry once with a short backoff, then fail over. Repeated retries during an overload extend the outage for your users and add load to the provider.

Does failing over to a different model change my output quality?

Yes. Prompts tuned for one model often behave differently on another, so run your evaluation set against the backup model before you rely on it in production.

Can I resume a streamed LLM response on another provider after it fails midway?

Not transparently. The backup model has no knowledge of the partial output, so you either restart the request or pass the partial text as context and accept a visible seam.

Bottom line

If you call two models and want to keep control, write the wrapper and classify errors carefully. If you are a platform team that needs shared keys, budgets, and many providers behind one endpoint, run LiteLLM. If you are a small team that wants one key and zero operations, accept the extra dependency and use OpenRouter. Whichever you choose, test the fallback by blocking the primary's hostname in staging, because an untested failover path is the one that fails during the incident.

SQLAlchemy MissingGreenlet in Async Code: Fix Lazy Loading With selectinload, awaitable_attrs, or lazy="raise"

Libme — Thu, 12 Nov 2026 21:57:00 +0000

sqlalchemy.exc.MissingGreenlet: greenlet_spawn has not been called; can't call await_only() here means you touched a relationship or expired column that wasn't loaded, and SQLAlchemy tried to run a lazy-load query from synchronous attribute access inside an async session. The durable fix is to decide at query time which relationships you need (selectinload or joinedload) instead of letting attribute access trigger I/O. Use awaitable_attrs for the occasional one-off, and lazy="raise" to turn the bug into a loud, early failure. (Written against SQLAlchemy 2.0.x as of late 2026.)

Why does MissingGreenlet happen?

In a sync session, author.books on an unloaded relationship quietly emits a SELECT and returns the list. That works because attribute access is a plain Python call and the driver call blocks.

In an async session, the driver call must be awaited, but author.books is not awaitable. SQLAlchemy's async layer runs sync-style ORM code inside a greenlet so that it can bridge to await. If you trigger the lazy load outside that bridge, for example in your route handler after the query returned, there is no greenlet to switch to and you get MissingGreenlet.

Here is a minimal reproduction:

import asyncio
from sqlalchemy import ForeignKey, select
from sqlalchemy.ext.asyncio import AsyncAttrs, async_sessionmaker, create_async_engine
from sqlalchemy.orm import DeclarativeBase, Mapped, mapped_column, relationship


class Base(AsyncAttrs, DeclarativeBase):
    pass


class Author(Base):
    __tablename__ = "author"
    id: Mapped[int] = mapped_column(primary_key=True)
    name: Mapped[str]
    books: Mapped[list["Book"]] = relationship(back_populates="author")


class Book(Base):
    __tablename__ = "book"
    id: Mapped[int] = mapped_column(primary_key=True)
    title: Mapped[str]
    author_id: Mapped[int] = mapped_column(ForeignKey("author.id"))
    author: Mapped[Author] = relationship(back_populates="books")


engine = create_async_engine("sqlite+aiosqlite:///:memory:")
Session = async_sessionmaker(engine)


async def main():
    async with engine.begin() as conn:
        await conn.run_sync(Base.metadata.create_all)

    async with Session() as session:
        session.add(Author(name="Ada", books=[Book(title="Notes")]))
        await session.commit()

    async with Session() as session:
        author = (await session.execute(select(Author))).scalars().one()
        print(author.books)  # MissingGreenlet


asyncio.run(main())

(This needs aiosqlite installed. Any async driver reproduces it.)

What tripped me up the first time: the failing line looks completely innocent. Nothing in it says "database". The query was written ten lines earlier.

Takeaway: in async SQLAlchemy, every attribute access that might hit the database must be loaded before you read it.

What are the three ways to fix it?

1. Eager-load at query time (default choice)

from sqlalchemy.orm import selectinload

stmt = select(Author).options(selectinload(Author.books))
author = (await session.execute(stmt)).scalars().one()
print([b.title for b in author.books])  # no extra I/O

selectinload issues one additional SELECT with an IN list of parent keys, which avoids the row duplication that a join creates for collections. joinedload fetches everything in one statement and is a better fit for many-to-one relationships like Book.author.

2. `awaitable_attrs` for one-off access

Because Base mixes in AsyncAttrs, you can await a lazy load explicitly:

books = await author.awaitable_attrs.books

This is honest about the I/O, which is nice. The drawback is that inside a loop it becomes an N+1 query pattern that is easy to write and hard to spot in review.

3. `lazy="raise"` to make the mistake impossible to miss

books: Mapped[list["Book"]] = relationship(
    back_populates="author", lazy="raise"
)

Now any unloaded access raises InvalidRequestError naming the attribute, rather than a vague greenlet message, and it fires in tests instead of in one rarely used endpoint. You opt in per query with selectinload or joinedload. The cost is some boilerplate on every query that needs the relationship.

Approach	Extra queries	Fails loudly if forgotten	Best for
`selectinload`	1 per relationship	Only with `lazy="raise"`	Collections (one-to-many, many-to-many)
`joinedload`	0 (one JOIN)	Only with `lazy="raise"`	Many-to-one, single related row
`awaitable_attrs`	1 per access	n/a (explicit)	Scripts, rare branches
`lazy="raise"` default	n/a	Yes	Teams that want N+1 caught in tests

Takeaway: pick loading strategy per query, and set lazy="raise" on the model so forgetting is a test failure rather than a production 500.

Why do I also get MissingGreenlet after commit()?

Same error, different trigger. By default commit() expires every loaded attribute on every object in the session. The next time you read author.name, SQLAlchemy wants to refresh it with a SELECT, which fails outside the greenlet bridge:

async with Session() as session:
    author = Author(name="Grace")
    session.add(author)
    await session.commit()
    print(author.name)  # MissingGreenlet: attribute was expired by commit

Two fixes. Either configure the sessionmaker so objects stay usable after commit:

Session = async_sessionmaker(engine, expire_on_commit=False)

or call await session.refresh(author) when you really need fresh values. With expire_on_commit=False the trade-off is that your in-memory objects can go stale relative to the database (server-side defaults and triggers won't show up until you refresh). For typical request-scoped sessions that live for one HTTP request, I have found that acceptable, but it is a real trade-off, not a free setting.

A related mistake: returning ORM objects from a handler after the session closes and letting a serializer walk the relationships. The serializer triggers the lazy load on a detached object. Convert to plain schemas inside the session scope, or eager-load everything the serializer will touch.

Takeaway: commit() expires attributes by default, so expire_on_commit=False is the standard async setting, with staleness as the price.

How do I debug which attribute triggered it?

The traceback usually points at the access site, but not at which relationship. Three quick moves:

Temporarily set lazy="raise" on all relationships. The exception names the exact attribute.
Turn on echo=True on the engine and look for a SELECT that appears where you expected none.
Check whether the object is coming from a different session, such as one cached or passed between tasks. Sessions are not safe to share across concurrent tasks.

If you use SQLModel or a FastAPI template that wraps SQLAlchemy, the same rules apply, since they sit on top of the same async layer.

FAQ

Why does SQLAlchemy say "greenlet_spawn has not been called"?

You accessed an unloaded attribute that requires a database query from synchronous code inside an async session. Eager-load the attribute with selectinload or joinedload, or await it through awaitable_attrs.

Should I use selectinload or joinedload with async SQLAlchemy?

Use selectinload for collections and joinedload for many-to-one relationships. Both work in async sessions because the loading happens inside the awaited execute() call.

Is expire_on_commit=False safe?

It is safe for short-lived, request-scoped sessions. It means objects keep their pre-commit values, so call await session.refresh(obj) when you need database-generated values.

Bottom line

If you are writing async SQLAlchemy, set lazy="raise" on relationships and expire_on_commit=False on the sessionmaker, then choose selectinload or joinedload explicitly in each query. Reach for awaitable_attrs in scripts and rare branches, not in loops. If your team is small and tests are thin, the lazy="raise" default is the single change that pays back fastest.

release-please vs Changesets vs semantic-release: Which Release Automation Fits a Solo Dev or Small Team?

Libme — Wed, 11 Nov 2026 18:32:09 +0000

If you merge mostly through squashed pull requests on GitHub, release-please is the lowest-friction choice: it opens a "release PR" you merge when you want to ship. If you maintain a monorepo with several packages, Changesets fits better because humans declare the bump. If you want zero human steps after merge and your team writes disciplined commit messages, semantic-release does that, at the cost of a plugin-heavy config.

All three solve the same problem: stop hand-editing version numbers and changelogs. They differ in where the decision to bump lives, and that is what you should pick on. (Behavior described as of late 2026; check each project's docs before copying config.)

Where does the version bump decision come from?

This is the whole comparison in one question.

Tool	Bump decided by	Release trigger	Human gate before publish
release-please	Conventional Commit messages	Merging an auto-maintained release PR	Yes, the release PR
Changesets	Markdown "changeset" files committed with each PR	Merging a "Version Packages" PR	Yes, the version PR
semantic-release	Conventional Commit messages	Every push to a release branch	No, fully automatic

The consequence people miss: the commit-message tools (release-please, semantic-release) move the discipline into the commit subject line. If your team squash-merges and the PR title becomes the commit, then the PR title is your release note. A sloppy title is a sloppy changelog, and no tool fixes that.

Takeaway: pick the tool whose input you are most willing to be careful about: commit subjects, changeset files, or nothing at all.

How does release-please work in practice?

It watches your default branch, reads commits since the last release, and keeps one open PR that bumps the version and updates CHANGELOG.md. Merging that PR creates the git tag and GitHub release. Nothing ships until you merge it, which is why I reach for it on side projects: batching is free.

A minimal GitHub Actions workflow:

# .github/workflows/release-please.yml
name: release-please
on:
  push:
    branches: [main]

permissions:
  contents: write
  pull-requests: write

jobs:
  release-please:
    runs-on: ubuntu-latest
    steps:
      - uses: googleapis/release-please-action@v4
        with:
          release-type: node

The permissions block matters. Without pull-requests: write the action fails when it tries to open the release PR, and in repos where the default GITHUB_TOKEN is read-only you will see a permissions error rather than a helpful hint. If this bites you, set the permissions explicitly as above.

What tripped me up: release-please does not publish to npm or PyPI by itself. It cuts the tag and release; you add a second job conditioned on the action's release_created output to run npm publish. Also, commits that don't follow Conventional Commits (feat:, fix:, feat!:) are simply ignored for versioning, so a repo with messy history produces an empty release PR, or none.

If you want the version bump to come from a reviewable PR you merge on your own schedule, release-please is the one that does that with the least config.

Takeaway: release-please trades full automation for a merge button, and for small teams that button is the feature.

When is Changesets the better fit?

Changesets inverts the model. Each PR that should affect a release includes a small markdown file created with npx changeset, stating which packages change and whether it is a patch, minor, or major, plus a human-written summary.

---
"@acme/ui": minor
"@acme/utils": patch
---

Add `size` prop to Button; fix off-by-one in `chunk()`.

A bot then opens a "Version Packages" PR that consumes those files, bumps versions (including dependents in the workspace), and writes changelogs. This is why it dominates JavaScript monorepos: bumping @acme/utils can automatically bump packages that depend on it, and you decide per package rather than per commit.

The drawbacks are real. Every contributor has to remember to add a changeset, and forgetting is the common failure; you can enforce it with a CI check, but that is more setup. It is also JavaScript-centric. You can bend it to other ecosystems, but that is not its home turf. For a single-package repo it is more ceremony than the problem deserves.

For a multi-package workspace where the person writing the change knows best what the bump should be, Changesets is the tool that gives you that control without parsing commit messages.

Takeaway: Changesets wins when the bump is a human judgment per package, and loses when nobody remembers to write the file.

Is semantic-release too much for a small project?

semantic-release publishes on every qualifying push: it analyzes commits, picks the version, tags, writes release notes, and publishes through plugins. There is no PR to merge. A basic .releaserc.json:

{
  "branches": ["main"],
  "plugins": [
    "@semantic-release/commit-analyzer",
    "@semantic-release/release-notes-generator",
    "@semantic-release/changelog",
    "@semantic-release/npm",
    "@semantic-release/github",
    ["@semantic-release/git", { "assets": ["CHANGELOG.md", "package.json"] }]
  ]
}

The upside is that releases are boring: merge, and it ships. The downside is the same sentence: merge, and it ships. A mistyped feat!: publishes a major version before anyone reviews it, and unpublishing from a public registry is restricted, so you generally cannot take it back cleanly. It also needs a token with permission to push the release commit back, which collides with branch protection rules; I have seen that setup consume an afternoon.

If you want fully automatic publishing and you trust your commit hygiene (for example via a commit-lint check in CI), semantic-release is the one that removes the last human step.

Takeaway: semantic-release is the right amount of automation only if a wrong release is cheap to recover from.

Decision table: which one for which situation?

Your situation	Pick	Reason
Single package, squash-merge PRs, GitHub	release-please	Release PR gives a free review step
Monorepo with interdependent packages	Changesets	Per-package, human-declared bumps
Library users depend on instant fixes	semantic-release	No batching delay
Team with inconsistent commit messages	Changesets	Doesn't depend on commit text
Non-JavaScript repo (Python, Go, etc.)	release-please	Supports several release types

FAQ

Do I need Conventional Commits to automate releases?
Only for release-please and semantic-release, which read commit messages to decide the bump. Changesets does not use commit messages at all.

Can release-please publish to npm automatically?
Not by itself. It creates the tag and GitHub release, and you add a follow-up job gated on its release_created output to run the publish step.

Why did my release PR not appear?
Almost always because no commit since the last release has a feat: or fix: type, or because the workflow lacks pull-requests: write permission.

Bottom line

For a solo developer or a two-to-five person team on a single package, start with release-please and enforce squash merges with clear PR titles. Move to Changesets when you have a workspace with several packages whose versions need independent control. Choose semantic-release only when you want hands-off publishing and have commit linting in place so a typo cannot ship a major version.

Postgres Full-Text Search vs Meilisearch vs Typesense: Do You Need a Search Service for a Small App?

Libme — Mon, 09 Nov 2026 21:59:04 +0000

If your app already runs on Postgres and you need a search box over a few hundred thousand rows, start with Postgres full-text search plus pg_trgm. Add Meilisearch or Typesense when you need typo tolerance as-you-type, faceted filtering with instant counts, or relevance tuning that non-engineers can adjust. Until one of those is a real requirement, a second datastore is mostly a second thing to keep in sync.

What does Postgres full-text search actually give you?

More than people assume. Stemming, stop words, weighted fields, phrase search, ranking, and a websearch_to_tsquery parser that understands quoted phrases and -exclusions the way users type them. As of late 2026 this is all core Postgres, no extension needed.

A setup that works well for a typical articles table:

ALTER TABLE articles ADD COLUMN search tsvector
GENERATED ALWAYS AS (
  setweight(to_tsvector('english', coalesce(title, '')), 'A') ||
  setweight(to_tsvector('english', coalesce(body,  '')), 'B')
) STORED;

CREATE INDEX articles_search_idx ON articles USING GIN (search);

SELECT id, title, ts_rank(search, q) AS rank
FROM articles, websearch_to_tsquery('english', 'postgres "connection pool" -mysql') AS q
WHERE search @@ q
ORDER BY rank DESC
LIMIT 20;

The generated column keeps the index in sync with the row inside the same transaction. That is the real advantage over an external service: there is no indexing pipeline that can fall behind or silently drop a document.

Takeaway: Postgres full-text search gives you transactionally consistent, weighted, stemmed search with zero new infrastructure.

Where does Postgres search fall short?

Three places, and they are the ones users notice.

Typos. Searching postgre or kuberentes returns nothing, because full-text search matches lexemes, not approximate strings. The standard patch is pg_trgm:

CREATE EXTENSION IF NOT EXISTS pg_trgm;
CREATE INDEX articles_title_trgm ON articles USING GIN (title gin_trgm_ops);

SELECT id, title, similarity(title, 'kuberentes') AS sim
FROM articles
WHERE title % 'kuberentes'
ORDER BY sim DESC
LIMIT 10;

This works for short fields like titles and names. It is a poor fit for fuzzy matching inside long bodies, and tuning pg_trgm.similarity_threshold is something you do by trial and error.

Ranking quality. ts_rank does not account for how rare a term is across the corpus in the way BM25-style engines do. For small, homogeneous data you may never care. For mixed content, results can feel arbitrary, and the fix is hand-written boosts in SQL.

Faceting. Counting results per category for every query means a GROUP BY over the whole match set. That is fine at tens of thousands of matches and gets slow beyond that.

Takeaway: Postgres fails at the user-facing polish (typos, ranking, facet counts), not at the correctness of matching.

How do Meilisearch and Typesense differ?

Both are single-binary, open-source search engines aimed at the "instant search box" use case, both ship typo tolerance on by default, and both have hosted cloud offerings if you do not want to run them. The differences that matter in practice:

	Postgres FTS + pg_trgm	Meilisearch	Typesense
New infrastructure	None	One service	One service (cluster for HA)
Typo tolerance	Only via `pg_trgm`, manual	Built in	Built in
Index lives in	Your database	Disk (memory-mapped)	RAM
Consistency with your DB	Transactional	Eventually, you sync it	Eventually, you sync it
Facets / filters	Hand-written SQL	Built in	Built in
Ranking control	SQL expressions	Ordered ranking rules	Per-field weights, sort expressions
Main drawback	Polish, tuning effort	Another sync pipeline	RAM sizing, another sync pipeline

Typesense holding its index in RAM is why it is fast, and it is also the cost driver: your dataset size becomes your memory bill. Meilisearch is lighter on RAM for the same corpus in my experience, but check the current license terms for any specific feature you depend on, since parts of the ecosystem have moved between licenses over time.

If you want the managed, zero-ops version of instant search with typo tolerance, Meilisearch Cloud is the option that removes the hosting question while keeping the engine simple. If you want predictable low-latency lookups and your dataset comfortably fits in memory, Typesense is the one built around that assumption.

Takeaway: Meilisearch and Typesense buy you polish, and the price is a sync pipeline plus one more thing to operate.

What is the hidden cost of adding a search service?

Keeping it in sync. Every create, update, and delete in Postgres must reach the engine, and the failure mode is quiet: a deleted record that still shows up in results, or a new one that never appears.

The minimum viable pattern is an outbox table written in the same transaction as the change, drained by a worker:

CREATE TABLE search_outbox (
  id bigserial PRIMARY KEY,
  article_id bigint NOT NULL,
  op text NOT NULL CHECK (op IN ('upsert', 'delete')),
  created_at timestamptz NOT NULL DEFAULT now()
);

The worker reads in batches, pushes to the engine, then deletes the processed rows. You also need a periodic full reindex, because you will eventually find drift. That is roughly a day of work plus ongoing monitoring, which is the honest comparison against the afternoon Postgres FTS takes.

Takeaway: The search engine is cheap; the sync pipeline and its drift detection are what you actually pay for.

Which one should you pick?

Your situation	Pick
Under roughly a few hundred thousand rows, users search by exact-ish terms	Postgres FTS
Short fields (names, titles) where typos matter	Postgres FTS + `pg_trgm`
Search-as-you-type with typo tolerance is a product requirement	Meilisearch or Typesense
Many facets with live counts	Meilisearch or Typesense
Dataset is small enough to fit in RAM and latency is critical	Typesense
You have no one to operate another service	Postgres, or a managed cloud plan

These size boundaries are rules of thumb, not benchmarks. Measure your own query latency with EXPLAIN (ANALYZE, BUFFERS) before deciding the database is the bottleneck.

FAQ

Is Postgres full-text search good enough for production?
Yes, for keyword search over moderate datasets it is production-grade, with GIN indexes and weighted ranking. It becomes a poor fit when you need typo tolerance in long text or live facet counts.

How do I add typo tolerance to Postgres search?
Enable the pg_trgm extension, add a GIN trigram index on the field, and query with the % similarity operator. It works best on short fields like titles.

Meilisearch or Typesense: which is easier to start with?
Both run as a single binary and take minutes to try locally. Meilisearch is usually lighter on memory, while Typesense keeps its index in RAM, so size your server to your data.

Bottom line

If you already have Postgres and a search box, use tsvector with a GIN index and add pg_trgm for names and titles; you can ship that in an afternoon and it stays consistent with your data. Move to Meilisearch or Typesense when typo-tolerant instant search or faceting becomes something users complain about, and budget for the outbox and reindex job from day one. Pick the engine after you have measured what your data size and latency actually need.

GitHub Actions "Resource not accessible by integration": Fix It With the Smallest GITHUB_TOKEN Permission

Libme — Sun, 08 Nov 2026 21:54:39 +0000

The error means the GITHUB_TOKEN your job runs with lacks the permission for the API call it just made. Fix it by adding a permissions: block to the specific job with only the scope that call needs (for example pull-requests: write to comment on a PR). Don't flip the whole repository to "read and write" just to make the red X go away.

What does the error actually look like?

Depending on the action or CLI, you will see one of these in the job log:

HttpError: Resource not accessible by integration
RequestError [HttpError]: Resource not accessible by integration - https://docs.github.com/rest/...
remote: Permission to org/repo.git denied to github-actions[bot].
fatal: unable to access 'https://github.com/org/repo/': The requested URL returned error: 403
gh: Resource not accessible by integration (HTTP 403)

All of them come down to the same thing: GitHub authenticated the request as the workflow's app token, and that token's permission set did not include what the endpoint requires. It is an authorization problem, not a bad token, so regenerating secrets will not help.

Takeaway: a 403 from github-actions[bot] is a missing permission scope, never an expired credential.

Why did it work last month and fail now?

Three things commonly change underneath you.

Default token permissions. Each repository (and organization) has a setting under Settings → Actions → General → Workflow permissions. Newer repositories default to read-only for the token; older ones may still have the permissive default. A repo created or migrated after the default changed behaves differently from your old one.
An explicit permissions: key. The moment a workflow or job declares any permission, every scope you did not list is set to none. Adding id-token: write for a cloud login quietly removes contents: read, and then actions/checkout fails on a private repo.
Who triggered the run. Workflows triggered by a pull request from a fork get a read-only token, and by default no secrets. Runs triggered by Dependabot also get a read-only token by default. No permissions: block can widen that for a pull_request event from a fork.

Takeaway: the same YAML can get different tokens depending on repo defaults, the trigger, and the PR author.

Which permission does each step need?

This is the part the error message never tells you. As of late 2026, these are the scopes I reach for most:

What the step does	Permission needed
`actions/checkout` on a private repo	`contents: read`
Push a commit, create a tag or release	`contents: write`
Comment on or label a pull request	`pull-requests: write`
Create or comment on an issue	`issues: write`
Push an image to GitHub Container Registry	`packages: write`
Request an OIDC token for cloud login	`id-token: write`
Upload SARIF to code scanning	`security-events: write`
Deploy to GitHub Pages	`pages: write` and `id-token: write`
Write commit statuses or check runs	`statuses: write` / `checks: write`

One trap: PR comments go through the issues API under the hood, so on some endpoints issues: write also works, but pull-requests: write is the one that matches intent and is what most actions document.

How do I scope it to one job?

Set the default low at the top of the workflow, then raise it only in the job that needs it:

name: pr-size-comment
on: pull_request

permissions:
  contents: read

jobs:
  comment:
    runs-on: ubuntu-latest
    permissions:
      contents: read
      pull-requests: write
    steps:
      - uses: actions/checkout@v4
      - name: Post a comment
        env:
          GH_TOKEN: ${{ github.token }}
        run: |
          gh pr comment "${{ github.event.pull_request.number }}" \
            --body "Checked by CI."

The workflow-level contents: read means any job you add later starts with the minimum. The job-level block replaces (not merges with) the workflow-level one, which is why contents: read is repeated there.

Takeaway: declare read-only at the top, grant write per job, and repeat every scope the job still needs.

Why does it still fail on fork pull requests?

Because the restriction is deliberate. A fork's code is untrusted; if it could run with a write token or your secrets, anyone could open a PR that exfiltrates them. So a pull_request run from a fork cannot comment, label, or push, whatever your YAML says. In my testing, the confusing part is that the same workflow succeeds on branches inside the repo and fails only for outside contributors.

Your options, from safest to riskiest:

Split the work. Let the untrusted pull_request run build and test, save the results as an artifact, and have a second workflow triggered by workflow_run (which runs in the base repo's context) download the artifact and post the comment. This keeps untrusted code away from the write token.
Skip the comment for forks. Guard the step with if: github.event.pull_request.head.repo.full_name == github.repository, so internal PRs get the comment and forks just pass quietly.
pull_request_target. It runs with a write token against the base branch, but if you check out and execute the PR's code in that job you have handed the keys to the contributor. Use it only for steps that never run PR-supplied code.

The same logic applies to Dependabot PRs: they get a read-only token and a separate Dependabot secrets store. If an auto-merge or comment workflow fails only on Dependabot branches, that is why.

Takeaway: for fork PRs, do not fight the read-only token; move the privileged step into a trusted workflow_run job.

What about reusable workflows and organization policy?

Two more causes that look identical in the log:

Reusable workflows can only reduce permissions. A called workflow inherits the caller's token and can downgrade it, but it cannot grant itself something the caller did not allow. Fix the permissions: on the calling job.
Org-level restrictions. An organization owner can cap the default token to read-only and prevent repositories from overriding it. If your permissions: block looks correct and the call still returns 403, check the org's Actions settings before you debug the YAML.

When a step needs more than the token can ever give, such as pushing to a different repository or triggering a workflow from a workflow (events created with GITHUB_TOKEN generally do not start new runs), use a GitHub App installation token or a fine-grained personal access token stored as a secret. A GitHub App is the better long-term choice because it is not tied to one person's account. The honest drawback: it adds setup (create the app, install it, mint a token in the job), so use it only when permissions: genuinely cannot cover the case.

Takeaway: if the YAML is right and it still 403s, the ceiling is set by the calling workflow or the organization, not by your job.

FAQ

Why does GitHub Actions say "Resource not accessible by integration"?

The GITHUB_TOKEN for that run does not have the permission scope the API endpoint requires. Add the matching scope, such as pull-requests: write, to the job's permissions: block.

Is it safe to set workflow permissions to "read and write" for all workflows?

It works, but it gives every job, including ones that run third-party actions, write access it does not need. Keep the repository default read-only and grant write per job.

Why can't my workflow comment on pull requests from forks?

Fork PRs run with a read-only GITHUB_TOKEN and no secrets by design. Post the comment from a separate workflow_run workflow instead.

Bottom line

If you own the repo and the failing job is a trusted internal one, add a job-level permissions: block with the single scope from the table and stop there. If the failure only happens for forks or Dependabot, restructure so the privileged step runs through workflow_run rather than loosening the token. If you need cross-repository writes or chained workflow triggers, move to a GitHub App token. Treat "read and write for everything" as a last resort, not a fix.

Vercel vs Netlify vs Cloudflare Pages: Which Meter Will Your Frontend Hosting Bill Actually Grow On?

Libme — Sat, 07 Nov 2026 15:14:07 +0000

The headline plan price rarely decides your frontend hosting bill. Three meters do: outbound bandwidth, server-side compute (functions, SSR, image optimization), and per-seat charges. Cloudflare Pages is the safest choice when bandwidth is the risk, Vercel is the smoothest when you run Next.js with SSR and a team, and Netlify sits in between with a model that has changed enough that you should re-read it before committing.

All plan details below are as of November 2026. These vendors revise pricing often, so use this post for the structure of the decision and confirm current numbers on each pricing page.

What is a frontend hosting bill actually made of?

Almost every overage I have seen traces back to one of four line items:

Bandwidth (egress): bytes served from the CDN to browsers. Big images, video, and source maps nobody needed to ship all land here.
Compute: serverless function invocations and duration, SSR rendering, middleware/edge execution, and image optimization requests.
Build minutes and concurrency: how long CI-style builds take and how many can run at once.
Seats: some plans charge per team member who can deploy, even if they only deploy once a month.

A static marketing site is dominated by item 1. A Next.js app with server components and personalised pages is dominated by item 2. A five-person agency with forty small sites is dominated by item 4. Pick the host that is cheapest on your dominant meter, not the one with the nicest free tier.

Takeaway: identify your dominant meter first; comparing plan sticker prices before that is comparing the wrong number.

How do the three differ on each meter?

Meter	Vercel	Netlify	Cloudflare Pages
Bandwidth	Included allowance per plan, metered overage beyond it	Included allowance, metered beyond it (pricing model revised in recent years, check current)	Static asset requests are not metered for bandwidth
Server compute	Metered functions (invocations, duration, memory); strongest Next.js support	Metered functions and edge functions	Pages Functions run on Workers and bill by request/CPU time on the Workers plan
Seats	Paid plans charge per deploying member	Paid plans have team-member pricing	Seats are not the lever; account-level plan
Fit for Next.js	Native, first-party	Supported through an adapter/runtime	Supported via adapters; some features lag or behave differently
Main drawback	Overages and per-seat cost surprise growing teams	Credit/plan model needs reading carefully; costs hard to predict	Smaller feature surface for framework-specific SSR behaviour

The pattern is that Cloudflare monetises compute and add-on products rather than static egress, while Vercel and Netlify meter the delivery itself. That is why a viral image-heavy page can be a non-event on one and a surprise invoice on another.

Takeaway: if your site is mostly static assets and traffic is unpredictable, an unmetered-egress host removes the worst tail risk.

When does SSR or image optimization become the real cost?

This is where I have seen Next.js projects get caught. Rendering a page on every request means every visitor consumes compute, and image optimization means each unique width/format combination can be a billable transformation on hosts that meter it.

Before you migrate or upgrade, find out which routes are actually dynamic. In a Next.js project, the build output lists them:

# After `next build`, the route table marks each route.
# ○ = static, ● = SSG with data, ƒ = dynamic (server-rendered on demand)
npx next build 2>&1 | grep -E '^\s*(○|●|ƒ)'

Every ƒ route is a compute meter. Often a route is dynamic only because it reads cookies() or headers() somewhere in the tree, or because a fetch is uncached. Converting accidental dynamic routes to static or revalidated ones cuts the compute meter on every host, which is a bigger lever than switching providers.

For images, two honest options exist: keep the host's optimizer and cap the variants (a limited deviceSizes list in next.config.js), or pre-generate sizes at build time and serve them as plain static files. The second removes the meter entirely at the cost of a build step you maintain.

Takeaway: audit which routes are dynamic before blaming the host; accidental SSR is the most common avoidable compute cost.

How do I estimate my bill before I commit?

Do not trust intuition; plug your own numbers into a model with the rates from each pricing page. This small script makes the structure explicit. The rates are placeholders you must fill in yourself, deliberately not real prices.

from dataclasses import dataclass

@dataclass
class Plan:
    name: str
    base_monthly: float        # plan fee
    seats_included: int
    per_extra_seat: float
    gb_included: float
    per_extra_gb: float        # 0 if egress is unmetered
    fn_million_included: float
    per_extra_million_fn: float

def monthly_cost(p: Plan, seats: int, gb_out: float, fn_millions: float) -> float:
    extra_seats = max(0, seats - p.seats_included)
    extra_gb = max(0.0, gb_out - p.gb_included)
    extra_fn = max(0.0, fn_millions - p.fn_million_included)
    return (p.base_monthly
            + extra_seats * p.per_extra_seat
            + extra_gb * p.per_extra_gb
            + extra_fn * p.per_extra_million_fn)

# Fill from each vendor's current pricing page. These zeros are placeholders.
plans = [
    Plan("host_a", 0, 1, 0, 0, 0, 0, 0),
    Plan("host_b", 0, 1, 0, 0, 0, 0, 0),
]

# Run three scenarios: today, 3x traffic, and a viral spike month.
scenarios = {"today": (3, 200, 2), "3x": (3, 600, 6), "spike": (3, 5000, 10)}
for p in plans:
    for label, (seats, gb, fn) in scenarios.items():
        print(p.name, label, round(monthly_cost(p, seats, gb, fn), 2))

The scenario that matters is the spike month, not today. A host with a slightly higher base fee and unmetered egress often wins the spike scenario by a wide margin, while a host with generous bundled compute wins the dynamic-app scenario.

Takeaway: model today, 3x, and a spike month; the host that wins only "today" is the one that surprises you later.

Is it worth paying for the platform that matches my framework?

Sometimes. If you run Next.js with server components, ISR and middleware, the first-party host removes a class of "works locally, behaves differently deployed" problems, and that time has real value. If you ship a static Astro, Vite, or Hugo site, you are paying a premium for features you do not use, and an unmetered-egress host is the rational default.

Each option has a real downside. Vercel's per-seat pricing punishes teams where many people occasionally deploy, and its overages can arrive without a hard stop unless you configure spend controls. Netlify's pricing structure has been revised enough that older blog posts (including comparisons from a couple of years ago) are unreliable. Cloudflare Pages gives you the least friction on cost but, for framework-specific SSR features, you may hit behavioural differences from Node-based hosting because the runtime is not Node by default.

If you want the managed Next.js experience and accept the metering, Vercel is the host that handles SSR, ISR and image optimization without extra adapter work. If your risk is egress on a mostly static site, Cloudflare Pages is the one that does not bill you for serving static assets.

Takeaway: pay for framework fit only if you use the framework features that need it.

FAQ

Is Cloudflare Pages really free for unlimited bandwidth?

Static asset requests on Pages are not metered for bandwidth as of November 2026, but Pages Functions run on Workers and are subject to Workers request and CPU limits and pricing. Check the current limits for your plan.

Why is my Vercel bill higher than the plan price?

Usually usage beyond the included allowances (bandwidth, function duration, image optimization) or additional paid seats. Open the usage dashboard and sort by line item to see which one grew.

Can I move a Next.js app off Vercel to avoid costs?

Yes, but test every dynamic feature (middleware, ISR, image optimization) on the target host first, since adapters and runtimes do not always match Vercel's behaviour exactly.

Bottom line

Static or image-heavy sites with unpredictable traffic fit Cloudflare Pages, because egress is the risk and it is not metered there. Next.js teams that use SSR, ISR and middleware heavily will usually spend less engineering time on Vercel, as long as they set spend limits and count seats honestly. Netlify is worth a fresh look if you value its build and form tooling, but read its current pricing model rather than relying on older comparisons. Whatever you choose, audit your dynamic routes first; that fixes more of the bill than any migration.

LLM Returns Invalid JSON Once in a While: Structured Outputs vs Validate-and-Retry

Libme — Fri, 06 Nov 2026 22:31:25 +0000

TL;DR: If json.loads() on model output fails only occasionally, there are three distinct causes: truncation, stray prose or code fences around the JSON, and schema drift (valid JSON, wrong shape). Provider-side structured outputs (constrained decoding) eliminate the second and third for supported schemas; a validate-and-retry loop with a typed model covers everything else, including the first. In production you usually want both.

What does the failure actually look like?

The symptom that sends people to a search engine is usually one of these:

json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
json.decoder.JSONDecodeError: Unterminated string starting at: line 1 column 140 (char 139)
pydantic_core._pydantic_core.ValidationError: 1 validation error for Ticket
priority
  Input should be less than or equal to 4 [type=less_than_equal, ...]

They look alike but have different fixes:

Symptom	Real cause	Right fix
`Expecting value: line 1 column 1`	Model wrapped the JSON in a code fence or added "Here is the JSON:"	Constrained decoding, or extract the outermost braces
`Unterminated string` / `Expecting ',' delimiter` near the end	Output hit `max_tokens` mid-object (truncation)	Raise the limit, shrink the schema, check the stop reason
Parses fine, then your code crashes on a missing key or odd enum value	Schema drift: valid JSON, wrong shape	Typed validation, or a strict schema on the provider side

The first thing to do before changing any prompt: log the raw response and the stop/finish reason for every failure. Most "the model is flaky" reports I have seen turned out to be truncation, which no prompt wording can fix.

Takeaway: classify the failure from the raw text and the stop reason before you touch the prompt.

Step 1: Validate with a typed model instead of `json.loads`

Parsing and validating in one step gives you an error message precise enough to feed back to the model. This uses Pydantic v2 and works with any provider, because call_model is just a function you supply.

from typing import Callable, Literal
from pydantic import BaseModel, Field


class Ticket(BaseModel):
    category: Literal["bug", "billing", "feature_request"]
    priority: int = Field(ge=1, le=4)
    summary: str = Field(min_length=1, max_length=200)


def extract_json_object(raw: str) -> str:
    """Strip code fences / chatter by taking the outermost braces."""
    start, end = raw.find("{"), raw.rfind("}")
    if start == -1 or end <= start:
        raise ValueError("no JSON object found in response")
    return raw[start : end + 1]


def parse_ticket(raw: str) -> Ticket:
    return Ticket.model_validate_json(extract_json_object(raw))

The brace extraction is a blunt tool: it will not rescue truncated output, and it can mis-slice if the model emits two objects. That is fine, because the retry loop below catches those cases.

Step 2: Retry with the error, not with the same prompt

Resending the identical request is a coin flip. Sending the validation error back gives the model something concrete to correct. In Pydantic v2, ValidationError subclasses ValueError, so one except covers both parse and schema failures.

Message = dict[str, str]


def extract_ticket(
    call_model: Callable[[list[Message]], str],
    prompt: str,
    max_attempts: int = 3,
) -> Ticket:
    messages: list[Message] = [{"role": "user", "content": prompt}]
    last_error: ValueError | None = None

    for _ in range(max_attempts):
        raw = call_model(messages)
        try:
            return parse_ticket(raw)
        except ValueError as exc:
            last_error = exc
            messages += [
                {"role": "assistant", "content": raw},
                {
                    "role": "user",
                    "content": (
                        "That response failed validation:\n"
                        f"{exc}\n"
                        "Reply with only a corrected JSON object."
                    ),
                },
            ]

    raise RuntimeError(f"no valid output after {max_attempts} attempts") from last_error

Two details that matter. Cap attempts at 2–3: each retry costs a full round trip and re-sends the growing conversation, so an uncapped loop turns one bad input into a bill spike. And if the failure was truncation, retrying with the same max_tokens will truncate again, so check the stop reason and raise the limit instead of looping.

Takeaway: a retry that includes the exact validation error converges; a retry that repeats the prompt just re-rolls the dice.

When do provider-side structured outputs replace the loop?

Most major providers now offer some form of schema-constrained output: OpenAI's Structured Outputs with a strict JSON schema, tool/function calling with an input schema on Anthropic's API, and grammar- or schema-constrained decoding in local runtimes such as llama.cpp and in libraries like Outlines. The model's token choices are restricted so the output cannot violate the schema's syntax. As of late 2026 the details differ per provider, so check current docs for supported schema features.

What this buys you: no code fences, no prose preamble, no invalid enum values, no missing required keys. If you want this without writing the plumbing yourself, OpenAI's Structured Outputs is the option that guarantees schema adherence on a strict schema, at the cost of being tied to that API.

What it does not buy you:

Truncation is still possible. If generation stops at the token limit, the object is still cut off. Strict mode does not make that safe.
Only a schema subset is supported. Constraints like minLength, maximum, or deeply recursive structures are often unsupported or ignored, so your Field(ge=1, le=4) check may still need to run in your own code.
Syntax is not semantics. A schema-valid priority: 4 can still be the wrong priority. Constrained decoding makes the shape certain, not the content correct.
First-request latency. Some providers compile the schema on first use and cache it, so a rarely used schema can be slower on its first call.

Takeaway: constrained decoding guarantees shape, not truncation-safety and not correctness, so keep your own validation behind it.

Which approach should you use?

Situation	Use
Hosted model, simple flat schema	Provider structured outputs plus Pydantic validation
Provider lacks strict mode, or schema uses unsupported constraints	Validate-and-retry loop
Local model (llama.cpp, Ollama-style runtimes)	Grammar/schema-constrained decoding if your runtime supports it, retry loop as backstop
Long outputs (big arrays, many records)	Split into smaller calls; truncation risk grows with output length

FAQ

Why does the LLM return invalid JSON only sometimes?
Sampling is probabilistic, so a model that follows your format 98% of the time will still occasionally add a code fence, stop mid-object at the token limit, or pick an off-schema value. Intermittent failures are normal without constrained decoding.

Does asking for JSON in the prompt guarantee valid JSON?
No. A prompt instruction raises the odds but guarantees nothing; only schema-constrained decoding restricts the output tokens, and even then truncation can break the object.

How many retries should I allow for LLM JSON parsing?
Two or three. If the third attempt still fails, the cause is usually truncation or an ambiguous schema, and more retries will not help.

Bottom line

If you call a hosted model with a simple schema, turn on the provider's strict structured output and keep a Pydantic model behind it for the constraints the schema cannot express. If you are on a provider or local model without it, the validate-and-retry loop above is enough for most small-team workloads. Either way, log the raw response and stop reason so you can tell truncation from drift. Cap retries, because the loop's failure mode is cost, not correctness.

Docker Build Secrets: Why ARG and COPY Leak Tokens Into Image History, and the BuildKit Fix

Libme — Thu, 05 Nov 2026 18:32:32 +0000

If you pass an npm token, pip index password, or SSH key into docker build with ARG, ENV, or COPY, it can be recovered from the image afterward, even if a later step deletes it. The fix is a BuildKit secret mount (RUN --mount=type=secret), which exposes the secret to one RUN step as a temporary file and never writes it into a layer. Below: how to prove the leak on your own image, the replacement pattern, and the traps that remain.

How do I check whether my image already leaks a secret?

Start with the evidence, not the theory. Take a Dockerfile that looks reasonable:

FROM node:22-slim
WORKDIR /app
ARG NPM_TOKEN
COPY package*.json ./
RUN echo "//registry.npmjs.org/:_authToken=${NPM_TOKEN}" > .npmrc \
 && npm ci \
 && rm .npmrc
COPY . .

The author deleted .npmrc in the same step, so it feels safe. Build it and inspect:

docker build --build-arg NPM_TOKEN=npm_fake_example_token -t leaky .
docker history --no-trunc leaky | grep -i npm_token

The ARG value appears in the recorded command of that layer in the history output, because the expanded RUN instruction is part of the image metadata. Anyone who can pull the image can run docker history or docker inspect on it.

Recent Docker versions also warn at build time with a check named SecretsUsedInArgOrEnv ("Do not use ARG or ENV instructions for sensitive data"). It is name-based, so it catches NPM_TOKEN or PASSWORD but will not catch a secret in a variable called X. Treat it as a smoke alarm, not an audit.

The same applies to COPY .npmrc . followed by RUN rm .npmrc: the COPY layer is a tarball of the file, and deleting it in a later layer only hides it from the merged filesystem. Extract the layers with docker save and the file is there.

Takeaway: deleting a file in a later layer does not remove it from the image; it only masks it.

What does the BuildKit secret mount do differently?

A secret mount makes a file (or env value) available only while one RUN instruction executes. It is mounted on a tmpfs-style path, not committed into the layer, and not recorded as a build argument.

# syntax=docker/dockerfile:1
FROM node:22-slim
WORKDIR /app
COPY package*.json ./
RUN --mount=type=secret,id=npmrc,target=/root/.npmrc \
    npm ci
COPY . .

Build by pointing the id at a source on the host:

docker build --secret id=npmrc,src=$HOME/.npmrc -t safe .

If the token is in an environment variable instead of a file, BuildKit can read it from the environment of the build command:

export NPM_TOKEN=npm_fake_example_token
docker build --secret id=NPM_TOKEN,env=NPM_TOKEN -t safe .

RUN --mount=type=secret,id=NPM_TOKEN \
    echo "//registry.npmjs.org/:_authToken=$(cat /run/secrets/NPM_TOKEN)" > /tmp/.npmrc \
 && NPM_CONFIG_USERCONFIG=/tmp/.npmrc npm ci \
 && rm /tmp/.npmrc

By default a secret mounted without target appears at /run/secrets/<id>. Note the second form writes a temp file inside the layer filesystem, but it is removed within the same RUN, and the token was never in the instruction text, so neither docker history nor the committed layer contains it. If you can use the first form with a real file at target, prefer it: there is nothing to clean up.

Takeaway: a secret mount keeps the value out of both the instruction text and the layer contents, which is the two places ARG and COPY leak.

Which method leaks what?

Method	In `docker history`	In a layer tarball	Survives to final image	Use for secrets?
`ARG` + `--build-arg`	Yes, when used in `RUN`	Possibly, if written to a file	Metadata yes	No
`ENV`	Yes	n/a	Yes, and in running containers	No
`COPY` then `rm`	No	Yes	Masked, recoverable	No
Multi-stage, secret only in builder stage	In builder only	In builder only	No, if final stage does not copy it	Acceptable, with caveats
`RUN --mount=type=secret`	No	No	No	Yes
`RUN --mount=type=ssh`	No	No	No	Yes, for git over SSH

The multi-stage row deserves the caveat: the final image is clean, but the builder stage's layers still exist in your local build cache, and if you export cache with --cache-to to a registry, those layers can go with it. Which is why "multi-stage" alone is not a secrets strategy.

What about private git dependencies and CI?

For git clone or pip install git+ssh://... from a private repo, use the SSH mount rather than copying a key:

# syntax=docker/dockerfile:1
RUN --mount=type=ssh \
    pip install "git+ssh://git@github.com/your-org/private-lib.git"

docker build --ssh default -t app .

This forwards your local SSH agent for that step only, so no private key lands in a layer. You will also need github.com in known_hosts inside the build, which is the usual first failure: Host key verification failed.

In GitHub Actions with docker/build-push-action, the secrets input takes the same id=value pairs:

- uses: docker/build-push-action@v6
  with:
    context: .
    push: true
    tags: ghcr.io/your-org/app:latest
    secrets: |
      npmrc=${{ secrets.NPMRC }}

As of November 2026 this works with BuildKit-based builds, which is the default builder in current Docker releases. Pin the action to the major version you have tested.

Takeaway: the pattern is identical locally and in CI; only the source of the secret changes.

What still goes wrong after you switch?

A secret echoed into build output. If a step fails verbosely (npm ci with debug logging, pip -vvv), the token can appear in CI logs. Mask it in your CI system and keep verbosity off.
Secrets written to a path the tool caches. Some package managers copy credentials into a cache directory inside the layer. Check with docker run --rm safe sh -c 'find / -name ".npmrc" 2>/dev/null' after the build.
Old images are already published. Switching Dockerfiles does not un-leak what is in your registry. If an image with a real token was ever pushed, rotate the token first; deleting the tag is secondary.
Inspecting tools have limits. dive is useful for browsing layer contents interactively, but it is a manual check and will not scan for credentials. A dedicated secret scanner run against the saved image tarball is a better gate in CI.

If you want automated detection rather than eyeballing layers, TruffleHog is the open-source scanner that can check container images for verified credentials; its limitation is that it finds what looks like a known credential format, so an internal password with no recognizable pattern slips through.

FAQ

Does deleting a file in a later Dockerfile step remove it from the image?
No. Each instruction creates a new layer, and a delete only adds a "whiteout" marker in the new layer. The earlier layer still contains the file and can be extracted from the image.

Is multi-stage build enough to keep secrets out of the final image?
It keeps them out of the final image's layers if the final stage never copies them, but the builder layers remain in your build cache and possibly in exported cache. Use a secret mount so the value never enters a layer at all.

How do I use a secret in docker build without BuildKit?
You can't do it safely with the legacy builder. Enable BuildKit (the default in current Docker Engine and Docker Desktop) and use --secret with RUN --mount=type=secret.

Bottom line

If a credential is needed only to install dependencies during the build, use RUN --mount=type=secret for it, and --mount=type=ssh for private git over SSH. Stop using ARG and ENV for anything you would not paste into a public chat. If an image with a real secret was ever pushed, rotate the secret before you fix the Dockerfile.

ENOSPC: System Limit for Number of File Watchers Reached — Raise the Limit, Shrink the Watch, or Poll?

Libme — Wed, 04 Nov 2026 15:13:09 +0000

If your dev server dies with Error: ENOSPC: System limit for number of file watchers reached, it has nothing to do with disk space. You've hit the Linux kernel's inotify watch limit. You have three fixes: raise fs.inotify.max_user_watches on the host, make your tools watch fewer directories, or switch to polling when inotify can't work at all (some Docker bind mounts, network drives).

The right choice depends on which of those situations you're in, and the order matters. Raising the limit is the fastest fix, but it hides the real problem if a tool is watching node_modules for no reason.

What does ENOSPC mean when there's plenty of disk space?

ENOSPC is "no space left on device", but the kernel reuses it for inotify. The message usually looks like this:

Error: ENOSPC: System limit for number of file watchers reached, watch '/home/dev/app/src/components'
    at FSWatcher.<computed> (node:internal/fs/watchers:247:19)

Two different limits produce it:

Limit	What it caps	Typical symptom
`fs.inotify.max_user_watches`	Total watches per user (roughly one per watched directory)	`...System limit for number of file watchers reached, watch '<path>'`
`fs.inotify.max_user_instances`	Number of inotify handles per user	Same ENOSPC, but no path, or `inotify_init` failing

Linux inotify watches directories, not individual files, so the cost scales with directory count. A monorepo with several thousand folders, plus a bundler, a test runner, a TypeScript server and an editor all watching the same tree, can use up the limit quickly. As of kernel 5.11 the default max_user_watches scales with system RAM (up to about a million); older kernels and many distro configs default to 8192, which is where most people hit it.

Takeaway: ENOSPC from a watch call is an inotify quota error, and the quota is per user, shared by every tool you have running.

How do I see who is using all the watches?

Before you change anything, check the current limits and find the biggest consumer:

cat /proc/sys/fs/inotify/max_user_watches
cat /proc/sys/fs/inotify/max_user_instances

# Watches per process (run with sudo to see other users' processes)
for fd in $(find /proc/*/fd -lname 'anon_inode:inotify' 2>/dev/null); do
  pid=$(echo "$fd" | cut -d/ -f3)
  n=$(grep -c '^inotify' "/proc/$pid/fdinfo/$(basename "$fd")" 2>/dev/null)
  echo "$n $pid $(tr '\0' ' ' < "/proc/$pid/cmdline" | cut -c1-80)"
done | sort -rn | head

In my experience the top entry is rarely the thing you expected. An editor's file watcher, a second dev server you forgot about, or a Jest watch mode running with the whole repo as its root are common culprits. If one process owns most of the watches and it's watching node_modules or .git, fix that first (next sections) rather than raising the limit.

To estimate what a single tool needs, count directories it would actually watch:

find . -type d -not -path '*/node_modules/*' -not -path '*/.git/*' | wc -l

Takeaway: one process typically owns most of the watches, and finding it takes thirty seconds.

How do I raise the inotify limit permanently?

This is the standard fix on a Linux dev machine or CI runner. A one-off change:

sudo sysctl fs.inotify.max_user_watches=524288
sudo sysctl fs.inotify.max_user_instances=512

To persist across reboots, put it in a drop-in file:

printf 'fs.inotify.max_user_watches=524288\nfs.inotify.max_user_instances=512\n' \
  | sudo tee /etc/sysctl.d/99-inotify.conf
sudo sysctl --system

The numbers above are common community values, not magic. Each watch costs kernel memory (on the order of a kilobyte or two on 64-bit systems), and that memory is unswappable, so setting the limit to the maximum on a small VM is not free. Size it to your tree plus headroom.

Where you apply it matters. The limit lives in the kernel, so:

On a Linux host running containers, set it on the host. Containers share the host kernel.
On Docker Desktop for Mac or Windows, the "host" is the Linux VM Docker runs, not your laptop OS.
In WSL2, set it inside the WSL distribution.
In managed CI, you often can't change it, which pushes you toward the next two options.

Takeaway: raise max_user_watches on the machine whose kernel is actually doing the watching, and write it to /etc/sysctl.d/ so it survives a reboot.

How do I make my tools watch fewer directories?

Raising the limit treats the symptom. Excluding directories that never change at dev time treats the cause. Settings for the common tools (as of late 2026, check your version's docs):

Tool	Setting
Vite	`server.watch.ignored`
webpack / webpack-dev-server	`watchOptions.ignored`
Next.js	Uses webpack/Turbopack watchers; keep generated and output directories out of the project root where possible
Jest	`watchPathIgnorePatterns`, or `--no-watchman` to avoid the Watchman crawler
VS Code	`files.watcherExclude`
tsc `--watch`	`watchOptions.excludeDirectories` in `tsconfig.json`

For example, a Vite config that stops watching build output and a vendored directory:

// vite.config.ts
import { defineConfig } from 'vite';

export default defineConfig({
  server: {
    watch: {
      ignored: ['**/dist/**', '**/.turbo/**', '**/vendor/**'],
    },
  },
});

And for the TypeScript watcher:

{
  "watchOptions": {
    "excludeDirectories": ["**/node_modules", "**/dist"]
  }
}

A caveat from experience: ignoring a directory means edits there will not trigger a rebuild. If you symlink a local package into node_modules for development, excluding node_modules will make that package's changes invisible, so check before you ignore.

If you run several tools that each crawl the repo, Watchman is the one that gives you a single shared watcher service instead of one crawler per tool, which Jest and Metro can use. Its drawback is that on Linux it still uses inotify underneath, so it reduces duplication but does not remove the need for a sane limit, and it adds a daemon to maintain.

Takeaway: if a tool is watching node_modules or build output, exclude them in its config before you touch the kernel setting.

When should I switch to polling instead?

Polling stat-checks files on a timer, so it uses no inotify watches at all. It trades CPU and latency for reliability. Use it when inotify genuinely can't see changes, not as a reflex:

Bind mounts from macOS or Windows into a Linux container often don't deliver filesystem events reliably, so hot reload silently never fires.
Network filesystems (NFS, SMB) generally don't produce inotify events for changes made on other machines.
Locked-down CI where you can't change sysctl values.

Common switches:

# chokidar-based tools (older webpack setups, nodemon alternatives)
CHOKIDAR_USEPOLLING=true npm run dev

# Next.js (webpack watcher)
WATCHPACK_POLLING=true npm run dev

// vite.config.ts
export default defineConfig({
  server: { watch: { usePolling: true, interval: 300 } },
});

Polling a large tree at a short interval will visibly burn CPU. Pair it with the ignore lists above so you only poll source directories, and keep the interval as long as you can tolerate.

Takeaway: polling is the fallback for places where inotify events never arrive, and it needs an ignore list to stay cheap.

FAQ

How do I fix "ENOSPC: System limit for number of file watchers reached"?
Run sudo sysctl fs.inotify.max_user_watches=524288 and persist it in /etc/sysctl.d/99-inotify.conf. If a tool is watching node_modules or build output, exclude those paths too.

Does ENOSPC in Node mean my disk is full?
Not when the message mentions "file watchers" or a watch call. That is the inotify watch or instance limit. A genuinely full disk shows up in write operations, and df -h will confirm it.

Do I need to set the inotify limit inside the Docker container?
No, the container shares the host kernel, so set it on the Linux host or in the Docker Desktop VM. Containers typically can't change it with their own sysctl settings.

Bottom line

On a Linux machine you control, raise fs.inotify.max_user_watches via /etc/sysctl.d/, but first spend a minute finding which process holds the watches and excluding node_modules and build output from it. If you're on a bind mount or network drive where events never arrive, use polling with a tight ignore list. If you can't change kernel settings, shrinking the watched set is your only real lever.

Docker Desktop Alternatives on Mac: OrbStack vs Colima vs Podman Desktop

Libme — Tue, 03 Nov 2026 22:15:35 +0000

If you want the lowest-friction swap, OrbStack is the closest drop-in for Docker Desktop on a Mac, with the licensing caveat that commercial use is paid. If you want free and open source with a CLI-first workflow, Colima keeps your existing docker and docker compose commands working. Podman Desktop is the right pick when rootless, daemonless containers are a requirement rather than a preference.

This post is for the moment you are about to pay for Docker Desktop, or a licensing email has landed in your inbox, and you need to know what actually changes when you leave. Details are as of November 2026; licensing terms and features move, so check each vendor's page before you commit a team.

Why look for an alternative at all?

Three reasons come up repeatedly: licensing, resource usage, and file-sharing speed. Docker Desktop's subscription requirement applies to larger companies (Docker's published terms tie it to company size and revenue), while personal use, education, and small businesses are exempt. If you are a solo developer, the licence probably does not apply to you, and you should switch only for the other two reasons.

On macOS every option here runs Linux containers inside a lightweight VM. The differences are which VM layer they use, how they share files with the host, and how much of the Docker developer experience they reproduce.

Takeaway: on a Mac, "which Docker alternative" is really "which Linux VM manager and which file-sharing layer".

How do the three compare?

	OrbStack	Colima	Podman Desktop
Licence model	Proprietary, free for personal use, paid for commercial (check current terms)	Open source (MIT)	Open source (Apache 2.0)
Engine	Docker-compatible engine	Docker (default) or containerd, via Lima	Podman, daemonless
GUI	Native Mac app	None (CLI)	Electron app
`docker` CLI compatibility	Works out of the box	Works via a Docker context	Needs the socket compatibility mode or `podman` alias
Compose	`docker compose`	`docker compose`	`podman compose` or a Docker-compatible socket
Local Kubernetes	Built in, optional	`colima start --kubernetes`	Via kind/minikube integrations
Rootless by default	No	No	Yes
Platform	macOS only	macOS and Linux	macOS, Windows, Linux

Two things the table hides. First, OrbStack is closed source, so you are trusting one vendor instead of a community project. Second, Podman is not Docker with a different logo: most commands translate, but anything that talks directly to /var/run/docker.sock needs configuration.

Takeaway: the further you move from the Docker daemon, the more you pay in compatibility work.

How do you set up Colima without breaking existing scripts?

Colima is the option I would try first if you are comfortable in a terminal, because your existing commands stay the same.

brew install colima docker docker-compose

# vz = Apple's Virtualization framework; virtiofs is the faster mount type
colima start --cpu 4 --memory 8 --disk 60 --vm-type vz --mount-type virtiofs

docker context ls          # a "colima" context should now be active
docker run --rm hello-world

Plan on running colima start after every reboot, or register it as a Homebrew service with brew services start colima. If you forget, every docker command fails with a connection error against the socket, which is a confusing first-day symptom.

The most common breakage is tooling that hardcodes the default socket path. Anything that does not read your Docker context needs to be pointed at Colima's socket explicitly:

export DOCKER_HOST="unix://$HOME/.colima/default/docker.sock"
# Testcontainers has its own override for the socket it mounts into containers
export TESTCONTAINERS_DOCKER_SOCKET_OVERRIDE=/var/run/docker.sock

Without those, the symptom I would look for is a test suite that works under Docker Desktop and fails at startup with "Could not find a valid Docker environment".

The honest drawback: Colima has no GUI and fewer guardrails. The VM's CPU, memory, and disk are fixed at start time, so if your builds run out of memory you restart the VM with new flags rather than dragging a slider.

For a developer who wants to keep the Docker CLI and pay nothing, Colima is the option that changes the least while removing the subscription question.

Takeaway: Colima keeps your commands, and the only migration cost is socket paths in tools that bypass your Docker context.

When is OrbStack worth it?

OrbStack's pitch is speed and a native feel: quick startup, lower idle resource use, and fast file sharing, with a Mac-native app instead of an Electron one. In my experience the practical difference is largest on bind-mount-heavy workloads, such as a Node project with a large node_modules mounted into a container. Benchmark it on your own project before believing anyone's numbers, including mine; I am not quoting figures because they vary heavily with the repo and the Mac.

If you want a fast, polished Docker Desktop replacement and your employer will pay for the commercial licence, OrbStack is the one that feels least like a compromise. Its limits are real: macOS only (so your team's Linux and Windows developers need something else), closed source, and a licence you have to track.

Takeaway: OrbStack buys polish and speed, and the price is a vendor dependency and a licence check for commercial use.

When does Podman Desktop make sense?

Choose Podman when the security model is the point: no root daemon, rootless containers by default, and a CLI that mirrors Docker closely enough that alias docker=podman covers most daily use.

brew install podman
podman machine init --cpus 4 --memory 8192
podman machine start
podman run --rm docker.io/library/hello-world

Note the fully qualified image name. Podman may prompt you to pick a registry for short names, which breaks non-interactive scripts until you configure unqualified-search-registries. Rootless mode also means you cannot bind ports below 1024 on Linux hosts without extra setup, and docker-compose-based projects sometimes need adjustment. If your team ships to environments where Podman is already the runtime, the parity benefit is genuine.

Takeaway: pick Podman for rootless and runtime parity, not as a licensing workaround, because it is the biggest behavioural change of the three.

How should you decide?

Your situation	Pick
Solo developer or small business under Docker's licence threshold	Stay on Docker Desktop, or Colima if you prefer CLI
Company that needs to cut licence cost, team comfortable in a terminal	Colima
Company that will pay for polish and speed, macOS-only team	OrbStack
Security or compliance requires rootless, or production runs Podman	Podman Desktop
Mixed Mac, Windows, Linux team needing one tool	Podman Desktop or Docker Desktop

FAQ

Is Colima a drop-in replacement for Docker Desktop?

For the docker and docker compose CLI, yes, since Colima runs a real Docker engine in a VM and exposes it through a Docker context. It is not a drop-in for the GUI, Docker extensions, or tools that hardcode the default socket path.

Is Docker Desktop free?

Docker's terms make it free for personal use, education, open source, and small businesses, and paid for larger companies. Check Docker's current subscription terms for the exact thresholds, since they are the ones that apply to you.

Is OrbStack faster than Docker Desktop?

It is commonly reported as faster at startup and file sharing, but the gain depends on your workload, so run your own build and test loop under both before deciding.

Bottom line

Solo developers under the licence threshold have no urgent reason to leave Docker Desktop. If you do want to leave, Colima is the safest default because your commands, Compose files, and mental model all carry over. If you want the smoothest Mac experience and someone else pays, OrbStack is the one to try. Choose Podman Desktop only when rootless containers or Podman parity in production is the actual requirement.

pip "externally-managed-environment" Error: venv, pipx, or uv?

Libme — Mon, 02 Nov 2026 22:16:10 +0000

If pip install fails with error: externally-managed-environment, your operating system's package manager owns that Python, and pip is refusing to write into it (PEP 668). The fix is almost never --break-system-packages: use a virtual environment for project dependencies, pipx or uv tool for command-line applications, and reserve the override flag for throwaway containers.

What does the error actually say, and why does it happen?

On a recent Debian, Ubuntu, or Homebrew-installed Python, this is what you get:

$ pip install requests
error: externally-managed-environment

× This environment is externally managed
╰─> To install Python packages system-wide, try apt install
    python3-xyz, where xyz is the package you are trying to
    install.

    If you wish to install a non-Debian-packaged Python package,
    create a virtual environment using python3 -m venv path/to/venv.
    ...
note: If you believe this is a mistake, please contact your Python installation or OS distribution provider. You can override this, at your own risk, by passing the --break-system-packages flag.

The mechanism is a marker file. Distributions that ship PEP 668 support place a file named EXTERNALLY-MANAGED in the interpreter's standard library directory, and pip checks for it before any install into that environment. You can see it yourself:

python3 -c "import sysconfig, pathlib; p = pathlib.Path(sysconfig.get_path('stdlib')) / 'EXTERNALLY-MANAGED'; print(p, p.exists())"

The reason it exists is real: apt and pip both write into the same site-packages, and when pip upgrades a library that an OS tool depends on, the OS tool can break in ways that are miserable to diagnose. As of late 2026 this behavior is the default on current Debian and Ubuntu releases, and Homebrew's Python applies it too. It is not a pip bug.

Takeaway: the error means "this interpreter belongs to your OS", and the right response is to stop installing into it, not to silence the check.

Which dead ends should you skip?

These are the ones I see suggested most often, and why they cost you later:

sudo pip install ... does not bypass the check, and if it did, you would be writing into the exact directory the protection exists for.
Deleting the EXTERNALLY-MANAGED file works until the next Python package update puts it back, and in the meantime you have turned off the guard for every user on the machine.
pip install --user is blocked by the same marker on most setups, so it does not help.
--break-system-packages works, and on a disposable machine that is fine. On a laptop or server you care about, it reintroduces the exact apt-versus-pip conflicts PEP 668 prevents.

Takeaway: every shortcut here trades a one-minute fix now for an unexplainable breakage of an OS tool later.

Which tool fits which job?

The right choice depends on what you are installing, not on personal taste.

Situation	Best fit	Why
Dependencies for a project you are developing	`venv` (or `uv venv`)	One isolated environment per project, reproducible from a requirements file
A CLI tool you just want on your PATH (e.g. a formatter, `httpie`)	`pipx` or `uv tool install`	Each app gets its own hidden environment, the command lands on PATH
Many projects, slow installs, lockfile wanted	`uv`	Fast resolver and installer, handles Python versions too
Docker image or CI job that is thrown away	`venv`, or the override flag	The OS-protection argument is weaker in a disposable root filesystem
A library packaged by your distro	`apt install python3-<name>`	Managed and patched by the OS, but often older

Takeaway: libraries go in a project venv, applications go in an isolated tool environment, and neither goes in the system interpreter.

How do you set up a venv correctly (including the Debian gotcha)?

The standard library route needs no extra tools:

python3 -m venv .venv
source .venv/bin/activate
pip install requests

On Debian and Ubuntu you may hit a second error first: The virtual environment was not created successfully because ensurepip is not available. Debian splits venv into its own package, so run sudo apt install python3-venv and retry.

Do not skip source .venv/bin/activate. If you forget, pip resolves to the system one and you are back at the original error. When in doubt, call the venv's interpreter explicitly: .venv/bin/python -m pip install requests.

If you want the managed version of this workflow, uv is the tool that creates the environment and installs from a requirements file in one fast step:

uv venv
uv pip install -r requirements.txt

A genuine drawback: uv is a separate binary you have to install and trust, and its uv pip interface follows pip closely but not identically, so edge cases in older requirements files can resolve differently than they did under pip. Test before you swap it into CI.

Takeaway: the venv is only protecting you if it is active or invoked by path; most "it still fails" reports are an unactivated shell.

How do you install command-line tools without polluting anything?

For applications, pipx creates one isolated environment per tool and exposes only its executables:

sudo apt install pipx   # or: brew install pipx
pipx ensurepath
pipx install httpie

If you want the one that handles isolated tool installs without a separate package from your distro, uv tool install httpie does the same job with the same one-environment-per-app model.

The limitation of both: they are for applications. If you pipx install a library and then import it in your own script, it will not be importable, because the library lives in its hidden environment. That is by design, and it is the most common confusion.

Takeaway: if you will type its name in a terminal, use pipx or uv tool; if you will import it, use a venv.

When is `--break-system-packages` acceptable?

In a Dockerfile or CI runner that is rebuilt from scratch every time, there is no OS tool left to protect, and a venv adds a layer for little benefit. Setting ENV PIP_BREAK_SYSTEM_PACKAGES=1 in the image is a reasonable, explicit choice there. Even so, a venv in the image keeps the same code path you use locally, which makes "works on my machine" bugs rarer. Choose the override flag knowingly, not because a forum answer said so.

Takeaway: the override flag is a tool for disposable filesystems, not for your daily laptop.

FAQ

How do I fix "error: externally-managed-environment" on Ubuntu or Debian?

Create a virtual environment with python3 -m venv .venv, activate it, and run pip inside it. If venv creation fails, install python3-venv with apt first.

Is `pip install --break-system-packages` safe?

It is safe on a throwaway container or CI runner. On a machine where apt manages Python tools, it can overwrite files that OS utilities depend on and cause breakage that is hard to trace.

Should I use pipx or uv to install Python command-line tools?

Either works: both give each application its own isolated environment and put its command on your PATH. Choose uv tool if you already use uv for projects, otherwise pipx is available from most package managers.

Bottom line

Use a venv (or uv venv) for anything you import, pipx or uv tool install for anything you run from a terminal, and apt install python3-<name> only when you want the distro-maintained version. Treat --break-system-packages as a Docker-and-CI setting. If you are on a team, put the one-line environment setup in the README so nobody rediscovers this error on day one.

DEV Community: Libme

asyncio.gather Raised an Exception but the Other Tasks Kept Running: TaskGroup vs return_exceptions

What actually happens when gather fails?

What do people try first (and why it falls short)?

How does TaskGroup change this?

When is return_exceptions=True the right choice?

Why do fire-and-forget tasks disappear mid-run?

What if you are stuck below Python 3.11?

FAQ

Bottom line

Related reading

LLM Provider Returns 529/503 Overloaded: LiteLLM vs OpenRouter vs a Hand-Rolled Fallback

Why does retrying the same provider not fix a 529 or 503?

What should trigger a failover, and what should not?

How does LiteLLM handle fallbacks?

How does OpenRouter handle fallbacks?

When is a hand-rolled fallback enough?

Comparison

FAQ

Should I retry or fail over when my LLM API returns 529 overloaded?

Does failing over to a different model change my output quality?

Can I resume a streamed LLM response on another provider after it fails midway?

Bottom line

Related reading

SQLAlchemy MissingGreenlet in Async Code: Fix Lazy Loading With selectinload, awaitable_attrs, or lazy="raise"

Why does MissingGreenlet happen?

What are the three ways to fix it?

1. Eager-load at query time (default choice)

2. awaitable_attrs for one-off access

3. lazy="raise" to make the mistake impossible to miss

Why do I also get MissingGreenlet after commit()?

How do I debug which attribute triggered it?

FAQ

Why does SQLAlchemy say "greenlet_spawn has not been called"?

Should I use selectinload or joinedload with async SQLAlchemy?

Is expire_on_commit=False safe?

Bottom line

Related reading

release-please vs Changesets vs semantic-release: Which Release Automation Fits a Solo Dev or Small Team?

Where does the version bump decision come from?

How does release-please work in practice?

When is Changesets the better fit?

Is semantic-release too much for a small project?

Decision table: which one for which situation?

FAQ

Bottom line

Related reading

Postgres Full-Text Search vs Meilisearch vs Typesense: Do You Need a Search Service for a Small App?

What does Postgres full-text search actually give you?

Where does Postgres search fall short?

How do Meilisearch and Typesense differ?

What is the hidden cost of adding a search service?

Which one should you pick?

FAQ

Bottom line

Related reading

GitHub Actions "Resource not accessible by integration": Fix It With the Smallest GITHUB_TOKEN Permission

What does the error actually look like?

Why did it work last month and fail now?

Which permission does each step need?

How do I scope it to one job?

Why does it still fail on fork pull requests?

What about reusable workflows and organization policy?

FAQ

Why does GitHub Actions say "Resource not accessible by integration"?

Is it safe to set workflow permissions to "read and write" for all workflows?

Why can't my workflow comment on pull requests from forks?

Bottom line

Related reading

Vercel vs Netlify vs Cloudflare Pages: Which Meter Will Your Frontend Hosting Bill Actually Grow On?

What is a frontend hosting bill actually made of?

How do the three differ on each meter?

When does SSR or image optimization become the real cost?

How do I estimate my bill before I commit?

Is it worth paying for the platform that matches my framework?

FAQ

Is Cloudflare Pages really free for unlimited bandwidth?

Why is my Vercel bill higher than the plan price?

Can I move a Next.js app off Vercel to avoid costs?

Bottom line

2. `awaitable_attrs` for one-off access

3. `lazy="raise"` to make the mistake impossible to miss

Step 1: Validate with a typed model instead of `json.loads`

When is `--break-system-packages` acceptable?

Is `pip install --break-system-packages` safe?