DEV Community: Sid Probstein

Moving Docker images between repos with crane (after imagetools wasted my afternoon)

Sid Probstein — Thu, 11 Jun 2026 17:55:15 +0000

I had a freshly-built, multi-arch dev image (linux/amd64 + linux/arm64) and one job: promote it into a private partner repo on Docker Hub, plus stamp a dated tag so :latest is always traceable back to a real build. Cross-repo. Should be five minutes.

It was not five minutes.

The problem

The obvious tool is docker buildx imagetools create... it's built for copying manifests between tags. So I reached for it. And it sat there. Then it 400'd. Then I retried, and it hung. Cross-repo blob copies on Docker Hub are reproducibly flaky with imagetools, and I burned the better part of an hour confirming that before I went looking for something else.

The fallback most people reach for next is worse: pull the image down, retag it, push it to the new repo. That round-trips the entire image — every layer, every arch - through your laptop's daemon and disk, only to push the same bits back up. And if you're not careful with how you tag, you flatten a multi-arch index down to whatever single arch your machine happens to be. No thanks.

Why crane

crane (from Google's go-containerregistry) does the copy registry-to-registry. It never pulls the image to your machine.

It preserves the multi-arch manifest. crane cp copies the whole image index by digest. Both arches come along. Nothing gets flattened.
It actually works cross-repo on Docker Hub. Where imagetools 400'd and hung, crane did the server-to-server copy in seconds.
No daemon, no disk. It talks to the registries directly. Your laptop just orchestrates.

flowchart LR
    subgraph slow["pull → tag → push"]
        A[source repo<br/>you/app:dev] --> L[your daemon + disk<br/>whole image] --> B[partner repo<br/>you/partner-app:latest]
    end
    subgraph crane["crane cp"]
        C[source repo<br/>you/app:dev] -->|manifest + blobs<br/>by digest| D[partner repo<br/>you/partner-app:latest]
    end

Run it as a container, no install

docker run --rm gcr.io/go-containerregistry/crane:debug <crane-args>

That's the whole installation story. Nothing on the host.

Auth issues (MacOS)

crane in the container reads ~/.docker/config.json. On macOS with Docker Desktop, your login isn't in that file — it's credsStore: osxkeychain, sitting in the macOS keychain. So if you naively mount ~/.docker/config.json into the container, crane sees no usable credential and hands you UNAUTHORIZED.

The fix: pull the credential out of the keychain on the host, write it into a temporary inline config, mount that, and delete it the moment you're done. Never print it, never commit it.

TMP=$(mktemp -d); trap 'rm -rf "$TMP"' EXIT
cat > "$TMP/mkcfg.py" <<'PY'
import json, sys, base64
d = json.load(sys.stdin)
auth = base64.b64encode((d["Username"] + ":" + d["Secret"]).encode()).decode()
json.dump({"auths": {"https://index.docker.io/v1/": {"auth": auth}}}, open(sys.argv[1], "w"))
PY
echo "https://index.docker.io/v1/" | docker-credential-osxkeychain get | python3 "$TMP/mkcfg.py" "$TMP/config.json"

CRANE() { docker run --rm -v "$TMP/config.json":/root/.docker/config.json:ro \
            gcr.io/go-containerregistry/crane:debug "$@"; }

The trap ... EXIT cleans up the temp config when your shell exits, so the credential doesn't linger.

On Linux (or anywhere docker login writes inline creds), skip all of that and mount ~/.docker/config.json directly. And obvious-but-worth-saying: you need a read/write login on the destination. A read-only token can't push.

Promote the image

# copy SRC -> DST, full multi-arch manifest, server to server:
CRANE cp you/app:dev \
         you/partner-app:latest

# also stamp a dated, immutable tag so :latest is always traceable to a build:
CRANE cp you/app:dev \
         you/partner-app:dev-2026-06-11

Two copies, both registry-side, both done before you can refill your coffee.

Verification

Don't trust that the copy worked. Prove it. The destination digest must equal the source digest — same bits, same manifest:

CRANE digest you/app:dev                 # source
CRANE digest you/partner-app:latest      # must equal source
CRANE digest you/partner-app:dev-2026-06-11   # must equal source

If those three lines match, the right image landed under both tags. If they don't, you copied the wrong thing — better to find out here than in a partner's deployment.

Bonus

CRANE ls you/partner-app                          # list tags
CRANE manifest you/app:dev                         # full manifest JSON, see the arches
CRANE tag you/app@sha256:<digest> newtag           # add a tag to an existing digest
CRANE copy <SRC> <DST>                             # alias of cp

One caveat worth knowing: tag deletion isn't a crane operation. Use the Docker Hub UI or the API for that (and it needs an admin-scoped token — a read/write PAT won't delete).

TL;DR

Reach for crane any time you're moving an image between registries or repos and you care about the manifest arriving intact — promotions, mirrors, cross-org handoffs. It skips the daemon, skips the disk, and it doesn't fall over on Docker Hub cross-repo copies the way imagetools does. And whatever you do, build the crane digest source-equals-dest check into the workflow. It costs two seconds.

Running PyTorch fork-safe in Celery on macOS

Sid Probstein — Mon, 01 Jun 2026 13:23:46 +0000

If you've ever seen this in your Celery logs:

Process 'ForkPoolWorker-7' pid:32839 exited with 'signal 11 (SIGSEGV)'
billiard.exceptions.WorkerLostError:
    Worker exited prematurely: signal 11 (SIGSEGV) Job: 0.

...and the macOS crash report buries the real message in a JSON blob:

"asi": {
  "CoreFoundation": ["*** multi-threaded process forked ***"],
  "libsystem_c.dylib": ["crashed on child side of fork pre-exec"]
}

...you've hit one of the classic fork-after-init traps. Here's what's going on and how to actually fix it.

The one-line fix (if you're in a hurry)

Set these env vars before the Celery worker's MainProcess imports anything heavy:

# <your-project>/celery.py ... the very first thing the worker imports
import os

for var in ("OPENBLAS_NUM_THREADS", "OMP_NUM_THREADS", "MKL_NUM_THREADS",
            "NUMEXPR_NUM_THREADS", "VECLIB_MAXIMUM_THREADS"):
    os.environ.setdefault(var, "1")

os.environ.setdefault("OBJC_DISABLE_INITIALIZE_FORK_SAFETY", "YES")

The first set forces every BLAS library to single-threaded mode. VECLIB_MAXIMUM_THREADS is the one most people forget; it covers Apple's Accelerate framework, which is what PyTorch uses by default on Apple Silicon. The last one tells the Objective-C runtime to skip its fork-safety abort.

Why this happens

PyTorch's nn.Linear on macOS arm64 calls into Apple Accelerate, which does its parallel matmuls via libdispatch (Grand Central Dispatch).

The first BLAS call lazily spins up a pool of libdispatch worker queues in the calling process.

If that "calling process" is your Celery worker's MainProcess (say, because something during boot does a tiny matmul: spaCy preload, an embedding warmup, anything that imports numpy and runs a real op), those queues now live in the parent.

When the prefork pool then fork()s a child, the child inherits broken queue handles. The next BLAS call from inside the child dereferences a stale pointer and you get the SIGSEGV.

The stack trace in the crash report makes it unambiguous:

0: _dispatch_apply_with_attr_f      (libdispatch)
1: dispatch_apply_with_attr         (libdispatch)
3: cblas_sgemm                      (Accelerate)
5: at::native::cpublas::gemm        (libtorch_cpu)
6: at::native::addmm_impl_cpu_      (libtorch_cpu)
7: at::native::linear               (libtorch_cpu)
8: torch::autograd::THPVariable_linear

What doesn't work

"Just lazy-load the model in the child." Even if you defer from_pretrained until you're inside a forked child, that first call still hits Accelerate BLAS, and the dispatch queues your child inherited from the parent are already broken.

"Just bypass sentence_transformers.CrossEncoder.predict() and use bare-torch." Same story. Whether you go through CrossEncoder or call AutoModelForSequenceClassification directly, the SIGSEGV is one frame down inside linear().

"Just don't import torch at the top of the module." Necessary but not sufficient. In our case, removing import torch from ai_provider.py was real progress, but then we discovered litellm transitively pulls torch the first time you call it. Every "warmup" preload that touched litellm still poisoned the parent. You have to audit every code path that runs before the first fork.

The defensive pattern that does work

Defer heavy imports. Don't import torch at module top in anything that's part of the Celery autodiscovery chain. Push it into the function that needs it:

# Bad ... taints anyone who imports this module
import torch

def rerank(query, documents):
    with torch.no_grad():
        ...

# Good ... torch only loads in workers that actually rerank
def rerank(query, documents):
    import torch
    with torch.no_grad():
        ...

Gate "warmup" preloads off the Celery worker. Preloading models at startup makes sense for an ASGI server like Daphne. It is actively harmful in a forking Celery worker, because the warmup runs in MainProcess:

class MyAppConfig(AppConfig):
    def ready(self):
        is_celery_worker = "celery" in sys.argv and "worker" in sys.argv
        if not is_celery_worker:
            self._preload_cross_encoder()

What about Linux / Docker?

Yes, this affects Linux too, just less dramatically.

OpenBLAS and MKL both spin up thread pools on first use that don't survive fork; the typical Linux failure mode is a hang or a deadlock rather than a SIGSEGV.

The good news: the same *_NUM_THREADS=1 env vars are the fix.

VECLIB_MAXIMUM_THREADS and OBJC_DISABLE_INITIALIZE_FORK_SAFETY are no-ops on Linux, so the snippet above is portable. The deferred-import and gate-off-warmup patterns apply unchanged.

The AI agent stack that’s quietly taking over enterprise workflows

Sid Probstein — Sat, 03 May 2025 02:58:27 +0000

Accenture, IBM, and AWS are all placing bets on Crew AI. Why? Because it makes building and deploying real AI agents possible.

With Crew AI, teams are spinning up agents that:

Launch predictive marketing campaigns
Automate financial back-office ops
Optimize inventory and logistics
And tackle 100+ other enterprise use cases

But here’s the catch: agents are only as good as the data they can reach. That’s where SWIRL comes in.

By pairing Crew AI with SWIRL, you get more than just agents—you get enterprise-ready, data-rich workflows that scale. No custom plumbing. No brittle integrations.

With Crew AI + SWIRL, your agents can:

Connect to 100+ enterprise data sources out-of-the-box
Fetch the most relevant structured/unstructured data across silos
Respect row-level permissions with real enterprise auth
Summarize and answer with your LLM of choice
Plug in easily via zero-code connectors

Want to see this in action?

Message me for a demo or check the open source edition here: https://github.com/swirlai/swirl-search

It was already indexed!

Sid Probstein — Sun, 13 Aug 2023 16:48:52 +0000

I recently had the pleasure of chatting with @dmitrykan on his Vector Podcast. Check it out: https://dmitry-kan.medium.com/vector-podcast-with-sid-probstein-search-in-siloed-data-with-swirl-f2b9595a2715

We talked about quite a few things, including:

The challenges of enterprise search in the post-cloud era
How cross-silo search is particularlyt tricky because of entitlements (aka permissions) across silos
Zero-code configuration of connectors in Swirl, where JSON path and developer API doc get the job done
How large language models contextually re-rank disparate search results

There was an interesting twist at the end of the call. Dmitry uses a service called Clearword to transcribe recordings. Dmitry asked: “how quickly can you index the transcript and search it with Swirl?”

Here is my answer: It was already indexed!

Since there is no audio and it goes by quickly, let me explain ... Clearword emailed the transcript to both of us shortly after recording ended. It was indexed by Microsoft Outlook within seconds of arriving in my inbox.

To verify this, I copied some text from the middle of the transcript and pasted it into Swirl, which returned the link to the email message with the transcript and the phrase I searched for.

That simple truth - that the average enterprise is awash in search forms - is the entire reason metasearch is such a game-changing approach. Instead of making yet another repository, Swirl sends queries to existing search APIs and re-ranks the results from everything. It saves users a huge amount of time without a major IT project.

Want to see for yourself? git it going with 2 commands via Docker here: https://github.com/swirlai/swirl-search

Swirl 2.5 released

Sid Probstein — Wed, 09 Aug 2023 14:37:03 +0000

I am delighted to announce availability of Swirl 2.5!

This version focused on performance. Configured with 12 SearchProviders, Swirl 2.5 supports ~15 queries/second on a Standard F16s v2 server (16 vcpus, 32 GiB memory) with a median response time of ~3 seconds.

Version 2.5 also includes SearchProviders for HubSpot contact, company, and deal records, plus improvements to the Galaxy search UI (shown above).

Check out the Release notes for full details: https://github.com/swirlai/swirl-search/releases/tag/v2.5.0

What is Swirl? A new open source metasearch engine; it queries anything with an API then uses spaCy to re-rank the unified results without copying any data! Includes zero-code configs for Apache Solr, ChatGPT, Elastic Search, OpenSearch, PostgreSQL, Google BigQuery, RequestsGet, Google PSE, NLResearch.com, Miro, Microsoft 365, HubSpot, Atlassian, YouTrack, GitHub & more!

I wrote a metasearch engine called Swirl

Sid Probstein — Thu, 03 Aug 2023 18:48:40 +0000

Hi all! Swirl sends queries to existing search engines, unifies the results and re-ranks them all using large language models. It solves cross-silo information access and search problems in a fraction of the time and effort required to copy, ingest and index data.

Here's a brief video intro: https://youtu.be/sfsBYyu6qDQ

Swirl was written in python atop the django/celery/redis stack with a choice of Sqlite3 or PostgreSQL back-ends. The source code is available under the Apache 2.0 license. The distribution includes zero-code configs for Apache Solr, ChatGPT, Elastic Search, OpenSearch, PostgreSQL, Google BigQuery, RequestsGet, Google PSE, NLResearch.com, Miro, Microsoft 365, Atlassian, YouTrack, GitHub, HubSpot & more. Plug-in your access tokens or use Microsoft 365 to login and users can stop searching and start Swirling!

Links:

We are seeking contributions and feedback from developers working on all kinds of search solutions... thanks!