DEV Community: u11d

MinIO Is Deprecated: Best Alternatives for S3-Compatible Storage

Bartek Gałęzowski — Mon, 28 Sep 2026 10:16:42 +0000

For years, MinIO was the obvious answer whenever an engineering team needed S3-compatible object storage outside AWS. You wanted an S3-compatible API on Kubernetes? You needed object storage for backups? You wanted something you could run locally with Docker Compose? The answer was the same - MinIO.

That recommendation has changed.

In 2026, the open-source MinIO project is no longer a maintained project in the way engineers historically expected it to be. The upstream GitHub repository was archived on April 25, 2026, and its README explicitly states that the repository is no longer maintained.

That raises an obvious question:

What should replace MinIO?

There isn't a single answer. Three projects are particularly worth looking at: Garage, RustFS, and SeaweedFS. All three provide S3-compatible storage, but they are built around quite different ideas. Garage focuses heavily on decentralized and geographically distributed object storage, RustFS is particularly interesting for engineers looking for something close to the MinIO model, while SeaweedFS takes a much broader approach and combines object storage with distributed file storage and other storage interfaces.

The important thing is therefore not simply to find "the next MinIO", but to understand what each project is actually designed to do.

What actually happened to MinIO?

The MinIO repository on GitHub is now archived and read-only. Its current README states:

"THIS REPOSITORY IS NO LONGER MAINTAINED."

For any production engineer, this should immediately raise a question: what are the alternatives? But let's be realistic. Thousands of projects are still running MinIO, and many of them are running perfectly fine on the latest version they deployed. A running MinIO cluster does not magically stop serving objects because the GitHub repository was archived. If your cluster is healthy and you already have the required binaries or container images, it can continue operating. The bigger problem is the future lifecycle of the installation.

In September 2026, the minio/minio repository on Docker Hub was archived, and users started reporting that both minio/minio and minio/mc could no longer be pulled from Docker Hub. This is exactly the kind of failure that makes infrastructure engineers nervous. Imagine that your application hasn't changed at all. Your Kubernetes manifests are still valid, your infrastructure code is still valid, and your production cluster is still running. Then a node fails. You need to recreate the workload, and Kubernetes tries to pull:

image: minio/minio:latest

But the image is no longer available.

If you haven't mirrored the image to your own registry, your deployment is blocked. You may end up relying on a colleague who still has the image locally, or trying to recover an old image from somewhere else.

Alternative #1: Garage

Garage is a distributed object storage system written in Rust that implements the S3 API.

Its philosophy is somewhat different from MinIO. Garage is particularly interesting if your goal isn't simply "I want a server that looks like MinIO", but rather "I want distributed object storage that can operate across multiple locations."

Garage uses a decentralized architecture where nodes participate in the distributed storage system. This makes it particularly interesting for deployments where you might have several relatively small nodes spread across different locations instead of one large storage cluster.

For example, you could have infrastructure distributed across several sites:

flowchart LR
    subgraph SiteA["Site A (Primary DC)"]
        A1["Node 1"] --- A2["Node 2"] --- A3["Node 3"]
    end

    subgraph SiteB["Site B (Secondary / Edge)"]
        B1["Node 4"] --- B2["Node 5"] --- B3["Node 6"]
    end

    SiteA <-->|Geo-distributed sync / WAN| SiteB

This is one of the areas where Garage differs significantly from the traditional MinIO mental model.

Alternative #2: RustFS

RustFS takes a different approach from Garage, although it shares some important characteristics. It is also written in Rust and provides an AWS S3-compatible API.

For MinIO users, the interesting part is its design philosophy. RustFS explicitly describes its architecture as being inspired by MinIO's design philosophy, which makes it particularly interesting when your starting point is: "I already know MinIO, and I want something that feels familiar."

RustFS uses a distributed model consisting of nodes and drives. Conceptually, it looks like:

flowchart LR
    Client["S3 Client Traffic"] --> N3 & N2 & N1

    subgraph Cluster["RustFS Cluster (Erasure Set)"]
        subgraph N3["Node 3"]
            D5["Drive 5"]
            D6["Drive 6"]
        end
        subgraph N2["Node 2"]
            D3["Drive 3"]
            D4["Drive 4"]
        end
        subgraph N1["Node 1"]
            D1["Drive 1"]
            D2["Drive 2"]
        end
    end

The cluster is divided into sets, with drives distributed across nodes. For engineers coming from MinIO, the mental model is relatively familiar. This is what makes RustFS one of the most obvious projects to investigate if the main objective is replacing an existing MinIO installation while keeping the application layer largely unchanged.

That doesn't mean that migration is completely transparent. S3 compatibility needs to be tested against the actual application workload, and the operational model of the new cluster still needs to be understood.

Alternative #3: SeaweedFS

SeaweedFS takes a broader approach than Garage and RustFS. While both are primarily distributed object-storage systems, SeaweedFS is designed as a distributed storage platform supporting S3, FUSE, HDFS, WebDAV, SFTP, and other interfaces.

For MinIO users, this makes SeaweedFS interesting when S3 is only part of the requirement. If you simply need an S3-compatible API, SeaweedFS may provide more infrastructure than necessary. But if you need multiple storage interfaces or have workloads involving very large numbers of small objects, its architecture becomes more compelling.

Conceptually, it looks like:

flowchart LR
    subgraph Interfaces["Storage Interfaces"]
        S3["S3 API"]
        FUSE["FUSE"]
        HDFS["HDFS"]
    end

    subgraph Cluster["SeaweedFS Cluster"]
        Storage["Storage Layer"]
    end

    Interfaces --> Cluster

SeaweedFS separates storage into master servers, volume servers, and the Filer. The master manages cluster metadata, volume servers store the data, and the Filer provides a filesystem-like namespace. The S3 gateway sits on top of this architecture and can be scaled independently.

Another notable feature is its support for both replication and erasure coding, together with a volume-based design that works well with large numbers of small objects. This makes SeaweedFS worth considering for datasets, thumbnails, generated files, and other high-object-count workloads.

The trade-off is complexity. SeaweedFS offers considerably more functionality than a simple S3-compatible object store, so the additional flexibility needs to be weighed against the operational overhead.

Garage vs RustFS vs SeaweedFS

At this point, it should be clear that these three projects aren't simply three interchangeable MinIO clones.

They all provide S3-compatible storage, but they target somewhat different requirements.

Garage is focused on decentralized object storage and is particularly interesting for geographically distributed or multi-site deployments.

RustFS is the most natural project to investigate when your primary requirement is replacing MinIO with another distributed S3-compatible object store while retaining a relatively familiar architecture.

SeaweedFS takes a broader approach and makes sense when object storage is only one part of the problem. Its ability to expose the same underlying storage through S3, filesystem, HDFS, and other interfaces can be valuable for larger or more diverse storage environments.

The important distinction is therefore not simply performance or feature count. It is the architecture you want to operate.

	Garage	RustFS	SeaweedFS
Primary language	Rust	Rust	Go
S3 compatibility	Yes	Yes	Yes
Architecture	Distributed, decentralized	Distributed, decentralized	Layered distributed storage
Primary focus	Distributed object storage	S3 object storage	Object + file storage platform
MinIO migration	Requires architectural changes	Conceptually closer	Requires architectural changes
Kubernetes	Yes	Yes	Yes
Geo-distribution	Strong focus	Supported	Supported
Filesystem access	No	No	Yes
FUSE	No	No	Yes
HDFS	No	No	Yes
Small-object workloads	Good fit	Good fit	Strong focus
Erasure coding	Supported	Supported	Supported
Operational complexity	Relatively focused	Relatively focused	Higher
Best suited for	Multi-site / decentralized storage	MinIO replacement	Multi-purpose distributed storage
Migration complexity	Medium	Low–medium	Medium–high

This table is not intended to declare a winner. The projects solve different problems, and the right choice depends on the workload and architecture you need to support.

If you have multiple locations and want decentralized storage that treats geographic distribution as a fundamental part of the architecture, Garage is worth investigating.

If your main requirement is replacing MinIO with a modern, Rust-based, distributed S3-compatible object store that follows a relatively familiar model, RustFS is an obvious candidate.

If you need more than S3 — for example, filesystem access, HDFS, huge numbers of small objects, or multiple storage interfaces — SeaweedFS deserves serious consideration.

The important thing is that you don't have to choose the project with the longest feature list. You have to choose the architecture that matches your workload.

Conclusion

MinIO may no longer be the default choice it once was, but that doesn't mean there is a single project waiting to take its place. Garage, RustFS, and SeaweedFS represent three different directions for self-hosted object storage. Garage focuses on decentralized, multi-site storage. RustFS is the closest fit for teams looking for a familiar distributed S3 model. SeaweedFS goes beyond object storage and provides a broader storage platform with multiple interfaces and a design particularly suited to large numbers of objects. The era of treating MinIO as the automatic default for self-hosted S3 is over. The right replacement depends on the operational model, workload, and architecture you want to maintain long after the migration is complete.

For an existing MinIO deployment, the decision should not start with "Which project is the best?". It should start with "What do we actually need from our storage layer?"

Once you answer that question, choosing between Garage, RustFS, and SeaweedFS becomes much less about finding the next MinIO and much more about choosing the storage architecture that actually fits your infrastructure.

TypeSafe AI Jev explained: examples, use cases, and LLM comparisons

Paweł Swiridow — Wed, 23 Sep 2026 19:00:00 +0000

TypeSafe AI Jev is a model for making bounded decisions inside software. You give it text or structured state plus questions whose valid answers you define. It returns typed choices, scores, probabilities, and confidence values instead of generating prose.

That makes Jev useful for classification, routing, scoring, and review gates. It does not replace GPT, Claude, or another generative model when you need writing, code, explanations, or open-ended reasoning. Think of it as a fuzzy decision function, not a chatbot wearing a smaller hat.

Quick answer

Use Jev when:

Every valid answer can be defined before the call.
The hard part is semantic judgment, not arithmetic or deterministic policy.
Software needs probabilities or confidence to decide whether to act, retry, or escalate.
The workload repeats often enough for latency and per-call cost to matter.

Use something else when:

The output must contain newly written text, code, or an explanation.
Correctness depends on exact math, dates, permissions, or database state.
The task requires long, multi-step reasoning.
The model would be the only security or authorization boundary.

What is TypeSafe AI Jev?

TypeSafe calls Jev its first System One model. That is the company's product category for models built to make fast, structured decisions that software can consume directly. It is not a new chat interface and it is not an industry-standard model category.

The contract is simple:

state + typed questions -> Jev -> typed answers + probabilities -> application code

The state can be a string, JSON object, or array containing text. Each question defines its answer space. Jev evaluates independent questions against the same state in parallel, then your code decides what happens next.

At the time of writing, jev-latest resolves to jev-1.13.0. TypeSafe's model documentation lists it at $0.042 per million input tokens with free output tokens. The documented context limit is 64,000 tokens per request, with a separate 32,000-token limit for the state plus the longest question. Input is text only, so images, audio, and video must be converted before they reach Jev.

Pin a version when tuning thresholds and rerun evaluation before upgrading. A moving alias is convenient until it changes a production route at 02:00.

TypeSafe's launch material describes Jev as unable to hallucinate because it cannot emit a value outside the schema. That claim needs a narrower reading. Jev cannot invent a fourth result when your Choice defines three, but it can choose the wrong one. Type safety protects the shape of an answer. It does not make the judgment correct. Independent analysis from Arize makes the same distinction.

How Choice, Score, and Noul work

Jev exposes three question types:

Primitive	Question it answers	Returned data	Good fit
Choice	Which predefined option fits best?	selected option, option probabilities, confidence	intent routing, taxonomy classification
Score	Where does this fall on an ordered rubric?	weighted score, level probabilities, legend, confidence	severity, quality, urgency
Noul	How likely is this yes/no proposition?	probability from 0 to 1	detection, verification, binary review signals

A Choice can contain up to 255 options. A Score accepts between two and ten ordered levels. Choice and Score return a confidence value derived from their probability distributions; Noul returns the probability of yes without a separate confidence field.

Questions in one request are independent. If question B depends on question A, use another call or express the dependency in code. Do not assume that two differently phrased questions will obey arithmetic identities either. TypeSafe documents cases where a Noul and an equivalent-looking Choice produce different numbers because they ask different statistical questions.

Keep questions atomic. Ask separately whether a refund was requested, score impact, classify intent, then let code apply policy. Models judge fuzzy meaning; code does not reinterpret accounting rules overnight.

Run a Jev decision in Python

Install the SDK

The Python SDK requires an API key in TYPESAFE_API_KEY:

python3 -m pip install typesafe-sdk
export TYPESAFE_API_KEY='replace-with-a-key-from-console.typesafe.ai'

Use your normal secret manager in CI and production. Committing the key beside the example would turn a quick start into an incident report.

Classify and route a support ticket

This example asks one Choice, one Score, and one Noul in a single call. It then applies routing thresholds in ordinary Python.

import os
import time

from typesafe_sdk import Choice, Noul, Score, TypeSafeClient

if "TYPESAFE_API_KEY" not in os.environ:
    raise SystemExit("Set TYPESAFE_API_KEY before running this example")

INPUT_USD_PER_MTOK = 0.042  # TypeSafe list price at the time of writing

ticket = {
    "subject": "Checkout fails after payment",
    "message": (
        "Our card was charged twice and checkout still shows an error. "
        "We need this fixed today because customers cannot place orders."
    ),
    "plan": "business",
}

started = time.perf_counter()

with TypeSafeClient(model="jev-1.13.0") as client:
    response = client.system_one(
        state=ticket,
        questions={
            "department": Choice(
                instructions="Which team should own this ticket first?",
                criteria={
                    "billing": "Charges, invoices, refunds, or subscriptions",
                    "technical": "Bugs, outages, failed integrations, or broken product behavior",
                    "account": "Login, identity, permissions, or account access",
                    "other": "The ticket does not fit the other categories",
                },
            ),
            "severity": Score(
                instructions="How severely does this issue block the customer's work?",
                criteria=[
                    "Minor inconvenience with a clear workaround",
                    "Important feature is degraded but work can continue",
                    "Critical workflow is blocked or customers are losing money",
                ],
            ),
            "requests_refund": Noul(
                instructions="Does the customer explicitly ask for a refund?",
                criteria={
                    "true": "The customer directly requests money back",
                    "false": "The customer reports a charge but does not request a refund",
                },
            ),
        },
    )

elapsed = time.perf_counter() - started

department = response.choices["department"]
severity = response.scores["severity"]
refund_probability = response.nouls["requests_refund"].noul

if department.confidence < 0.65 or severity.confidence < 0.55:
    destination = "human-review"
elif severity.score > 1.5:
    destination = "incident-response"
else:
    destination = department.choice

input_tokens = response.usage.input_tokens or 0
estimated_cost = input_tokens / 1_000_000 * INPUT_USD_PER_MTOK

print(f"model={response.model}")
print(f"destination={destination}")
print(f"department_confidence={department.confidence:.2f}")
print(f"severity={severity.score:.2f}")
print(f"refund_probability={refund_probability:.2f}")
print(f"input_tokens={input_tokens}")
print(f"output_tokens={response.usage.output_tokens}")
print(f"elapsed_seconds={elapsed:.2f}")
print(f"estimated_cost_usd={estimated_cost:.8f}")

The SDK exposes separate collections for Choice, Score, and Noul answers. The thresholds are examples, not TypeSafe defaults. Set them from errors and review costs measured on your own data.

Pinning jev-1.13.0 prevents alias drift. In production, log the criteria version, probability distributions, chosen route, and human outcome.

Sample output and cost

A real run of the example above against jev-1.13.0 printed this:

model=jev-1.13.0
destination=incident-response
department_confidence=0.95
severity=2.00
refund_probability=0.06
input_tokens=528
output_tokens=77
elapsed_seconds=0.64
estimated_cost_usd=0.00002218

The usage block in the response is what makes the cost measurable without opening a billing dashboard. That call moved 528 input tokens. TypeSafe bills $0.042 per million input tokens and does not bill output tokens, so the run cost 528 / 1,000,000 * 0.042, or roughly $0.000022 at current list prices. A thousand calls of this size cost about two cents, and one dollar buys roughly 45,000 of them.

elapsed_seconds covers client setup plus the round trip, not just server time. Values move with the input, criteria, and model version, so treat the transcript as a real example rather than a fixed contract.

Practical Jev use cases

Route support requests

Support routing is the natural first pilot. Put the ticket text and relevant account context in state. Use Choice for the owning queue, Score for impact, and Noul for independent conditions such as an explicit refund request:

{
  "state": {
    "channel": "email",
    "subject": "Wrong item delivered, replacement needed before Friday",
    "message": "Order 48213 arrived with two left boots. I need the replacement before the conference on Friday or I want to cancel.",
    "account_tier": "enterprise",
    "earlier_tickets": 2
  },
  "model": "jev-1.13.0",
  "questions": {
    "queue": {
      "type": "choice",
      "instructions": "Which queue should own this ticket first?",
      "criteria": {
        "returns": "Wrong, damaged, or missing items, replacements, and shipping problems",
        "billing": "Charges, invoices, refunds, and subscriptions",
        "technical": "Product failures, errors, and defects",
        "account": "Access, identity, and permissions",
        "other": "None of the other queues fit"
      }
    },
    "impact": {
      "type": "score",
      "instructions": "How much does the problem block the customer?",
      "criteria": [
        "Minor inconvenience with a workaround",
        "The item or order cannot be used as delivered",
        "Time-critical situation or total loss of the purchase"
      ]
    },
    "wants_cancel": {
      "type": "noul",
      "instructions": "Is the customer asking to cancel an order or close the account?"
    }
  }
}

Code can accept a confident route, send uncertain tickets to a person, and apply exact entitlement rules from the account database.

Review agent actions

An agent can propose a tool call while Jev supplies a bounded semantic risk signal:

{
  "state": {
    "user_goal": "Remove generated build artifacts",
    "tool": "shell",
    "command": "rm -rf dist"
  },
  "model": "jev-1.13.0",
  "questions": {
    "risk": {
      "type": "choice",
      "instructions": "Classify the operational risk of the proposed action",
      "criteria": {
        "read_only": "Does not modify state",
        "reversible_write": "Changes state but has a reliable rollback",
        "irreversible_write": "Deletes or overwrites state without a reliable rollback",
        "unclear": "The available state is insufficient"
      }
    },
    "matches_goal": {
      "type": "noul",
      "instructions": "Does the proposed action directly match the user's stated goal?"
    }
  }
}

This is a review signal, not permission to execute. Allowlist enforcement, path validation, sandboxing, and user confirmation still belong in code. Jev's own documentation warns that adversarial text in state can move an answer, so it must not become the sole security boundary.

Check RAG evidence before generation

Jev can classify whether a retrieved passage supports a claim before a generative model writes the answer:

{
  "state": {
    "question": "Does the service retain customer prompts for training?",
    "claim": "Customer requests are not used to train Jev.",
    "passage": "Jev is not trained on customer requests or responses."
  },
  "model": "jev-1.13.0",
  "questions": {
    "support": {
      "type": "choice",
      "instructions": "How well does the passage support the claim?",
      "criteria": {
        "supported": "The passage directly supports the claim",
        "contradicted": "The passage conflicts with the claim",
        "insufficient": "The passage does not settle the claim"
      }
    },
    "contains_instruction": {
      "type": "noul",
      "instructions": "Does the passage contain an instruction aimed at changing the evaluator's behavior?"
    }
  }
}

The application still needs to preserve the source URL and quote. Jev judges the relationship between claim and passage; it does not browse, establish provenance, or write the final cited answer.

Triage moderation and operations queues

Moderation reports, invoices, incidents, and agent traces all end up in queues. Jev can classify the queue, score severity, and flag missing evidence:

{
  "state": {
    "source": "monitoring alert",
    "service": "checkout-api",
    "summary": "p99 latency rose from 180 ms to 2.4 s over the last 20 minutes while the error rate stayed near 0.2%",
    "context": "Deploy checkout-api v2.31.0 finished 25 minutes ago"
  },
  "model": "jev-1.13.0",
  "questions": {
    "owner": {
      "type": "choice",
      "instructions": "Which team should take the first response for this alert?",
      "criteria": {
        "service_team": "The reporting service's own code or configuration",
        "platform": "Shared infrastructure, networking, or managed dependencies",
        "database": "Storage, replication, or query performance",
        "unclear": "The available evidence does not point to one owner"
      }
    },
    "severity": {
      "type": "score",
      "instructions": "How much customer impact does the summary describe?",
      "criteria": [
        "No confirmed customer impact, worth watching",
        "Degraded latency for some customers",
        "Customers blocked or data at risk"
      ]
    },
    "missing_evidence": {
      "type": "noul",
      "instructions": "Is the alert missing context a first responder would need before acting?"
    }
  }
}

Low-confidence or high-impact items, and anything flagged for missing evidence, go to a human reviewer. Exact totals, due dates, and account permissions stay in deterministic code because Jev 1.13 is explicitly weak at counting, arithmetic, and date comparison.

Route work to the right model or human

Use Jev in front of more expensive handlers. A Choice can select deterministic lookup, a small generative model, a reasoning model, or human review:

{
  "state": {
    "request": "Summarize the attached vendor contract and flag the clauses that change our data-retention obligations",
    "attachments": ["vendor-contract-v7.pdf"],
    "requester": "procurement"
  },
  "model": "jev-1.13.0",
  "questions": {
    "route": {
      "type": "choice",
      "instructions": "Which handler should receive this request?",
      "criteria": {
        "deterministic": "A lookup, calculation, or template can answer it exactly",
        "small_model": "Short generation or extraction a cheap model handles well",
        "reasoning_model": "Long-document analysis or multi-step reasoning",
        "human": "Judgment, negotiation, or accountability beyond the model stack"
      }
    },
    "in_scope": {
      "type": "noul",
      "instructions": "Does the request fall inside the documented scope of the available handlers?"
    }
  }
}

Confidence determines whether the route is safe to accept. Below the threshold, send the request to a human instead of accepting a cheap guess that the matching handler will only have to redo later. This architecture spends generation tokens only when the task needs generation, rather than asking a novelist to sort every envelope.

Jev vs LLMs, classifiers, and rules

Jev is not automatically better than the alternatives. It occupies a specific slot:

Dimension	Rules and code	Traditional classifier	LLM structured output	TypeSafe Jev
Best job	exact deterministic checks	stable prediction task with labeled data	generation, extraction, explanation, broad reasoning	bounded semantic decisions
Output	exact programmed value	fixed trained labels and scores	generated text constrained to a schema	predefined Choice, Score, or Noul answers
New taxonomy	edit code	usually retrain	change prompt or schema	change request criteria
Probability signal	not applicable	often available, calibration varies	usually not a reliable application probability	first-class probabilities; local validation still required
Explanation	code is inspectable	usually limited	can generate one	no generated explanation
Exact math and policy	best option	depends on trained features	possible but unreliable	documented poor fit; keep it in code
Deployment tradeoff	engineering maintenance	labels, training, serving	latency, cost, parsing, model variance	hosted early access, bounded interface, no generation

Plain JSON mode guarantees valid JSON, not adherence to a supplied schema. Strict structured-output APIs can enforce supported JSON schemas, including required keys and enum values.

Schema enforcement is therefore not unique to Jev. Its distinction is a decision-specific interface built around Choice, Score, and Noul, with probability distributions and confidence for bounded judgments.

A conventional classifier may be the better choice for a stable, high-volume task when you already have good labels, training infrastructure, and predictable categories. In one early spam experiment, Jev reached 98.6% accuracy on a 5,733-email ham, spam, and phishing test without task-specific fine-tuning or labeled examples in its requests, while a TF-IDF logistic-regression baseline trained on labeled messages reached 98.9% (full writeup). The criteria had been refined after reviewing labeled examples, so this was useful early evidence, not magic zero-data victory.

Rules win whenever the answer can be computed exactly. Do not ask any model whether an invoice is overdue if two parsed dates and a comparison operator can settle it.

The cost and latency story is still notable.

TypeSafe reports 70 to 500 ms end-to-end latency and $0.042 per million input tokens for suitable requests.

Its launch workflow evaluation produced headline claims of 193.6 times faster and 444.6 times cheaper than its selected LLM comparison, but TypeSafe says those figures are likely near the high end of real-world gains and notes possible benchmark bias.

When not to use TypeSafe AI Jev

Jev 1.13 is not trained to write text. It will not draft a customer reply, produce code, summarize a report, or explain why it chose an answer. Use a generative model when the output itself must be created. The missing explanation also matters for evaluation work: an LLM judge can describe a failure, while Jev returns probabilities that tell you where to inspect, not how to fix the problem.

Keep arithmetic, counting, numeric interpolation, and date comparisons in code. TypeSafe's limitations page says Jev can read dates as text but does not reliably treat them as ordered values. It also warns about literal phrasing, multi-hop indirection, irrelevant context, contradictory criteria, and adversarial content.

More context is not automatically better. Retrieve and filter first, then send the fields needed for the decision. A 64,000-token limit is capacity, not a challenge.

Test non-English workloads separately. TypeSafe says English is Jev's strongest training language and other languages are not handled equally well.

Most importantly, separate a valid answer from a correct answer. A typed output prevents parser failures and out-of-schema labels. It does not prevent a confident, valid, expensive mistake.

How to evaluate Jev on your workload

Start in shadow mode:

Pick one reversible decision with a closed answer set.
Freeze a representative labeled dataset and the exact criteria text.
Run Jev and the current baseline on identical records.
Measure per-class errors, false automatic actions, human-review rate, calibration, end-to-end latency, and cost per completed decision.
Tune thresholds by consequence. Sending a ticket to the wrong queue and approving a destructive action do not deserve the same confidence floor.
Pin the model version and rerun the suite before changing either model or criteria.
Keep permissions, arithmetic, and state changes in deterministic code.

Do not optimize only for overall accuracy. A model can look good while failing badly on a rare queue that matters. Track confusion by class and inspect probability bands: cases scored near 0.9 should be correct more often than cases scored near 0.6. If that relationship does not hold on your data, the threshold is decoration.

Include review capacity in the evaluation. A cautious threshold that sends 70 percent of cases to a human may be safe, but it has not automated much. The useful operating point balances wrong automatic actions against the cost of review.

Should you use TypeSafe AI Jev?

Pilot TypeSafe AI Jev when the answer space is bounded, the judgment is semantic, and software consumes the result often enough for decision latency and cost to matter. Support routing, relevance checks, moderation labels, model routing, and broad evaluation coverage are credible starting points.

Keep an LLM when the output must be written, explained, or reasoned through. Keep code when the answer can be computed exactly. Keep a human in the path when a wrong but valid decision has serious consequences and your own calibration data is thin.

Jev's useful idea is not that every LLM should be replaced. It is that generation and judgment do not have to be the same model call. Split them, measure them independently, and make code responsible for the final action.

Implementing Postgres Row-Level Security in Next.js: The Drizzle Multi-Tenant Pattern

uninterrupted — Wed, 16 Sep 2026 07:00:00 +0000

The first time we turned on Row-Level Security, nothing happened. Policies were in place, ENABLE ROW LEVEL SECURITY had run, and every query still returned every tenant's rows. Took a minute to remember: we were connected as postgres, the superuser, and superusers ignore RLS entirely. That one gotcha basically shaped the rest of this setup.

The problem we were actually trying to solve is boring and everyone's had it: a route handler forgets WHERE organization_id = ?, code review doesn't catch it because the query still "works," and Tenant A quietly gets to see Tenant B's data. Nobody notices until someone does.

Our answer was to stop trusting application code to remember the filter and push isolation into Postgres itself with Row-Level Security. The pitch is simple: if a query forgets tenant scoping, it should get zero rows back, not someone else's data. Here's the setup we landed on — two DB roles, a SET LOCAL session variable, a Next.js wrapper so route handlers don't have to think about any of this — plus the four things that actually bit us building it.

Two roles, two connections

Back to that first surprise: superusers and table owners bypass RLS by default. The docs say it plainly — superuser or BYPASSRLS, policies don't apply, full stop. Table owners get the same pass unless you explicitly add FORCE ROW LEVEL SECURITY. If your app connects as postgres, which is what basically every local Postgres setup defaults to, your carefully written policies are just decoration.

So we run two roles now:

Role	RLS	Used by
`postgres` (superuser)	Bypassed	Auth library, admin routes, migrations
`app_user` (non-superuser)	Enforced	All tenant-scoped queries

Each gets its own pool, wrapped as two Drizzle instances:

// db.ts
export function getDbAdmin(): PostgresJsDatabase<Schema> {
  if (!_dbAdmin) {
    const client = postgres({
      ...adminConnectionConfig, // DATABASE_URL, superuser
      max: 2,
      connection: { application_name: "app-admin" },
    });
    _dbAdmin = drizzle(client, { schema });
  }
  return _dbAdmin;
}

export function getDb(): PostgresJsDatabase<Schema> {
  if (!_db) {
    const client = postgres({
      ...appConnectionConfig, // APP_DATABASE_URL, app_user
      max: 4,
      connection: { application_name: "app-tenant" },
    });
    _db = drizzle(client, { schema });
  }
  return _db;
}

Worth flagging two things here. First, the lazy init isn't stylistic — Next.js runs module scope at build time, and if you open a connection pool at import time, next build needs a live database to succeed. That bit us once in CI. Wrapping construction in a getter defers the actual connection until something queries it.

We also wrap both in a Proxy so call sites can still just import db and use it like a normal object:

export const db = new Proxy({} as PostgresJsDatabase<Schema>, {
  get(_, prop) {
    return Reflect.get(getDb(), prop);
  },
});

I won't pretend this is elegant. It works, and it means nobody importing db has to know it's lazy underneath. Fine trade.

Second thing: we set a distinct application_name per pool. Costs nothing, and the first time something's connection-starved you'll be very glad pg_stat_activity tells you exactly which pool is choking instead of guessing.

The policy itself

Every tenant table gets the same shape:

ALTER TABLE projects ENABLE ROW LEVEL SECURITY;

CREATE POLICY tenant_isolation ON projects
  USING (organization_id = current_setting('app.organization_id', true)::uuid)
  WITH CHECK (organization_id = current_setting('app.organization_id', true)::uuid);

Two things matter more than they look like they should.

That true as the second argument to current_setting means "don't error if the setting's unset, just give me NULL." NULL compared to anything is NULL, and Postgres treats a NULL policy result as "exclude this row" — same as false. So a query that runs without tenant context isn't a crash and isn't a leak, it's just an empty table. Someone forgets to set context, they get a confusing empty result, they go debug it. Nobody's data went anywhere in the meantime. That's the property I actually care about here.

And don't skip WITH CHECK. It's easy to write the USING clause, ship it, and feel done — but USING only governs reads. Without WITH CHECK, nothing stops a tenant from INSERTing or UPDATEing a row with someone else's organization_id in it. We missed this on the first pass of one migration and caught it in review, which is exactly the kind of thing you don't want to catch in review.

Getting context into the transaction: SET LOCAL

Policies check current_setting('app.organization_id'), so something has to set it per request. The obvious first attempt — plain SET app.organization_id = '...' — is wrong, and it's wrong in a way that won't show up until you're under load. SET is session-scoped, connections are pooled, and a session-scoped setting sticks around for whoever grabs that connection next. That's tenant bleed waiting to happen the moment your pool gets reused across requests, which it will.

SET LOCAL is the fix — scoped to the current transaction, reverts automatically on commit or rollback. Open transaction, SET LOCAL, run your queries, done, connection goes back to the pool clean.

In Next.js we wrapped this into one function every tenant route goes through:

export function withAuth(handler: AuthenticatedHandler) {
  return async (request: NextRequest, ctx?: RouteContext) => {
    const result = await authenticate(); // session lookup
    if (!result.success) {
      return NextResponse.json({ error: result.error }, { status: result.status });
    }

    const { user, organizationId } = result;

    return db.transaction(async (tx) => {
      await setOrgContext(tx, organizationId);
      return handler(request, { user, organizationId, tx: castTx(tx) }, ctx);
    });
  };
}

Route handlers never see any of this. They just get a tx that's already scoped:

export const GET = withAuth(async (request, { tx }) => {
  const rows = await tx.select().from(projects); // only this org's rows, guaranteed
  return NextResponse.json({ data: rows });
});

No WHERE organizationId = ? anywhere in that query, and it's still safe. We keep the explicit filters in most places anyway — belt and suspenders, and it documents intent for the next person reading the code — but the point stands: if you forgot it, RLS still has you.

Server components got a matching helper since they're not going through route handlers at all:

export async function withOrgDb<T>(
  organizationId: string,
  callback: (tx: DbInstance) => Promise<T>,
): Promise<T> {
  return getDb().transaction(async (tx) => {
    await setOrgContext(tx, organizationId);
    return callback(castTx(tx));
  });
}

Gotcha #1: you can't parameterize SET

setOrgContext looks like this:

const UUID_RE =
  /^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$/i;

export async function setOrgContext(tx: DrizzleTx, organizationId: string) {
  if (!UUID_RE.test(organizationId)) {
    throw new Error("Invalid organization ID format");
  }
  await tx.execute(
    sql`SET LOCAL app.organization_id = '${sql.raw(organizationId)}'`,
  );
}

That sql.raw() should bother you a little — it bothered me too, first time I wrote it. Here's the thing though: SET is a utility command, not a regular statement, and utility commands in Postgres just don't take $1-style parameters. You cannot write SET LOCAL app.organization_id = $1. It'll error. You've got two real options — set_config('app.organization_id', $1, true), which is a normal function call and does accept a parameter, or raw string interpolation gated behind strict validation. We went with the second, gated on a UUID regex: the value either looks exactly like a UUID or the function throws before it ever touches SQL. I still don't love it stylistically, but there's no injection surface left once that regex has to match.

Point either way: you're going to hit this wall the first time you try to parameterize a SET, and it's not a bug in your code, it's just how the command works.

Gotcha #2: Drizzle's transaction type doesn't match its client type

Drizzle's transaction callback hands you a PostgresJsTransaction, and that type is not structurally the same as PostgresJsDatabase — even though every method you'd actually call (select, insert, update, delete) is identical between them. If AuthContext.tx gets typed as the transaction class, every shared data function downstream now needs that same internal type, which means importing a class Drizzle never really meant for you to reach into.

We just centralized the cast once and moved on:

// Extract the tx type without importing internal Drizzle classes
type DrizzleTx = Parameters<Parameters<DbInstance["transaction"]>[0]>[0];

export function castTx(tx: DrizzleTx): DbInstance {
  return tx as unknown as DbInstance;
}

Now shared data-layer code just takes a DbInstance and doesn't care if it's the root client, the admin client, or a live transaction. One cast, one comment explaining why, easy to delete later if Drizzle ever unifies the types properly.

Gotcha #3: don't let a stream hold your transaction

withAuth holds a transaction — and the pooled connection under it — for as long as the request runs. For a JSON endpoint that returns in 40ms, fine. For a Server-Sent Events endpoint that stays open for twenty minutes, that's a problem: with a pool of 4, three people leaving a dashboard tab open exhausts it and everyone else starts timing out.

We learned this the hard way and split off a separate wrapper for streaming routes — it authenticates, but deliberately never opens a transaction:

export const GET = withAuthStream(async (request, { organizationId }) => {
  // Short transaction for the initial state read
  const job = await withOrgDb(organizationId, (tx) =>
    tx.select().from(jobs).where(eq(jobs.id, jobId)),
  );

  const stream = new ReadableStream({
    /* ...on each event, use withOrgDb() again for any query... */
  });
  return new Response(stream, {
    headers: { "Content-Type": "text/event-stream" },
  });
});

The rule we now follow: RLS context only lives as long as a transaction, so keep the transaction as short as the query needs — never as long as the request.

Gotcha #4: knowing which connection you're on

Two Drizzle instances means the failure mode shifted from "forgot a WHERE clause" to "grabbed the wrong client." We ended up needing a cheat sheet:

Scenario	Connection
Tenant API route	`tx` from `withAuth`
SSE route	`withOrgDb()` per query
Server component, tenant data	`withOrgDb()`
Server component, auth tables only	`dbAdmin`
Admin route	`dbAdmin`
Auth library internals	`dbAdmin`
Migrations / background workers	superuser

The reassuring part: the mistake in the safe direction fails loudly. Query tenant tables through db with no context set, and you get zero rows — a visibly broken feature in dev, not a security incident. Using dbAdmin where you meant tenant scoping is the dangerous direction, and that's the one worth actually watching for in review. At least it's greppable.

Fluctum: Open-Source Dynamic Pricing Solution for Stores Selling Spot-Priced Goods

Michał Miler — Mon, 14 Sep 2026 06:35:04 +0000

If you sell gold, silver, or anything else priced off a live market, you've probably already run into this problem: the price you show a customer and the actual market price start drifting apart the moment you set it. Update prices too rarely and you're either giving away margin or overcharging people - and both cost you customers eventually, just in different ways.

Most store owners assume there are only two options: pay developers to build a custom pricing system, or keep manually updating prices and accept the risk. There's a third option that's easy to miss - using an existing, already-built open-source solution instead of starting from scratch.

This post lays out what each path actually costs, in plain terms, so you can decide which one makes sense for your store.

Where This Problem Shows Up

Precious metals dealers feel this first, because gold and silver prices move constantly and the margins are thin enough that a stale price matters. But the same problem shows up anywhere a product's fair price is tied to a live index rather than something you set once and leave alone:

Precious and industrial metals - bullion, plus copper, platinum, and palladium for wholesale operations.
B2B catalogs synced to an ERP - pricing that has to reflect what your internal systems say right now, not what a database field said last week.
Currency-sensitive goods - high-value items where the price also has to move with exchange rates, not just the underlying market.

The same logic extends further, too - anything priced off a live sale index rather than a fixed catalog value (think collectible trading cards or limited-run goods pegged to recent resale prices) runs into the same drift problem, even if the tooling for those categories is less mature today.

The Real Cost of Building This Yourself

"Just show a live price" sounds simple, but a live-pricing checkout has to get several things right at once, and getting any one of them wrong costs real money:

Pulling a live price feed reliably, including handling the moments the feed is slow or unavailable.
Updating the price a customer sees in real time, without expensive full-page reloads.
Freezing the price at the right moment during checkout - if the price can still move after the customer clicks "buy," you're back to the original problem.
Refusing to complete an order if the frozen price has gone stale - otherwise a customer could complete a purchase minutes after the quoted price expired.

None of these are exotic, but each one is a place a custom build can quietly get wrong, and the mistakes are the expensive kind: a wrong price that either loses margin on every sale or drives away customers who feel overcharged. This is why a live-pricing checkout is realistically a multi-week engineering project, not a quick add-on - even though the end result looks simple to a shopper.

What a Ready-Made Solution Looks Like

Fluctum is a free, open-source plugin built on top of Medusa, an open-source e-commerce platform. Because it's a Medusa plugin rather than a separate system, it drops into an existing Medusa store and keeps everything else Medusa already gives you - multi-region and multi-currency support, promotions, customer accounts, taxes, shipping, and payments - while adding live pricing on top. It's installed by a developer once, and after that it runs on its own.

You can try the live demo yourself at fluctum.medusajs.site - add a product to the cart, watch the price update, and go through checkout to see the price lock in place.

Which one should you choose?

If you already have engineering resources and want a pricing system that's deeply custom to your business (unusual pricing rules, an existing in-house platform you can't move off of), building it yourself is a legitimate option - just budget for the ongoing maintenance, not only the initial build.

For most stores, especially ones already running or considering Medusa, starting from a free, working solution and customizing it only where needed is the faster and lower-risk path. Because Fluctum is open-source and built on Medusa, you own the resulting codebase outright rather than licensing access to someone else's product - the same store can be extended to sync with an ERP system, connect to warehouse or inventory systems, switch or add payment providers like Stripe or PayPal, or bolt on whatever custom business logic your store needs later. You're never locked into someone else's roadmap if your store eventually needs something different.

Quick Reference

	Build it yourself	Use Fluctum (free, open-source)
Time to launch	Months of developer time, plus testing	Installed and configured in hours by a developer
Ongoing maintenance	Your team owns it - feed outages, checkout bugs, every future fix	Maintained upstream; you install updates when you want them
Upfront cost	Developer time (custom-built, one-off)	Free - MIT-licensed, no license fee
Risk of a checkout pricing mistake	You design and test the "lock the price at checkout" logic yourself	Already built to handle the common failure modes (expired prices, stale quotes at payment time)
Customization	Full control, by definition	Open-source and built on Medusa - you own the code, so you can extend it to ERP, storage/inventory, payments, or custom logic as your store needs

Conclusion

Static prices and live markets don't mix, but that doesn't mean the fix has to be an expensive custom build. Fluctum gives Medusa stores selling precious metals - or any other spot-priced goods - a free, working starting point, with the option to extend it later if your business needs something more specific.

How to get fixed IP addresses for an AWS ALB with Global Accelerator

Paweł Swiridow — Wed, 09 Sep 2026 11:00:00 +0000

An AWS Application Load Balancer gives you a DNS name, not client-facing addresses you can promise to another company forever. The practical workaround is to publish the two anycast IPv4 addresses from AWS Global Accelerator and route that traffic to the ALB behind the scenes.

Application Load Balancers are great until someone asks, "What IP addresses should we whitelist?"

That is where the fun starts. The addresses behind the ALB name can change as AWS scales and maintains the service. Telling a customer to whitelist whatever resolves today is not architecture. It is a calendar reminder for a future outage.

This keeps the firewall conversation simple without replacing your load balancer, target groups, certificates, listener rules, or the other ALB machinery you already use.

Quick answer

Use this pattern when the other side only accepts IP-based allowlists, but your service still needs ALB features such as HTTPS listeners, host routing, path routing, target groups, and health checks.

Implementation checklist:

Create a Global Accelerator accelerator.
Add a TCP listener, usually on port 443.
Create an endpoint group in the same Region as the ALB.
Register the ALB as the endpoint.
Send both accelerator IPs to the firewall owner.
Add a Route 53 alias to the accelerator if clients can connect by name.

The pattern looks like this:

Client network
  -> allowlist 2 Global Accelerator IPs
  -> AWS edge location
  -> Global Accelerator listener :443
  -> regional endpoint group
  -> Application Load Balancer
  -> targets

Whitelisting, allowlisting, firewall exception, "please make networking stop yelling" - same practical problem.

Why ALB alone does not work

An ALB is addressed by DNS:

my-api-1234567890.eu-west-1.elb.amazonaws.com

That DNS name resolves to AWS-managed addresses. They are not yours, they are not fixed, and they are not a contract.

You can put a friendly name in Route 53:

api.example.com -> my-api-1234567890.eu-west-1.elb.amazonaws.com

That helps humans. It does not help a firewall that only accepts destination IPs.

Do not solve this by resolving the ALB name and sending the current addresses to a customer. Those addresses can change. The customer will eventually block your production traffic and then everyone gets to pretend DNS was the surprising part.

How Global Accelerator gives an ALB fixed inbound IPs

AWS Global Accelerator creates a fixed entry point for your application.

For an IPv4 accelerator, AWS gives you two static anycast IP addresses:

203.0.113.10
203.0.113.11

Those are the addresses your customer allowlists as outbound destinations. You can point your own DNS name at them:

api.example.com -> Global Accelerator alias

or, when the other side insists on raw IPs, hand them the two accelerator addresses.

Global Accelerator can use an Application Load Balancer as a standard accelerator endpoint. It also monitors endpoint health and can route to another Region if you add one later.

The important detail: the two Global Accelerator IPs are client-facing destination IPs. They are not the source IPs your ALB sees. Do not create an ALB security group rule that says "allow inbound from these two Global Accelerator IPs" and expect it to work. That is the common foot-gun, and it wears steel-toed boots.

Configure AWS Global Accelerator for an ALB with Terraform

Assume you already have an ALB listening on HTTPS.

The Global Accelerator pieces are:

An accelerator
A listener on TCP 443
An endpoint group in the ALB Region
The ALB as an endpoint

"aws_globalaccelerator_accelerator" "api" {
  name            = "api-prod"
  ip_address_type = "IPV4"
  enabled         = true

  attributes {
    flow_logs_enabled = false
  }
}

"aws_globalaccelerator_listener" "https" {
  accelerator_arn = aws_globalaccelerator_accelerator.api.id
  protocol        = "TCP"
  client_affinity = "NONE"

  port_range {
    from_port = 443
    to_port   = 443
  }
}

"aws_globalaccelerator_endpoint_group" "eu_west_1" {
  listener_arn          = aws_globalaccelerator_listener.https.id
  endpoint_group_region = "eu-west-1"
  traffic_dial_percentage = 100

  endpoint_configuration {
    endpoint_id                    = aws_lb.api.arn
    weight                         = 128
    client_ip_preservation_enabled = true
  }
}

After terraform apply, output the addresses:

"global_accelerator_ips" {
  value = aws_globalaccelerator_accelerator.api.ip_sets[0].ip_addresses
}

Example output:

global_accelerator_ips = [
  "203.0.113.10",
  "203.0.113.11",
]

These are the addresses to give to the other side for inbound access to your service from their point of view. They allow outbound traffic to those destination IPs. You keep your ALB behind the accelerator.

Add DNS without losing the fixed IPs

Create an alias record for the accelerator DNS name.

"aws_route53_record" "api" {
  zone_id = aws_route53_zone.example.zone_id
  name    = "api.example.com"
  type    = "A"

  alias {
    name                   = aws_globalaccelerator_accelerator.api.dns_name
    zone_id                = aws_globalaccelerator_accelerator.api.hosted_zone_id
    evaluate_target_health = false
  }
}

Customers that can allowlist by FQDN can use:

api.example.com

Customers that require IPs use the two accelerator addresses. Yes, both. Sending only one is how you turn high availability into a coin toss with extra paperwork.

Protect the ALB from direct access

Global Accelerator solves the fixed destination IP problem. It does not automatically make an internet-facing ALB impossible to reach directly through its own DNS name.

If the ALB is internet-facing, someone can still try:

curl https://my-api-1234567890.eu-west-1.elb.amazonaws.com

That bypasses your pretty api.example.com name and goes straight to the ALB. Security groups and AWS WAF still apply, but the request did not enter through the Global Accelerator IPs.

The cleaner pattern is to use an internal ALB as the accelerator endpoint. Global Accelerator supports internal Application Load Balancer endpoints and creates elastic network interfaces in your VPC to deliver the traffic.

"aws_lb" "api" {
  name               = "api-prod"
  internal           = true
  load_balancer_type = "application"
  subnets            = module.vpc.private_subnets
  security_groups    = [aws_security_group.alb.id]
}

Then allow the application port on the ALB security group. With client IP preservation enabled for an ALB endpoint, your application can still read the original client address from X-Forwarded-For.

"aws_security_group_rule" "alb_https_in" {
  type              = "ingress"
  security_group_id = aws_security_group.alb.id
  protocol          = "tcp"
  from_port         = 443
  to_port           = 443
  cidr_blocks       = ["0.0.0.0/0"]
}

That rule may look too open, but the ALB is internal. The point is that the ALB does not have a public route of its own. Traffic arrives through Global Accelerator into the VPC path AWS creates for the endpoint.

If you must keep the ALB internet-facing, add compensating controls:

Use AWS WAF on the ALB.
Require valid host headers.
Require authentication at the application layer.
Restrict known customer source ranges in the ALB security group if the client list is small and stable.

Do not rely on obscuring the ALB DNS name. Obscurity is not a control. It is a shrug with a certificate.

Keep the real client IP

For ALB endpoints, Global Accelerator and ALB preserve the original client IP in X-Forwarded-For.

Your application will receive a header like this:

X-Forwarded-For: 198.51.100.25

If there are multiple proxies, the header can contain multiple addresses:

X-Forwarded-For: 198.51.100.25, 10.0.12.34

Make sure your app or ingress layer knows which proxy chain it trusts. Blindly trusting every X-Forwarded-For value from the internet is how logs become fan fiction.

For nginx behind the ALB, configure trusted proxy ranges for your VPC or ALB layer, not the whole internet:

set_real_ip_from 10.0.0.0/8;
real_ip_header X-Forwarded-For;
real_ip_recursive on;

Adjust the CIDR to your actual network.

Verify the fixed IP setup

Get the accelerator IPs:

aws globalaccelerator list-accelerators \
  --query 'Accelerators[?Name==`api-prod`].IpSets[].IpAddresses[]' \
  --output text

Test HTTPS against each IP while sending the expected host header:

curl -v --resolve api.example.com:443:203.0.113.10 https://api.example.com/health
curl -v --resolve api.example.com:443:203.0.113.11 https://api.example.com/health

Check the accelerator health:

aws globalaccelerator describe-endpoint-group \
  --endpoint-group-arn "$ENDPOINT_GROUP_ARN" \
  --query 'EndpointGroup.EndpointDescriptions[*].{EndpointId:EndpointId,HealthState:HealthState,Weight:Weight}' \
  --output table

Check your application logs for the client IP:

grep 'X-Forwarded-For' /var/log/app/access.log

Or query ALB access logs in Athena:

SELECT
  client_ip,
  request_url,
  target_status_code,
  time
FROM alb_logs
WHERE request_url LIKE '%/health%'
ORDER BY time DESC
LIMIT 20;

Troubleshooting common errors

If traffic does not reach the ALB, check these first:

Symptom	Likely cause	What to check
`curl --resolve` times out	The accelerator endpoint is unhealthy or the ALB security group blocks traffic	Check `describe-endpoint-group`, ALB listener rules, target group health, and security group ingress
TLS certificate mismatch	The request reached the ALB with the wrong host name	Use `--resolve api.example.com:443:<accelerator-ip>` and make sure the ALB certificate covers `api.example.com`
Customer allowlisted one IP but traffic still fails sometimes	Only one Global Accelerator IP was added to the firewall	Give them both accelerator IPs
ALB is still reachable directly	The ALB is internet-facing	Use an internal ALB endpoint or add WAF, host-header checks, authentication, and source restrictions
App logs show a proxy IP instead of the client IP	The app does not trust the right proxy chain	Configure trusted proxy ranges and read `X-Forwarded-For` correctly

The most common mistake is treating the two Global Accelerator IPs as source IPs for the ALB security group. They are destination IPs for clients. Your ALB will not see requests as coming from those two addresses.

Multi-Region is the obvious next step

You do not need multiple Regions to justify Global Accelerator. Fixed ingress IPs are already enough.

But once it is in place, adding another ALB in another Region is straightforward:

"aws_globalaccelerator_endpoint_group" "us_east_1" {
  listener_arn          = aws_globalaccelerator_listener.https.id
  endpoint_group_region = "us-east-1"
  traffic_dial_percentage = 0

  endpoint_configuration {
    endpoint_id                    = aws_lb.api_us_east_1.arn
    weight                         = 128
    client_ip_preservation_enabled = true
  }
}

Start with traffic_dial_percentage = 0, validate the stack, then dial traffic up when you are ready:

traffic_dial_percentage = 10

This is useful for migrations, blue-green deployments, and disaster recovery plans that contain fewer hopes and prayers.

Things to remember

Global Accelerator is a good fit when external parties need stable destination IPs for your AWS-hosted service.

Use this pattern when:

Clients must allowlist fixed IPs.
You want an ALB behind those fixed IPs.
You may want regional failover later.
You want to avoid managing proxy instances or static Elastic IP hacks.

Be clear about what it does not do:

It does not give the ALB fixed IPs.
It does not make an internet-facing ALB private.
It does not let you use the two accelerator IPs as ALB security group sources.
It does not replace application authentication or WAF.

The short version: if you need fixed IP addresses for an AWS ALB, use AWS Global Accelerator. ALB gives you flexible HTTP load balancing. Global Accelerator gives you two stable anycast IPs in front of it. Put them together when the network team on the other side says, "We need IPs, not DNS," and you would prefer not to build your own little networking sadness machine.

Building a Multilingual Shop with Next.js

Paweł Sobolewski — Tue, 01 Sep 2026 22:00:00 +0000

Adding multiple languages to a web application often starts with a simple step: replacing hardcoded strings with translation keys and adding a language switcher.

In an e-commerce application, however, internationalization quickly becomes a broader architectural challenge.

Language affects far more than the text displayed on the page:

It becomes part of the URL,
it influences routing and redirects,
it determines which translation resources are loaded,
it affects server and client rendering,
it changes how data such as prices and dates should be formatted.

In this article, we’ll look at how to approach these problems in a Next.js application using the App Router. Rather than focusing on the basics of translating strings, we’ll focus on the technical decisions behind the implementation: how the language is detected, how it’s propagated through the application, how translations are loaded on the server and client, and how localized routes are handled.

The implementation presented here is based on a small e-commerce shop, with a product page as its main piece of domain content. It supports English, Polish, and German, and it intentionally stays focused on the internationalization layer rather than on building a complete commerce platform.

The goal is to use this relatively simple application to explore the architecture behind multilingual Next.js applications and the trade-offs involved in choosing and implementing an i18n solution.

Implementation Remarks

Before diving into the implementation, it’s worth noting a few deliberate simplifications in this project.

The repository is intentionally designed to demonstrate the technical aspects of internationalization in Next.js, not to provide a complete, production-ready e-commerce application. As a result, some parts of the implementation are intentionally simplified.

Repository link

Product data is part of the translation resources

To keep the example self-contained, product data such as names and descriptions is stored alongside the translation resources.

In a real e-commerce application, this data would typically come from a domain-specific API or commerce backend:

Product page
    ↓
Commerce API
    ↓
Product data

In that setup, the i18n layer would primarily handle UI translations, while localized product content would be part of the product data model.

All languages are stored in a single file

The example keeps translations for all supported languages in one file to make the structure easier to explore.

In a larger application, each language would typically have its own resource files:

languages/
├── en/
│   ├── common.json
│   ├── home.json
│   └── product.json
├── pl/
│   ├── common.json
│   ├── home.json
│   └── product.json
└── de/
    ├── common.json
    ├── home.json
    └── product.json

This makes translation files easier to manage independently and fits better with external localization workflows.

No external translation management system

The project uses local resources and an API route to demonstrate the resource-loading boundary.

In a production application, you might instead use a CMS, a translation management platform, or another external service. The resourceLoader abstraction is intentionally designed so the underlying source can be swapped out without changing the components that consume translations.

Choosing an i18n Solution for Next.js

Before implementing internationalization, the first step is choosing the right tool.

Next.js provides the routing primitives needed to build a multilingual application, but it doesn’t prescribe a translation library. That leaves several options: building a small custom solution, using a Next.js-focused library such as next-intl, or adopting the broader i18next ecosystem through next-i18next.

For this project, I chose next-i18next.

That decision wasn’t based solely on the number of translated strings. The application needs an i18n layer that works across Server and Client Components, supports namespaces, and gives us control over how translation resources are loaded. These are important capabilities for a shop where translations may eventually move from local files to an external CMS or another backend service.

next-i18next is built on top of i18next and react-i18next, while adding the Next.js-specific integration. The version used in this project supports the App Router, including Server Components, Client Components, and proxy-based language detection*.* It also provides separate APIs such as getT() for Server Components and useT() for Client Components.

Why not build it ourselves?

A custom implementation might initially look attractive. With a small application, a simple dictionary and a React context can be enough:

const translations = { en: { ... }, pl: { ... }, de: { ... }, };

The problem is that translation lookup is only one part of the equation.

Once the application needs server-side translations, client-side hydration, namespaces, fallback languages, language detection, or external translation resources, the custom solution starts accumulating infrastructure that an established i18n library already provides.

There’s another important advantage: a mature i18n library gives us much more than a simple key → string lookup.

For example, i18next provides built-in support for interpolation:

t("welcome", { name: "John" });

It also supports pluralization without requiring us to implement language-specific rules ourselves, and react-i18next provides the <Trans> component for translations that contain React elements or more complex markup.

These features may seem like minor conveniences at first, but they quickly become important as the number and complexity of translations grow. Pluralization is particularly difficult to implement correctly because different languages can have completely different grammatical rules.

Language-Based Routing

Once we have chosen the i18n library, the next decision is where the current language should live.

For this application, the language is part of the URL:

/en
/pl
/de

The same applies to nested routes:

/en/product/1
/pl/product/1
/de/product/1

This is implemented using a dynamic segment in the App Router:

app/
└── [lng]/
    ├── page.tsx
    └── product/
        └── [id]/
            └── page.tsx

The [lng] segment becomes a route parameter, so a product page receives both the language and the product identifier:

const { id, lng } = await params;

A small App Router detail: Starting with Next.js 16.3.0, you can also access route parameters via next/root-params. For example, instead of passing lng through multiple component levels, a component can use:
import { lng } from "next/root-params";
This can reduce prop drilling when deeper components need the language. It’s especially useful for components that rely on route parameters but don’t otherwise need props from their parent.

Why put the language in the URL?

The first benefit is that each language has its own URL.

A user can navigate directly to:

/pl/product/1

and the server immediately knows which language to use to render the page.

The URL is also shareable and bookmarkable. If someone sends a Polish product page to another user, they’re sharing the Polish version explicitly rather than relying on the recipient’s browser preferences.

There is also a clear relationship between URLs:

/en/product/1
/pl/product/1
/de/product/1

They represent the same resource in different languages.

This is especially useful for SEO, where language-specific URLs can be indexed and connected using mechanisms such as hreflang.

Detecting the User's Language with Proxy

Putting the language in the URL solves the routing problem, but it raises another question: what should happen when a user visits the application without specifying a language?

For example:

GET /

There is no language in the URL, so the application needs a way to determine the most appropriate one.

In this project, that logic lives in the Next.js Proxy and follows a simple priority order:

URL
 ↓
Cookie
 ↓
Accept-Language
 ↓
Fallback

The key point is that these sources are not treated equally.

URL comes first

If the requested URL already contains a supported language, we treat it as an explicit user choice.

For example:

/pl/product/1

should always be rendered in Polish, even if the user's browser prefers German.

The Proxy first extracts the first segment of the pathname and checks whether it is one of the supported languages:

const { pathname } = request.nextUrl;

const urlLanguage = pathname.split("/")[1];

if (supportedLanguages.includes(urlLanguage)) {
  // ...
}

When a valid language is found, the request can continue normally.

The Proxy also updates the language cookie to remember the user's explicit choice for subsequent visits.

Cookie as persistent preference

If there is no language in the URL, the next source is the language cookie.

The cookie allows us to remember a language selected during a previous visit without requiring the user to select it again.

However, the cookie is deliberately lower in the priority chain than the URL.

This means that:

URL:    /de
Cookie: pl

results in German.

The URL represents the current request, while the cookie represents a previously stored preference.

Keeping this distinction prevents a stale cookie from overriding an explicit URL.

Falling back to `Accept-Language`

If neither the URL nor the cookie provides a supported language, we can use the browser's Accept-Language header.

A browser might send something like:

pl-PL,pl;q=0.9,en-US;q=0.8,en;q=0.7

The application extracts the language codes and looks for the first one it supports.

For this project, the supported languages are:

en
pl
de

so the example header would result in:

pl

It is important to treat Accept-Language as a preference, rather than as a guaranteed language selection. The browser may request a language that the application does not support.

Finally, use a fallback

There is always a case where none of the detected languages are supported.

For example:

URL:    /
Cookie: fr
Header: ja-JP,ja;q=0.9

If the application only supports English, Polish and German, there is no match.

This is where the configured fallback language is used:

en

Translation Resources and Namespaces

Once the language is established, the next question is how the application retrieves the translations.

The project uses three namespaces:

common
home
product

Rather than keeping every translation in a single large dictionary, namespaces let us group resources by responsibility.

For example:

common
├── navigation
├── language switcher
└── shared UI

home
├── hero
└── product grid

product
├── product details
├── add to cart
└── stock information

This becomes increasingly useful as the application grows. A single translation file works well at first, but over time it becomes harder to maintain and to understand which part of the application owns a particular key.

Namespaces also provide a useful loading boundary. A product page doesn’t necessarily need every translation used by the homepage.

Loading resources through `resourceLoader`

The interesting part of this implementation is how those resources are loaded.

Instead of importing the translation JSON directly into the application, next-i18next is configured with a custom resourceLoader:

resourceLoader: async (language, namespace) => {
  const baseUrl =
    process.env.NEXT_PUBLIC_BASE_URL ?? "http://localhost:3000";

  const res = await fetch(
    `${baseUrl}/api/fetchTranslations?lang=${language}&ns=${namespace}`,
  );

  if (!res.ok) {
    throw new Error(
      `Failed to load translations: ${language}/${namespace}`,
    );
  }

  return res.json();
},

Why introduce an API layer?

For local JSON files, calling an API might seem unnecessary—we could simply import them.

The API is included here to illustrate a more realistic setup, where translations are managed outside the application bundle.

Imagine replacing the current implementation with:

resourceLoader
     ↓
    CMS

or:

resourceLoader
     ↓
translation service

The rest of the application can remain unchanged.

This separation is especially useful for e-commerce applications, where translations are often managed by content or localization teams rather than developers.

Server and Client Components

With the language and translation resources in place, we still need to answer one key question: how do Server Components and Client Components access translations?

The App Router provides two execution environments, and next-i18next offers an API for each.

On the server, translations can be accessed with getT():

const { t } = await getT("product");

const title = t("title");

On the client, translations are accessed through useT():

const { t } = useT("common");

return <button>{t("addToCart")}</button>;

This distinction matters because using translations doesn’t automatically mean a component needs to become a Client Component.

Keep translated content on the server

A product page is a good example.

There’s usually no reason to move the entire page to the client just because it contains translated text. The product name, description, labels, and other static content can be translated as the page is rendered on the server.

That way, the application can send translated HTML in the initial response instead of waiting for client-side JavaScript to initialize the translation layer.

Providing the resources from the layout

The [lng]/layout.tsx file is where the server-loaded translation resources are passed into the client-side i18n context.

The layout initializes the server-side i18n instance and obtains the loaded resources:

const { i18n } = await getT();
const resources = getResources(i18n);

Those resources are then passed to the provider:

<I18nProvider
  language={language}
  resources={resources}
>
  {children}
</I18nProvider>

The layout therefore establishes the translation context once for the entire language-specific part of the application.

Why this separation matters

With the App Router, we should generally default to Server Components.

The reason is simple: turning a component into a Client Component moves its logic into the browser and can add JavaScript that isn’t otherwise necessary.

Internationalization shouldn’t be the reason to do that.

If a component only needs to render translated content, it can remain a Server Component and use getT().

useT() should be reserved for components that already need to run on the client — for example, because they handle user interaction, browser events, local state, or other client-side behavior.

Formatting Prices

Internationalization is not limited to translating text. Numbers, dates, currencies, and other values often need to be formatted according to the user's language as well.

Prices are a simple example.

The same value:

1999.99

can be displayed differently depending on the language:

$1,999
1 999 zł
1.999 €

In this project, the language used in the URL is a short code such as en, pl, or de. However, the Intl APIs work with more specific locale identifiers, such as en-US or pl-PL.

The formatPrice utility handles this mapping:

const LOCALE_CURRENCY: Record<string, string> = {
  en: "USD",
  pl: "PLN",
  de: "EUR",
};

const LOCALE_BCP47: Record<string, string> = {
  en: "en-US",
  pl: "pl-PL",
  de: "de-DE",
};

There are two separate mappings here.

LOCALE_CURRENCY determines which currency should be displayed, while LOCALE_BCP47 determines the regional formatting conventions used by Intl.NumberFormat.

The actual formatting is then kept in a small utility:

export function formatPrice(price: number, language: string): string {
  const currency = LOCALE_CURRENCY[language] ?? "USD";
  const bcp47 = LOCALE_BCP47[language] ?? "en-US";

  return new Intl.NumberFormat(bcp47, {
    style: "currency",
    currency,
    maximumFractionDigits: 0,
  }).format(price);
}

This gives components a simple API:

formatPrice(product.price, lng);

without requiring them to know anything about currencies, BCP 47 locale identifiers, or Intl.NumberFormat.

Localized Navigation

As language becomes part of the URL, navigation is one of the first areas where i18n can start leaking into your application code.

Without an abstraction, every link needs to be aware of the current language:

<Link href={`/${lng}/product/${id}`}>
  Product
</Link>

This works, but it couples the component directly to the routing structure.

It also gets repetitive: every developer has to remember that the first URL segment represents the current language.

To avoid this, the project introduces a LocalizedLink component.

LocalizedLink

The idea is simple: components provide a path relative to the language root, and LocalizedLink adds the current language automatically.

Instead of:

<Link href={`/${lng}/product/${id}`}>
  Product
</Link>

we can write:

<LocalizedLink href={`/product/${id}`}>
  Product
</LocalizedLink>

If the current language is Polish, the resulting URL is:

/pl/product/1

For German, it becomes:

/de/product/1

This way, the component hides a routing detail that’s irrelevant to the component using it.

Switching between languages

The language switcher is a slightly different case.

Here, we want to change the language while preserving the current route.

For example, if the user is currently on:

/pl/product/1

switching to German should result in:

/de/product/1

rather than simply navigating to /de.

The switcher can therefore treat the current pathname as a resource-independent path and replace only the language segment.

Conceptually:

Current URL
/pl/product/1
      │
      │ replace language
      ▼
/de/product/1

This is important for product pages because changing the language should not interrupt the user's current context.

Production Considerations

The implementation covers the core architecture, but a production application would still need to address a few additional concerns.

Caching translation resources

Translation resources are usually much more stable than application data. Fetching the same translations on every request would therefore be wasteful.

The resourceLoader is a natural place to introduce caching and revalidation:

Component
    ↓
next-i18next
    ↓
resourceLoader
    ↓
cache
    ↓
translation source

The exact strategy depends on where the translations are stored. With a CMS or external translation service, the application could use Next.js caching and revalidation to avoid unnecessary requests while still allowing updated translations to reach users without rebuilding the entire application.

SEO and `hreflang`

Language-specific URLs also make SEO an important consideration.

For example:

/en/product/1
/pl/product/1
/de/product/1

These URLs represent different language versions of the same page. A production application should expose this relationship to search engines using appropriate hreflang metadata.

Canonical URLs should also be generated consistently so that each language version points to itself as the canonical URL, while the hreflang links connect the available alternatives.

The routing architecture already provides a clear URL for each language version. SEO metadata can build on top of that structure instead of inferring the language from cookies or client-side state.

Conclusion

Building a multilingual application with Next.js isn’t just about translating strings. Language affects routing, request handling, resource loading, rendering, and navigation, so internationalization quickly becomes an architectural concern.

The implementation presented here addresses those concerns at well-defined boundaries. The language is part of the URL, Proxy handles language detection, next-i18next provides the translation layer, and Server and Client Components use different APIs depending on where translations are needed. Application-specific abstractions such as LocalizedLink then keep language-related details out of regular components.

Real-Time Dynamic Pricing in Medusa — A Developer's Guide to Fluctum

Michał Miler — Wed, 26 Aug 2026 07:37:20 +0000

Selling gold, silver, or other spot-priced goods online creates a pricing challenge most e-commerce platforms don't handle: market quotes move continuously, and a product's listed price can change between add-to-cart and checkout. That mismatch forces merchants to either update prices constantly (impractical) or accept losses and unhappy customers.

Medusa's module system makes it straightforward to add custom pricing logic without forking the platform. You even don't have to design and maintain a custom pricing engine yourself — a community-built Medusa plugin handles the heavy lifting. Fluctum is an open-source Medusa-based solution that implements the full stack you need: scheduled spot-price ingestion, real-time storefront updates via Server-Sent Events (SSE), and auditable price locks at checkout.

The Solution: Fluctum

Fluctum fetches live spot prices on a schedule, streams them to the storefront, and locks the computed price for a cart when checkout begins. It ships as two pieces: the installable plugin (@u11d/medusa-dynamic-pricing) for adding dynamic pricing to an existing Medusa store, and fluctum_starter, a ready-to-use template repository (Medusa backend + Next.js storefront + Docker Compose) for running the whole flow locally.

This guide uses the starter because it's the fastest way to see live pricing, SSE updates, and checkout locking working end to end. For full configuration and option details, see the plugin npm page. Fluctum is MIT-licensed and the starter ships with four metals (XAU, XAG, XPT, XPD) out of the box, and it can be extended to price any volatile asset.

In this article, we'll walk through setting up the starter locally, verifying the live pricing and checkout-lock flow end to end, how Fluctum computes the final price, and deploying it to production.

Prerequisites

Node.js v24+
pnpm v11+ (corepack enable picks up the pinned version automatically)
Docker & Docker Compose (for local PostgreSQL and Redis)

Quick Start

Step 1: Create your repo from the template

Click Use this template on GitHub to create your own repo pre-wired with the plugin, or clone it directly:

git clone https://github.com/u11d-com/fluctum_starter.git my-store
cd my-store

Step 2: Install dependencies

Run the project's package manager to install dependencies:

pnpm install

Step 3: Configure environment variables

cp backend/.env.template backend/.env
cp storefront/.env.template storefront/.env

backend/.env already points DATABASE_URL and REDIS_URL at the local Docker services, and JWT_SECRET/COOKIE_SECRET are pre-filled, so no changes are needed there.

Add GOLD_API_KEY if you have a goldapi.io key to pull real spot prices. Otherwise, leave it unset and the plugin falls back to randomProvider, which generates realistic-looking fluctuating prices - useful for local development and for testing the checkout-lock flow without a live feed dependency. You'll set NEXT_PUBLIC_MEDUSA_PUBLISHABLE_KEY in storefront/.env in step 6, after the admin user exists.

Step 4: Start infrastructure

docker compose up -d

This starts PostgreSQL on port 5432 (database fluctum) and Redis on port 6379.

Step 5: Run migrations

pnpm run backend:migrate

This runs Medusa's database migrations and seeds initial data.

Step 6: Create an admin user and start the backend

pnpm run backend:create-admin
pnpm run backend:dev

Open http://localhost:9000/app → Settings → API Keys, copy the Publishable API key, and set it as NEXT_PUBLIC_MEDUSA_PUBLISHABLE_KEY in storefront/.env:

NEXT_PUBLIC_MEDUSA_PUBLISHABLE_KEY=pk_...

Step 7: Start the storefront

In a separate terminal, with the backend still running:

pnpm run storefront:dev

Backend: http://localhost:9000
Admin panel: http://localhost:9000/app
Storefront: http://localhost:8000

Once both are confirmed working, stop them and use pnpm run dev going forward to start backend and storefront together.

Testing & Verification

Visit the storefront and add a gold or silver product to the cart.

Stay on the cart page for a few seconds and watch the price update on its own - that's the live SSE feed, not a page refresh.

Click "Go to checkout." This calls lockCartPrices(cartId, force=true), which discards any existing locks for the cart and unconditionally recreates them - the price itself is computed the exact same way regardless of force.

Submit the address form and continue through delivery and payment. The price stays fixed across these steps - it won't drift even if the underlying spot price moves in the meantime.
Click "Place order." completeCartWorkflow.hooks.validate checks that every dynamically-priced line item has a valid, unexpired lock before letting the order complete.

Deployment

Once you've verified the flow locally, the starter is ready to deploy - either to Medusa Cloud, the simplest option, or to your own infrastructure. If you'd rather self-host, see deploymedusa.com for a managed self-hosting option.

Price Logic

Fluctum never stores a dynamic price on the Medusa variant itself - it computes the final price on the fly from two of its own data models, plus the variant's weight:

SpotPrice - a snapshot per material (XAU, XAG, ...), refreshed on every scheduled fetch. It stores ask, bid, and price. The pricing formula uses price.
PricingRule - the merchant-configured margin for a product or variant: spread_factor (a multiplicative margin, e.g. 1.02 for a 2% spread), spread_fixed (a flat fee applied after the spread), premium_percentage, and premium_fixed (a second flat fee, e.g. a minting or handling charge).
weight_oz - the variant's weight in troy ounces, linked in via a module link rather than stored on either model above.

The formula, rounded to two decimal places:

base          = weight_oz × spot_price × spread_factor × currency_conversion
after_premium = base × (1 + premium_percentage / 100)
final_price   = after_premium + spread_fixed + premium_fixed

For example, a 1 oz gold coin with a spot price of $2,000 and a 2% spread (spread_factor: 1.02, no premiums or fixed fees) prices at 1.0 × 2000.00 × 1.02 = 2040.00. Add a 1.5% premium, a $5 fixed spread, and a $10 fixed premium, and the same coin prices at 2040 × 1.015 + 5.00 + 10.00 = 2085.60.

This calculation lives in a single function, computeFinalPrice() in dynamic-pricing-plugin/src/utils/price-formula.ts, exported from both the plugin's main entry point and a ./client subpath - so the storefront can import the exact same logic the backend uses without pulling in server-only Medusa dependencies. Every CartPriceLock row created at checkout stores the resulting unit_price alongside the spot_price and all spread/premium factors used to compute it, so a locked price is always auditable after the fact. See docs/pricing-formula.md for the full breakdown and more worked examples.

Price-Lock Lifecycle

Prices in the cart are live - they update continuously via SSE and are never locked while a customer is just browsing. Locking only happens at two points, both handled by the same store API route. force doesn't change where the price comes from or how it's computed - it's purely a switch between reusing and recreating CartPriceLock rows:

POST /store/dynamic-pricing/carts/:id/price-lock
    ?force=true   - unconditionally discards existing locks and creates fresh ones
    ?force=false  - reuses existing locks if they still match the cart's current line items (same variant IDs); otherwise creates new ones

Trigger	Lock behavior	`force`
Click "Go to checkout" on the cart page	Always creates fresh locks	`true`
Click "Refresh prices" on checkout	Always creates fresh locks	`true`
Checkout page mounts (`CheckoutSummary`)	Reuses existing valid locks, or creates them if none exist	`false`
Submit address form	`CheckoutSummary` remounts, existing valid locks reused	`false`
Select delivery/payment (client-side `router.push()`)	No effect - `CheckoutSummary` stays mounted	-
Click "Place order"	Does not create locks - reuses mount-time locks	-

At order completion, completeCartWorkflow.hooks.validate queries the cart_price_lock table directly and rejects the order if any dynamically-priced variant has a missing or expired lock.

Conclusion

Static prices and volatile-priced goods don't mix - one side always ends up paying for the gap between the listed price and the real market. Fluctum closes that gap without requiring a custom pricing engine: live SSE updates keep the storefront honest, and price locks at checkout entry and completion keep both the customer and the merchant protected from a price moving mid-transaction.

Resources

Landing page: fluctum.io
Live demo: fluctum.medusajs.site/us
npm package: @u11d/medusa-dynamic-pricing
Starter template: u11d-com/fluctum_starter
Plugin source: u11d-com/fluctum_medusa-dynamic-pricing-plugin

Stop One EKS Node Group From Blocking Scale-Down in Another

Daniel Kraszewski — Wed, 19 Aug 2026 10:00:00 +0000

If idle EKS nodes keep running while other node groups scale up around them, Cluster Autoscaler's global scale-down cooldown is the likely cause, and --scale-down-delay-type-local=true is the fix. The flag makes Cluster Autoscaler track post-scale-up cooldowns per node group instead of cluster-wide, so a GPU pool adding capacity no longer resets the clock on an idle node in an unrelated memory pool. The default is false, which means every cluster running multiple ASGs has this behavior until someone turns it off.

We hit this with Dagster. Jobs used separate pools for general work, large CPU and memory tasks, and GPU tasks, each with min_size = 0, so we expected an unused pool to disappear on its own schedule. It didn't. A job starting in one pool delayed scale-down in every other pool, and with jobs starting and ending throughout the day, nodes that were no longer needed stayed alive far longer than they should have.

Why idle nodes stay running

Take two node groups, one for memory-heavy batch jobs and one for GPU jobs, both with min_size = 0. At noon the last memory-heavy job completes, its node goes empty, and Cluster Autoscaler starts the normal waiting period before deletion. Ten minutes later that node has waited long enough - and at almost the same moment a GPU job appears, its pod can't schedule, and Cluster Autoscaler grows the GPU Auto Scaling Group. You'd expect the memory node to disappear now: it belongs to another group, has no useful work, and has already passed its own waiting period. It stays.

Time	Memory group	GPU group	Result
`12:00`	Last job ends	Idle	Memory node becomes unneeded
`12:10`	Node has waited 10 minutes	New job triggers scale-up	Global cooldown starts
`12:11`	Node is eligible for deletion	New node is starting	Memory node cannot scale down
`12:20`	Node has been idle 20 minutes	Cooldown expires	Deletion can proceed

One extra ten-minute wait doesn't look serious. Bursty workloads turn it into a loop: a CPU pool grows, then a GPU pool grows, then a general pool grows, and each event postpones unrelated scale-down work. The problem is easy to miss because every node group looks correctly configured - minimum size is zero, jobs finish, pods disappear, and the autoscaler still reports nodes as unneeded. Yet the EC2 instances stay active, and nothing in your Terraform or your Helm values looks wrong. We didn't identify the shared cooldown from configuration alone; it took finding upstream issue #4872, which describes the same cross-group delay.

What Cluster Autoscaler actually tracks

Cluster Autoscaler treats an AWS Auto Scaling Group as a Kubernetes node group and decides both when to grow that group and when nodes can be safely removed from it. Scale-down has two separate waiting mechanisms that are easy to confuse: one governs each node's eligibility, the other protects the cluster after recent scaling actions.

Setting	Default	Meaning
`--scale-down-unneeded-time`	`10m`	How long a node must remain unneeded
`--scale-down-delay-after-add`	`10m`	How long scale-down waits after scale-up
`--scale-down-delay-after-delete`	`0s`	How long scale-down waits after deletion
`--scale-down-delay-after-failure`	`3m`	How long scale-down waits after failed scale-down
`--scale-down-delay-type-local`	`false`	Whether post-action delays are per node group

scale-down-unneeded-time answers "has this node been unnecessary for long enough?" - Cluster Autoscaler checks resource requests, movable pods, scheduling rules, PodDisruptionBudgets, and other safety conditions to decide. The scale-down-delay-after-* settings answer a completely different question: "is scale-down allowed at all right now, given recent autoscaler activity?" A node can pass its unneeded timer while a cooldown still blocks deletion, and that's exactly what happens here. Another node group scaling up does not erase the idle node's unneeded history - it adds a separate gate on top of it. Before 2024 that gate was always global, and with the default false it still is. Support for per-group cooldowns arrived through kubernetes/autoscaler#5729, which lets one process remember scaling activity per node group instead of treating every action as cluster-wide.

Enable per-node-group cooldowns

Keep one Deployment managing all node groups and set the flag. If you install Cluster Autoscaler with Helm, the change belongs in your values file:

cloudProvider: aws

autoDiscovery:
  clusterName: production-eks

awsRegion: eu-west-1

extraArgs:
  scale-down-delay-type-local: "true"
  scale-down-delay-after-add: 10m
  scale-down-unneeded-time: 10m

A GPU group scale-up now starts a cooldown for the GPU group only, and an eligible node in the memory group can be removed while it runs. Keep the delay values at their defaults unless you have measurements that justify changing them - new nodes need time to register, start DaemonSets, advertise GPUs, and receive pending pods, and local scope fixes the isolation problem without removing that protection. Setting scale-down-delay-after-add: 0 is not an equivalent shortcut. It does avoid cross-group delay, but it does so by removing the cooldown everywhere, including from the group that just added nodes, which is the one group that actually needs it.

Check your binary supports the flag

The feature merged into the main branch in January 2024 and was backported to the 1.29 release branch that March through #6484. The original 1.29.0 release does not include it, so do not assume every binary labeled 1.29 supports the flag. Check the actual image and the rendered command:

kubectl -n kube-system get deployment cluster-autoscaler \
  -o jsonpath='{.spec.template.spec.containers[0].image}{"\n"}'

kubectl -n kube-system get deployment cluster-autoscaler \
  -o jsonpath='{range .spec.template.spec.containers[0].command[*]}{.}{"\n"}{end}'

If the logs contain unknown flag: --scale-down-delay-type-local, the image lacks the feature - upgrade the binary rather than quietly dropping the setting. That upgrade has a constraint of its own: Cluster Autoscaler simulates Kubernetes scheduling, so its minor version must match your cluster's Kubernetes minor version, and cross-version combinations are not supported. AWS repeats this in its EKS guidance. One more trap sits between you and the right binary: Helm chart version and Cluster Autoscaler application version are different numbers. Inspect the chart's appVersion or pin image.tag explicitly, so a chart that looks current doesn't deploy a binary that isn't.

Local cooldowns only isolate delays within a single Cluster Autoscaler process, which makes process ownership the next thing to confirm.

Make sure one autoscaler owns the ASGs

The guarantee is worthless if a second, unmanaged autoscaler also watches some of your ASGs, or if one process never discovers all of them. On AWS, auto-discovery selects ASGs by tags, and the standard Helm values look for two keys:

autoDiscovery:
  clusterName: production-eks
  tags:
    - k8s.io/cluster-autoscaler/enabled
    - k8s.io/cluster-autoscaler/{{ .Values.autoDiscovery.clusterName }}

Every ASG the autoscaler should own needs both keys attached before discovery picks it up. Tag values are ignored unless you include them in the discovery expression, but values like true and owned make ownership readable and give you something to write IAM conditions against.

Use separate node groups when capacity or scheduling actually differs, and give each one real labels, real taints, and scale limits that allow it to reach zero. With terraform-aws-modules/eks/aws (~> 21.0), the node group definition is the part that matters:

eks_managed_node_groups = {
  batch = {
    instance_types = ["m7i.2xlarge"]

    min_size     = 0   # allows the group to empty completely
    max_size     = 20
    desired_size = 0

    labels = {
      "node-role" = "batch"
    }

    taints = {
      dedicated = {
        key    = "dedicated"
        value  = "batch"
        effect = "NO_SCHEDULE"
      }
    }
  }
}

The ASG this creates is the resource your discovery tags must reach, and managed node groups generate ASG names you don't control. Module inputs and outputs also shift between versions, so verify the tags actually landed. Tags on launch templates, security groups, or EC2 instances do not count - auto-discovery reads ASG tags and nothing else:

aws autoscaling describe-auto-scaling-groups \
  --query 'AutoScalingGroups[].{Name:AutoScalingGroupName,Tags:Tags[?starts_with(Key, `k8s.io/cluster-autoscaler/`)].{Key:Key,Value:Value}}' \
  --output table

If your module doesn't attach the discovery tags, read the generated ASG name and attach them yourself:

locals {
  batch_asg_name = module.eks.eks_managed_node_groups["batch"].node_group_resources[0].autoscaling_groups[0].name

  autoscaler_tags = {
    "k8s.io/cluster-autoscaler/enabled"        = "true"
    "k8s.io/cluster-autoscaler/production-eks" = "owned"
  }
}

resource "aws_autoscaling_group_tag" "autoscaler" {
  for_each = local.autoscaler_tags

  autoscaling_group_name = local.batch_asg_name

  tag {
    key                 = each.key
    value               = each.value
    propagate_at_launch = false  # this tag is for the ASG itself, not the instances it launches
  }
}

Do not combine static --nodes arguments with auto-discovery. Mixing the two makes ownership hard to reason about and will eventually have you debugging the state of a node group the process was never managing. The AWS provider documentation recommends auto-discovery and warns against the combination. Discovery controls what the process can see; IAM controls what it can change. Use IRSA or EKS Pod Identity and constrain SetDesiredCapacity and TerminateInstanceInAutoScalingGroup to ASGs carrying the cluster tags - AWS publishes a tag-conditioned IAM example.

Scale node groups back up from zero

Scale-to-zero introduces a different failure mode with the same symptom: a pool that sits at the wrong size and won't move. When a node group is empty, Kubernetes has no live Node object advertising that group's labels, taints, or custom resources, and Cluster Autoscaler still has to decide whether a pending pod would fit there. This bites hardest on pools selected by labels, protected by taints, or backed by GPUs.

Cluster Autoscaler infers CPU, memory, and GPU capacity from AWS configuration, and for EKS managed node groups it can also read labels and taints through eks:DescribeNodegroup, including at zero nodes. If that covers everything your pod's scheduling rules reference, you may not need explicit node-template tags at all. When it doesn't, put the metadata on the ASG, where it describes the simulated node used for scale-from-zero decisions:

resource "aws_autoscaling_group_tag" "batch_label" {
  autoscaling_group_name = local.batch_asg_name

  tag {
    key                 = "k8s.io/cluster-autoscaler/node-template/label/node-role"
    value               = "batch"
    propagate_at_launch = false
  }
}

resource "aws_autoscaling_group_tag" "batch_taint" {
  autoscaling_group_name = local.batch_asg_name

  tag {
    key                 = "k8s.io/cluster-autoscaler/node-template/taint/dedicated"
    value               = "batch:NoSchedule"  # value and effect packed into one string
    propagate_at_launch = false
  }
}

Node-template tags do not configure real nodes. They describe a hypothetical node for the simulator, so the same labels and taints must also exist on the managed node group itself - the labels and taints blocks shown earlier - and the pending pod must request that same contract:

spec:
  nodeSelector:
    node-role: batch
  tolerations:
    - key: dedicated
      operator: Equal
      value: batch
      effect: NoSchedule
  containers:
    - name: job
      image: example.com/data-job:2026-07-28
      resources:
        requests:
          cpu: "2"
          memory: 6Gi

Break that three-way match and you get one of two outcomes, both of which look like a stuck queue. Either the pool never scales up, because Cluster Autoscaler can't tell the pending pod would fit, or it scales up a node that still can't schedule the pod, because the real labels don't match what the pod asked for. Whichever source the metadata comes from, simulated and real must agree.

Diagnose and verify

Start by confirming the nodes are genuinely idle from Cluster Autoscaler's perspective. Low observed CPU is not enough - Cluster Autoscaler reasons about pod resource requests and whether pods can move elsewhere, not utilization. Run it at verbosity 4 while investigating and follow the Deployment logs:

kubectl -n kube-system logs deployment/cluster-autoscaler --follow \
  | grep -E 'unneeded|scale down|scaled up recently|removing'

You're looking for evidence that Cluster Autoscaler already considers the node removable, followed by a cooldown message instead of a deletion. Exact wording varies by version, but the pattern holds: one line marks a node unneeded or eligible, a later line skips scale-down because something scaled up recently. Compare those timestamps against ASG scaling activity in the other group - a scale-up in group B shortly before an eligible node in group A fails to disappear is the signature:

aws autoscaling describe-scaling-activities \
  --auto-scaling-group-name <group-b-asg-name> \
  --max-items 20

aws autoscaling describe-auto-scaling-groups \
  --query 'AutoScalingGroups[].{Name:AutoScalingGroupName,Desired:DesiredCapacity,Min:MinSize,Max:MaxSize}' \
  --output table

Rule out the ordinary blockers before blaming the cooldown. A node legitimately stays when it hosts local storage, a restrictive PodDisruptionBudget, an unreplicated pod, or safe-to-evict: "false". Scheduling constraints do the same: if the pods on that node can't fit anywhere else because of selectors, affinity, host ports, or resource requests, Cluster Autoscaler is keeping the node for good reason.

Once the flag is deployed, confirm that exactly one Deployment manages the intended ASGs and that the rendered command carries the setting:

kubectl -n kube-system get deployments | grep cluster-autoscaler

kubectl -n kube-system get deployment cluster-autoscaler \
  -o jsonpath='{range .spec.template.spec.containers[0].command[*]}{.}{"\n"}{end}' \
  | grep scale-down-delay

Then run a controlled two-group test. Let a node in group A become unneeded, wait for the logs to show it as a candidate, and create a pod only group B can run - here a batch pool labeled node-role=batch and tainted dedicated=batch:NoSchedule:

apiVersion: v1
kind: Pod
metadata:
  name: force-batch-scale-up
spec:
  restartPolicy: Never
  nodeSelector:
    node-role: batch
  tolerations:
    - key: dedicated
      operator: Equal
      value: batch
      effect: NoSchedule
  containers:
    - name: pause
      image: public.ecr.aws/eks-distro/kubernetes/pause:3.10
      resources:
        requests:
          cpu: "2"
          memory: 6Gi

Group B's scale-up should no longer postpone the eligible node in group A. Deletion still depends on scan interval, drain time, AWS API latency, and the node staying unneeded throughout, so give it a couple of scan cycles before declaring the test failed.

Legacy: one autoscaler per ASG

Before per-node-group cooldowns existed, sharding was the practical workaround: run one Cluster Autoscaler per ASG, each process watching a disjoint discovery tag. Each shard then kept its own cooldown state, so a GPU scale-up couldn't delay the process responsible for memory nodes.

It's heavier than it looks. Every shard needs its own ASG ownership boundary, leader election, service account identity, and IAM scope, and the operational surface grows linearly with node group count. Worse, shards watch the same unschedulable pods without coordinating, so two autoscalers can each decide their ASG can help and add duplicate capacity for a single pending pod. AWS recommends sharding only as a last resort, and the only case that still qualifies is a cluster where the binary matching your Kubernetes minor version predates the feature and the upgrade is blocked.

What's next

Run one Cluster Autoscaler with --scale-down-delay-type-local enabled, keep the default safety delays, and let them apply to the group that earned them. Verify three things afterward: one process owns every ASG you care about, the deployed binary actually accepts the flag, and any group that reaches zero has matching metadata across ASG node-template tags, real node configuration, and pod scheduling rules.

Cluster Autoscaler's behavior shifts between releases, and flags have been added, renamed, and defaulted differently over time. Check the AWS provider documentation against the source tag matching your image before you ship, then test it with real workloads rather than a single pause pod.

Postgres LISTEN/NOTIFY in Production: Safe Reconnects & SSE

uninterrupted — Fri, 14 Aug 2026 07:24:09 +0000

If your app needs to push live updates to connected clients — a job progress bar, a balance that changes when a background worker finishes, a "this record was deleted elsewhere" toast — the reflex is to reach for Redis Pub/Sub or a message broker. But if you already run PostgreSQL, it ships with a pub/sub primitive built in: LISTEN / NOTIFY. No extra infrastructure, no extra failure domain.

What is Postgres LISTEN/NOTIFY? It's a built-in pub/sub mechanism: one session runs pg_notify(channel, payload) inside a transaction, and any other session that has issued LISTEN <channel> on a persistent connection receives the payload once that transaction commits. There's no persistence and no delivery guarantee — it's a live signal, not a queue — so it fits best as a "something changed, go re-check" nudge rather than a system of record.

This post walks through the implementation we actually run in production: a single shared LISTEN connection multiplexing many logical channels, wired into per-client Server-Sent Events (SSE) streams. Along the way: the reconnect logic and the race conditions that come with it.

How Postgres LISTEN/NOTIFY works

Postgres gives you two SQL-level building blocks:

SELECT pg_notify('job_status_123', '{"status":"done"}');

and, on a persistent connection:

LISTEN job_status_123;

Any session that has issued LISTEN on a channel receives the payload once the transaction that ran pg_notify commits, not the instant the call executes. Multiple identical notifications on the same channel/payload fired within one transaction get collapsed into one delivery. Several constraints shape everything that follows (see the Postgres NOTIFY reference for the authoritative spec):

Payload is capped at 8000 bytes. Fine for status pings, not for shipping large blobs.
Channel names are Postgres identifiers, truncated at 63 characters. Patterns like job_status_<uuid> need to fit under that limit or get silently truncated and collide.
No persistence, no delivery guarantee. If nobody is listening when pg_notify fires, the message is gone. This is a live signal, not a queue: any client relying on it needs a way to independently reconcile state, e.g. re-fetch from the DB.
The notification queue itself is bounded (8GB by default, per the Postgres async messaging docs). If a listening session sits idle inside an open transaction, it stops draining the queue for everyone, and pg_notify starts erroring once the queue fills. Keep LISTEN sessions out of long-running transactions.

Because LISTEN requires a session-scoped, long-lived connection, and most apps talk to Postgres through a connection pool, you can't just LISTEN from your normal query pool. Pooled connections get recycled and your subscription vanishes with them. You need one connection dedicated to listening, held open for the life of the process.

This also rules out transaction-mode poolers like pgbouncer or RDS Proxy for the LISTEN connection specifically. They can hand your session's underlying socket to a different client between transactions, which silently kills the subscription. The dedicated LISTEN connection needs to bypass the pooler and go straight to Postgres. If you run multiple app instances, each one holds its own LISTEN connection and each receives every NOTIFY independently, which is usually what you want: every instance can push updates to its own locally-connected clients.

One connection, many channels

Rather than open a Postgres connection per subscriber, a single physical LISTEN connection can multiplex an arbitrary number of logical channels. In-process, that's just a map from channel name to a set of callbacks:

class NotificationHub {
  private client: ReturnType<typeof postgres> | null = null;
  private readonly listeners = new Map<Channel, Set<MessageCallback>>();
  private readonly unlistenFns = new Map<Channel, () => Promise<void>>();
  ...

The connection itself is configured to never idle out and to identify itself for observability:

this.client = postgres({
  ...getAppConnectionConfig(),
  max: 1,
  idle_timeout: 0,
  connect_timeout: 10,
  connection: { application_name: "app-listen" },
  onclose: () => {
    /* ... */
  },
});

This pool is deliberately capped at one connection: it exists purely to host a dedicated LISTEN session, not to serve queries. Mixing it into a general-purpose pool would mean losing subscriptions every time the pool recycles a connection.

Handling reconnects: capped exponential backoff

A long-lived TCP connection to a database will eventually drop: network blips, database failover, load balancer idle timeouts. The interesting design question isn't "will it disconnect" but "what happens to every registered channel when it does."

onclose: () => {
  if (!this.closed) {
    dbLogger.warn("LISTEN connection closed unexpectedly");
    this.setState("disconnected");
    this.scheduleReconnect();
  }
},

Reconnect uses a capped exponential backoff so a flapping database doesn't get hammered with reconnect attempts:

const RECONNECT_DELAYS = [1_000, 2_000, 4_000, 8_000, 15_000] as const;

private scheduleReconnect(): void {
  if (this.closed || this.reconnectTimer) return;
  const delayIndex = Math.min(this.reconnectAttempt, RECONNECT_DELAYS.length - 1);
  const delay = RECONNECT_DELAYS[delayIndex];
  this.reconnectAttempt++;
  this.reconnectTimer = setTimeout(() => {
    this.reconnectTimer = null;
    void this.connect();
  }, delay);
}

reconnectAttempt resets to 0 on a successful connect(), so backoff only escalates across consecutive failures. A single flaky reconnect doesn't leave future reconnects artificially delayed.

On reconnect, every channel that had at least one active subscriber gets re-LISTENed from scratch:

const channels = Array.from(this.listeners.keys());
const failedChannels: Channel[] = [];
for (const channel of channels) {
  try {
    await this.listenToChannel(channel);
  } catch (err) {
    failedChannels.push(channel);
  }
}

if (failedChannels.length > 0) {
  this.setState("disconnected");
  this.scheduleReconnect();
  return;
}

Note the all-or-nothing treatment: if even one channel fails to re-LISTEN, the whole reconnect is considered failed and retried. A hub that's "half connected," silently missing notifications on some channels, is worse than one that's visibly disconnected and retrying. At least then the failure isn't invisible to callers.

The unsubscribe race, solved by ordering, not locking

Multiple parts of an app can subscribe to the same channel concurrently (two browser tabs watching the same job, for instance). The tricky moment is when the last subscriber for a channel unsubscribes at the same time a new subscriber for that same channel is arriving:

unsubscribe: async () => {
  for (const channel of channels) {
    const callbacks = this.listeners.get(channel);
    if (!callbacks) continue;
    callbacks.delete(onMessage);

    if (callbacks.size === 0) {
      this.listeners.delete(channel);
      const unlisten = this.unlistenFns.get(channel);
      // Delete before awaiting so a concurrent subscribe() for this
      // channel calls listenToChannel() again. PostgreSQL silently
      // ignores a duplicate LISTEN on the same connection, so the race
      // is harmless — the hub will have an active LISTEN either way.
      this.unlistenFns.delete(channel);
      if (unlisten) {
        try {
          await this.timeoutUnlisten(unlisten);
        } catch { /* timed out or connection gone — safe to ignore */ }
      }
    }
  }
},

The naive version would await UNLISTEN first, then clean up bookkeeping. But between "decide to unlisten" and "UNLISTEN finishes," a new subscriber could arrive, see the channel already has a bookkeeping entry, and skip re-LISTENing. That leaves the hub with no active LISTEN for a channel it thinks it's subscribed to.

By deleting the map entry before awaiting UNLISTEN, a concurrent subscribe() call always sees "no entry" and issues its own LISTEN unconditionally. Postgres treats a duplicate LISTEN on the same session as a no-op, so the worst case is two harmless LISTEN calls in flight instead of a silently dropped subscription. When you can't easily lock across an async boundary, look for an operation idempotent enough to make the race safe by construction.

Callers should never worry about connection state

The public subscribe() API registers callbacks unconditionally, regardless of whether the underlying Postgres connection is currently up. It only rejects if an opportunistic LISTEN call fails outright while attempting an already-connected session, and that case is recovered by the reconnect logic on its next attempt:

async subscribe(channels: Channel[], onMessage: MessageCallback): Promise<Subscription> {
  // Register callbacks regardless of connection state
  for (const channel of channels) {
    let callbacks = this.listeners.get(channel);
    if (!callbacks) {
      callbacks = new Set();
      this.listeners.set(channel, callbacks);
    }
    callbacks.add(onMessage);
  }

  if (this.state === "connected" && this.client) {
    for (const channel of channels) {
      await this.listenToChannel(channel); // no-op if already LISTENing
    }
  }

  if (this.state === "disconnected" && !this.connecting && !this.reconnectTimer) {
    void this.connect();
  }

  return { unsubscribe: async () => { /* ... */ } };
}

Callbacks are registered in memory first; the actual LISTEN happens opportunistically if connected, or gets picked up automatically on the next successful reconnect. Every consumer of the hub stays decoupled from the connection's health. They just subscribe, and messages show up when the pipe is open.

Wiring it into Server-Sent Events

The hub by itself is transport-agnostic; a thin SSE layer turns it into per-client streams with keep-alives and cleanup on disconnect:

const stream = new ReadableStream({
  start(controller) {
    let subscription: Subscription | null = null;

    const onAbort = () => {
      void cleanup();
    };
    options.request.signal.addEventListener("abort", onAbort);

    keepAliveInterval = setInterval(
      () => safeEnqueue(`: ping\n\n`),
      keepAliveMs,
    );

    const init = async () => {
      await options.onStart(safeEnqueue, () => {
        void cleanup();
      });
      if (!options.subscribe || isClosed) return;

      const { channels, onMessage } = options.subscribe;

      unsubscribeState = onHubStateChange((state) => {
        safeEnqueue(
          `data: ${JSON.stringify({ type: "info", liveUpdates: state === "connected" })}\n\n`,
        );
      });

      subscription = await subscribe(channels, (ch, message) => {
        if (!isClosed)
          onMessage(ch, message, safeEnqueue, () => {
            void cleanup();
          });
      });

      if (options.onAfterSubscribe) {
        await options.onAfterSubscribe(safeEnqueue, () => {
          void cleanup();
        });
      }
    };

    init().catch(() => {
      void cleanup();
    });
  },
  cancel() {
    return cleanupFn?.();
  },
});

Two details worth calling out.

The hub's connection state is surfaced to the client. When onHubStateChange reports "disconnected", an {"type":"info","liveUpdates":false} frame goes out over SSE so the UI can show "reconnecting..." instead of silently going stale. Since NOTIFY gives no delivery guarantee, telling the client when it might be missing updates is almost as important as delivering the updates themselves.

There's also a read-then-subscribe race window. onStart typically does an initial DB read to send current state, and only afterwards does the code call subscribe() to start receiving live updates. If a NOTIFY fires in the gap between that read and the LISTEN actually being established, it's lost for good: the classic "check-then-watch" race in any pub/sub system. The onAfterSubscribe hook closes that window. After the subscription is confirmed active, it re-checks state one more time, so any update that landed in the gap gets picked up on the second read instead of being silently missed. The client-side handler needs to tolerate receiving the same update twice (once from the catch-up read, once from a live NOTIFY that arrives moments later). Treat updates as idempotent, not as a strictly-once stream.

When this pattern fits

Postgres LISTEN/NOTIFY works well when:

Update volume is modest (dashboards, job status, per-user notifications), not a high-throughput event bus.
Clients treat NOTIFY as a nudge to re-sync, not as the sole source of truth. Always have a path to reconstruct state via a normal query if a notification is missed.
You want one fewer piece of infrastructure to run and monitor.

It's the wrong tool when you need guaranteed delivery, durable queues, replay, or very high fan-out. That's what a real message broker is for. But for "tell connected clients something changed," Postgres has had a message bus built in the whole time.

Keycloak at 100% CPU After Deploy: Root Cause and Fix

Bartek Gałęzowski — Fri, 14 Aug 2026 07:08:30 +0000

The Symptom: Recognizing This CPU Spike

You deploy a containerized Keycloak instance — even the official quay.io/keycloak/keycloak image, unmodified — to a managed container platform, and within seconds CPU utilization climbs to 100% and stays there — regardless of traffic. No requests are being served, no realm imports are running, and nothing in the application logs looks wrong. Health checks pass. HTTP endpoints respond normally.

This pattern shows up most often on:

DigitalOcean App Platform
Google Cloud Run
AWS ECS Fargate
Heroku
Self-managed or managed Kubernetes clusters without a multicast-aware CNI

Root Cause: Infinispan, JGroups, and the Multicast Problem

The Default: Distributed Caching via Infinispan

By default, Keycloak ships with a distributed cache mode built on Infinispan, an embedded data grid used to replicate session state, authentication sessions, and other transient data across cluster nodes. This is the ispn cache provider, and it's the default unless explicitly overridden — either at build time (kc.sh build --cache=<mode>) or at runtime (KC_CACHE).

Why JGroups Can't Discover Peers

Infinispan's clustering layer relies on JGroups, a toolkit for reliable group communication. JGroups needs a discovery protocol to find other cluster members, and out of the box, Keycloak's default JGroups stack uses UDP-based discovery (PING/MPING), which depends on IP multicast to broadcast "who else is out there?" messages across the network.

Managed container platforms don't expose multicast-capable networking. Their networking layers are software-defined, ephemeral, and built around dynamic scheduling of containers across shared infrastructure — a model that's fundamentally incompatible with IP multicast, which assumes a flat, broadcast-capable network segment. This holds almost universally: PaaS offerings, serverless container runtimes, and even many managed Kubernetes configurations without a multicast-aware CNI all impose the same restriction.

The Result: A Silent, Load-Independent CPU Burn

JGroups doesn't fail gracefully when discovery is unreachable. Instead, it enters an aggressive retry loop — broadcasting discovery packets, waiting for a timeout, retrying, re-forming view negotiations, and repeating indefinitely. Each cycle consumes CPU on packet construction, timeout scheduling, and thread contention, multiplied across the JGroups protocol stack (transport, discovery, failure detection, and flow control protocols all layered on top of one another).

None of this is visible from an application standpoint. A background thread pool is perpetually spinning, trying to solve a problem — finding cluster peers via multicast — that's structurally impossible in this environment. The result is sustained, load-independent CPU burn with no connection to request volume.

Setting KC_CACHE=local eliminates the problem — not by fixing discovery, but by disabling clustering altogether. In local mode, Keycloak uses in-memory, non-replicated caches with no Infinispan clustering, no JGroups stack, and therefore no discovery loop to fail.

Why It Happens After Redeploys, Not on First Boot

First Boot: A Clean, Uncontested Singleton

On the very first deploy, exactly one container exists. When JGroups' discovery protocol fires, it queries whatever peer list mechanism is configured — raw UDP multicast, or more commonly on managed platforms, a DNS-based or static-host discovery protocol that resolves the platform's internal service name to the currently running container IPs. Since this is the first and only container, that resolution returns exactly one address: itself. There's nothing to negotiate and no peer to merge with. JGroups times out once on the discovery request, finds no one else, and installs itself as a singleton cluster view — a cheap, one-shot operation that doesn't manifest as sustained CPU load.

Redeploy or Restart: A Two-Node Race That Never Resolves

A redeploy or restart on most managed platforms is a rolling operation, not an instant swap. The platform starts the new container before fully tearing down the old one, to avoid downtime. For a short window, both the outgoing and incoming containers are alive simultaneously, and depending on how peer discovery is configured, both may be resolvable under the same internal service identity.

This is where things diverge from the first-boot case: JGroups on the new node now believes there is another cluster member to reach. It attempts to establish a channel, negotiate a merged view, and start heartbeat-based failure detection with that peer — but the platform's networking still doesn't permit the actual cache-cluster transport traffic between containers (multicast is blocked, and often the cache port isn't even routed between sibling containers in the first place). The handshake can never complete: the peer is addressable enough to be discovered, but not reachable enough to actually communicate.

The result is a protocol deadlock rather than a clean failure. JGroups cycles through:

periodic heartbeat attempts to the discovered peer
failure-suspicion timers when heartbeats go unanswered
merge-coordinator election attempts to decide which node's view should win
state-transfer request/response handshakes that time out without completing

Each sub-protocol retries independently and indefinitely, because there's no terminal "peer confirmed dead, drop it forever" signal — the old container is still technically registered as existing from the discovery layer's point of view, even though it's unreachable at the transport layer. This composite retry storm is considerably more CPU-intensive than the one-shot singleton bootstrap on first boot, which is why the spike reliably appears after a redeploy or restart, but not on the initial one.

How to Fix It: Step-by-Step

Confirm you're hitting this issue. Check CPU metrics with no active traffic — a sustained 100% with zero requests is the signature. Confirm KC_CACHE is unset or set to ispn (the default).
Decide whether you actually need clustering. If you're running a single Keycloak instance, which covers most small and mid-sized deployments, you don't need Infinispan's distributed cache at all.
Single instance → set KC_CACHE=local. Add it as a runtime environment variable, or bake it in at build time with -cache=local. Redeploy and confirm CPU returns to baseline.
Multiple instances / HA → switch to JDBC_PING. Build with -cache-stack=jdbc-ping (or supply a custom cache-ispn.xml), pointing at your existing database. This keeps session replication working without depending on multicast.
Verify after a redeploy, not just first boot. Because the issue specifically reproduces on rolling restarts, don't declare victory after the first boot alone — trigger a redeploy and confirm CPU stays flat afterward too.

The Trade-Offs of KC_CACHE=local

KC_CACHE=local is a valid and common fix, but it comes with real trade-offs:

No safe horizontal scaling. Run more than one replica behind a load balancer, and users can be routed to a different instance mid-session — silently logged out, losing SSO state, or hitting inconsistent authorization decisions.
Restarts are lossy. A container restart — which managed platforms perform routinely during deploys, scaling events, or health-check failures — wipes all in-memory session state for that instance.

For a single-instance deployment, none of this is a compromise — there's no cluster to form in the first place, so disabling clustering is simply correct.

Need High Availability? Use JDBC_PING Instead

If horizontal scaling or high availability is a requirement, the architecturally correct fix isn't to disable clustering — it's to replace the discovery mechanism with one that doesn't depend on multicast.

Keycloak, via Infinispan/JGroups, supports a JDBC_PING discovery protocol that uses your existing relational database as a shared coordination point: each node writes its cluster address to a database table, and peers discover each other by querying that table instead of broadcasting UDP packets.

This is configured via a custom JGroups stack — --cache-stack=jdbc-ping at build time, or a custom cache-ispn.xml descriptor — and it works reliably on any managed platform, regardless of vendor, because it never depends on network-layer broadcast at all. The cost is a small, constant amount of additional load on your database (heartbeat writes/reads at a configurable interval) in exchange for genuine session replication across nodes.

KC_CACHE Settings: Quick Reference

Scenario	Recommended Setting	Why
Single Keycloak instance, no plans to scale horizontally	`KC_CACHE=local`	No cluster to form — disables Infinispan/JGroups entirely
Multiple instances, need session replication / HA	`KC_CACHE=ispn` with `--cache-stack=jdbc-ping`	Keeps clustering, swaps multicast discovery for database-backed discovery
Multiple instances, occasional failover session loss acceptable (rare)	`KC_CACHE=ispn` with default stack	Not recommended on managed platforms — reproduces the CPU issue

Faster Geospatial Animations with PMTiles: Multi-Layer MVT Time Series in OpenLayers

uninterrupted — Wed, 08 Jul 2026 06:00:00 +0000

Most people try to animate time series by stacking separate tile sources or flipping CSS styles mid-frame. That causes flicker and a surprising amount of wasted CPU/GPU work. The core takeaway is simple: bake your timesteps into a single vector-tile archive (PMTiles) with per-timestep named layers, load it once, and drive animation by re-styling already-decoded tiles.

I tested this with OpenLayers and noticed the difference right away: playback became smoother and far less CPU-bound once tiles were pre-baked and cached. The hard part is server-side — how you generate, package, and serve tiles matters more than which client library you pick.

When you animate a map by switching between separate tile sources, each frame causes one of two expensive operations: a network fetch of new tiles, or client-side re-decoding and re-styling. Baking timesteps into one archive avoids that by turning each frame into a cheap style re-evaluation on already-decoded features.

The architecture: single archive, dynamic layers

MVT (Mapbox Vector Tile) is a compact protobuf-based vector tile format that encodes geometry and attributes per z/x/y tile. PMTiles packages MVT tiles into a single file for efficient distribution and partial HTTP range retrieval.

The key workflow:

Server side — each timestep is a separate GeoJSON file. tippecanoe joins them into one PMTiles archive, giving each timestep its own named MVT layer (timestep_00, timestep_01, …).
Client side — one VectorTileSource loads the archive once. One VectorTileLayer uses a style function that reads a mutable currentLayerName variable. Advancing a frame means: update the variable, call layer.changed(). OpenLayers re-runs the style function on the already-decoded tile features and repaints — no network request, no re-decode.

Complete working example

!animation.gif

This section contains everything needed to reproduce the demo from scratch: 11 per-timestep GeoJSON files, a tile generation script, and the client HTML.

Step 1 — Per-timestep GeoJSON files

Create 11 files, one per timestep. Each contains a single large polygon with a value property. The temporal identity comes from the file name (and the MVT layer name tippecanoe will assign), not from a property on the feature.

for i in $(seq 0 10); do
  pad=$(printf "%02d" $i)
  value=$((i * 10))
  cat > "square_t${pad}.geojson" << EOF
{
  "type": "FeatureCollection",
  "features": [{
    "type": "Feature",
    "geometry": {
      "type": "Polygon",
      "coordinates": [[[-60,-50],[60,-50],[60,50],[-60,50],[-60,-50]]]
    },
    "properties": { "value": ${value} }
  }]
}
EOF
done

This produces square_t00.geojson (value=0) through square_t10.geojson (value=100).

Step 2 — Generate PMTiles with named layers (`generate-tiles.sh`)

#!/usr/bin/env bash
set -euo pipefail

SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"

tippecanoe \
  --named-layer=timestep_00:"$SCRIPT_DIR/square_t00.geojson" \
  --named-layer=timestep_01:"$SCRIPT_DIR/square_t01.geojson" \
  --named-layer=timestep_02:"$SCRIPT_DIR/square_t02.geojson" \
  --named-layer=timestep_03:"$SCRIPT_DIR/square_t03.geojson" \
  --named-layer=timestep_04:"$SCRIPT_DIR/square_t04.geojson" \
  --named-layer=timestep_05:"$SCRIPT_DIR/square_t05.geojson" \
  --named-layer=timestep_06:"$SCRIPT_DIR/square_t06.geojson" \
  --named-layer=timestep_07:"$SCRIPT_DIR/square_t07.geojson" \
  --named-layer=timestep_08:"$SCRIPT_DIR/square_t08.geojson" \
  --named-layer=timestep_09:"$SCRIPT_DIR/square_t09.geojson" \
  --named-layer=timestep_10:"$SCRIPT_DIR/square_t10.geojson" \
  -o "$SCRIPT_DIR/square.pmtiles" \
  -z 8 -Z 0 \
  --no-simplification \
  --no-tile-size-limit \
  --no-feature-limit \
  -f

Run with: bash generate-tiles.sh

Step 3 — Start the server

npx serve

Step 4 — Client animation (`index.html`)

The full self-contained HTML file. No build step, no npm install — all dependencies come from esm.sh, which rewrites bare module specifiers (like pbf imported inside ol/format/MVT) to absolute CDN URLs that the browser can resolve.

<!DOCTYPE html>
<html lang="en">
<head>
  <meta charset="UTF-8" />
  <meta name="viewport" content="width=device-width, initial-scale=1.0" />
  <title>MVT Animation</title>
  <link rel="stylesheet" href="<https://cdn.jsdelivr.net/npm/ol@10.9.0/ol.css>" />
  <style>
    * { box-sizing: border-box; margin: 0; padding: 0; }
    body { font-family: monospace; background: #111; color: #eee;
           display: flex; flex-direction: column; height: 100vh; }
    #map  { flex: 1; }
    #hud  { padding: 12px 20px; background: #1a1a1a; border-top: 1px solid #333;
            display: flex; align-items: center; gap: 24px; }
    #hud label { font-size: 13px; color: #aaa; }
    #hud span  { font-size: 16px; font-weight: bold; }
    #timestep-val { color: #7af; }
    #value-val    { color: #fa7; }
    #color-swatch { width: 28px; height: 28px; border-radius: 4px;
                    border: 1px solid #555; display: inline-block; }
    #progress-bar { flex: 1; height: 8px; background: #333; border-radius: 4px; overflow: hidden; }
    #progress-fill { height: 100%; background: linear-gradient(to right, #00f, #f00); border-radius: 4px; }
    #controls { display: flex; gap: 8px; }
    button { background: #333; color: #eee; border: 1px solid #555;
             padding: 5px 14px; border-radius: 4px; cursor: pointer; font-size: 13px; }
    button:hover { background: #444; }
    #loading { position: absolute; inset: 0; display: flex; align-items: center;
               justify-content: center; background: rgba(0,0,0,0.55); color: #fff;
               font-size: 15px; pointer-events: none; z-index: 999; }
    #loading.hidden { display: none; }
  </style>
</head>
<body>
  <div id="map"><div id="loading">Loading tiles…</div></div>
  <div id="hud">
    <div><label>Timestep </label><span id="timestep-val"></span></div>
    <div><label>Value </label><span id="value-val"></span></div>
    <div style="display:flex;align-items:center;gap:8px;">
      <label>Color </label><span id="color-swatch"></span>
    </div>
    <div id="progress-bar"><div id="progress-fill" style="width:0%"></div></div>
    <div id="controls">
      <button id="btn-playpause">Pause</button>
      <button id="btn-reset">Reset</button>
    </div>
  </div>

  <script type="module">
    import Map              from '<https://esm.sh/ol@10.9.0/Map>';
    import View             from '<https://esm.sh/ol@10.9.0/View>';
    import TileLayer        from '<https://esm.sh/ol@10.9.0/layer/Tile>';
    import OSM              from '<https://esm.sh/ol@10.9.0/source/OSM>';
    import VectorTileLayer  from '<https://esm.sh/ol@10.9.0/layer/VectorTile>';
    import VectorTileSource from '<https://esm.sh/ol@10.9.0/source/VectorTile>';
    import MVT              from '<https://esm.sh/ol@10.9.0/format/MVT>';
    import Style            from '<https://esm.sh/ol@10.9.0/style/Style>';
    import Fill             from '<https://esm.sh/ol@10.9.0/style/Fill>';
    import Stroke           from '<https://esm.sh/ol@10.9.0/style/Stroke>';
    import { fromLonLat }   from '<https://esm.sh/ol@10.9.0/proj>';
    import { PMTiles }      from '<https://esm.sh/pmtiles@4.4.1>';

    const LAYER_NAMES = [
      'timestep_00', 'timestep_01', 'timestep_02', 'timestep_03',
      'timestep_04', 'timestep_05', 'timestep_06', 'timestep_07',
      'timestep_08', 'timestep_09', 'timestep_10',
    ];
    const VALUES = LAYER_NAMES.map((_, i) => i * 10);

    const STYLES = LAYER_NAMES.map((_, i) => new Style({
      fill:   new Fill({ color: `rgb(${Math.round(255*i/10)},0,${Math.round(255*(1-i/10))})` }),
      stroke: new Stroke({ color: '#fff', width: 2 }),
    }));

    const archive = new PMTiles('square.pmtiles');

    const source = new VectorTileSource({
      format: new MVT(),
      url: 'square.pmtiles',
      maxZoom: 8,
    });

    let currentLayerIndex = 0;

    const squareLayer = new VectorTileLayer({
      source,
      style: (feature) => {
        return feature.get('layer') === LAYER_NAMES[currentLayerIndex]
          ? STYLES[currentLayerIndex] : null;
      },
    });

    const map = new Map({
      target: 'map',
      layers: [ new TileLayer({ source: new OSM() }), squareLayer ],
      view: new View({ center: fromLonLat([0, 0]), zoom: 2 }),
    });

    const FRAME_MS = 500;

    function animate(frames, onFrame = () => {}) {
      let i = 0;
      let intervalId = setInterval(() => {
        currentLayerIndex = (currentLayerIndex + 1) % frames.length;
        squareLayer.changed();
        onFrame(frames[i]);
        i = (i + 1) % frames.length;
      }, FRAME_MS);

      return {
        pause()  { clearInterval(intervalId); },
        resume() { intervalId = setInterval(() => {
          currentLayerIndex = (currentLayerIndex + 1) % frames.length;
          squareLayer.changed();
          onFrame(frames[i]);
          i = (i + 1) % frames.length;
        }, FRAME_MS); },
        reset()  { i = 0; },
      };
    }

    const timestepEl   = document.getElementById('timestep-val');
    const valueEl      = document.getElementById('value-val');
    const swatchEl     = document.getElementById('color-swatch');
    const progressFill = document.getElementById('progress-fill');

    function onFrame(layerName) {
      const i = LAYER_NAMES.indexOf(layerName);
      timestepEl.textContent    = layerName;
      valueEl.textContent       = VALUES[i];
      swatchEl.style.background = STYLES[i].getFill().getColor();
      progressFill.style.width  = `${(i / (LAYER_NAMES.length - 1)) * 100}%`;
    }

    onFrame(LAYER_NAMES[0]);

    const loadingEl = document.getElementById('loading');
    let player = null;
    let started = false;

    function startWhenReady() {
      if (started) return;
      started = true;
      loadingEl.classList.add('hidden');
      player = animate(LAYER_NAMES, onFrame);
    }

    source.once('tileloadend', startWhenReady);
    map.once('rendercomplete', startWhenReady);
    setTimeout(startWhenReady, 10000);

    let playing = true;
    document.getElementById('btn-playpause').addEventListener('click', (e) => {
      if (!player) return;
      playing = !playing;
      e.target.textContent = playing ? 'Pause' : 'Play';
      playing ? player.resume() : player.pause();
    });
    document.getElementById('btn-reset').addEventListener('click', () => {
      if (!player) return;
      player.reset();
      if (!playing) {
        playing = true;
        document.getElementById('btn-playpause').textContent = 'Pause';
        player.resume();
      }
    });
  </script>
</body>
</html>

Running the full setup

# 1. Install tippecanoe (macOS)
brew install tippecanoe

# 2. Create the 11 per-timestep GeoJSON files
for i in $(seq 0 10); do
  pad=$(printf "%02d" $i); value=$((i * 10))
  echo '{"type":"FeatureCollection","features":[{"type":"Feature","geometry":{"type":"Polygon","coordinates":[[[-60,-50],[60,-50],[60,50],[-60,50],[-60,-50]]]},"properties":{"value":'$value'}}]}' \
    > square_t${pad}.geojson
done

# 3. Generate the PMTiles archive
bash generate-tiles.sh

# 4. Start the server
npx serve

# 5. Open the demo
open <http://localhost:3000/index.html>

Conclusion: paint time, not flip it

Don't stack separate tile sources for time-series animation. Bake timesteps into a single PMTiles archive as named MVT layers, load it once, and drive playback by updating a closure variable and calling layer.changed(). Serve the archive with npx serve. Start animation only after the first tile arrives.

The result: smooth playback from first load, no flicker on hard refresh, no blank map with cache disabled.

Argo CD GitLab Authentication: A Complete GitOps-Friendly Setup

Paweł Swiridow — Wed, 01 Jul 2026 10:00:00 +0000

Argo CD can use GitLab as an authentication provider, so users log in with their GitLab accounts instead of local Argo CD users. That gives you centralized identity, cleaner access control, and one less password database waiting to become a security incident with a logo.

The setup uses OIDC through Dex, stores the GitLab OAuth credentials as encrypted Kubernetes secrets with SOPS and age, and lets Argo CD decrypt them with KSOPS during reconciliation.

How authentication works

Argo CD does not talk to GitLab directly for user login. It uses Dex as an identity broker.

The flow looks like this:

User -> Argo CD -> Dex -> GitLab -> Dex -> Argo CD -> RBAC

The short version:

The user opens Argo CD.
Argo CD redirects the user to Dex.
Dex redirects the user to GitLab.
GitLab authenticates the user and returns OIDC claims.
Argo CD reads those claims and applies RBAC rules.

Authentication tells Argo CD who the user is. RBAC tells Argo CD what damage they are allowed to do. Ideally, not much.

Create a GitLab OAuth application

First, create an OAuth application in GitLab.

In GitLab, go to your user, group, or admin application settings and create a new application with these values:

Name: Argo CD
Redirect URI: https://argocd.example.com/api/dex/callback
Scopes: openid, read_user

GitLab gives you two important values:

Application ID
Secret

In the Argo CD Dex config, these become:

dex.gitlab.clientID: "<gitlab-application-id>"
dex.gitlab.clientSecret: "<gitlab-application-secret>"
dex.gitlab.redirectURI: "https://argocd.example.com/api/dex/callback"

Do not commit the secret in plain YAML. That is not GitOps. That is just leaving your keys under the doormat and calling it automation.

Encrypt the Argo CD secret with SOPS and age

Use SOPS with age to encrypt the Kubernetes secret before committing it.

Generate an age key if you do not have one yet:

age-keygen -o age.key

Store the public key in .sops.yaml:

creation_rules:
  - encrypted_regex: '^(data|stringData)$'
    age: age1examplepublickeyreplacewithyourrealpublickey

Create argocd-secret.yml:

apiVersion: v1
kind: Secret
metadata:
  name: argocd-secret
  namespace: argocd
type: Opaque
stringData:
  dex.gitlab.clientID: "<gitlab-application-id>"
  dex.gitlab.clientSecret: "<gitlab-application-secret>"
  dex.gitlab.redirectURI: "https://argocd.example.com/api/dex/callback"

Encrypt it:

sops --encrypt argocd-secret.yml > argocd-secret.sops.yml

Then remove the unencrypted file:

rm argocd-secret.yml

Yes, really remove it. Git has a long memory and an even longer blame history.

Add KSOPS to the Argo CD repo server

Argo CD does not decrypt SOPS files by default. To make this work with Kustomize, install the KSOPS plugin in the Argo CD repo server.

The plugin lives here:

https://github.com/viaduct-ai/kustomize-sops

If you deploy Argo CD with Helm, add this to your values:

repoServer:
  volumeMounts:
    - name: custom-tools
      mountPath: /usr/local/bin/kustomize
      subPath: kustomize
    - name: custom-tools
      mountPath: /usr/local/bin/ksops
      subPath: ksops
  volumes:
    - name: custom-tools
      emptyDir: {}
  initContainers:
    - name: install-ksops
      image: viaductoss/ksops:v4.5.1
      command:
        [
          "/usr/local/bin/ksops",
          "install",
          "--with-kustomize",
          "/custom-tools",
        ]
      volumeMounts:
        - mountPath: /custom-tools
          name: custom-tools

This installs ksops and a Kustomize binary with plugin support into the repo server container.

Provide the age private key to Argo CD

The repo server needs the age private key to decrypt the SOPS file.

Create a Kubernetes secret from your age key:

kubectl -n argocd create secret generic sops-age-key \
  --from-literal=key="$(cat age.key)"

Expose it to the repo server as SOPS_AGE_KEY:

repoServer:
  env:
    - name: SOPS_AGE_KEY
      valueFrom:
        secretKeyRef:
          name: sops-age-key
          key: key
  volumeMounts:
    - name: custom-tools
      mountPath: /usr/local/bin/kustomize
      subPath: kustomize
    - name: custom-tools
      mountPath: /usr/local/bin/ksops
      subPath: ksops
  volumes:
    - name: custom-tools
      emptyDir: {}

Protect this key properly. Anyone with the private key can decrypt the secrets. Kubernetes secrets are base64 encoded, not magically protected by tiny YAML elves.

Create the KSOPS generator

Create secret-generator.yml:

apiVersion: viaduct.ai/v1
kind: ksops
metadata:
  name: argocd-secret-generator
  annotations:
    config.kubernetes.io/function: |
      exec:
        path: ksops
files:
  - argocd-secret.sops.yml

This tells Kustomize to call ksops and decrypt argocd-secret.sops.yml during the build.

Reference the generator in Kustomize

In kustomization.yml, add the generator:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

generators:
  - secret-generator.yml

When Argo CD renders this application, Kustomize runs the KSOPS generator and produces the decrypted argocd-secret resource.

Configure Dex in Argo CD

Now configure Argo CD to use GitLab through Dex.

In Helm values, configure argocd-cm like this:

configs:
  cm:
    kustomize.buildOptions: "--enable-alpha-plugins --enable-exec"
    dex.config: |
      connectors:
        - type: gitlab
          id: gitlab
          name: GitLab
          config:
            baseURL: https://gitlab.example.com
            clientID: $dex.gitlab.clientID
            clientSecret: $dex.gitlab.clientSecret

The $dex.gitlab.* values are loaded from argocd-secret. That is why the secret names must match exactly.

The Kustomize build options matter:

kustomize.buildOptions: "--enable-alpha-plugins --enable-exec"

This allows Kustomize to execute the KSOPS plugin.

Do not enable exec plugins for repositories you do not trust. Otherwise, you have invented remote command execution with extra YAML. Security teams love that. Briefly.

Configure Argo CD RBAC

Authentication is not authorization. After GitLab login works, configure Argo CD RBAC.

For simple user mapping by email:

helm:
  valuesObject:
    configs:
      rbac:
        policy.csv: |
          g, alice@example.com, role:admin
          g, bob@example.com, role:readonly
        scopes: "[email,groups]"

This maps:

alice@example.com -> role:admin
bob@example.com -> role:readonly

For larger teams, prefer GitLab groups instead of individual users. User by user RBAC starts cute and ends as spreadsheet archaeology.

Apply the configuration

If Argo CD is managed by another Argo CD application, commit the files and let Argo CD sync them.

If you are applying locally for a test, run:

kustomize build --enable-alpha-plugins --enable-exec . | kubectl apply -f -

Then restart the Argo CD components if needed:

kubectl -n argocd rollout restart deployment argocd-server
kubectl -n argocd rollout restart deployment argocd-dex-server
kubectl -n argocd rollout restart deployment argocd-repo-server

Verify the setup

Check that the secret exists:

kubectl -n argocd get secret argocd-secret

Check the Dex logs:

kubectl -n argocd logs deploy/argocd-dex-server

Check the Argo CD server logs:

kubectl -n argocd logs deploy/argocd-server

Check the repo server if decryption fails:

kubectl -n argocd logs deploy/argocd-repo-server

Open Argo CD in the browser:

https://argocd.example.com

You should see a GitLab login option. After login, the user should land in Argo CD with the expected role.

Troubleshoot common failures

If GitLab returns a redirect error, check the redirect URI.

It must match exactly:

https://argocd.example.com/api/dex/callback

If login works but RBAC does not, check which claim Argo CD receives. The email in policy.csv must match the user claim.

If the secret does not render, check the repo server pod:

kubectl -n argocd describe pod -l app.kubernetes.io/name=argocd-repo-server

Common causes:

KSOPS is not installed in the repo server.
-enable-alpha-plugins --enable-exec is missing.
The age private key is not available in the repo server environment.
SOPS_AGE_KEY is missing or points to the wrong Kubernetes secret key.
The encrypted file name does not match secret-generator.yml.
The secret keys do not match $dex.gitlab.clientID, $dex.gitlab.clientSecret, or $dex.gitlab.redirectURI.

You can test decryption locally if you have the age key:

SOPS_AGE_KEY="$(cat age.key)" sops --decrypt argocd-secret.sops.yml

Security notes

Keep the boring security parts boring. Boring is good. Boring means nobody is explaining an outage in a meeting with too many directors.

Encrypt the GitLab OAuth secret with SOPS.
Protect the age private key.
Limit who can change Argo CD applications that use Kustomize exec plugins.
Use GitLab groups for larger teams.
Rotate the GitLab OAuth secret periodically.
Audit role:admin assignments.
Avoid giving admin access to users because they asked nicely in Slack.

Final result

With this setup, GitLab becomes the login provider for Argo CD, secrets stay encrypted in Git, and Argo CD RBAC controls what authenticated users can do.

The moving parts are not complicated:

GitLab OAuth app
SOPS encrypted argocd-secret
KSOPS in repo server
Dex GitLab connector
Argo CD RBAC policy

That is a reasonable amount of YAML for centralized authentication. Not small, obviously. This is Kubernetes. We measure configuration in geological layers.

DEV Community: u11d

MinIO Is Deprecated: Best Alternatives for S3-Compatible Storage

What actually happened to MinIO?

Alternative #1: Garage

Alternative #2: RustFS

Alternative #3: SeaweedFS

Garage vs RustFS vs SeaweedFS

Conclusion

TypeSafe AI Jev explained: examples, use cases, and LLM comparisons

Quick answer

What is TypeSafe AI Jev?

How Choice, Score, and Noul work

Run a Jev decision in Python

Install the SDK

Classify and route a support ticket

Sample output and cost

Practical Jev use cases

Route support requests

Review agent actions

Check RAG evidence before generation

Triage moderation and operations queues

Route work to the right model or human

Jev vs LLMs, classifiers, and rules

When not to use TypeSafe AI Jev

How to evaluate Jev on your workload

Should you use TypeSafe AI Jev?

Implementing Postgres Row-Level Security in Next.js: The Drizzle Multi-Tenant Pattern

Two roles, two connections

The policy itself

Getting context into the transaction: SET LOCAL

Gotcha #1: you can't parameterize SET

Gotcha #2: Drizzle's transaction type doesn't match its client type

Gotcha #3: don't let a stream hold your transaction

Gotcha #4: knowing which connection you're on

Fluctum: Open-Source Dynamic Pricing Solution for Stores Selling Spot-Priced Goods

Where This Problem Shows Up

The Real Cost of Building This Yourself

What a Ready-Made Solution Looks Like

Which one should you choose?

Quick Reference

Conclusion

How to get fixed IP addresses for an AWS ALB with Global Accelerator

Quick answer

Why ALB alone does not work

How Global Accelerator gives an ALB fixed inbound IPs

Configure AWS Global Accelerator for an ALB with Terraform

Add DNS without losing the fixed IPs

Protect the ALB from direct access

Keep the real client IP

Verify the fixed IP setup

Troubleshooting common errors

Multi-Region is the obvious next step

Things to remember

Building a Multilingual Shop with Next.js

Implementation Remarks

Product data is part of the translation resources

All languages are stored in a single file

No external translation management system

Choosing an i18n Solution for Next.js

Why not build it ourselves?

Language-Based Routing

Why put the language in the URL?

Detecting the User's Language with Proxy

URL comes first

Cookie as persistent preference

Falling back to Accept-Language

Finally, use a fallback

Translation Resources and Namespaces

Loading resources through resourceLoader

Why introduce an API layer?

Server and Client Components

Keep translated content on the server

Providing the resources from the layout

Why this separation matters

Formatting Prices

Localized Navigation

LocalizedLink

Switching between languages

Production Considerations

Caching translation resources

Falling back to `Accept-Language`

Loading resources through `resourceLoader`

SEO and `hreflang`

Step 2 — Generate PMTiles with named layers (`generate-tiles.sh`)

Step 4 — Client animation (`index.html`)