DEV Community: 𝗝𝗼𝗵𝗻

Just received a DEV Contributor badge for suggesting changes to the forem documentation! Hell yeah!

𝗝𝗼𝗵𝗻 — Tue, 20 Oct 2026 19:09:37 +0000

The Hidden Cost of AI-Generated PRs: Why Most Teams Are Missing Silent Breakages

𝗝𝗼𝗵𝗻 — Tue, 20 Oct 2026 14:30:00 +0000

Your team is shipping pull requests twice as fast as last year, the CI pipeline is green, and every diff looks cleanly formatted. Yet your production incident rate is quietly climbing.

According to Google Cloud's DORA State of AI-Assisted Software Development research, while roughly 90% of engineering teams now use AI tools daily to accelerate coding, increased AI adoption without matching quality guardrails correlates directly with a measurable drop in software delivery stability. Code generation is no longer the bottleneck—code verification is.

When pull request volume doubles while human review capacity stays flat, engineering teams fall into the "LGTM trap": rubber-stamping large, plausible-looking diffs that harbor silent, state-corrupting bugs.

The Problem: Plausible Code Is Harder to Debug Than Broken Code

When a junior developer writes a bug from scratch, the code usually looks suspicious. Naming is inconsistent, edge cases are visibly missing, or the syntax feels clunky. A senior reviewer spots the smell in seconds.

AI-generated code flips this heuristic on its head. Modern coding assistants produce code that is:

Syntactically idiomatic and cleanly formatted.
Accompanied by confident, well-structured PR descriptions.
Paired with unit tests that pass—because the tests were generated from the exact same flawed implementation context.

The Reviewer's Paradox: The cleaner and more idiomatic an AI-generated diff looks on the surface, the less cognitive effort a fatigued human reviewer spends tracing its underlying state transitions and failure modes.

The Common Mistake: Local Correctness vs. System Invariants

Coding agents excel at local context (the open file and immediate type definitions), but they frequently miss system-wide invariants—such as idempotency, transactional boundaries, permission scoping, and partial failure recovery.

Consider a real-world backend scenario: updating a user's subscription tier and syncing billing credits.

What Gets Approved in a Hurry (The Silent Bug)

An engineer prompts an assistant to "Add an endpoint handler that upgrades a user's plan and grants bonus credits, with unit tests." The PR arrives looking clean:

// ❌ Looks clean in a 600-line PR diff, but breaks production invariants
export async function upgradeSubscription(
  userId: string,
  newPlanId: string
): Promise<{ status: string; credits: number }> {
  const user = await db.users.findUnique({ where: { id: userId } });
  if (!user) {
    throw new Error("User not found");
  }

  // 1. External call inside business flow without idempotency key
  await billingClient.chargeUpgradeFee(user.stripeCustomerId, newPlanId);

  // 2. Non-atomic read-modify-write (race condition under concurrent requests)
  const updatedCredits = user.credits + 500;

  // 3. Separate DB writes without a transaction
  await db.subscriptions.update({
    where: { userId },
    data: { planId: newPlanId, updatedAt: new Date() },
  });

  await db.users.update({
    where: { id: userId },
    data: { credits: updatedCredits },
  });

  return { status: "upgraded", credits: updatedCredits };
}

Why did this pass review?

The TypeScript types compile cleanly.
The auto-generated unit test mocked billingClient and db, asserting that credits equaled user.credits + 500.
It was buried inside a 14-file PR alongside boilerplate route definitions and DTOs.

The Consequence in Production

Two weeks later, a client network timeout triggers an automatic HTTP retry:

Double charging: billingClient.chargeUpgradeFee runs twice because no idempotency key was passed.
Lost updates / Race condition: Concurrent requests read the same user.credits snapshot in memory before writing.
Partial state corruption: If db.users.update fails due to a transient lock timeout, the user is billed and their plan is updated, but their credits vanish.

The Improved Approach: Designing for Verification, Not Just Generation

Fixing the code requires enforcing transactional boundaries, atomic increments, and idempotency—things a reviewer only catches when diffs are small and architectural constraints are explicit:

// ✅ Production-safe: Idempotent, atomic, and transactionally bounded
export async function upgradeSubscription(
  userId: string,
  newPlanId: string,
  idempotencyKey: string
): Promise<{ status: string; credits: number }> {
  return await db.$transaction(async (tx) => {
    // 1. Prevent duplicate execution via idempotency record lock
    const existingOp = await tx.idempotencyKeys.findUnique({
      where: { key: idempotencyKey },
    });
    if (existingOp) {
      return existingOp.responsePayload as { status: string; credits: number };
    }

    const user = await tx.users.findUniqueOrThrow({ where: { id: userId } });

    // 2. Pass deterministic idempotency key to external provider
    await billingClient.chargeUpgradeFee(user.stripeCustomerId, newPlanId, {
      idempotencyKey,
    });

    await tx.subscriptions.update({
      where: { userId },
      data: { planId: newPlanId, updatedAt: new Date() },
    });

    // 3. Atomic database increment instead of in-memory calculation
    const updatedUser = await tx.users.update({
      where: { id: userId },
      data: { credits: { increment: 500 } },
    });

    const result = { status: "upgraded", credits: updatedUser.credits };

    await tx.idempotencyKeys.create({
      data: { key: idempotencyKey, responsePayload: result },
    });

    return result;
  });
}

Click to view the hidden test smell that masks AI bugs
When an AI assistant writes both the implementation and the test suite in the same prompt, it generates mirror tests—tests that assert what the code does rather than what the business contract requires.

If the implementation forgets to wrap two database calls in a transaction, the generated test simply mocks both calls and asserts each was called once. To break this loop:

Write test cases (or failure-mode assertions) before generating implementation code.
Require at least one test per PR that simulates a partial failure (e.g., the second database write throwing an error) and asserts that state rolls back cleanly.

Shifting the Workflow: How High-Performing Teams Adapt

As highlighted in Google's engineering practices on The Standard of Code Review, the primary purpose of code review is ensuring that the overall code health of the system improves over time. When code generation becomes virtually free, your review standards must tighten, not loosen.

Use this checklist before opening or approving any AI-assisted pull request:

[ ] Cap PR size at 250 lines of logic: If AI helped you write 800 lines in 20 minutes, split it into three stacked PRs so a human can actually reason about the state transitions.
[ ] Separate boilerplate from business logic: Never mix auto-generated migrations, DTOs, or formatting changes in the same PR as core domain logic.
[ ] Audit the tests before the implementation: Check whether the tests verify edge cases and failure rollbacks, or if they merely assert that mocked functions were called.
[ ] Check system boundaries first: Look directly at external I/O, database transactions, concurrency locks, and authorization checks—where local context models fail most often.
[ ] Require the author to explain the failure mode: Every PR description should answer one human-written question: "What happens if this code fails halfway through execution?"

Explore more production open-source projects on my GitHub

The Takeaway

AI coding assistants don't create bad engineering cultures—they amplify your existing bottlenecks. If your team treats pull request velocity as the primary metric of productivity, AI will help you accumulate technical debt and silent production bugs at record speed. Keep batch sizes small, separate test design from implementation generation, and treat human code review as an architectural safeguard rather than a syntax check.

What is the most subtle bug or production failure mode that slipped through an AI-assisted PR in your team recently, and what guardrail did you add to catch it next time?

Sources & Further Reading

Connect with Me

If you found this article helpful, let's connect!

💻 GitHub: johnnylemonny

𝗝𝗼𝗵𝗻Follow

Freelance fullstack developer creating fast, modern, and refined applications. I focus on simplicity, performance, and real value for users. I also write tech-related articles here.

Building Mythic Scribe: An RPG Character Generator Powered by Gemini

𝗝𝗼𝗵𝗻 — Sun, 18 Oct 2026 16:00:00 +0000

This post is my submission for DEV Education Track: Build Apps with Google AI Studio.

What I Built

I built an RPG Character Portrait & Lore Generator using the "Build" feature in Google AI Studio. The application turns user-selected RPG traits (race, class, visual style, and equipment) into a complete character sheet with ability scores, backstory, and a visual portrait.

It integrates the Imagen API for high-resolution portrait generation alongside Gemini Flash for lore generation. Additionally, because the Imagen API currently requires a billed API key, I implemented an automatic Canvas Portrait Fallback Generator powered by Gemini Flash so the app remains 100% functional and visually engaging even on a free-tier API key.

Prompt Used to Generate the App:

Build an interactive "RPG Character Portrait & Lore Generator" web application that integrates both the Gemini model for text generation and the Imagen API for image generation.

App Features & Workflow:

User Input Form:

Character Name (optional, with a "Randomize" button)

Race (e.g., Elf, Dwarf, Cyborg, Tiefling, Human, Orc)

Class/Role (e.g., Necromancer, Paladin, Netrunner, Rogue, Bard)

Visual Art Style (dropdown: Dark Fantasy Oil Painting, Anime/Cel-Shaded, Pixel Art, Watercolor Concept Art, Cyberpunk Neon)

Key Traits / Equipment (short text input, e.g., "glowing blue runes on armor, scarred face, holding a crystal staff")

AI Generation Logic:
a) A compelling 3-sentence character backstory and personality trait.
b) Basic RPG ability scores (Strength, Dexterity, Constitution, Intelligence, Wisdom, Charisma) formatted neatly.
c) A highly detailed, optimized image generation prompt based on the user's selections.

Use the Imagen API to generate a high-quality 1:1 character portrait using the optimized visual prompt and selected art style.

UI/UX Design:

Dark-mode RPG tavern/grimoire aesthetic (deep slate background, gold/amber accents, clean typography).

Responsive two-column layout: configuration panel on the left, generated Character Sheet (portrait, stats badges, and backstory) on the right.

Include loading states while the text and image are being generated, and a button to download the portrait.

Features Utilized:

Build apps with Gemini in Google AI Studio for full-stack application generation and iterative refinement.
Imagen API for primary character portrait generation.
Gemini Flash for structured RPG lore, stat generation, prompt expansion, and real-time SVG code synthesis as a fallback mechanism.

Demo

🔗 Link to the deployed app

Screenshot:

Generated RPG character sheet with stats, lore, and custom HTML5 Canvas fallback portrait.

My Experience

Working through this track was a great hands-on introduction to building multimodal web applications rapidly in Google AI Studio.

Key Takeaways & What Surprised Me:

Handling API Tier Constraints Gracefully: The biggest surprise during development was discovering that calls to the Imagen API strictly require a paid/billed API key. Instead of letting the app fail with an error for free-tier users, I iterated on the prompt in Google AI Studio to build an automatic HTML5 Canvas portrait fallback generator. When the app detects that Imagen API billing is unavailable, Gemini Flash dynamically writes custom, layered SVG code to render the character's portrait based on their race, class, and equipment.
Rapid Iteration in AI Studio: Adding error handling and a completely secondary visual rendering pipeline (HTML5 Canvas generation to PNG) took just a quick conversational follow-up in the Build tool, showing how effective Gemini is at both UI scaffolding and resilient API integration.
Iterative Refinement & Error Recovery: The initial scaffold generated by AI Studio felt like a demo and needed polish. The generated text was originally overly complex and bogged down with technical jargon, cluttered labels, and unnecessary descriptions. Through quick follow-up prompts, I stripped out the noise to focus on clean, immersive RPG storytelling. What made this process painless was Gemini’s ability to catch runtime errors, inspect stack traces, and apply automatic fixes on the fly, making iterative UI and prompt tuning remarkably smooth.

AI-Generated Tests Can Agree With the Bug

𝗝𝗼𝗵𝗻 — Thu, 15 Oct 2026 14:30:00 +0000

An AI coding agent implements a feature, generates tests, runs them, and opens a pull request with a green build.

That looks like independent verification. It often is not.

If the implementation and the tests came from the same prompt, repository context, and interpretation of the requirement, they can repeat the same mistake. The code is wrong. The tests confidently prove that the wrong behavior works exactly as implemented.

The failure is not that AI cannot write tests. The failure is treating correlated output as independent evidence.

The Test Passed Because It Copied the Assumption

Imagine a billing requirement:

Apply a 20% discount to the order subtotal, then add tax to the discounted amount.

The intended calculation is:

subtotal = 100.00
discount = 20.00
taxable amount = 80.00
tax = 8.00
final total = 88.00

An agent misreads the requirement and calculates tax on the undiscounted subtotal instead of the discounted amount:

export function calculateTotal(subtotal, taxRate, discountRate) {
  const tax = subtotal * taxRate;
  const discount = subtotal * discountRate;

  return subtotal + tax - discount;
}

When the same agent generates the unit test in the same session, it applies the exact same faulty mental model (100 + 10 - 20 = 90). The generated test passes:

import { describe, expect, it } from "vitest";
import { calculateTotal } from "./billing.js";

describe("calculateTotal", () => {
  it("applies a discount and tax", () => {
    expect(calculateTotal(100, 0.1, 0.2)).toBe(90);
  });
});

The test looks reasonable at a glance. It uses concrete inputs, asserts a specific number, and produces a green build.

Because the $2.00 gap (subtotal * taxRate * discountRate) is small, a reviewer skimming the pull request can easily miss that 90 came from the agent's own implementation rather than the requirement (88). The shared assumption becomes a production billing bug.

Coverage Is Not an Oracle

Code coverage answers questions such as:

Was this line executed?
Was this branch visited?
Was this function called?

It does not answer:

Was the expected value correct?
Did the test encode the product requirement?
Would the test fail for a plausible wrong implementation?
Did the system preserve a business invariant?

A generated test suite can reach 100% line coverage while asserting the implementation's current behavior back at itself.

This pattern is especially weak:

const result = calculateTotal(order);

expect(result.subtotal).toBe(order.subtotal);
expect(result.tax).toBe(result.taxableAmount * order.taxRate);
expect(result.total).toBe(
  result.subtotal + result.tax - result.discount
);

The test reconstructs the same formula used by the implementation. If the formula is wrong, both sides remain wrong together.

A test needs an oracle: an independent reason to believe the expected result is correct.

Shared Context Creates Correlated Failures

When one agent writes the code and tests in the same session, both artifacts may inherit:

the same ambiguous requirement
the same incomplete repository context
the same mistaken domain assumption
the same missing edge case
the same naming-driven interpretation
the same outdated example

Regenerating the test in the same conversation may not add meaningful independence. The model still sees its earlier reasoning and implementation.

Even using a second model is not automatically enough. If both models receive the same ambiguous prompt and derive expectations from the same implementation, they can converge on the same attractive mistake.

The goal is not merely a different generator. The goal is a different source of truth.

The Dangerous Test Patterns

1. Asserting the implementation against itself

expect(formatPrice(input)).toBe(formatPrice(input));

This extreme example is obvious, but indirect versions are common. A test may call the same helper, parser, mapper, or formula as the production path.

2. Snapshotting unknown behavior

expect(calculateInvoice(order)).toMatchSnapshot();

The first snapshot records whatever the implementation produced. Approving it proves that someone accepted the output, not that the output matches the requirement.

Snapshots are useful for detecting change. They are weak at establishing initial correctness when nobody verifies their meaning.

3. Mirroring private implementation details

expect(cache.internalEntries.size).toBe(1);

The test confirms structure rather than observable behavior. It may pass while users receive stale data, and fail during a safe refactor.

4. Testing only the example from the prompt

If an issue contains one example, an agent will often reproduce it as one happy-path test. That example may not distinguish competing interpretations.

5. Mocking away the failure

paymentGateway.charge = vi.fn().mockResolvedValue({ status: "paid" });

The test proves behavior for a dependency that always succeeds instantly. It says nothing about timeouts, duplicated responses, uncertain outcomes, or retries after the remote side processed the charge.

A quick review question For every generated assertion, ask: > Where did this expected value come from? If the answer is "from the implementation," "from the agent's earlier output," or "from a snapshot we did not inspect," the test is not independent enough for a critical behavior.

Start With Invariants, Not Examples

Examples verify selected points. Invariants describe properties that must hold across many inputs.

For a discount calculation, useful invariants might include:

a discount cannot increase the pre-tax amount
a 0% discount changes nothing
a 100% discount produces no taxable amount
the total cannot be negative
changing item order cannot change the total
rounding happens at the boundary defined by the business rules

Property-based tests can exercise these rules across generated inputs:

import fc from "fast-check";
import { expect, it } from "vitest";
import { calculateTotal } from "./billing.js";

it("never increases the total when the discount increases", () => {
  fc.assert(
    fc.property(
      fc.double({ min: 0, max: 100_000, noNaN: true }),
      fc.double({ min: 0, max: 0.3, noNaN: true }),
      fc.double({ min: 0, max: 1, noNaN: true }),
      fc.double({ min: 0, max: 1, noNaN: true }),
      (subtotal, taxRate, firstDiscount, secondDiscount) => {
        const lower = Math.min(firstDiscount, secondDiscount);
        const higher = Math.max(firstDiscount, secondDiscount);

        const totalWithLowerDiscount = calculateTotal(
          subtotal,
          taxRate,
          lower
        );

        const totalWithHigherDiscount = calculateTotal(
          subtotal,
          taxRate,
          higher
        );

        expect(totalWithHigherDiscount).toBeLessThanOrEqual(
          totalWithLowerDiscount
        );
      }
    )
  );
});

This still does not prove every domain rule. It does make it harder for one convenient example to hide a broken relationship.

Derive Acceptance Tests From a Separate Source

The strongest acceptance tests should come from artifacts that existed independently of the implementation:

product acceptance criteria
API contracts
regulatory rules
protocol specifications
previously verified behavior
production examples approved by domain experts
compatibility fixtures from another implementation

For the billing example, create a decision table before asking an agent to implement it:

Case: percentage discount before tax
Subtotal: 125.00
Discount: 20%
Tax rate: 8%
Expected discount: 25.00
Expected taxable amount: 100.00
Expected tax: 8.00
Expected total: 108.00

Then convert that independently reviewed fixture into a test:

it("taxes the amount after the percentage discount", () => {
  expect(calculateTotal(125, 0.08, 0.2)).toBe(108);
});

Because the expected value (108) was derived from the specification before the code existed, the test immediately rejects the 110 produced by taxing the undiscounted subtotal.

Make the Test Fight the Implementation

Mutation testing changes the code in small ways and checks whether the tests fail.

For example, a mutation tool might replace:

const taxableAmount = subtotal - discount;

with:

const taxableAmount = subtotal + discount;

or change:

if (total >= freeShippingThreshold)

into:

if (total > freeShippingThreshold)

If the test suite stays green, it did not protect that behavior.

A mutation score is not a perfect quality metric. Equivalent mutations and costly test runs are real constraints. But targeted mutation testing is valuable for authorization, billing, retries, idempotency, limits, and other logic where a plausible one-character error matters.

Meta reported a related change-aware approach in its 2026 work on Just-in-Time catching tests. Instead of only generating tests that pass on the proposed code, catching tests are designed to surface regressions before a change lands. In an evaluation of 22,126 generated tests, change-aware methods improved candidate catch generation, and the reported process found confirmed serious failures. The important idea is practical: generated tests should actively try to falsify the change, not merely document it.

Use Differential Testing When You Have Two Implementations

Differential testing runs the same inputs through two independent implementations and compares the outputs.

This is useful during rewrites, migrations, and optimizations:

for (const order of generatedOrders) {
  const legacyResult = legacyCalculator(order);
  const newResult = newCalculator(order);

  expect(newResult).toEqual(legacyResult);
}

The reference implementation does not need to be fast or elegant. It needs to be trusted for the behavior under comparison.

Be careful: if the legacy behavior contains known bugs, encode those exceptions explicitly. Otherwise, differential testing can preserve defects just as snapshots do.

Separate the Roles in the Workflow

A safer AI-assisted workflow creates friction between implementation and validation.

Step 1: Extract the contract

Before generating code, write down:

inputs and outputs
invariants
boundary conditions
error behavior
security constraints
examples that distinguish plausible interpretations

Step 2: Generate the implementation

Give the coding agent the contract, but do not let its implementation become the source of expected values.

Step 3: Generate adversarial tests in a fresh context

Provide another agent or session with:

the contract
public interfaces (without the implementation diff, to avoid priming the tester on the author's code structure)
known failure classes

Ask it to find counterexamples that fail on the pre-patch branch first, not to improve coverage.

A useful prompt is:

Act as an adversarial test engineer.

Given the requirement and public interface:
1. List plausible incorrect interpretations.
2. Create inputs that distinguish them.
3. Identify boundary conditions and invariants.
4. Write tests that would fail for each plausible wrong implementation.
5. Do not derive expected values from the implementation under test.

Step 4: Run targeted mutation testing

Focus on critical decisions rather than mutating the entire repository on every commit.

Step 5: Review the oracle

Reviewers should spend less time asking whether the test is readable and more time asking:

Why is this expected value correct?
Which wrong implementation would this test reject?
Is the test independent of production helpers?
What happens at the exact boundary?
Which requirement remains untested?

Step 6: Keep merge controls

A green agent-generated pull request still needs repository protections, required checks, and human review. GitHub explicitly requires human review before merging draft pull requests produced by its coding agent and layers security scanning and branch restrictions around generated changes.

That is not ceremony. It is recognition that generated code plus generated tests is evidence, not authority.

A Practical Review Example

Suppose an agent adds this rate limiter:

export function isAllowed(requestCount, limit) {
  return requestCount <= limit;
}

It also generates:

it("allows requests within the limit", () => {
  expect(isAllowed(3, 5)).toBe(true);
});

it("blocks requests above the limit", () => {
  expect(isAllowed(6, 5)).toBe(false);
});

Both tests pass. Coverage may be complete.

But the critical question is whether the fifth request is allowed or whether limit represents the number of requests already consumed. The tests avoid the exact boundary where the interpretation matters.

Add the distinguishing case:

it("defines behavior at the exact limit", () => {
  expect(isAllowed(5, 5)).toBe(false);
});

Whether false is correct depends on the contract. The key is that a human or independent specification must decide it. The implementation cannot decide its own requirement.

Actionable Checklist

Before trusting AI-generated tests, verify that:

[ ] Expected values come from a specification, fixture, reference implementation, or reviewed calculation.
[ ] Tests do not call production helpers to calculate their own expectations.
[ ] Boundary values are tested explicitly.
[ ] At least one case distinguishes plausible interpretations of the requirement.
[ ] Important behavior is expressed as an invariant where possible.
[ ] Snapshots were inspected for meaning before approval.
[ ] Mocks include failures, delays, retries, and uncertain outcomes.
[ ] Generated tests attempt to falsify the change, not only confirm it.
[ ] Critical logic receives targeted mutation testing.
[ ] Rewrites are compared against a trusted implementation where appropriate.
[ ] A reviewer examines the test oracle, not only coverage.
[ ] Agent-created pull requests remain subject to required checks and human review.

CTA: Pick one AI-generated test from your current project and trace every expected value back to its source. If the trail ends at the implementation itself, replace that assertion with an independent fixture, invariant, or boundary case.

The Takeaway

AI-generated tests are useful. They can produce scaffolding, cover routine branches, explore edge cases, and shorten feedback loops.

But tests generated from the same context as the implementation are not automatically independent validation.

They may encode the same misunderstanding twice:

incorrect requirement interpretation
              ↓
      incorrect implementation
              ↓
     matching incorrect tests
              ↓
          green build

Do not measure confidence only by the number of generated tests or the coverage percentage.

Increase independence instead:

derive expectations from reviewed contracts
test invariants and exact boundaries
use fresh adversarial context
apply mutation and differential testing
review where each oracle came from
keep humans responsible for the merge decision

A test suite is valuable when it can disagree with the implementation.

What is the most convincing green test suite you have seen that still allowed a production bug through?

Sources

Connect with Me

If you found this article helpful, let's connect!

💻 GitHub: johnnylemonny

𝗝𝗼𝗵𝗻Follow

Freelance fullstack developer creating fast, modern, and refined applications. I focus on simplicity, performance, and real value for users. I also write tech-related articles here.

Your Metrics Are Lying After They Hit the Cardinality Limit

𝗝𝗼𝗵𝗻 — Tue, 13 Oct 2026 14:30:00 +0000

Your dashboard says request traffic is normal.

Your error-rate alert is quiet.

But users are reporting failures.

This is not always a bad query or a delayed exporter. Your metrics may have exceeded their cardinality limit, causing OpenTelemetry to preserve the total while silently removing the attributes your dashboards depend on.

That is a dangerous failure mode because the data still looks believable.

The Problem Is Not Request Volume

Metric cardinality is the number of unique attribute combinations recorded for a metric.

Consider this request counter:

requestCounter.add(1, {
  "http.route": "/checkout",
  "http.request.method": "POST",
  success: false
});

OpenTelemetry maintains aggregation state for each unique combination of attributes.

These are four different combinations:

{ route="/checkout", method="POST", success="true"  }
{ route="/checkout", method="POST", success="false" }
{ route="/cart",     method="GET",  success="true"  }
{ route="/cart",     method="GET",  success="false" }

Recording one million requests with the same combination does not create one million metric series.

Recording one request for each of one million user IDs potentially does.

The dangerous part is therefore not necessarily request volume. It is how many different attribute combinations those requests produce.

The Common Mistake: Treating Metrics Like Logs

A developer wants more context during an incident, so they add a few useful-looking attributes:

requestCounter.add(1, {
  route: req.originalUrl,
  user_id: req.user.id,
  tenant_id: req.user.tenantId,
  request_id: req.id,
  success: response.statusCode < 500
});

This looks helpful. Every measurement contains enough information to identify the exact request.

It is also a cardinality explosion.

Values such as these are usually unbounded:

user IDs
request IDs
session IDs
raw URL paths
search queries
email addresses
exception messages
dynamically generated resource names

They can be appropriate in logs or traces, where individual events matter.

Metrics are designed for aggregation. Their dimensions should normally come from small, predictable sets.

If an attribute helps you find one specific request, it probably belongs in a trace or log, not in a metric.

Raw Paths Are a Frequent Source of Trouble

These URLs may look like one endpoint:

/users/10482/orders/9001
/users/25118/orders/9002
/users/78341/orders/9003

But as raw metric attribute values, they are three separate combinations.

The bounded route template is:

/users/{userId}/orders/{orderId}

That difference matters.

OpenTelemetry distinguishes a raw URL path from a route representing the matched application endpoint. A route template groups requests that execute the same code path, while a raw path can contain identifiers supplied by users.

A safer measurement looks like this:

requestCounter.add(1, {
  "http.route": "/users/{userId}/orders/{orderId}",
  "http.request.method": "GET",
  success: true
});

The identifiers can remain on the corresponding span:

span.setAttributes({
  "user.id": userId,
  "order.id": orderId
});

Now metrics answer aggregate questions, while traces preserve request-level context.

What Happens When the Limit Is Reached?

OpenTelemetry metrics SDKs use a cardinality limit to prevent unlimited aggregation state from consuming process memory.

The default aggregation cardinality limit defined by OpenTelemetry is 2,000 combinations per metric stream, although SDK support and configuration details can vary.

When the limit is reached, additional combinations are aggregated into an overflow data point:

{ otel.metric.overflow=true }

The measurement value is not simply discarded.

The original measurement attributes are.

Suppose this measurement arrives after the stream reaches its limit:

{
  route="/checkout",
  tenant_id="tenant-48291",
  success="false"
} 1

It may be incorporated into:

{ otel.metric.overflow=true } 1

Notice what disappeared:

route
tenant_id
success

The unbounded tenant_id caused the explosion, but OpenTelemetry does not preserve the other attributes on the overflow point.

That includes the low-cardinality success boolean.

Your Total Can Be Correct While Your Alert Is Wrong

Imagine the service processed 100,000 requests:

98,000 successful requests
 2,000 failed requests

The total query can still return:

100,000

But if 1,400 failed requests were folded into overflow without their original attributes, a filtered query might only see:

success="false" -> 600

Your total request count remains correct.

Your calculated error rate becomes:

600 / 100000 = 0.6%

The actual error rate is:

2000 / 100000 = 2%

If the alert threshold is 1%, the alert does not fire.

This can affect any query that filters or groups by measurement attributes:

error-rate alerts
route-level latency dashboards
per-tenant SLOs
status-code breakdowns
regional traffic analysis
autoscaling signals
capacity reports

This is what makes overflow so deceptive. The monitoring system continues returning numbers. Only some dimensions have become incomplete.

Cardinality Multiplies Faster Than It Looks

Assume a metric contains:

120 routes
5 HTTP methods
6 status-code groups
2 success values
4 deployment regions

The theoretical maximum is:

120 × 5 × 6 × 2 × 4 = 28,800 combinations

Not every combination will necessarily occur, but this calculation exposes the risk.

Now add 10,000 tenants:

28,800 × 10,000 = 288,000,000 combinations

Adding “just one label” can change the metric from bounded to effectively unbounded.

Histograms deserve additional attention because their exported representation can produce multiple time series per attribute combination, depending on the aggregation and backend.

Why local testing rarely reveals this problem

Development traffic usually exercises:

a small number of routes
one or two users
a few tenants
short process lifetimes
limited parallel activity

Production introduces the full value space. A metric may behave correctly in tests and staging, then overflow only after enough unique combinations have accumulated.

With cumulative temporality, a process can retain combinations across collection cycles. A long-running production instance may therefore hit a limit that a frequently restarted staging instance never reaches.

Detect Overflow Before Debugging the Dashboard

Overflow points use this attribute:

otel.metric.overflow=true

Prometheus-compatible systems often normalize dots to underscores, exposing it as:

otel_metric_overflow="true"

A PromQL query for recent overflow can look like this:

count by (__name__, job) (
  last_over_time(
    {otel_metric_overflow="true"}[5m]
  )
) > 0

Adjust resource labels such as job, service_name, cluster, or namespace to match your environment.

For an OTLP-native backend, use its query language to:

filter on otel.metric.overflow = true
group by metric name
group by service name
identify the first deployment or time window where overflow appeared

Do not treat an empty result as absolute proof that cardinality is safe. Confirm that your SDK implements the relevant limit and that your exporter and backend retain the overflow attribute.

Do Not Fix It by Blindly Raising the Limit

Increasing the limit can stop overflow temporarily.

It can also allow substantially more aggregation state to accumulate in application memory and push more expensive series into your observability backend.

A higher limit is appropriate when:

every attribute is intentional
the dimensions are bounded
the expected number of combinations is known
the process has sufficient memory
the backend can handle the resulting series
overflow is continuously monitored

It is not the right first response to a request ID, raw path, or exception message appearing in a metric.

Fix the attribute model first.

Build a Cardinality Budget

Before shipping a metric, list every attribute and estimate its value space.

For example:

http.route             80 known values
http.request.method     5 known values
status_class             5 known values
region                   3 known values

A pessimistic upper bound is:

80 × 5 × 5 × 3 = 6,000 combinations

This does not mean all 6,000 combinations will exist simultaneously. It gives you a useful boundary to challenge before deployment.

Ask these questions for every attribute:

Is its value space bounded?
Can a user create new values?
Does it contain an identifier?
Can it include raw input?
Do we actually group or filter by it?
Would the value be more useful on a span or log?
What happens if one process sees every possible value?

If you cannot estimate the value space, treat the attribute as high-cardinality until proven otherwise.

Use Views to Remove Dangerous Attributes

OpenTelemetry Views can customize metric streams, including which attributes are retained.

The exact API differs between SDK languages, but the intended transformation is simple:

Input attributes:
  http.route
  http.request.method
  http.response.status_code
  user_id
  request_id

Retained metric attributes:
  http.route
  http.request.method
  http.response.status_code

The identifiers remain available in traces or logs.

The metric keeps only dimensions that are useful for aggregation.

This separation gives you both:

affordable, reliable aggregate monitoring
detailed context when investigating individual requests

Prefer Categories Over Raw Values

Sometimes the problematic attribute contains useful information but has too many possible values.

Replace the raw value with a bounded category.

Instead of recording the full exception message:

errorCounter.add(1, {
  error_message: error.message
});

Classify the failure:

errorCounter.add(1, {
  error_type: classifyError(error)
});

Where the classifier returns a controlled set:

function classifyError(error) {
  if (error instanceof TimeoutError) {
    return "timeout";
  }

  if (error instanceof ValidationError) {
    return "validation";
  }

  if (error instanceof DependencyError) {
    return "dependency";
  }

  return "unknown";
}

The complete error message belongs in a log or span event:

span.recordException(error);

The metric answers:

Are timeout failures increasing?

The trace answers:

What happened to this request?

Trying to make one telemetry signal answer both questions is what creates the problem.

Add Overflow to Your Observability Tests

Instrumentation deserves tests just like business logic.

A useful test should generate many unique values and verify that the exported metric does not retain an unbounded attribute.

Conceptually:

for (let index = 0; index < 10_000; index += 1) {
  requestCounter.add(1, {
    route: `/users/${index}`,
    request_id: `request-${index}`,
    success: true
  });
}

The test should fail if route contains raw identifiers or if request_id reaches the exported metric stream.

A corrected version records a normalized route:

for (let index = 0; index < 10_000; index += 1) {
  requestCounter.add(1, {
    "http.route": "/users/{userId}",
    success: true
  });
}

You can also add CI checks that inspect instrumentation changes for suspicious attribute names:

user_id
request_id
session_id
email
query
url.path
error.message

This is not a perfect detector, but it forces deliberate review before an unbounded dimension reaches production.

Production Checklist

Before deploying a new or modified metric:

[ ] List every metric attribute.
[ ] Estimate the number of possible values for each attribute.
[ ] Calculate a pessimistic combination count.
[ ] Replace raw paths with route templates.
[ ] Keep request, user, and session IDs out of metrics.
[ ] Replace raw error messages with bounded error categories.
[ ] Move request-level context to traces or logs.
[ ] Verify the SDK's cardinality-limit behavior.
[ ] Query for otel.metric.overflow=true.
[ ] Alert on overflow for metrics used by SLOs or paging.
[ ] Review cumulative versus delta temporality.
[ ] Raise the limit only after validating the attribute model.
[ ] Load-test instrumentation with realistic value diversity.
[ ] Recheck cardinality after framework or instrumentation upgrades.

Before changing the limit: Review the OpenTelemetry cardinality-limit documentation and verify that the metric attributes are intentionally bounded.

The Takeaway

A cardinality limit protects your application from unbounded metric state.

It does not guarantee that every dashboard, SLO, or alert remains accurate after overflow.

Once measurements are folded into an overflow point:

totals can remain correct
attributes can disappear
filtered queries can undercount
error alerts can stay quiet
dashboards can continue looking believable

The safest approach is not to maximize the limit. It is to design metric dimensions deliberately:

use bounded categories
normalize routes
keep identifiers in traces and logs
define a cardinality budget
monitor overflow as a production failure

Your monitoring should never fail silently while continuing to display convincing numbers.

Which high-cardinality attribute has caused the hardest observability problem in one of your production systems?

Sources

Connect with Me

If you found this article helpful, let's connect!

💻 GitHub: johnnylemonny

𝗝𝗼𝗵𝗻Follow

Freelance fullstack developer creating fast, modern, and refined applications. I focus on simplicity, performance, and real value for users. I also write tech-related articles here.

Honored by the shoutout from @nyaomaru! We broke down how changelog-bot extracts the "Why" behind PRs: treating it as evidence selection over source text, backed by our 50-case evaluation harness. Check out the writeup!

𝗝𝗼𝗵𝗻 — Sat, 10 Oct 2026 16:50:39 +0000

Balancing strict LLM constraints with raw PR text

nyaomaru

Oct 10

How Do We Extract the “Why” from a PR into a CHANGELOG with Jev? 🤔

#opensource #githubactions #ai #typescript

114

6 min read

Your Page Loaded Fast. So Why Does It Still Feel Slow?

𝗝𝗼𝗵𝗻 — Thu, 08 Oct 2026 14:30:00 +0000

The page appears in less than a second.

The user clicks a button.

Nothing happens.

The content was visible, the network request had finished, and the load report looked respectable. The main thread was simply too busy parsing, executing, and hydrating JavaScript to respond.

A page can load quickly and still feel slow.

If you optimize only the first render, you may be measuring the moment the interface becomes visible rather than the moment it becomes usable.

Visible is not the same as interactive.

Measure what happens after the user clicks, types, opens a menu, or submits a form. A fast screenshot does not guarantee a responsive application.

Start with a real interaction

Do not begin with a bundle-size target.

Begin with an action a user actually performs:

opening the main navigation,
typing into search,
applying a filter,
expanding a product panel,
adding an item to a cart,
submitting a form.

Then record what the user experiences:

Click
  ↓
Visible response begins
  ↓
Interface finishes updating

If there is a noticeable pause after the input, the page has a responsiveness problem even when initial content appeared quickly.

Interaction to Next Paint, or INP, measures the latency of user interactions across a page visit. Web performance guidance recommends an INP of 200 milliseconds or less at the 75th percentile for a good user experience.

Field data is the strongest evidence because it reflects real devices, networks, and interactions. Lab tools help reproduce and explain the problem.

Find long tasks in the Performance panel

Open browser DevTools, switch to the Performance panel, and record the interaction that feels slow.

During the recording:

wait for the page to settle,
click or type exactly as a user would,
stop recording after the interface updates,
inspect the main-thread timeline around the interaction.

A long task is an uninterrupted period where the main UI thread stays busy for at least 50 milliseconds. While that task runs, the browser cannot process another main-thread task, including input handling and rendering work.

Common causes include:

expensive event handlers,
parsing and evaluating large scripts,
synchronous data processing,
repeated rendering,
forced layout,
third-party scripts,
hydration work,
initializing features not yet needed.

In Chromium-based DevTools, long tasks are marked visually in the performance trace. Expand the task and inspect the call tree or bottom-up view to find the functions consuming time.

Do not optimize the first large function name you see without checking the complete stack. Framework code may be executing because application code requested too much work.

Measure JavaScript, not only transfer size

Compression can make a script small on the network while leaving substantial work for the browser.

JavaScript must still be:

Downloaded
   ↓
Decompressed
   ↓
Parsed
   ↓
Compiled
   ↓
Executed

A package that adds only a modest compressed payload can still perform expensive initialization, walk a large DOM, register many listeners, or schedule repeated work.

Use the Network panel to understand transfer size, but use the Performance panel to understand main-thread cost.

Ask these questions:

Is this script needed before the first interaction?
Is the complete library used?
Does initialization run for components not present on the page?
Does the script execute during every navigation?
Is the same calculation repeated?
Is third-party code running at the moment of input?

Remove work before splitting work

Code splitting is useful, but the best delayed JavaScript is often JavaScript the page no longer needs.

Look for:

unused libraries,
duplicated utilities,
obsolete analytics integrations,
polyfills outside the supported browser policy,
large dependencies used for one small helper,
components loaded globally but rendered rarely,
initialization that can be replaced with HTML or CSS.

For example, do not load a charting library on every page if only one route displays a chart.

const { renderChart } = await import("./chart.js");
renderChart(data);

Dynamic import moves the module behind the point where it becomes necessary.

That improves responsiveness only if the feature genuinely does not need the code earlier. Moving required work directly into the click handler may replace load delay with interaction delay.

A better option can be to load an optional feature during idle time or after primary content becomes interactive.

Break long work into smaller tasks

Sometimes the work is necessary but does not need to block the main thread in one uninterrupted chunk.

Imagine processing a large collection:

function processAllItems(items) {
  for (const item of items) {
    processItem(item);
  }
}

If the loop becomes a long task, yield periodically:

async function processItems(items) {
  for (let index = 0; index < items.length; index++) {
    processItem(items[index]);

    if (index % 100 === 0) {
      await scheduler.yield();
    }
  }
}

Yielding gives the browser an opportunity to process higher-priority work such as input and rendering before continuing.

The best chunk size depends on the cost of each item and target devices. A fixed item count is easy to demonstrate but should be measured in the real application.

If scheduler.yield() is not available in your browser support range, use an appropriate scheduling fallback and keep the behavior progressively enhanced.

Move CPU-heavy work off the main thread

For expensive calculations that do not need direct DOM access, consider a Web Worker.

Good candidates include:

parsing large datasets,
image manipulation,
compression,
complex search indexing,
numerical calculations.

Workers add message-passing and lifecycle complexity. Use them when measurement shows CPU work is blocking interactions, not as a default wrapper for every function.

Keep interaction handlers focused

An event handler should perform the minimum work required to produce the next useful visual response.

This handler does too much synchronously:

button.addEventListener("click", () => {
  calculateLargeReport();
  storeAnalyticsPayload();
  updateRecommendations();
  openPanel();
});

The user waits for unrelated work before seeing the panel.

Prioritize the visible response:

button.addEventListener("click", () => {
  openPanel();

  setTimeout(() => {
    calculateLargeReport();
    storeAnalyticsPayload();
    updateRecommendations();
  }, 0);
});

This simplified example separates tasks, but setTimeout(..., 0) does not make expensive work cheap. If the deferred work is still large, split it further, move it to a worker, or remove it.

Also avoid reading layout information immediately after changing styles when possible. Repeated read-write cycles can force synchronous layout:

// Repeated read/write pattern
for (const item of items) {
  item.style.width = `${container.offsetWidth / 2}px`;
}

Read shared measurements once, then apply updates:

const width = container.offsetWidth / 2;

for (const item of items) {
  item.style.width = `${width}px`;
}

Audit third-party scripts

Your application is not the only code using the main thread.

Analytics, experimentation, advertising, chat, support widgets, personalization, and tag managers can all execute around user interactions.

For each third-party script, document:

who owns it,
why it exists,
where it loads,
whether it is required on every page,
how much main-thread time it consumes,
what user data it accesses,
what happens when it fails.

Remove scripts without a current owner or measurable purpose.

Load nonessential tools after the primary experience is usable. Be careful with a tag manager that can silently restore removed scripts later.

A third-party script may not be large enough to dominate network transfer and can still create long tasks through initialization or repeated callbacks.

Watch for hydration delays

Server-rendered HTML can appear before client-side JavaScript finishes attaching behavior.

That creates a frustrating state:

The button is visible
The button looks interactive
The click handler is not ready

To reduce the gap:

ship less client-side JavaScript,
avoid hydrating static sections,
isolate interactive components,
defer low-priority widgets,
keep initial client state small,
prefer server or platform behavior when interaction is unnecessary.

Framework-specific tools differ, but the principle is stable: do not make the browser hydrate parts of the page that never needed client-side ownership.

Verify the improvement

Repeat the same interaction under the same conditions.

Compare:

Before
- interaction delay
- longest task
- total main-thread work
- JavaScript evaluation time

After
- interaction delay
- longest task
- total main-thread work
- JavaScript evaluation time

Use CPU throttling to reveal problems hidden by a fast development machine. Then verify with field data, because real users have different devices, extensions, background apps, and interaction patterns.

A lower bundle size is useful evidence, but it is not proof that the slow interaction is fixed.

Responsiveness audit checklist

[ ] Start with a real click, tap, or keypress.
[ ] Check field INP data when available.
[ ] Record the interaction in the Performance panel.
[ ] Find long tasks around the input.
[ ] Inspect the call stack before changing code.
[ ] Remove unnecessary JavaScript first.
[ ] Delay optional features away from critical interactions.
[ ] Split necessary long work into smaller tasks.
[ ] Move suitable CPU work to a Web Worker.
[ ] Audit third-party scripts.
[ ] Check for visible-but-not-hydrated controls.
[ ] Reproduce the same interaction after the change.

The takeaway

Initial load speed and interaction speed are related, but they are not the same problem.

A responsive page needs room on the main thread for user input and rendering.

The practical workflow is:

Choose a real interaction
          ↓
Record it in DevTools
          ↓
Find the blocking task
          ↓
Remove unnecessary work
          ↓
Delay, split, or move necessary work
          ↓
Repeat the same interaction
          ↓
Verify with field data

Do not optimize only the moment the page becomes visible.

Optimize the moment the user tries to use it.

Which interaction in your application feels slower than the initial page load?

Read the web.dev guide to optimizing INP

Sources and further reading

MDN: JavaScript performance optimization
MDN: PerformanceLongTaskTiming
web.dev: Optimize long tasks
web.dev: Optimize Interaction to Next Paint
W3C: Long Tasks API

Connect with Me

If you found this article helpful, let's connect!

💻 GitHub: johnnylemonny

𝗝𝗼𝗵𝗻Follow

Freelance fullstack developer creating fast, modern, and refined applications. I focus on simplicity, performance, and real value for users. I also write tech-related articles here.

Stop Retrying Every Failed Request: Your Resilience Code May Be Causing the Outage

𝗝𝗼𝗵𝗻 — Tue, 06 Oct 2026 14:30:00 +0000

The API returns 503 Service Unavailable.

Your client immediately retries three times.

That sounds resilient.

Now imagine ten thousand clients doing the same thing at the same time.

The struggling service does not receive less traffic. It receives several times more traffic precisely when it has the least capacity to handle it.

A retry is not automatically a recovery mechanism.

Without timeouts, backoff, jitter, retry limits, and safe operation semantics, retry code can turn a temporary failure into a longer outage.

Retry only when all three conditions are true:

The failure is likely to be temporary.
Repeating the operation is safe.
The next attempt will not happen immediately.

The retry loop that makes failures worse

This implementation looks reasonable at first glance:

async function requestWithRetry(url) {
  for (let attempt = 0; attempt < 3; attempt++) {
    try {
      const response = await fetch(url);

      if (response.ok) {
        return response;
      }
    } catch {
      // Try again immediately
    }
  }

  throw new Error("Request failed");
}

It has several problems:

every attempt can wait indefinitely,
all failures are treated as retryable,
retries happen immediately,
every client follows the same timing,
the total operation has no time budget,
the method does not consider whether repeating the request is safe,
the final error loses useful context.

The fix is not simply “add more retries.”

The fix is to make every retry earn its place.

Start with a timeout

A request that never completes can occupy connections, memory, and other limited resources. Set a maximum wait for each attempt.

Modern JavaScript provides AbortSignal.timeout():

const response = await fetch(url, {
  signal: AbortSignal.timeout(5_000),
});

The signal aborts the request after the configured active time and rejects with a TimeoutError DOM exception.

If the caller also needs to cancel the operation, combine signals:

async function fetchWithTimeout(url, callerSignal) {
  const signal = AbortSignal.any([
    callerSignal,
    AbortSignal.timeout(5_000),
  ]);

  return fetch(url, { signal });
}

A timeout does not tell you whether the server completed an operation before the client stopped waiting. That distinction becomes critical for requests with side effects.

Retry only failures that may be temporary

A 400 Bad Request usually will not improve after a pause. Neither will an authentication failure caused by an invalid token.

Good retry candidates often include:

408 Request Timeout,
429 Too Many Requests,
selected 5xx responses,
a temporary network failure,
a per-attempt timeout when repeating the operation is safe.

Treat this list as a policy for the API you are calling, not a universal law.

const RETRYABLE_STATUS_CODES = new Set([
  408,
  429,
  500,
  502,
  503,
  504,
]);

function isRetryableResponse(response) {
  return RETRYABLE_STATUS_CODES.has(response.status);
}

Do not retry every 5xx response automatically without considering the service contract. Some APIs document narrower behavior, while others provide a specific Retry-After response.

Add exponential backoff

Immediate retries create more load when the service is already struggling.

Exponential backoff increases the delay after each failed attempt:

function backoffDelay(attempt, baseDelay = 250) {
  return baseDelay * 2 ** attempt;
}

That produces delays such as:

250 ms
500 ms
1000 ms
2000 ms

Backoff gives the dependency time to recover and reduces repeated pressure from one client.

It does not solve synchronization across many clients.

Add jitter so clients do not retry together

If thousands of clients fail at the same moment and use the same deterministic backoff, they wake up together after the same delay. The retry spike has merely moved.

Jitter adds randomness:

function fullJitterDelay(attempt, baseDelay = 250, cap = 8_000) {
  const maximum = Math.min(cap, baseDelay * 2 ** attempt);
  return Math.floor(Math.random() * maximum);
}

Clients now spread attempts across a time window rather than forming another synchronized wave.

A small sleep helper keeps the retry loop readable:

function sleep(milliseconds, signal) {
  return new Promise((resolve, reject) => {
    const timeout = setTimeout(resolve, milliseconds);

    signal?.addEventListener(
      "abort",
      () => {
        clearTimeout(timeout);
        reject(signal.reason);
      },
      { once: true },
    );
  });
}

In production code, remove abort listeners when they are no longer needed or use a well-tested utility that already handles cleanup.

Respect `Retry-After`

A server may tell the client when to try again:

Retry-After: 5

The value can be a delay in seconds or an HTTP date.

function parseRetryAfter(value) {
  if (!value) return null;

  const seconds = Number(value);

  if (Number.isFinite(seconds)) {
    return Math.max(0, seconds * 1_000);
  }

  const date = Date.parse(value);

  if (Number.isNaN(date)) {
    return null;
  }

  return Math.max(0, date - Date.now());
}

When the server provides a valid instruction, prefer it over an arbitrary client delay, while still enforcing your own maximum wait and total operation budget.

Do not retry unsafe operations blindly

Retrying a read is usually easier to reason about than retrying a payment, order creation, email send, or database mutation.

Consider this sequence:

Client sends payment request
Server charges the card
Response is lost
Client times out
Client retries
Server charges the card again

A timeout proves that the client did not receive a response. It does not prove that the server did nothing.

Operations with side effects need an idempotency strategy. A common pattern is a unique key representing one logical operation:

const response = await fetch("/payments", {
  method: "POST",
  headers: {
    "Content-Type": "application/json",
    "Idempotency-Key": crypto.randomUUID(),
  },
  body: JSON.stringify(payment),
});

The server stores the key and returns the original result when the same logical operation is submitted again.

This works only when the server actually implements idempotency. A client-generated header alone does not make an endpoint safe.

Build a bounded retry helper

Here is a compact implementation that combines the main ideas:

const RETRYABLE_STATUS_CODES = new Set([
  408,
  429,
  500,
  502,
  503,
  504,
]);

function fullJitterDelay(attempt, baseDelay, cap) {
  const maximum = Math.min(cap, baseDelay * 2 ** attempt);
  return Math.floor(Math.random() * maximum);
}

function parseRetryAfter(value) {
  if (!value) return null;

  const seconds = Number(value);
  if (Number.isFinite(seconds)) {
    return Math.max(0, seconds * 1_000);
  }

  const date = Date.parse(value);
  return Number.isNaN(date) ? null : Math.max(0, date - Date.now());
}

function sleep(milliseconds) {
  return new Promise((resolve) => setTimeout(resolve, milliseconds));
}

export async function fetchWithRetry(
  url,
  options = {},
  {
    attempts = 3,
    timeoutMs = 5_000,
    baseDelayMs = 250,
    maxDelayMs = 8_000,
  } = {},
) {
  let lastError;

  for (let attempt = 0; attempt < attempts; attempt++) {
    try {
      const response = await fetch(url, {
        ...options,
        signal: AbortSignal.timeout(timeoutMs),
      });

      if (response.ok || !RETRYABLE_STATUS_CODES.has(response.status)) {
        return response;
      }

      if (attempt === attempts - 1) {
        return response;
      }

      const retryAfter = parseRetryAfter(
        response.headers.get("Retry-After"),
      );

      const delay = Math.min(
        maxDelayMs,
        retryAfter ?? fullJitterDelay(
          attempt,
          baseDelayMs,
          maxDelayMs,
        ),
      );

      await sleep(delay);
    } catch (error) {
      lastError = error;

      if (attempt === attempts - 1) {
        throw error;
      }

      await sleep(
        fullJitterDelay(attempt, baseDelayMs, maxDelayMs),
      );
    }
  }

  throw lastError ?? new Error("Request failed");
}

This is a starting point, not a universal networking library. A production client may also need:

caller cancellation,
a total time budget across all attempts,
domain-specific retry rules,
observability,
circuit breaking,
connection pooling,
rate-limit coordination,
idempotency keys,
body replay rules.

Measure retries as part of the system

A retry that eventually succeeds can hide a degrading dependency.

Track at least:

requests by final outcome,
attempts per logical operation,
time spent waiting between attempts,
status codes that triggered retries,
requests that exhausted the retry budget,
operations completed after one or more retries,
duplicate-operation prevention,
total latency including backoff.

A dashboard showing only successful final responses may report a healthy service while clients quietly perform two or three times the expected traffic.

A practical retry checklist

[ ] Every attempt has a timeout.
[ ] The whole operation has a bounded time budget.
[ ] Only temporary failures are retried.
[ ] Backoff increases between attempts.
[ ] Jitter prevents synchronized retry waves.
[ ] Retry-After is respected when appropriate.
[ ] Attempts are strictly limited.
[ ] Side-effecting operations are idempotent before retrying.
[ ] Retry metrics are visible separately from final success rates.
[ ] The client can stop retrying when the caller cancels.

The takeaway

Retries are useful because distributed systems fail temporarily.

Retries are dangerous for exactly the same reason: a struggling dependency has less capacity available, not more.

A safer sequence is:

Set a timeout
      ↓
Classify the failure
      ↓
Check whether repeating is safe
      ↓
Wait with backoff and jitter
      ↓
Respect the retry budget
      ↓
Measure the result

Do not ask only whether the request eventually succeeded.

Ask how many attempts it took, how much extra load it created, and whether repeating the operation could have caused a second side effect.

What is the hardest retry failure you have diagnosed: synchronized clients, duplicate writes, ignored rate limits, or something else?

Read the AWS retry with backoff guidance

Sources and further reading

AWS Prescriptive Guidance: Retry with backoff pattern
AWS Builder Center: Timeouts, retries, and backoff with jitter
MDN: AbortSignal.timeout()
MDN: AbortSignal

Connect with Me

If you found this article helpful, let's connect!

💻 GitHub: johnnylemonny

𝗝𝗼𝗵𝗻Follow

Freelance fullstack developer creating fast, modern, and refined applications. I focus on simplicity, performance, and real value for users. I also write tech-related articles here.

TypeScript 7 Is Up to 10x Faster. Should You Upgrade Now?

𝗝𝗼𝗵𝗻 — Thu, 01 Oct 2026 15:30:00 +0000

TypeScript 7 makes an unusually attractive promise:

The same kind of type checking, much faster.

The compiler and language service have been ported to Go. Microsoft reports typical full-build improvements between 8x and 12x in large codebases, along with lower aggregate memory use in its published benchmarks.

That sounds like an automatic upgrade.

It is not.

A faster compiler is useful only if your framework build, declaration output, editor, project references, linting, and surrounding tools still behave correctly.

Fortunately, you do not need a week-long migration project to find out.

You need a temporary branch, your existing verification commands, and two comparable timings.

The goal of this test is not to reproduce Microsoft's benchmark.

The goal is to learn whether TypeScript 7 improves your project without breaking the workflow around it.

Why TypeScript 7 is different

This is not a routine compiler release.

TypeScript 7 replaces the previous JavaScript-based compiler and language service with a native implementation written in Go. The new implementation uses native code, shared-memory multithreading, and other optimizations intended to preserve familiar TypeScript behavior while substantially reducing wait times.

Microsoft's published full-build results include:

VS Code: 125.7 seconds to 10.6 seconds
Sentry: 139.8 seconds to 15.7 seconds
Playwright: 12.8 seconds to 1.47 seconds
tldraw: 11.2 seconds to 1.46 seconds

Those are impressive results, but they are not a promise about every repository. A small application that spends most of its build time bundling assets or running tests may see a less dramatic end-to-end improvement.

The useful question is not:

Is TypeScript 7 fast?

It is:

How much of my development time is actually controlled by TypeScript?

Benchmark your current project

Start from a clean working tree and record the current compiler version:

npx tsc --version

Then time the command your project already uses for type checking.

On macOS or Linux:

time npx tsc --noEmit

For a project using references:

time npx tsc --build --force

Also run the commands that represent the complete project workflow:

npm run lint
npm test
npm run build

Do not compare a cold first run with a warm second run and call the difference a compiler improvement. Run each measurement several times under similar conditions.

A small note is enough:

Current TypeScript version:
Cold type check:
Second type check:
Production build:
Tests:
Declaration build:
Editor project load:

The editor measurement can be approximate. Open the same workspace and observe how long it takes before diagnostics, references, and completions become useful.

Test the upgrade on a branch

Create an isolated branch:

git switch -c test/typescript-7

Install TypeScript 7:

npm install --save-dev typescript@latest

Verify that the local executable changed:

npx tsc --version

Now repeat the same commands you recorded earlier:

time npx tsc --noEmit
npm run lint
npm test
npm run build

If the repository uses project references, declaration generation, API extraction, code generation, or framework-specific builds, run those too.

A successful tsc --noEmit result is necessary, but it is not the whole migration test.

Compare more than type-check time

Record the results without polishing them into a marketing claim.

| Check | Before | TypeScript 7 | Result |
|---|---:|---:|---|
| Cold type check | your result | your result | compare |
| Second type check | your result | your result | compare |
| Production build | your result | your result | compare |
| Tests | passed/failed | passed/failed | compare |
| Declarations | passed/failed | passed/failed | compare |

The most useful observations may not fit in a timing table:

Did the editor load the project faster?
Did auto-imports behave normally?
Did find-all-references return complete results?
Did declaration files change unexpectedly?
Did the framework build use the new compiler?
Did linting or another tool report an incompatible peer dependency?
Did CI use the same TypeScript version as the local test?

For many developers, faster diagnostics in the editor will matter more than shaving another second from an already short production build.

Know the main compatibility boundary

TypeScript 7.0 does not ship a stable programmatic compiler API.

That matters when a tool imports TypeScript as a library instead of simply invoking tsc. Custom transformers, compiler plugins, embedded-language tooling, declaration tools, and some framework integrations may still depend on the TypeScript 6 API.

Microsoft provides the @typescript/typescript6 compatibility package for tools that still need programmatic access while TypeScript 7 handles compilation.

A side-by-side setup can look like this:

{
  "devDependencies": {
    "@typescript/native": "npm:typescript@^7.0.2",
    "typescript": "npm:@typescript/typescript6@^6.0.2"
  }
}

In this arrangement, the TypeScript 6 compatibility package remains available to tools expecting the established API, while the aliased native package supplies the TypeScript 7 compiler.

Do not add this workaround automatically. First identify whether any tool in your project actually needs it, then follow that tool's current migration guidance.

Projects that should test especially carefully

Take extra care if your project uses:

custom TypeScript transformers,
tools that import the compiler API,
framework-specific template type checking,
MDX or embedded languages,
declaration bundlers,
compiler-based code generation,
complex project references,
editor plugins tied to the older language service,
build systems that pin a particular TypeScript range.

This does not mean the project cannot upgrade. It means npx tsc --noEmit is not enough evidence by itself.

Check the editor separately

TypeScript 7 uses the Language Server Protocol, which makes the new language service available beyond one editor. Microsoft also provides dedicated TypeScript 7 support for VS Code, while other editors may have their own enablement instructions.

Verify the editor rather than assuming the workspace dependency controls everything.

Test a few ordinary actions:

Open a large file with a known type error.
Trigger auto-completion.
Find all references for a widely used symbol.
Rename a symbol across several files.
Add an import using a quick fix.
Restart the editor and observe project-load time.

The best upgrade result is not merely a faster CI number. It is a shorter feedback loop throughout the day.

Keep it or roll it back

Keep the upgrade when:

the same checks pass,
declaration output remains acceptable,
the framework build behaves correctly,
editor features work as expected,
CI uses the intended compiler,
the measured improvement matters to the project.

Roll it back for now when:

a critical tool needs the older compiler API,
generated output changes unexpectedly,
editor support is incomplete for your setup,
the framework has not yet documented support,
the performance gain does not justify the migration work.

Rolling back the experiment is simple:

git switch -
git branch -D test/typescript-7

Alternatively, keep the branch and open an issue listing the blockers. A measured “not yet” is a useful result.

A ten-minute upgrade checklist

[ ] Record the current TypeScript version.
[ ] Time the existing type-check command more than once.
[ ] Run tests, linting, and the production build.
[ ] Create a separate upgrade branch.
[ ] Install TypeScript 7 and verify the active executable.
[ ] Repeat the same measurements.
[ ] Compare editor behavior.
[ ] Check declaration output and project references.
[ ] Identify tools that import TypeScript programmatically.
[ ] Confirm that CI uses the intended version.
[ ] Keep the change only when correctness and tooling still hold.

The takeaway

TypeScript 7 is exciting because the improvement is not a small optimization around the edges. It replaces the compiler and language service with a native implementation designed for speed and parallelism.

But a headline benchmark is not a migration plan.

The safest approach is short and practical:

Measure the current project
          ↓
Create a temporary branch
          ↓
Install TypeScript 7
          ↓
Run the same checks
          ↓
Verify editor and tooling behavior
          ↓
Keep the upgrade or document the blocker

If you test TypeScript 7, share four details in the comments:

approximate project size,
previous TypeScript version,
old type-check time,
new type-check time.

Small applications and large monorepos may tell very different stories.

Read the official TypeScript 7 announcement

Sources and further reading

Microsoft TypeScript Blog: Announcing TypeScript 7.0
Microsoft TypeScript Blog: Progress on TypeScript 7

Connect with Me

If you found this article helpful, let's connect!

💻 GitHub: johnnylemonny

𝗝𝗼𝗵𝗻Follow

Freelance fullstack developer creating fast, modern, and refined applications. I focus on simplicity, performance, and real value for users. I also write tech-related articles here.

Stop Waiting for GitHub Actions: Make Your CI Faster Today

𝗝𝗼𝗵𝗻 — Tue, 29 Sep 2026 15:30:00 +0000

You push a small README correction.

GitHub Actions starts from a clean runner, downloads every dependency, runs the linter, checks every type, executes the complete test suite, builds the application, and uploads an artifact nobody will download.

Before it finishes, you notice another typo and push again.

Now both workflows are running.

Nothing is technically broken. The workflow is simply doing more work than the change requires.

A slow CI pipeline is not only an infrastructure problem. It delays code review, interrupts concentration, consumes runner time, and makes every small pull request feel heavier than it should.

The good news is that the first improvements are usually small.

This guide starts with a deliberately ordinary Node.js workflow and improves it step by step. The same principles apply to other ecosystems, even when the exact setup and cache commands differ.

The goal is not the shortest possible workflow.

The goal is fast, trustworthy feedback with no unnecessary work.

Start with evidence, not guesses
The workflow we are fixing
Fix 1: cancel outdated runs
Fix 2: cache dependency downloads
Fix 3: use deterministic installs
Fix 4: put cheap failures first
Fix 5: parallelize only independent work
Fix 6: avoid irrelevant runs
Fix 7: stop uploading unused artifacts
Add limits and minimum permissions
Treat caches as untrusted input
The complete improved workflow
A practical optimization checklist

Start with evidence, not guesses

Do not optimize a workflow because one step looks slow.

Open a representative run in the Actions tab and record:

total workflow duration,
queue time,
duration of each job,
duration of dependency installation,
duration of linting, type checking, tests, and builds,
how often runs are canceled or superseded,
cache hit and miss behavior,
artifact size and whether anyone downloads it,
repeated work across jobs.

Use several runs rather than one unusual result. A temporary package registry delay or cold runner can distort a single measurement.

A simple baseline is enough:

Dependency installation:  52 seconds
Lint and type check:       24 seconds
Tests:                    104 seconds
Build:                     47 seconds
Artifact upload:           18 seconds
Total workflow:           245 seconds

Your numbers will be different. The important part is knowing where the time goes before changing the YAML.

Optimize feedback time as well as total time

Two workflows can have the same total duration but feel very different.

If a formatting error is reported after four minutes of testing, the developer waits four minutes for information that could have arrived in twenty seconds. Moving fast checks earlier may improve the development loop even if a successful run is not dramatically shorter.

Useful measurements include:

Time to first useful failure
Time to all required checks passing
Runner minutes consumed per pull request
Percentage of runs canceled as outdated

A fast green run is useful. A fast, precise failure is often more useful.

The workflow we are fixing

Here is a common first version:

name: CI

on:
  push:
  pull_request:

jobs:
  ci:
    runs-on: ubuntu-latest

    steps:
      - name: Check out repository
        uses: actions/checkout@v5

      - name: Set up Node.js
        uses: actions/setup-node@v6
        with:
          node-version: 24

      - name: Install dependencies
        run: npm install

      - name: Lint
        run: npm run lint

      - name: Type check
        run: npm run typecheck

      - name: Test
        run: npm test

      - name: Build
        run: npm run build

      - name: Upload build
        uses: actions/upload-artifact@v5
        with:
          name: application-build
          path: dist

This workflow is understandable, which is a good start. It also has several opportunities for improvement:

every push starts another run,
dependency downloads are not cached,
npm install may change the resolved dependency tree,
every check executes in one long sequence,
documentation-only changes run the full pipeline,
every successful run uploads a build,
there is no timeout,
permissions are implicit.

We will improve these one at a time.

Fix 1: cancel outdated runs

Suppose a pull request receives three commits in ten minutes. In most CI workflows, only the newest commit matters. Finishing tests for the first two revisions provides little value once a newer revision exists.

Add workflow-level concurrency:

concurrency:
  group: ci-${{ github.workflow }}-${{ github.ref }}
  cancel-in-progress: true

Full context:

name: CI

on:
  push:
    branches:
      - main
  pull_request:

concurrency:
  group: ci-${{ github.workflow }}-${{ github.ref }}
  cancel-in-progress: true

Runs sharing the same concurrency group will not continue in parallel. With cancel-in-progress: true, a new run cancels the older active run in that group.

Including github.workflow prevents unrelated workflows from accidentally sharing the same group. Including github.ref keeps different branches or pull requests separate.

When not to cancel

Cancellation is a good fit for validation of changing pull requests. It may be wrong for:

deployments that must complete in order,
database migrations,
release publishing,
stateful integration processes,
jobs that perform irreversible external actions.

For those cases, use a separate concurrency group and consider queueing rather than cancellation.

Do not apply one concurrency policy to every workflow without considering what interruption means.

Fix 2: cache dependency downloads

GitHub-hosted jobs start on clean runner images. Without caching, package managers repeatedly download the same dependency archives.

For npm, setup-node can manage the package-manager cache:

- name: Set up Node.js
  uses: actions/setup-node@v6
  with:
    node-version: 24
    cache: npm

- name: Install dependencies
  run: npm ci

This caches npm's download cache, not the completed node_modules directory.

That distinction matters.

npm ci still reconstructs the dependency installation from the lockfile. It simply has a better chance of retrieving packages from a restored local cache instead of downloading every archive again.

Cache the package manager, not `node_modules`

Caching node_modules can look faster in a quick experiment, but it couples the cached directory to details such as:

operating system,
architecture,
Node.js version,
native module compilation,
package-manager behavior,
lifecycle scripts.

A package-manager cache is usually more portable and easier to reason about. The workflow should also remain correct on a cache miss.

Monorepos need the correct lockfile

If the lockfile is not in the repository root, declare the dependency path:

- name: Set up Node.js
  uses: actions/setup-node@v6
  with:
    node-version: 24
    cache: npm
    cache-dependency-path: apps/web/package-lock.json

For multiple lockfiles:

cache-dependency-path: |
  apps/web/package-lock.json
  apps/api/package-lock.json

A cache is only useful when its key changes with the dependency definition that matters.

Verify that the cache actually helps

After enabling caching, compare several runs:

the first run should usually miss and populate the cache,
later runs with the same lockfile should restore it,
a changed lockfile should create a new applicable cache entry,
dependency installation must still succeed when no cache exists.

If install time barely changes, the bottleneck may be lifecycle scripts, native compilation, network access outside the package manager, or work performed after download.

Fix 3: use deterministic installs

In CI, prefer:

npm ci

over:

npm install

npm ci expects a lockfile, removes an existing node_modules directory, and installs the dependency tree represented by the lockfile. It also fails when package.json and the lockfile disagree instead of silently updating the lockfile.

That gives CI a clearer job:

Verify the dependency state committed to the repository.

It also avoids a workflow that succeeds with one dependency resolution while developers or production use another.

Equivalent deterministic commands exist in other package managers. Use the command recommended for reproducible CI installs in your ecosystem.

Keep runtime versions explicit

Do not rely on whatever runtime happens to be preinstalled on the current runner image.

- uses: actions/setup-node@v6
  with:
    node-version: 24
    cache: npm

If the project supports several active Node.js versions, test them intentionally with a matrix rather than accepting accidental variation.

Fix 4: put cheap failures first

A workflow should report obvious problems before starting expensive work.

A reasonable order for many projects is:

Install
  ↓
Formatting or linting
  ↓
Type checking
  ↓
Unit tests
  ↓
Integration tests
  ↓
Production build

That does not mean every step must be sequential. It means the workflow should consider cost and diagnostic value.

Sequential approach

One job is simple and reuses one dependency installation:

steps:
  - uses: actions/checkout@v5

  - uses: actions/setup-node@v6
    with:
      node-version: 24
      cache: npm

  - run: npm ci
  - run: npm run lint
  - run: npm run typecheck
  - run: npm test
  - run: npm run build

Advantages:

one checkout,
one runtime setup,
one dependency installation,
easy logs,
later work stops after an earlier failure.

Disadvantage:

independent checks cannot run in parallel.

For small repositories, this may be the fastest and easiest design.

Gated jobs

If tests are expensive and linting commonly fails, use a cheap quality gate:

jobs:
  quality:
    runs-on: ubuntu-latest
    timeout-minutes: 10

    steps:
      - uses: actions/checkout@v5
      - uses: actions/setup-node@v6
        with:
          node-version: 24
          cache: npm
      - run: npm ci
      - run: npm run lint
      - run: npm run typecheck

  test:
    needs: quality
    runs-on: ubuntu-latest
    timeout-minutes: 20

    steps:
      - uses: actions/checkout@v5
      - uses: actions/setup-node@v6
        with:
          node-version: 24
          cache: npm
      - run: npm ci
      - run: npm test

This avoids running tests after a quality failure, but each job checks out the repository and installs dependencies separately.

The trade-off is real. Measure it.

How should I choose between one job and several jobs?

Prefer one job when:

the repository is small,
dependency installation is significant,
checks are short,
a simple log is valuable,
later steps should stop after the first failure.

Consider several jobs when:

tasks are long and independent,
parallel work meaningfully reduces feedback time,
different jobs need different runners or permissions,
branch protection should show distinct required checks,
one cheap gate can prevent substantial expensive work.

Do not split jobs only because a longer YAML file looks more sophisticated.

Fix 5: parallelize only independent work

Parallel jobs can reduce wall-clock time, but they can also increase total runner usage.

Suppose linting takes one minute, tests take four minutes, and the build takes three minutes.

Sequential execution takes roughly eight minutes after setup. Running all three in parallel may finish in about four minutes, but each job repeats setup and dependency installation.

That can be a good trade when developer feedback is the priority. It can be wasteful when jobs are short or runner usage is constrained.

A balanced structure

Use one lightweight required job and parallelize the expensive independent work after it:

jobs:
  quality:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v5
      - uses: actions/setup-node@v6
        with:
          node-version: 24
          cache: npm
      - run: npm ci
      - run: npm run lint
      - run: npm run typecheck

  test:
    needs: quality
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v5
      - uses: actions/setup-node@v6
        with:
          node-version: 24
          cache: npm
      - run: npm ci
      - run: npm test

  build:
    needs: quality
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v5
      - uses: actions/setup-node@v6
        with:
          node-version: 24
          cache: npm
      - run: npm ci
      - run: npm run build

After quality succeeds, test and build can run at the same time.

Avoid unnecessary matrices

A matrix is valuable when the project genuinely supports several runtimes or platforms:

strategy:
  matrix:
    node-version: [22, 24]

Every matrix combination creates another job. Do not test combinations your project does not claim to support.

For pull requests, it may be reasonable to test the primary runtime and reserve the full compatibility matrix for pushes to main or scheduled runs. Make that choice explicit and ensure changes cannot merge without the coverage your project actually requires.

Fix 6: avoid irrelevant runs

If a workflow validates only an application under src/, a change to unrelated documentation may not require the full test and build pipeline.

Path filters can narrow the trigger:

on:
  pull_request:
    paths:
      - "src/**"
      - "tests/**"
      - "package.json"
      - "package-lock.json"
      - "tsconfig.json"
      - ".github/workflows/ci.yml"

For a monorepo, a service-specific workflow might use:

on:
  pull_request:
    paths:
      - "apps/api/**"
      - "packages/shared/**"
      - ".github/workflows/api-ci.yml"

Path filters can create merge problems

GitHub warns that when a required workflow is skipped because of path or branch filtering, its associated check can remain pending and block the pull request.

Before adding filters, review:

branch protection rules,
required check names,
shared packages that affect several applications,
generated files,
root configuration,
workflow files themselves.

For complex monorepos, a small always-running classification job may be safer than completely skipping a required workflow. That job can determine which test jobs need to run while still producing a stable required check.

Be cautious with skip commands

Commit messages such as [skip ci] can suppress workflows triggered by push or pull_request, but skipped required checks may remain pending. A manual skip should not become the normal optimization strategy.

Use trigger design and job conditions that are understandable to the whole team.

Fix 7: stop uploading unused artifacts

Artifacts are useful for files that must survive after a job finishes:

packaged applications,
test reports,
screenshots from failed browser tests,
coverage reports,
debugging logs,
release candidates.

They are not free decoration for every successful run.

If nobody downloads the build from ordinary pull requests, do not upload it on every pull request.

Upload only on main:

- name: Upload build
  if: github.event_name == 'push' && github.ref == 'refs/heads/main'
  uses: actions/upload-artifact@v5
  with:
    name: application-build
    path: dist
    retention-days: 7

Or upload diagnostic material only after failure:

- name: Upload test diagnostics
  if: failure()
  uses: actions/upload-artifact@v5
  with:
    name: test-diagnostics
    path: test-results
    retention-days: 7

Cache and artifacts solve different problems

Use a cache for reusable data that can be regenerated, such as downloaded dependencies.

Use an artifact for output that someone or another job needs to inspect or consume.

A cache miss should make the workflow slower, not incorrect. Losing a required release artifact may be a real failure.

Choose retention deliberately. Keeping every temporary report for the maximum period creates storage without necessarily creating value.

Add limits and minimum permissions

A hung command should not consume runner time indefinitely.

Set a timeout for every job:

jobs:
  quality:
    runs-on: ubuntu-latest
    timeout-minutes: 10

Use a value based on observed normal duration, with enough room for ordinary variance. A job that usually finishes in two minutes probably does not need a six-hour ceiling.

Make permissions explicit

A CI workflow that only reads repository content can often start with:

permissions:
  contents: read

Place this at workflow level, then increase permissions only for the specific job that needs them.

Minimal permissions improve security more than speed, but they also make the workflow easier to audit. Optimization should not trade away safety for a few seconds.

Pin actions according to your security policy

Version tags are easy to read:

uses: actions/checkout@v5

Security-sensitive organizations may require actions to be pinned to full commit SHAs. GitHub provides repository and organization policies for restricting which actions can run and requiring full-length SHA pinning.

Follow the policy appropriate to your project, and use automated dependency updates so action references do not become stale.

Treat caches as untrusted input

Caching is an optimization boundary, not a secret store.

Never place these in a cache path:

API keys,
access tokens,
environment files containing secrets,
signing keys,
production credentials,
unprotected deployment configuration.

GitHub warns that workflows able to read a cache restore its contents as-is. Fork-based pull requests and low-trust workflow events make cache permissions especially important, and poisoned caches can affect later trusted work.

In September 2026, GitHub made cache-mode generally available across plans. It lets workflows or jobs declare cache access as:

read       restore but do not save
write      restore and save
write-only save but do not restore
none       disable cache access

The secure default depends on the event. Low-trust events receive restricted behavior, and explicitly granting write access can reintroduce cache-poisoning risk.

Do not add a write-capable cache mode to an untrusted event merely to avoid a cache miss. Read the current GitHub syntax documentation and grant only the access the job needs.

A faster compromised workflow is not an optimization.

Treat restored cache content as untrusted, keep secrets out of caches, and use the least cache access required by each event.

The complete improved workflow

The best final structure depends on your repository. The following example favors clear feedback and manageable complexity for a typical Node.js project.

name: CI

on:
  push:
    branches:
      - main
  pull_request:

permissions:
  contents: read

concurrency:
  group: ci-${{ github.workflow }}-${{ github.ref }}
  cancel-in-progress: true

jobs:
  quality:
    name: Lint and type check
    runs-on: ubuntu-latest
    timeout-minutes: 10

    steps:
      - name: Check out repository
        uses: actions/checkout@v5

      - name: Set up Node.js
        uses: actions/setup-node@v6
        with:
          node-version: 24
          cache: npm

      - name: Install dependencies
        run: npm ci

      - name: Lint
        run: npm run lint

      - name: Type check
        run: npm run typecheck

  test:
    name: Test
    needs: quality
    runs-on: ubuntu-latest
    timeout-minutes: 20

    steps:
      - name: Check out repository
        uses: actions/checkout@v5

      - name: Set up Node.js
        uses: actions/setup-node@v6
        with:
          node-version: 24
          cache: npm

      - name: Install dependencies
        run: npm ci

      - name: Run tests
        run: npm test

      - name: Upload test diagnostics
        if: failure()
        uses: actions/upload-artifact@v5
        with:
          name: test-diagnostics-${{ github.run_id }}
          path: test-results
          if-no-files-found: ignore
          retention-days: 7

  build:
    name: Build
    needs: quality
    runs-on: ubuntu-latest
    timeout-minutes: 15

    steps:
      - name: Check out repository
        uses: actions/checkout@v5

      - name: Set up Node.js
        uses: actions/setup-node@v6
        with:
          node-version: 24
          cache: npm

      - name: Install dependencies
        run: npm ci

      - name: Build
        run: npm run build

      - name: Upload main-branch build
        if: github.event_name == 'push' && github.ref == 'refs/heads/main'
        uses: actions/upload-artifact@v5
        with:
          name: application-build-${{ github.sha }}
          path: dist
          if-no-files-found: error
          retention-days: 7

What changed:

pushes to feature branches no longer duplicate pull request validation,
older runs for the same ref are canceled,
npm download caching is enabled,
npm ci produces deterministic installs,
cheap quality checks run before expensive jobs,
tests and builds run in parallel after the quality gate,
artifacts are uploaded only when useful,
failed tests can preserve diagnostics,
every job has a timeout,
repository permissions are explicit.

Is this always faster?

No.

For a small project, three separate npm ci executions may cost more than the parallelism saves. In that case, keep one job and retain the other improvements:

jobs:
  ci:
    runs-on: ubuntu-latest
    timeout-minutes: 20

    steps:
      - uses: actions/checkout@v5
      - uses: actions/setup-node@v6
        with:
          node-version: 24
          cache: npm
      - run: npm ci
      - run: npm run lint
      - run: npm run typecheck
      - run: npm test
      - run: npm run build

The article cannot choose for your repository. Your measurements can.

Changes that look fast but often backfire

Caching everything

A huge, unstable cache may take longer to restore than the work it replaces. It also increases invalidation and security complexity.

Splitting every command into its own job

Parallel YAML is not automatically efficient. Each job has setup overhead and consumes runner time.

Skipping tests based on weak path rules

A root configuration or shared package may affect more applications than the filter suggests.

Removing verification to save time

A workflow that finishes quickly because it no longer checks important behavior is not improved.

Using broad restore keys without understanding them

A partial cache match can restore older content. That may be acceptable for package downloads, but risky for build outputs that depend on exact source, compiler, or environment state.

Running the full compatibility matrix on every draft commit

Test what is necessary for pull request feedback, then run broader compatibility checks at the stage where they provide value.

Uploading every output forever

Artifacts need a consumer and a retention policy.

A practical optimization checklist

Measure

[ ] Record total duration across several representative runs.
[ ] Identify the slowest steps and repeated setup.
[ ] Measure time to first useful failure.
[ ] Check queue time separately from execution time.
[ ] Review runner minutes and artifact storage.

Eliminate outdated work

[ ] Add an appropriate concurrency group.
[ ] Cancel superseded pull request runs.
[ ] Keep deployments and irreversible jobs on a safer policy.
[ ] Avoid duplicate push and pull request runs for the same feature branch.

Improve dependency setup

[ ] Use the deterministic install command for the package manager.
[ ] Cache package-manager downloads.
[ ] Include the correct lockfile in cache configuration.
[ ] Verify behavior on both cache hits and misses.
[ ] Keep secrets and sensitive files out of cache paths.

Improve feedback

[ ] Put cheap, precise checks before expensive work when useful.
[ ] Parallelize only independent tasks.
[ ] Keep job boundaries understandable.
[ ] Use the full compatibility matrix only where it adds value.
[ ] Preserve useful diagnostics after failure.

Reduce irrelevant execution

[ ] Review event and branch triggers.
[ ] Use path filters only after checking required-check behavior.
[ ] Include shared configuration and packages in relevant filters.
[ ] Do not make manual skip messages the main workflow strategy.

Control resources

[ ] Add realistic job timeouts.
[ ] Upload artifacts only when someone or another job needs them.
[ ] Set intentional artifact retention periods.
[ ] Use explicit minimum permissions.
[ ] Review action pinning and update policy.

The takeaway

A faster GitHub Actions workflow usually does not require faster hardware.

It requires less unnecessary work.

Start with the changes that have the clearest value:

Measure the current run
          ↓
Cancel outdated work
          ↓
Cache dependency downloads
          ↓
Use deterministic installs
          ↓
Report cheap failures early
          ↓
Parallelize only where it helps
          ↓
Skip irrelevant work carefully
          ↓
Retain only useful artifacts

Then measure again.

Do not optimize for an impressive YAML file. Optimize for reliable feedback that arrives while the developer still remembers the change.

Which step consumes the most time in your current GitHub Actions workflow?

Explore the GitHub Actions documentation

Sources and further reading

GitHub Docs: Concurrency
GitHub Docs: Control the concurrency of workflows and jobs
GitHub Docs: Dependency caching
GitHub Docs: Dependency caching reference
GitHub Docs: Workflow syntax for GitHub Actions
GitHub Docs: Skipping workflow runs
GitHub Docs: Managing GitHub Actions settings
GitHub Changelog: Control GitHub Actions cache access with cache-mode

Connect with Me

If you found this article helpful, let's connect!

💻 GitHub: johnnylemonny

𝗝𝗼𝗵𝗻Follow

Freelance fullstack developer creating fast, modern, and refined applications. I focus on simplicity, performance, and real value for users. I also write tech-related articles here.

Good read. I think a lot of people confuse confidence with probability, so this distinction really matters in practice.

𝗝𝗼𝗵𝗻 — Mon, 28 Sep 2026 20:06:46 +0000

Raju Dandigam

Sep 28

A Confidence Score Is Not a Probability: Act, Ask, or Abstain

#ai #machinelearning #testing #architecture

3 min read

Stop Pushing Secrets to GitHub: Protect Your API Keys Before They Leak

𝗝𝗼𝗵𝗻 — Tue, 22 Sep 2026 15:30:00 +0000

You notice the mistake a few seconds after pushing:

.env
config.ts
service-account.json

One of those files contains a real credential.

Your first instinct may be to delete the value, create another commit, and push again.

Do not make that your first response.

The credential may already exist in Git history, a clone, a fork, a pull request, an automated log, or a security alert. Removing it from the latest version of a file does not invalidate it.

If a real secret reaches GitHub, assume it has been exposed.

Revoke or rotate it first. Investigate the exposure second. Clean the repository only when necessary. Then add controls that make the same mistake harder to repeat.

If you just pushed a real secret

Revoke or rotate the credential immediately.
Check the provider's access and billing logs.
Identify every location where the secret appeared.
Replace it with a newly issued credential stored outside the repository.
Decide whether Git history must also be rewritten.
Add preventive controls before continuing normal work.

What counts as a secret?
The correct recovery order
Why deleting the file is not enough
Check what Git is already tracking
Use .gitignore correctly
Commit .env.example, not .env
Enable GitHub's protection
Store secrets safely in GitHub Actions
Add local checks before every push
Use the final checklist

What counts as a secret?

A secret is any value that grants access or proves identity and should not be available to everyone who can read the repository.

Common examples include:

API keys,
personal access tokens,
OAuth client secrets,
database passwords,
private SSH keys,
cloud access credentials,
webhook signing secrets,
service-account credentials,
package registry tokens,
session-signing keys,
production connection strings.

Not every configuration value is a secret. A public API base URL, feature flag name, region, or build mode can usually be stored as ordinary configuration.

A simple test is:

If someone copied this value from a public repository, could that person access data, impersonate a service, trigger an action, or create a cost?

If the answer might be yes, treat the value as sensitive.

A secret name is not a secret

This is safe to commit:

PAYMENT_API_KEY=
DATABASE_URL=
EMAIL_PROVIDER_TOKEN=

This is not:

PAYMENT_API_KEY=live_example_value
DATABASE_URL=postgres://user:real-password@host/database
EMAIL_PROVIDER_TOKEN=real_token_value

The first file documents the required configuration. The second file contains credentials that can be used.

The correct recovery order

When a secret is exposed, sequence matters.

Revoke or rotate
        ↓
Assess the exposure
        ↓
Replace the credential
        ↓
Remove it from current code
        ↓
Decide whether history cleanup is necessary
        ↓
Add preventive controls

1. Revoke or rotate the credential

Go to the service that issued the credential and make the exposed value unusable.

Depending on the provider, this may mean:

deleting the key,
rotating the key,
revoking a token,
resetting a password,
disabling a service account,
replacing a signing secret.

If the service supports overlapping credentials, create a replacement, update the application, verify the new credential, and then revoke the exposed one. If immediate abuse is possible, revoke first and accept the temporary interruption.

GitHub's remediation guidance says leaked secrets should be treated as compromised and revoked or rotated. Deleting the value from the current code is not considered sufficient remediation.

2. Check the provider's logs

Look for unexpected activity after the earliest possible exposure time:

unfamiliar IP addresses,
unusual API calls,
new resources,
unexpected downloads,
permission changes,
increased usage,
billing spikes.

The secret provider is the strongest source of truth for whether a credential remains valid and how it was used.

3. Find every exposed location

The value may appear in more places than the file you first noticed:

older commits,
another branch,
an open or closed pull request,
issue text or comments,
workflow logs,
generated artifacts,
documentation,
copied sample files,
forks or local clones.

GitHub secret scanning examines Git history and can also scan other GitHub surfaces such as issues, pull requests, discussions, wikis, and secret gists for supported patterns.

4. Replace the credential safely

Store the replacement in an appropriate secret manager, deployment platform, local environment, or GitHub Actions secret. Do not paste the replacement into the same tracked file.

5. Decide whether history cleanup is necessary

Rotating the credential removes its ability to grant access. Rewriting history serves a different purpose: removing the sensitive value from repository history.

History cleanup may be justified when the exposed material is not easily revocable, contains private data, or must be removed for policy or legal reasons. It is not automatically the first or safest step for every leaked API key.

Rotation limits access. History rewriting removes stored copies.

These are different operations, and rotation usually comes first.

Why deleting the file is not enough

Git stores snapshots through commits. If a secret appears in one commit and is deleted in the next, the earlier commit still contains it.

This sequence does not solve the incident:

git rm .env
git commit -m "Remove environment file"
git push

It removes .env from the latest version of the branch, but the exposed credential remains in previous history and is still usable until revoked.

Deleting and recreating the repository is not a substitute for revoking the credential either. Copies may already exist elsewhere.

Why history rewriting requires care

Tools such as git-filter-repo can remove sensitive paths or values from history, but rewriting shared history has side effects:

commit hashes change,
collaborators must clean or replace existing clones,
outdated clones can accidentally restore the secret,
pull request diffs and comments can be disrupted,
branch protections may need temporary changes,
automation that depends on commit hashes may break.

GitHub explicitly recommends coordinating with collaborators and understanding these consequences before rewriting history.

Do not paste a destructive history-rewrite command from a random snippet and run it against an active repository. Read GitHub's current procedure, back up what must be preserved, coordinate the maintenance window, and verify every branch and tag afterward.

When is history rewriting worth considering?

Consider it when:

the exposed data cannot be revoked,
the repository contains a private key or sensitive personal data,
policy requires removal from all reachable history,
the value appears across many commits or branches,
the repository owner has assessed the operational impact,
collaborators can coordinate replacement or cleanup of clones.

Rotation may be enough for a standard API token when the old value is definitely invalid and no separate policy requires complete removal. Make that decision based on the type of data, repository visibility, exposure, and organizational requirements.

Check what Git is already tracking

A common misunderstanding is that adding a filename to .gitignore makes Git forget it.

It does not.

.gitignore prevents matching untracked files from being added by ordinary commands. It does not stop Git from tracking a file that is already in the index.

Check whether .env is tracked:

git ls-files --error-unmatch .env

If Git prints .env, the file is tracked. If it returns an error, the exact path is not tracked in the current index.

You can search for a wider group of environment files:

git ls-files | grep -E '(^|/)\.env($|\.)'

To stop tracking the current file while keeping the local copy:

git rm --cached .env

Then commit the index change and the updated ignore rule:

git add .gitignore
git commit -m "Stop tracking local environment files"

This is a repository-hygiene step. It does not revoke a leaked credential or erase older commits.

Check the staged snapshot before committing

Before every sensitive commit, inspect what is about to become part of history:

git status --short

git diff --cached

The second command shows the staged diff. This catches many accidental additions before they become commits.

If the staged diff is large, do not skim it blindly. Split unrelated work into smaller commits so that unexpected configuration files and credentials are easier to notice.

Use `.gitignore` correctly

A sensible starting point for a repository that uses local environment files is:

# Local environment files
.env
.env.*
!.env.example

# Local credentials and certificates
*.pem
*.key
service-account*.json

# Tool-specific local configuration
.envrc

Do not copy this block without adapting it.

For example, a project might intentionally track a public test certificate, a harmless fixture named service-account.example.json, or a platform-specific environment template. Ignore rules are repository policy, not universal truth.

Ignore by purpose, not by panic

Avoid broad rules such as:

*.json
*.yaml

Those rules can hide legitimate source files. A good .gitignore excludes local or generated data without making important project files invisible.

Verify the matching rule

If Git ignores a file and you do not know why, run:

git check-ignore -v .env.local

Git will show the ignore file and rule responsible for the match.

Remember global ignore files

A developer may also have a global Git ignore file. That can be useful for editor files and system artifacts, but repository-critical safety rules should still live in the repository's own .gitignore so every contributor receives them.

Commit `.env.example`, not `.env`

A project needs to document its required configuration without publishing real values.

Create .env.example:

# Application
APP_ENV=development
APP_PORT=3000

# Database
DATABASE_URL=

# External services
PAYMENT_API_KEY=
EMAIL_PROVIDER_TOKEN=

# Session security
SESSION_SECRET=

Then document setup:

cp .env.example .env

The developer fills .env locally or retrieves values from the team's approved secret manager.

Use obviously fake examples

Avoid realistic-looking placeholder credentials. A detector may flag them, and a contributor may not know whether they are safe.

Prefer:

PAYMENT_API_KEY=replace_with_local_value

or leave the value blank.

Do not copy a real key and alter only the last few characters. Even a partially exposed credential may reveal information or be reconstructed incorrectly in logs and screenshots.

Fail clearly when required configuration is missing

A missing variable should cause an understandable startup error rather than a mysterious failure later.

const requiredVariables = [
  "DATABASE_URL",
  "PAYMENT_API_KEY",
  "SESSION_SECRET",
];

for (const name of requiredVariables) {
  if (!process.env[name]) {
    throw new Error(`Missing required environment variable: ${name}`);
  }
}

This validates presence, not safety. It cannot determine whether a value has been committed elsewhere or granted excessive permissions.

What is safe to commit?

Usually safe

variable names,
empty placeholders,
public URLs,
non-sensitive feature flags,
documented local defaults,
sample configuration using fake values.

Usually unsafe

live API keys,
access and refresh tokens,
private keys,
production database URLs with credentials,
cloud credentials,
signing secrets,
passwords,
complete service-account files.

Context dependent

public client identifiers,
analytics IDs,
test credentials,
webhook URLs,
certificate files,
internal hostnames.

A value being visible in frontend code does not automatically make it harmless. Check the provider's security model and intended usage.

Enable GitHub's protection

GitHub provides two related defenses: secret scanning and push protection.

Secret scanning

Secret scanning searches for supported credentials in repository history and other supported GitHub content. Public repositories receive secret scanning automatically for free. Availability for private and internal repositories depends on repository ownership, plan, and GitHub Secret Protection settings.

When an alert identifies a real credential, rotate the credential immediately. A scanner can find an exposure, but it cannot undo access that has already occurred.

Push protection

Push protection attempts to stop supported secrets before they reach the repository.

GitHub documents coverage for several entry paths, including:

command-line pushes,
commits made through the GitHub UI,
file uploads,
REST API requests,
certain GitHub MCP server interactions for public repositories.

User push protection is enabled by default on GitHub.com and protects pushes of supported secrets to public repositories. Repository-level protection provides broader administrative controls and depends on GitHub Secret Protection.

If GitHub blocks your push

Do not automatically bypass the warning.

Read the detected secret type and every location listed in the error. If the value is real, remove it from all affected commits before pushing again.

If the secret was introduced in the latest local commit:

# Remove the secret from the file first, then stage the correction
git add path/to/file

git commit --amend --no-edit
git push

Amending changes the commit that introduced the secret instead of adding a second commit that still leaves the original one in the branch history.

If the secret appears in earlier local commits, resolving the block may require an interactive rebase. Follow GitHub's current instructions and create a backup branch before changing local history.

Bypass only when you have verified that the value is a false positive, a documented test value, or another value explicitly safe to publish. “I will fix it later” is not a safe default for a live credential.

Push protection recognizes supported patterns. It is an important safety net, not proof that a repository contains no secrets.

Store secrets safely in GitHub Actions

Workflow files are committed to the repository, so never place a credential directly in YAML.

Wrong:

- name: Deploy
  env:
    DEPLOY_TOKEN: live_token_value
  run: ./deploy.sh

Use a GitHub Actions secret:

- name: Deploy
  env:
    DEPLOY_TOKEN: ${{ secrets.DEPLOY_TOKEN }}
  run: ./deploy.sh

Create the secret in:

Repository settings
→ Secrets and variables
→ Actions
→ New repository secret

Or use GitHub CLI:

gh secret set DEPLOY_TOKEN

GitHub supports secrets at repository, environment, and organization scope. Choose the narrowest scope that matches the workflow.

Secrets and variables are not interchangeable

Use a configuration variable for non-sensitive data:

env:
  DEPLOY_REGION: ${{ vars.DEPLOY_REGION }}

Use a secret for sensitive data:

env:
  DEPLOY_TOKEN: ${{ secrets.DEPLOY_TOKEN }}

GitHub warns that variables are not masked in build output by default. Sensitive values belong in secrets.

Apply least privilege

A deployment credential should not automatically have administrator access to every repository or environment.

Prefer:

short-lived credentials where available,
read-only access when write access is unnecessary,
environment secrets for protected deployments,
fine-grained tokens over broad personal tokens,
explicit GITHUB_TOKEN permissions,
required reviewers for sensitive environments.

A minimal workflow permission block might look like:

permissions:
  contents: read

Increase permissions only for the job that needs them.

Do not rely completely on log redaction

GitHub masks many known secret values, but transformations, structured values, generated tokens, and accidental output can still create risk.

Avoid commands such as:

echo "$DEPLOY_TOKEN"

Do not place JSON or YAML blobs into one secret if individual values can be stored separately. GitHub notes that structured secret values can be harder to redact reliably because masking often depends on exact matches.

If an unredacted credential reaches a workflow log, delete the affected log and rotate the credential.

Add local checks before every push

Remote protection is valuable, but earlier feedback is better.

A local secret scanner can inspect staged content or commits before they leave the workstation. Common open-source options include tools such as Gitleaks and TruffleHog. Select a tool based on your language ecosystem, CI environment, supported patterns, maintenance policy, and false-positive handling.

The workflow matters more than the brand:

Edit
  ↓
Review staged diff
  ↓
Run local secret scan
  ↓
Commit
  ↓
Run CI scan
  ↓
Push protection
  ↓
Repository secret scanning

No layer is complete on its own.

Add a pre-commit check carefully

A local hook can scan staged changes, but hooks stored only under .git/hooks are not automatically shared with every clone. If the team depends on a hook, manage it through a documented tool or repository setup process.

The hook should:

scan only relevant staged content for speed,
show the file and rule that triggered,
provide a documented false-positive process,
fail safely,
avoid uploading source code to an unapproved service.

Repeat the check in CI

Local hooks can be skipped or misconfigured. CI provides a consistent second boundary for pull requests and protected branches.

A failing CI scan should explain how to remediate the finding. Security gates that only display “failed” encourage bypasses rather than safe behavior.

Avoid these common mistakes

“It is private, so the key is safe”

Private repositories reduce exposure but do not make hardcoded credentials appropriate. Access can expand, logs can leak, repositories can change visibility, and credentials often outlive the code that contains them.

“Base64 hides the value”

Base64 is encoding, not encryption. Anyone with the string can decode it.

“The key is only for testing”

Test credentials can still access shared data, trigger paid services, or become production credentials later. Use clearly scoped, disposable values with strict limits.

“I deleted the repository”

Deletion does not revoke the credential or delete existing copies.

“`.gitignore` protects everything”

.gitignore does not remove tracked files, scan arbitrary content, prevent manual force-adds, or invalidate exposed credentials.

“Push protection will catch every secret”

Detectors cover many known and generic patterns, but no scanner can identify every custom password, encoded value, or context-specific credential.

Final repository security checklist

Local development

[ ] Real secrets are stored outside tracked files.
[ ] .env and local credential files are ignored.
[ ] .env.example contains names and safe placeholders only.
[ ] git diff --cached is reviewed before sensitive commits.
[ ] Required configuration fails with a clear startup error.
[ ] Local scanning is documented and repeatable.

GitHub repository

[ ] Repository visibility is intentional.
[ ] Secret scanning availability and alerts have been reviewed.
[ ] Push protection is enabled where available.
[ ] Bypass permissions are limited and monitored.
[ ] Security alerts have a named owner.
[ ] Contributors know how to report an accidental exposure.

GitHub Actions

[ ] Sensitive values use Actions secrets, not workflow literals.
[ ] Non-sensitive configuration uses variables.
[ ] GITHUB_TOKEN permissions are minimal.
[ ] Deployment secrets use protected environments where appropriate.
[ ] Workflows do not print secrets or transformed secret values.
[ ] Third-party actions are reviewed and pinned according to team policy.

Incident response

[ ] The exposed credential is revoked or rotated first.
[ ] Provider logs and billing are reviewed.
[ ] Every known location is identified.
[ ] History rewriting is treated as a coordinated operation.
[ ] Existing clones are addressed after a rewrite.
[ ] Preventive controls are added after the incident.

The takeaway

The most dangerous misconception is that removing a key from the latest file removes the exposure.

It does not.

A safer response is straightforward:

Assume exposure
      ↓
Revoke or rotate
      ↓
Check usage
      ↓
Replace securely
      ↓
Clean current code
      ↓
Evaluate history cleanup
      ↓
Strengthen prevention

The strongest setup is also layered:

.gitignore keeps ordinary local files out of new commits,
.env.example documents configuration safely,
staged-diff review catches mistakes before commit,
local and CI scanning provide early feedback,
push protection blocks many supported secrets,
secret scanning finds supported exposures,
least-privilege credentials reduce the impact of a mistake.

None of these controls replaces the others.

If you discover a live secret in a repository, do not spend the first minutes trying to make the commit disappear. Make the credential useless first.

What is the most useful secret-protection check you have added to a project?

Read GitHub's secret remediation guide

Sources and further reading

GitHub Docs: Remediating a leaked secret in your repository
GitHub Docs: Removing sensitive data from a repository
GitHub Docs: Push protection
GitHub Docs: Working with push protection from the command line
GitHub Docs: Secret scanning
GitHub Docs: Using secrets in GitHub Actions
GitHub Docs: Secure use reference for GitHub Actions

Connect with Me

If you found this article helpful, let's connect!

💻 GitHub: johnnylemonny

𝗝𝗼𝗵𝗻Follow

Freelance fullstack developer creating fast, modern, and refined applications. I focus on simplicity, performance, and real value for users. I also write tech-related articles here.