DEV Community: Benoit COUETIL 💫

🦊 GitLab CI: Automated Testing of Job Rules

Benoit COUETIL 💫 — Fri, 20 Mar 2026 16:52:47 +0000

Initial thoughts
1. The problem: CI rules complexity
2. The solution: automated job list validation
- gitlab-ci-local: the cornerstone
- How it works
3. Setting up the test infrastructure
- Directory structure
- Test case definition
- Reference CSV files
4. The validation script
5. The CI job
6. Workflow: adding or modifying jobs
7. Testing with rules:changes
- The problem with rules:changes and gitlab-ci-local
- The force-build label workaround
- Per-module test cases
8. Documentation that writes itself
Wrapping up
Further reading

Initial thoughts

As a CICD engineer, you've likely experienced this frustrating scenario: you modify a job's rules: to optimize pipeline execution, only to discover later that some jobs no longer trigger in specific situations. Or worse, jobs that should be mutually exclusive now run together, wasting resources and causing confusion.

GitLab CI's rules: syntax is powerful but complex. With workflow:rules, job-level rules:, extends:, !reference, and changes: all interacting, predicting which jobs will run for a given pipeline type becomes increasingly difficult as your CI configuration grows.

What if we could automatically test that the right jobs appear for each pipeline type? This article presents a practical solution: using gitlab-ci-local to validate job presence across all your pipeline variants, with reference files that serve as both tests and documentation.

1. The problem: CI rules complexity

In a mature GitLab CI setup, you typically have multiple pipeline types — especially in mono-repos with complex workflow:rules and module-based builds. Here, we focus on testing that your rules produce the expected jobs.

Type	Trigger	Example
A	Merge Request	Developer pushes to a feature branch with an open MR
B	Protected Branch	Push to `main`, `develop`, or release branches
C	Manual Pipeline	Triggered from GitLab UI with variables
D	Tag (auto)	Tag push triggering preprod deployment
E	Tag (manual)	Tag pipeline for production deployment
F	Scheduled	Nightly builds, cache warmup, full test suites

Each pipeline type should have a specific set of jobs. For example:

MR pipelines should run tests related to changed modules only
Protected branch pipelines should run all tests and prepare deployable artifacts
Scheduled pipelines might run expensive security scans or full regression tests

When you have 50+ jobs with complex rules:, maintaining this becomes a nightmare. Change one rule, break three jobs, sometimes in further pipeline types — the butterfly effect, CI edition. And if you've ever endured the slow feedback loop of CI YAML modifications, you know that discovering these regressions through push-and-pray is not sustainable.

2. The solution: automated job list validation

The solution is surprisingly simple and will be detailed step by step in this article:

Define test cases as variable files simulating each pipeline type
Use gitlab-ci-local --list-csv to get the jobs that would run
Compare with reference CSV files committed to the repository
Fail the CI if the job list differs from the expected reference

gitlab-ci-local: the cornerstone

gitlab-ci-local is an open-source tool that parses GitLab CI YAML locally. Among its many features, --list-csv outputs the jobs that would run given a set of variables, without actually executing anything.

This is perfect for our use case: we simulate pipeline conditions and check the resulting job list. Think of it as a flight simulator for your CI — all the turbulence, none of the crashes.

# Install globally
npm install -g gitlab-ci-local

# List jobs as CSV with custom variables
gitlab-ci-local --list-csv --variables-file ci/test/D-tag-preprod.variables.yml

How it works

Load the test case
Run gitlab-ci-local
Capture the CSV output listing all jobs that would run:

   name;stage;when;allowFailure;needs
   📦🌐api-build;📦 Package;on_success;false;[]
   🗄️✅back-unit-tests;✅ Test;on_success;false;[]
   ...

Compare with the reference file ci/test/D-tag-preprod.csv:
- ✅ Match → Test passes
- ❌ Diff → Show diff, fail test

When job name / stage / when condition / allowFailure flag / needed jobs change, the CI will let you know, and you will commit the change or fix the CI regression.

3. Setting up the test infrastructure

Directory structure

Create a ci/test/ directory to store test definitions and reference files:

ci/
└── test/
    [...]
    ├── D-tag-preprod.variables.yml      # Tag triggering preprod deployment
    ├── D-tag-preprod.csv                 # Expected jobs (reference)
    ├── F-scheduled.variables.yml         # Scheduled nightly pipeline
    └── F-scheduled.csv                   # Expected jobs (reference)

The naming convention {Type}-{Description}.variables.yml makes it easy to identify test scenarios.

Test case definition

Each .variables.yml file contains the GitLab CI predefined variables that simulate a specific pipeline context:

D-tag-preprod.variables.yml - Tag triggering preprod deployment:

CI_COMMIT_TAG: "v1.2.3"

F-scheduled.variables.yml - Scheduled nightly pipeline:

CI_PIPELINE_SOURCE: "schedule"
CI_COMMIT_BRANCH: "main"

The key is to set the variables that your workflow:rules and job rules: check to determine pipeline behavior.

Reference CSV files

The reference CSV files contain the expected job list. They're auto-generated on first run and then committed. Here's an example:

F-scheduled.csv:

name;description;stage;when;allowFailure;needs
🔎🖥️front-lint;"";🔎 Check;on_success;false;[]
✅🖥️front-unit-tests;"";🔎 Check;on_success;false;[]
📦🖥️front-build;"";📦 Package;on_success;false;[]
📦🌐api-build;"";📦 Package;on_success;false;[]
🗄️✅back-unit-tests;"";✅ Test;on_success;false;[]
🗄️🧩✅back-integration-tests;"";✅ Test;on_success;false;[]
🕵️💯security-scan;"";🕵 Quality;on_success;false;[]
🕵ᯤ🌐api-sonar;"";🕵 Quality;on_success;false;[📦🌐api-build]

This file serves as documentation and test oracle simultaneously:

Developers can quickly see which jobs run for scheduled pipelines
CI validates that reality matches expectations

4. The validation script

Here's a Bash script that automates the validation. It discovers test cases, runs gitlab-ci-local, and compares with reference files:

#!/bin/bash
# ci/test-ci-jobs-list.sh
#
# Tests CI non-regression by comparing gitlab-ci-local --list-csv
# output with committed reference files.
# Creates/updates reference files when differences are detected.
#
# Usage: ./ci/test-ci-jobs-list.sh [test-case-name]

set -e
FAILED=0
TOTAL=0

# Optional: run a single test case
if [ -n "$1" ]; then
    FILES="ci/test/$1.variables.yml"
else
    FILES="ci/test/*.variables.yml"
fi

for varsFile in $FILES; do
    [ -f "$varsFile" ] || continue

    baseName=$(basename "$varsFile" .variables.yml)
    referenceFile="ci/test/$baseName.csv"

    echo "🔍 Testing $baseName..."
    ((TOTAL++))

    # Generate current job list (filter out logs, keep only CSV)
    gitlab-ci-local --list-csv --variables-file "$varsFile" 2>/dev/null | \
        grep -E '^[^[]' | grep -v '^$' > "$referenceFile.tmp" || true

    # Ensure header is present
    if ! head -1 "$referenceFile.tmp" | grep -q "^name;"; then
        gitlab-ci-local --list-csv --variables-file "$varsFile" 2>&1 | \
            grep -E "^name;" > "$referenceFile.header"
        cat "$referenceFile.header" "$referenceFile.tmp" > "$referenceFile.new"
        mv "$referenceFile.new" "$referenceFile.tmp"
        rm -f "$referenceFile.header"
    fi

    mv "$referenceFile.tmp" "$referenceFile"

    # Check for git differences
    if ! git diff --exit-code -- "$referenceFile"; then
        echo "❌ FAILED: $baseName - jobs list has changed"
        ((FAILED++)) || true
    else
        echo "✅ PASSED: $baseName"
    fi
done

echo ""
echo "=========================================="
echo "Results: $((TOTAL - FAILED))/$TOTAL passed"
echo "=========================================="

if [ $FAILED -gt 0 ]; then
    echo "Review the diffs above and commit updated reference files."
    exit 1
fi

💡 Note: A PowerShell version is available on request for Windows-based runners.

5. The CI job

Add a job resembling this to your .gitlab-ci.yml:

✅🦊validate-ci-jobs-list:
  stage: ✅ Test
  resource_group: avoid-gcl-concurrency  # gitlab-ci-local may not be thread-safe
  tags: [your-runner-tag]
  image: node:20-alpine
  before_script:
    - npm install -g gitlab-ci-local
  script:
    - ./ci/test-ci-jobs-list.sh

Key points:

resource_group: Prevent concurrent executions that might conflict
rules:: you may want to run this on merge request when CI has changed, and on long-lived branches

6. Workflow: adding or modifying jobs

When you modify CI rules, follow this workflow:

Make your changes to .gitlab-ci.yml or included files
Run locally (optional but recommended):

   ./ci/test-ci-jobs-list.sh

Review the diffs - the script shows exactly what changed:

  🗄️✅back-unit-tests;"";✅ Test;on_success;false;[]
  🗄️🧩✅back-integration-tests;"";✅ Test;on_success;false;[]
− 🕵ᯤsonar;"";✅ Test;manual;true;[]
+ 🕵ᯤsonar;"";✅ Test;on_success;false;[]
  🕵ᯤ🌐api-sonar;"";🕵 Quality;on_success;false;[📦🌐api-build]

If intentional, commit the updated CSV files along with your CI changes
If unintentional, fix your rules before committing

This creates a self-documenting system: the git history of CSV files shows exactly when and why job behavior changed.

7. Testing with rules:changes

The examples above cover pipeline types where all jobs of a category run (protected branches, tags, scheduled). But on a real project with module-based rules: using changes:, MR pipelines only trigger jobs for modules whose files were modified. This is where things get spicy — and where regressions love to hide.

The problem with rules:changes and gitlab-ci-local

gitlab-ci-local does evaluate rules:changes — it has access to the git diff. But since our test job runs inside the MR pipeline, the diff contains whatever files happen to be modified in the current MR. A test for "backend-only pipeline" would suddenly include frontend jobs if someone touched a frontend file in the same branch. The results depend on the MR content, not on CI configuration — the opposite of a reliable test.

The solution is to guard changes: rules with $GITLAB_CI == "true":

.api-mr-rules:
  rules:
    - if: $CI_MERGE_REQUEST_ID && $GITLAB_CI == "true"
      changes:
        - src/Api/**/*
        - src/Commons/**/*

In real GitLab CI, $GITLAB_CI is always "true", so the rule works normally — changes: is evaluated against the MR diff. But gitlab-ci-local sets $GITLAB_CI to "false", which makes the if: condition fail before changes: is even evaluated. The entire rule is cleanly skipped, no ambiguity. Schrödinger's job: it exists in your YAML but never appears in the output — by design.

This is what makes test results deterministic: regardless of which files are modified in the current MR, the changes: rules are consistently neutralized, and only the label-based fallback (below) controls which jobs appear.

The force-build label workaround

The trick is to add a label-based bypass to every module's rules. In your actual CI, this serves double duty: developers use it when they need to force-rebuild a module (dependencies changed outside the changes: scope, cache issues, cosmic rays), and tests use it to simulate "this module has changes":

.api-mr-rules:
  rules:
    - if: $CI_MERGE_REQUEST_ID && $GITLAB_CI == "true"
      changes:
        - src/Api/**/*
        - src/Commons/**/*
    - if: $CI_MERGE_REQUEST_LABELS =~ /force-build-back/

In real CI, the first rule handles normal operation (job runs when matching files change). In gitlab-ci-local, the first rule is dead — only the label matters. Now the test variables file simply sets the right label:

A-MR-back.variables.yml — simulating backend changes:

CI_MERGE_REQUEST_ID: "12345"
CI_MERGE_REQUEST_LABELS: "force-build-back"

A-MR-front.variables.yml — simulating frontend changes:

CI_MERGE_REQUEST_ID: "12345"
CI_MERGE_REQUEST_LABELS: "force-build-front"

A-MR-no-change.variables.yml — the MR where only non-module files changed (docs, README, etc.):

CI_MERGE_REQUEST_ID: "12345"
CI_MERGE_REQUEST_LABELS: ""

This last one is particularly valuable: it verifies that common jobs (config generation, post-deploy checks) still run even when no module was touched. The CI equivalent of checking that the lights still work when nobody's home.

💡 We won't dive deeper into label-based pipeline control here — that topic deserves its own article. Stay tuned.

Per-module test cases

With this approach, the ci/test/ directory naturally reflects your module structure:

ci/test/
├── A-MR-no-change.variables.yml          # MR with no module changes
├── A-MR-no-change.csv
├── A-MR-back.variables.yml               # Backend module changes
├── A-MR-back.csv
├── A-MR-front.variables.yml              # Frontend module changes
├── A-MR-front.csv
├── A-MR-migrations.variables.yml         # Database migrations
├── A-MR-migrations.csv
├── A-MR-all-force-build-labels.variables.yml  # Everything activated
├── A-MR-all-force-build-labels.csv
├── B-protected-branch.csv                # Non-MR types (no changes: involved)
└── ...

The resulting CSV diffs tell a precise story. For instance, A-MR-back.csv might list 33 jobs while A-MR-front.csv lists only 18 — you can instantly verify that backend Sonar jobs don't sneak into frontend-only pipelines, and that shared deployment jobs appear in both.

On the project that inspired this approach (7 modules, 50+ jobs, 5 pipeline types), we ended up with 11 test cases covering every meaningful combination. The entire suite runs in under 10 seconds. That's less time than it takes to explain to a colleague why their MR pipeline is mysteriously empty.

8. Documentation that writes itself

The CSV reference files serve a dual purpose: automated validation and always up-to-date documentation.

If you use a doc-as-code tool like VitePress, Asciidoctor, or Docusaurus, you can render CSV files as HTML tables at build time — either natively or through a custom plugin. Most tools support CSV includes out of the box or with minimal effort, and any AI assistant can generate the glue code for your specific stack in seconds.

In your documentation markdown, it would look something like this:

## Pipeline Type D: Tag (preprod deployment)

```csv-table ci/test/D-tag-preprod.csv
```

## Pipeline Type F: Scheduled (nightly)

```csv-table ci/test/F-scheduled.csv
```

💡 Note: Adapt the csv-table syntax to your tool.

The same ci/test/*.csv files then serve two purposes:

CI Validation: gitlab-ci-local compares actual job lists against these reference files → pipeline passes or fails
Documentation build: your doc tool renders these CSV files as HTML tables → always accurate documentation

No more outdated documentation. No more "the wiki says X but CI does Y". The CSV files are the single source of truth — enforced by CI, displayed by docs, and immune to the corporate amnesia that plagues most wikis.

Wrapping up

This approach provides several benefits:

Benefit	Description
🛡️ Regression prevention	Catch unintended rule changes before they reach production
📖 Living documentation	CSV files document expected behavior for each pipeline type
🔍 Clear diffs	See exactly which jobs were added, removed, or modified
⚡ Fast feedback	Tests run in seconds, not minutes
🎯 Granular testing	Test specific pipeline variants independently

The key insight is treating your CI configuration as code that deserves its own tests. Just as you wouldn't ship application code without tests, you shouldn't ship CI changes without validating their effects.

This technique has saved countless hours of debugging "why doesn't this job run anymore?" issues. The upfront investment in setting up the test infrastructure pays off quickly as your CI configuration grows in complexity. Combined with other CI best practices to avoid widespread anti-patterns, it builds a solid foundation for maintainable pipelines.

📝 Dev.to Writers Community: One Command to Maintain Them All

Benoit COUETIL 💫 — Sat, 28 Feb 2026 17:19:00 +0000

The Community
What I Built
Demo
- Content enrichment
- Bulk operations
- Environment
Code
How I Built It
Further reading

This is a submission for the DEV Weekend Challenge: Community

The Community

The people writing technical articles right here on dev.to — the ones who keep a folder full of markdown files and occasionally wonder why they didn't just use the browser editor. Occasionally.

What I Built

A fork of devto-cli — a Node.js CLI that turns local markdown files into dev.to articles. The original tool by Yohan Lasorsa handles article creation, push to dev.to via the API, stats, GitHub-hosted images with automatic URL rewriting, and a GitHub Action for automated publishing.

I just kept running into small things it didn't cover, so I forked it and added a bunch of features. Here's what came out.

Demo

Content enrichment

Diagrams as code with Kroki

Write a diagram in your markdown:

<!-- diagram-name: shell-runner-well-sized-before-optimization -->
'''mermaid
%%{init: {'theme':'forest'}}%%
gantt
    title Shell Executor - Well-Sized Server (Before Extreme Optimization)
    dateFormat X
    axisFormat %s
    section Waiting
    Sufficient capacity                    :active, wait, 0, 1s
    section Server
    Resource allocation                    :active, prep, after wait, 1s
    section OS
    Native system (no container)           :active, env, after prep, 1s
    section Source Code
    Git fetch (local reuse)                :active, git, after env, 2s
    section Cache
    Restore (local filesystem)             :active, cache, after git, 2s
    section Artifacts
    Download artifacts                     :done, art, after cache, 3s
    section Execution
    Scripts                                :done, exec, after art, 10s
    section Termination
    Upload cache (local)                   :active, save1, after exec, 1s
    Upload artifacts (GitLab network)      :done, save2, after save1, 4s
'''

Run:

dev diaggen my-article.md

The CLI sends the block to Kroki, gets back an image, and saves it locally. After committing the image to your assets repo, dev push replaces the code block with an image link pointing to GitHub — dev.to never sees the mermaid source. But your local source is preserved, letting you fix the diagram when needed. Supports mermaid, PlantUML, BlockDiag, and many others.

See 13 diagrams generated this way in GitLab Runners: Which Topology for Fastest Job Execution?

Automatic table of contents

Add two markers anywhere in your article:

<!-- TOC start -->
<!-- TOC end -->

Then:

dev push --update-toc my-article.md

The CLI scans headings and generates anchored links between the markers. The current article uses it.

ANSI-styled terminal output

Choosing colors in a displayed log is a must. The CLI converts a simple pseudo-ANSI syntax into styled HTML blocks.

In your markdown, use {{TAG}} to start a color. The {{/}} closing tag is optional — without it, the color continues until the next tag or end of block. This means multiline content stays colored naturally:

'''ansi
{{RED}}The quick brown fox{{BOLD_RED}} jumps over the lazy dog
{{GREEN}}The quick brown fox{{BOLD_GREEN}} jumps over the lazy dog
{{YELLOW}}The quick brown fox{{BOLD_YELLOW}} jumps over the lazy dog
{{ORANGE}}The quick brown fox{{BOLD_ORANGE}} jumps over the lazy dog
{{BLUE}}The quick brown fox
{{BOLD_BLUE}}jumps over the lazy dog
{{MAGENTA}}The quick brown fox{{BOLD_MAGENTA}} jumps over the lazy dog
{{CYAN}}The quick brown fox{{BOLD_CYAN}} jumps over the lazy dog
{{WHITE}}The quick brown fox{{BOLD_WHITE}} jumps over the lazy dog
{{GRAY}}The quick brown fox{{BOLD_GRAY}} jumps over the lazy dog
{{BG_RED}}The quick brown fox{{/}} jumps over the lazy dog
{{BG_GREEN}}The quick brown fox{{BG_YELLOW}} jumps over the lazy dog
{{BG_ORANGE}}The quick brown fox
jumps over the lazy dog
{{BG_BLUE}}The quick brown fox{{BG_MAGENTA}} jumps over the lazy dog
{{BG_CYAN}}The quick brown fox{{BG_WHITE}} jumps over the lazy dog
{{BG_GRAY}}The quick brown fox{{/}} jumps over the lazy dog
'''

After dev push, this renders as a properly colored terminal block on dev.to — 9 base colors × 3 variants (plain, bold, background). No closing tag needed: colors flow until the next tag, even across lines. All colors are overridable via .env variables (ANSI_RED=#ff6161, etc.).

The quick brown fox jumps over the lazy dog
The quick brown fox jumps over the lazy dog
The quick brown fox jumps over the lazy dog
The quick brown fox jumps over the lazy dog
The quick brown fox
jumps over the lazy dog
The quick brown fox jumps over the lazy dog
The quick brown fox jumps over the lazy dog
The quick brown fox jumps over the lazy dog
The quick brown fox jumps over the lazy dog
The quick brown fox jumps over the lazy dog
The quick brown fox jumps over the lazy dog
The quick brown fox
jumps over the lazy dog
The quick brown fox jumps over the lazy dog
The quick brown fox jumps over the lazy dog
The quick brown fox jumps over the lazy dog

Bulk operations

Run one command, affect all your articles at once.

A shared footer file (e.g. "Further reading" links) kept in sync across all articles. Update the footer file once, then push all — the CLI finds the # Further reading marker in each article and replaces everything below it with the shared footer content:

dev push --update-toc "*.md"

Smart file renaming

When an article title changes, the filename should follow:

dev rename "*.md"

🦊 GitLab CI: A Battle-Tested Mono-Repo CI/CD Architecture
  GITLAB_mono-repo-architecture.md → GITLAB_ci-battle-tested-mono-repo-ci-cd-architecture.md

🔍 Every Developer Should Review Code — Not Just Seniors
  MISC_every-developer-should-review-code.md → MISC_every-developer-review-code-seniors.md

Strips emoji, filters 80+ English stop words (a, the, and, how…), keeps the first 5 significant words, preserves the date/category prefix.

Article badges generation

Generate visual badges for a GitHub profile README:

dev badges --jpg "*.md"

Each badge overlays the article's cover image with its title, publication date, view count, and reading time. Articles are auto-grouped by category (extracted from filename prefix) and sorted by date within each group.

The most viewed articles get golden stars ⭐ — popularity is measured relative to the 2nd most viewed article (to avoid one viral post skewing everything). ≥90% of that reference = 3 stars, ≥50% = 2, ≥25% = 1. Stars and stats refresh on every run.

Produces a _ARTICLES.md file with badge images linking to each article. Here's my GitLab section:

Embed the file in a pinned article and all your articles become browsable from one place. Badges stay up to date with every dev badges run.

Broken link checking

Before publishing:

dev checklinks my-article.md

Catches dead URLs before readers do. Smart enough to treat HTTP 403/429 as "probably fine" (bot blocking, not a dead link). Also validates cover image and canonical URL from front matter. Works on all articles at once with dev checklinks "*.md".

Environment

Configure once, forget about it.

Organization publishing

One line in .env:

DEVTO_ORG=zenika

The CLI adds organization: zenika to the front matter and resolves the org ID automatically on push.

Proxy support for corporate environments

export HTTPS_PROXY=http://proxy.company.com:8080
dev push

All API calls, Kroki requests, and link checks go through the proxy. Self-signed certificates handled too.

Code

github.com/bcouetil/devto-cli

How I Built It

This is a TypeScript / Node.js CLI, forked from sinedied/devto-cli. Each feature was added incrementally — proxy support first, then Kroki diagrams, TOC generation, organization publishing, footer management, link checking, file renaming, ANSI blocks, and finally badge generation.

This fork started as a one-line proxy fix and kept growing from there. Every feature was added because I needed it, not because it looked good on a feature list.

Illustrations generated locally by Draw Things using Flux.1 [Schnell] model

🦊 GitLab CI: Achieving 3-Second Jobs on Million-Line Codebases

Benoit COUETIL 💫 — Thu, 19 Feb 2026 21:17:00 +0000

Initial thoughts
Prerequisites: The right foundation
Classic optimizations first
Extreme optimizations: going beyond
Breaking down the 3-second job
- Job execution on a well-sized shell runner
- Phase 1: Waiting (~0s)
- Phase 2: Server (~0s)
- Phase 3: OS / Shell (~1s)
- Phase 4: Git Clone/Fetch (~1s)
- Phase 5: Cache (~0s)
- Phase 6: Artifacts (none)
- Phase 7: Script (~1s)
- Phase 8: Termination (~0s)
Final timing breakdown
Real project results
Wrapping up
Further reading

Initial thoughts

Is it really possible to run GitLab CI jobs in just 3 seconds on a codebase with several million lines of code? The answer is yes, and this article will show you how.

After comparing different runner topologies in GitLab Runners: Which Topology for Fastest Job Execution?, we found that Shell and Docker executors offer the best potential for fast job execution. Now it's time to push those runners to their absolute limits with extreme optimizations.

This isn't about theoretical performance—these are real, production results achieved on actual projects. The 3-second example is our lightest job (a JIRA/MR synchronization check), and it consistently runs in ~3-5 seconds on our 1,500,000+ line mono-repo actively maintained by 15 developers. We have jobs at various fixed speeds: one at ~3-5s, some at a few seconds, others at ~2min for resource-heavy builds, and end-to-end tests at ~15min. The optimizations in this article apply to all jobs.

Time to hunt down every millisecond of overhead — no mercy. Because, as French president Macron said, "Pipeline sometimes is too slow".

Prerequisites: The right foundation

Before diving into extreme optimizations, we need the right infrastructure foundation. Based on our topology comparison, this means:

Infrastructure choice:

✅ Shell executor (fastest) or Docker executor (fast with isolation)
✅ Single well-provisioned server (not autoscaling)
✅ Local SSD storage (NVMe preferred)
✅ Sufficient CPU/RAM for concurrent jobs
✅ Fast network connection to GitLab instance

Why this matters: Every other topology adds fundamental latencies (VM provisioning, pod scheduling, remote cache, shared resources) that cannot be eliminated through configuration. Starting with the wrong topology means you've already lost.

Classic optimizations first

Before going extreme, apply standard GitLab CI optimizations from GitLab CI Optimization: 15+ Tips for Faster Pipelines.

But these optimizations alone won't get you to 3 seconds. They'll improve your average pipeline duration. To reach sub-10-second jobs, we need to dig deeper into GitLab runners arcanes.

Extreme optimizations: going beyond

Now we enter extreme optimization territory. These techniques are specific to Shell/Docker runners and exploit their local filesystem advantages.

Here's what we'll optimize to near-zero overhead:

Waiting time → Proper server sizing
Server provisioning → Already exists (no VM/pod creation)
OS/Container startup → Native shell (1s) or cached images
Git operations → Shallow fetches, reused local clones
Cache → Local filesystem, preserved directories
Artifacts → None in our fastest jobs
Script execution → Minimal in our fastest jobs
Termination → No cache/artifact uploads in our fastest jobs

Let's break down each phase and see how to optimize it.

Breaking down the 3-second job

Job execution on a well-sized shell runner

First, let's visualize what a well-optimized shell runner job timeline looks like before extreme optimizations:

Total: already fast on fastest jobs scripts. Now let's optimize each phase to the extreme.

Phase 1: Waiting (~0s)

The problem: Jobs queue when runners are saturated. Your developers are staring at a spinner. Somewhere, a PM is asking "is the pipeline stuck again?".

The solution : proper server sizing

Size the VM for peak concurrent load (not average)
CPU: 16 cores for a 15 developers team on a mono-repo. Not the smallest, but cheap comparing to salaries
Disk I/O: NVMe SSD essential for concurrent git/cache operations
Concurrency tuned (~16 concurrent jobs for 16 CPU)

⏱️ Result: ~0s waiting (when properly sized). No more "grab a coffee" excuses — the job finishes before you stand up.

Phase 2: Server (~0s)

Server already exists — shocking, right?

Shell runner: no provisioning at all
Docker runner: no VM creation, just container scheduling
Resources immediately available
No cloud API calls needed

This is a fundamental advantage of single-server topologies over autoscaling. While Kubernetes is busy scheduling pods, our job is already done.

Phase 3: OS / Shell (~1s)

Think of it this way: Shell executor is a barefoot sprinter, Docker is a sprinter with fancy shoes. Both are fast, but one has less to put on.

For Shell executor (fastest, ~1s):

No Docker image needed
No container startup
Direct shell execution
Native OS environment

For Docker executor (fast with isolation ~1-4s):

Pre-pull images to server
Use lightweight base images (alpine)
Layer caching on local Docker
Keep containers warm when possible

Phase 4: Git Clone/Fetch (~1s)

This is where the magic happens. Or rather, where the magic doesn't happen — because the best git operation is the one you barely do.

Strategy and depth

Obviously, to achieve 3s on a large codebase, a close version of the code must already be present locally. The default algorithm is a fetch (which is perfect), meaning that the runner will just reconstruct a few commits in a build directory that already has been used by a previous job, preferably with nearly same code but different commits.

The default is 20 commits reconstructed. In fact, you only need one most of the time (when not doing git stuff).

variables:
  GIT_DEPTH: 1 # default to 20

Fetch flags

The default fetch flags are --force --prune --tags. force and prune are very useful to handle git fetch problems and keep local repo size reasonable. You can experiment with these on short-lived runners, at your own risks. We tried this but had to step back for consistency, even on our daily runners.

But fetching tags is almost never a good idea, at least by default. Even though we heavily rely on tag pipelines, we still don't need to fetch any tag. Except for a job that uses older tags to extract versions; for it we manually fetch tags.

Optionally, we can define the refmap to fetch, avoiding unnecessary fetching.

Here are our flags for merge requests :

  GIT_FETCH_EXTRA_FLAGS: --force --prune --no-tags
    --refmap "+refs/merge-requests/${CI_MERGE_REQUEST_IID}/head:refs/remotes/origin/merge-requests/${CI_MERGE_REQUEST_IID}/head"

Note: add --verbose to show what's happening, and be sure to know what takes time and what not. We still have this flag because there only a few lines added when fetch is optimised, and it does not slow down the job.

Situational: tailored clone path

When long-lived branches have a fair amount of difference (the longer the branch and/or the higher the developer count, the more differences), fetch takes time. On our projects we have thousands commits of differences between long-lived branches at worse time of our cycle!

To keep a sub-second fetch time, we configure clone in paths depending on the target branch. And yes, MR pipelines secret mode is required for this.

So our clone path depend on the pipeline type :

  # for MRs. Concurrent ID is after the branch name, to ease old branches cleaning
  GIT_CLONE_PATH: $CI_BUILDS_DIR/$CI_PROJECT_NAME/$CI_MERGE_REQUEST_TARGET_BRANCH_NAME/$CI_CONCURRENT_ID
  # for long-lived branches (same path, different variable)
  GIT_CLONE_PATH: $CI_BUILDS_DIR/$CI_PROJECT_NAME/$CI_COMMIT_REF_NAME/$CI_CONCURRENT_ID
  # for tags (rare, no need for distinction)
  GIT_CLONE_PATH: $CI_BUILDS_DIR/$CI_PROJECT_NAME/tags/$CI_CONCURRENT_ID

This can only be used when custom_build_dir is enabled in the runner’s configuration.

Phase 5: Cache (~0s)

Downloading and extracting remote cache takes time. Even with local GitLab cache, you still have to zip/unzip the folder elsewhere. That's like packing and unpacking your suitcase every time you go to the kitchen.

Shared package managers folders

Shared package managers folders on the server will reuse downloaded assets without overhead. When local cache is warm, there are no download/unzip and no zip/upload.

variables:
  NPM_CONFIG_CACHE: /cache/gitlab-runner/.npm
  NUGET_PACKAGES: $CI_BUILDS_DIR/NuGetPackages

Note: For safety, it is still better to have different folders for dev/staging/prod.

Optional : do not clean unpacked assets

Before executing the scripts, GitLab deletes any past produced files. We can save precious seconds, even minutes, by not deleting them, and, most importantly, reuse them. This is especially useful for node_modules :

variables:
  GIT_CLEAN_FLAGS: "-ffdx --exclude=**/node_modules/"

This works best for custom GIT_CLONE_PATH discussed earlier. This could lead to strange behavior in theory : we took the risk for feature branches, and never encountered problems for our 15 developers mono-repo.

Note: For safety, it is still better to clean for staging/prod environment.

Phase 6: Artifacts (none)

The fastest jobs in pipelines do not handle artifacts. We consider none here. Zero. Nada. The fastest artifact is the one that doesn't exist.

Phase 7: Script (~1s)

For the 3-second target, we're specifically measuring fast jobs like linting, formatting checks. Any standard job will naturally take longer. But hey, even our "slow" jobs appreciate having 2 seconds less overhead — that's 2 seconds of their life they'll never get back.

Phase 8: Termination (~0s)

At the end, the job handles artifacts and the cache. The fastest jobs produce no artifact and use local custom cache or none. The job exits so fast, it doesn't even say goodbye.

Final timing breakdown

Here's what the fully optimized timeline looks like:

⚡ Total: ~3 seconds per job (with sub-1s script)

The key insight: We've reduced overhead to ~2s, leaving all remaining time for actual work. Your CI is now faster than your npm start.

Real project results

Again, these aren't theoretical numbers: we experience this extreme speed on a daily basis.

Shell runner - 3 seconds

Docker runner - 13 seconds

Takeaway: If you need isolation, Docker is still very fast with these optimizations. But Shell executor is unbeatable for raw speed.

Example development pipeline - 1min45 for 20 jobs

Most jobs run in parallel on each stage. The pipeline spends minimal time on overhead and maximum time on actual work.

Wrapping up

Achieving 3-second jobs on a multi-million line codebase is possible with the right combination.

These techniques show that single-server Shell/Docker runners, when properly optimized, vastly outperform autoscaling solutions for typical development workflows. The local filesystem advantages are impossible to beat.

Not every job can be 3 seconds—builds and full test suites will always take longer. But for fast-feedback jobs, sub-10-second execution is absolutely achievable and dramatically improves developer experience. Your developers will wonder if the pipeline is broken... because it's too fast.

Illustrations generated locally by Draw Things using Flux.1 [Schnell] model

All Articles by Theme

Benoit COUETIL 💫 — Thu, 19 Feb 2026 19:04:39 +0000

Here are my articles, grouped by theme and sorted by publication date (newest first), with view counts and highlights for the most read ones ⭐

🦊 GitLab / 🔀 Git

☸️ Kubernetes

📝 Miscellaneous

Generated by a fork of devto-cli on 2026-03-20

🦊 GitLab Runners: Which Topology for Fastest Job Execution?

Benoit COUETIL 💫 — Fri, 30 Jan 2026 17:28:43 +0000

Initial thoughts
Understanding job execution phases
Comparing runner types
- GitLab Shared Runners (SaaS)
- Kubernetes Executor (Fixed Cluster)
- Kubernetes Executor (Autoscaling Cluster)
- Docker Executor (Single Server)
- Shell Executor (Single Server)
- Docker Autoscaler (Fleeting)
Job speed comparison summary
Recommendation: Shell or Docker executors for fastest jobs
- Why single-server executors win on speed
- Shell vs Docker trade-offs
- The performance/price/maintenance sweet spot
Wrapping up
Further reading

With multiple executor types available—from shared SaaS runners to Kubernetes clusters to single-server setups—understanding how each impacts job speed helps you make informed architectural decisions. Spoiler: the simplest topologies often deliver the fastest jobs.

Initial thoughts

Choosing the right GitLab Runner topology is crucial for fast job execution. With multiple executor types available—from shared SaaS runners to self-hosted solutions—understanding how each impacts job speed helps you make informed architectural decisions.

This article focuses on a single question: which runner topology executes jobs the fastest? We analyze different topologies through the lens of job execution phases, comparing the time overhead each infrastructure type adds before, during, and after script execution. Whether you're starting fresh or optimizing an existing setup, understanding these trade-offs is essential to minimize job duration.

📖 Note: This article focuses on intrinsic speed characteristics. Some disadvantages mentioned here can be mitigated through configuration tricks (idle pools, spot instances, over-provisioning, etc.). For detailed pros/cons and mitigation strategies, see GitLab Runners Topologies: Pros and Cons.

Understanding job execution phases

From runner selection to final cleanup, every job goes through multiple technical phases. Understanding these phases is crucial to optimizing pipeline performance.

Here's a detailed breakdown of all the technical steps involved in running a single GitLab CI job:

Each phase adds latency to job execution. Your script might take 10 seconds, but the overhead can easily triple that — death by a thousand papercuts. Different runner topologies handle these phases very differently.

Comparing runner types

Let's analyze the performance characteristics of each GitLab Runner infrastructure type, examining how they handle the execution phases differently.

📊 Reading the charts: Throughout this article, the durations shown in Gantt diagrams are relative and illustrative—actual times vary based on project size, network conditions, and infrastructure specs. What matters is the color coding:

🟢 Green: Faster than average across runner types

⚪ Grey: Average performance

🔴 Red: Slower than average across runner types

GitLab Shared Runners (SaaS)

GitLab.com provides shared runners available to all users without any setup. While convenient, these runners compete for resources with thousands of other projects, making them the slowest option for job execution.

Topology: Shared SaaS infrastructure - all users share the same pool of runners and cache storage.

Performance characteristics:

Speed advantages:

✅ No wait for infrastructure provisioning

Speed disadvantages:

❌ Slowest overall performance (all resources shared)
❌ Every job pulls images from scratch
❌ Shared network bandwidth slows git operations
❌ Unpredictable performance due to multi-tenancy
❌ Slow artifact uploads/downloads

Best for: When speed is not a priority and zero maintenance is essential.

Kubernetes Executor (Fixed Cluster)

A fixed-size Kubernetes cluster provides consistent resources for CI jobs. While more complex to set up than shared runners, it offers better performance through image caching and dedicated resources—though still limited by network-based cache and pod startup overhead.

Topology: Fixed-size cluster with shared remote cache (S3/MinIO).

Performance characteristics:

Speed advantages:

✅ Fast when warm (images cached on nodes)
✅ Pod creation relatively quick on existing nodes
✅ Consistent performance with dedicated resources

Speed disadvantages:

❌ Jobs queue when capacity is reached
❌ Remote cache adds network latency
❌ Pod startup overhead (even on warm nodes)

Best for: Teams with existing Kubernetes infrastructure where moderate speed is acceptable.

Kubernetes Executor (Autoscaling Cluster)

Autoscaling Kubernetes clusters dynamically add nodes when demand increases. This eliminates queuing issues but introduces significant cold-start delays—new nodes take time to provision, and each job starts with a fresh environment requiring full image pulls and git clones.

Topology: Cluster with autoscaler that adds/removes nodes based on load, pods distributed dynamically.

Performance characteristics:

Speed advantages:

✅ No queue time when scaling up
✅ Dedicated resources per job once running

Speed disadvantages:

❌ Very slow cold starts (node provisioning 30s-2min)
❌ Full image pulls on new nodes
❌ Complete git clones on ephemeral pods
❌ Remote cache network latency

Best for: Variable workloads where cold-start delays are acceptable trade-offs for unlimited capacity.

Docker Executor (Single Server)

A single server running Docker provides the sweet spot between isolation and performance. Containers start quickly when images are cached, git repositories are reused locally, and cache access is lightning-fast through the local filesystem—all while maintaining job isolation.

Topology: Single server with Docker Engine - container isolation but shared local cache via volumes.

Performance characteristics:

Speed advantages:

✅ Very fast local cache access (filesystem)
✅ Image layers cached locally
✅ Quick container startup when warm
✅ Local git repository reuse
✅ No network latency for cache

Speed disadvantages:

❌ Jobs queue when server capacity is reached
❌ Container overhead (minimal but present)

Best for: Teams needing container isolation with near-optimal speed.

Shell Executor (Single Server)

The shell executor is the fastest possible configuration—no containers, no image pulls, no pod scheduling. Everything runs directly on the host system with instant access to local git repos and filesystem cache. The trade-off? No job isolation, requiring trust in your codebase.

Topology: Everything is local on a single server - runner, shell execution, and cache share the same filesystem.

Performance characteristics:

Speed advantages:

✅ Absolute fastest (zero container overhead)
✅ Instant cache access (direct filesystem)
✅ Fastest git operations (local clones reused)
✅ No image pulls ever
✅ Immediate job start (no container creation)
✅ Best optimization potential

Speed disadvantages:

❌ Jobs queue when server capacity is reached

Best for: Maximum speed when job isolation is not required.

Docker Autoscaler (Fleeting)

Docker Autoscaler uses the Fleeting plugin system to spawn VMs on cloud providers. It dynamically provisions VMs when demand increases and terminates them when idle, providing unlimited capacity at the cost of cold-start delays.

Topology: Autoscaling VMs - plugin-based architecture supporting multiple cloud providers.

Performance characteristics:

Speed advantages:

✅ No queue time (infinite capacity)
✅ Dedicated resources once VM is ready

Speed disadvantages:

❌ Slow cold starts (15+ seconds VM provisioning)
❌ Full git clone on each VM
❌ Full image pull on each VM
❌ Cloud cache network latency

Best for: Variable workloads requiring autoscaling where cold-start delays are acceptable.

Job speed comparison summary

Here's a comprehensive comparison of job execution speed across all execution phases:

Legend: 🟢 Fast · ⚪ Medium · 🔴 Slow

Runner Type	Wait	VM/Pod	OS	Git	Script	Cache	Artifacts
GitLab Shared	🟢	🟢	🔴	🔴	🔴	🔴	🔴
K8S Fixed	⚪	🟢	⚪	⚪	⚪	⚪	⚪
K8S Autoscaling	🟢	⚪	🔴	🔴	🟢	⚪	⚪
Docker	🔴	🟢	⚪	🟢	🔴	🟢	⚪
Shell	🔴	🟢	🟢	🟢	🔴	🟢	⚪
Docker Autoscaler	🟢	🔴	🔴	🔴	🟢	⚪	⚪

Key insights:

GitLab Shared: Great for zero maintenance but slowest overall
K8S Fixed: Balanced approach with good warm performance
K8S Autoscaling: Best for variable loads but cold starts are slow
Docker Single Server: Fast local operations with isolation
Shell Single Server: Fastest local operations, best optimization potential
Docker Autoscaler: Maximum scalability but slowest cold starts

Recommendation: Shell or Docker executors for fastest jobs

After analyzing all runner types, Shell and Docker executors on single servers offer the fastest job execution for most teams.

Why single-server executors win on speed

1. Local everything = minimal latency

Git repositories cached locally
Dependencies and cache on local filesystem
No network round-trips for cache operations
Instant resource availability (no VM/pod provisioning)

2. Vertical scaling is underrated

Modern servers can handle dozens of concurrent jobs
SSD storage makes local caching extremely fast
RAM caching for frequently accessed data
CPU cores scale linearly for parallel jobs

3. Warm infrastructure advantage

Docker images already pulled and cached
Git repos incrementally fetched (not full clones)
Dependencies preserved between jobs
No cold-start penalties

Shell vs Docker trade-offs

Choose Shell when:

You need maximum speed
Your codebase is trusted
You can maintain consistent tooling
Security isolation is less critical
You want to push performance limits

Choose Docker when:

You need job isolation
Multiple projects with different dependencies
Security/multi-tenancy matters
Slightly slower speed is acceptable trade-off

The performance/price/maintenance sweet spot

📈 Vertical scalability compensates for the lack of horizontal scalability

💪 A properly sized server with local SSD can handle dozens of simultaneous jobs

💰 Predictable costs: No per-job pricing, one server cost

🔧 Low maintenance: Simple architecture, fewer moving parts

📖 Reference: GitLab CI: The Majestic Single Server Runner

Wrapping up

Choosing the right GitLab Runner topology depends on your specific needs, but if minimizing job execution time is your primary concern, Shell or Docker executors on well-provisioned single servers consistently deliver the fastest jobs.

Key takeaways:

Shared runners are great for getting started but have performance limitations due to multi-tenancy
Kubernetes solutions offer good isolation and scalability but add network latency
Single-server executors (Shell/Docker) provide the fastest local operations and best optimization potential
Autoscaling solutions handle variable loads well but suffer from cold-start penalties
Vertical scaling of single servers is often more effective than complex horizontal scaling

The comparison shows that while autoscaling solutions offer flexibility, a properly configured single-server runner often provides the best performance for teams with predictable workloads or those willing to size for peak capacity.

In our follow-up article GitLab CI: Achieving 3-Second Jobs on Million-Line Codebases, we dive deep into extreme optimizations you can apply to Shell and Docker runners to push performance even further—achieving job times as low as 3 seconds on multi-million line codebases!

Illustrations generated locally by Draw Things using Flux.1 [Schnell] model

🔍 Every Developer Should Review Code — Not Just Seniors

Benoit COUETIL 💫 — Sat, 03 Jan 2026 15:37:07 +0000

Initial thoughts
1. Reviewing accelerates learning and skill development
2. Junior developers provide unique value and grow fastest
3. Reviewing enhances your own code quality and habits
4. It fosters system-wide understanding and reduces silos
5. Universal review enforces standards, consistency, and security
6. More reviewers distribute workload, reducing bottlenecks and accelerating delivery
7. It builds psychological safety, team cohesion, and positive culture
8. Code reviews make for better estimates
9. Google's Review Culture: A Proven Model for Excellence
10. "But we don't have time for everyone to review" — Common objections answered
How to Review Effectively at Any Level
Making Universal Review Work: Practical Implementation
Wrapping up
Further reading

Code review shouldn't be reserved for a handful of senior developers. Every team member — regardless of experience level — must participate in reviewing code. Junior developers bring fresh perspectives that catch issues seniors miss, while simultaneously accelerating their own learning. Universal review distributes knowledge, prevents bottlenecks, and transforms code quality from a gate into a shared team responsibility.

Initial thoughts

Picture this: A team of 10 developers where only 2–3 senior engineers handle all code reviews. Those seniors become overwhelmed gatekeepers, review queues grow longer, and junior developers submit code without ever learning to critically evaluate others' work. Sound familiar? Welcome to the review bottleneck club.

Mandatory code review consistently separates high-performing engineering teams from the rest. This approach mirrors the rigorous review culture that propelled Google to become one of the world's leading software providers in its early days.

Codacy.com 2024 article Code review process: how to improve developer productivity shows dominance of code review on code quality:

But code review is too often perceived as a quality gate run by seniors to catch mistakes. This is fundamentally wrong. In reality, the primary beneficiary of reviewing someone else's code is the reviewer themselves — and the entire team benefits from the shared knowledge and improved practices.

The core message: Every developer should review code. Not just approve it with a quick scan, but genuinely engage with it, ask questions, and provide feedback. Here's why.

1. Reviewing accelerates learning and skill development

Reading production code written by others exposes patterns, architectural decisions, edge-case handling, and new techniques at a rate no amount of Stack Overflow browsing can match.

A developer who only writes code sees one mental model. A developer who reviews 5–10 merge requests per week sees dozens of approaches, trade-offs, and real-world solutions every sprint. This bidirectional mentoring — juniors learning from seniors, and vice versa — grows skills across the team.

2. Junior developers provide unique value and grow fastest

The objection "I'm too junior to review" is consistently proven wrong in practice.

Engineers new to a codebase are uniquely positioned to detect:

Ambiguous naming (they still have to think about what something means)
Missing documentation or comments (context that seniors assume is obvious)
Overly clever constructs that violate the principle of least astonishment
Inconsistent style that seniors have become blind to

Fresh eyes often catch security vulnerabilities, performance issues, or logic errors overlooked due to familiarity. Meanwhile, reviewing senior code is one of the most powerful learning tools for juniors — they see production-quality patterns in context, not just in documentation.

3. Reviewing enhances your own code quality and habits

Developers who regularly review begin anticipating reviewer questions before submitting changes:

Commit messages become descriptive
Changes are split into logical, reviewable chunks
Tests are written upfront
Public APIs receive extra care

The "Hawthorne effect" kicks in: knowing your code will be reviewed magically makes you write it better the first time. Over time, this feedback loop is one of the most reliable skill accelerators observed.

4. It fosters system-wide understanding and reduces silos

When every engineer reviews code across the codebase, knowledge silos disappear.

Team members gain:

Insight into why certain retry policies or architectural choices exist
Understanding of how authentication, authorization, and critical paths actually work
Familiarity with the full system, reducing the "bus factor"

This shared context is essential for resilience, refactoring, fast incident response, better estimates, and true team flexibility. When knowledge is distributed, team members can take time off without the project grinding to a halt — no single developer holds the codebase hostage.

5. Universal review enforces standards, consistency, and security

Reviews ensure adherence to coding standards, patterns, and best practices, unifying the codebase and preventing style drift.

They catch:

Security vulnerabilities (e.g., input validation failures or duplicated code risks)
Readability and maintainability issues
Opportunities for design improvements

This early-cycle quality control promotes secure development and a maintainable codebase.

6. More reviewers distribute workload, reducing bottlenecks and accelerating delivery

The higher the number of reviewers involved, the faster code reviews tend to complete. Distributing review requests across the team minimizes bottlenecks — someone available can pick up the review promptly.

This leads to quicker merges, faster time to market, and improved overall team velocity — no more waiting three days for a single thumbs-up.

7. It builds psychological safety, team cohesion, and positive culture

When everyone both gives and receives feedback regularly, code review stops feeling personal.

Teams that practice universal review report:

Fewer surprises in production
Faster decision making
Higher engineering satisfaction and motivation
Stronger bonds through constructive, positive feedback

8. Code reviews make for better estimates

For teams performing estimation, this is a team exercise, and the team makes better estimates as product knowledge is spread across the team.

As new features are added to the existing code, the original developer can provide good feedback and estimation. In addition, any code reviewer is also exposed to the complexity, known issues, and concerns of that area of the code base.

The code reviewer, then, shares in the knowledge of the original developer of that part of the code base. This practice creates multiple, informed inputs which, when used for a final estimate, always make that estimate stronger and more reliable.

9. Google's Review Culture: A Proven Model for Excellence

Google's early dominance as a software provider was deeply rooted in its mandatory pre-commit code review process. Every change required a "Looks Good To Me" (LGTM) approval from a qualified engineer before landing.

This culture:

Caught bugs early, preventing production incidents
Maintained exceptionally high code quality standards
Avoided the "broken windows" effect
Promoted openness, teamwork, and security awareness
Enabled organic knowledge transfer and rapid onboarding
Ensured consistency across a rapidly growing codebase

These factors produced scalable, reliable products (Search, Maps, Gmail) and laid the foundation for Google's engineering reputation that still endures today.

10. "But we don't have time for everyone to review" — Common objections answered

Let's address the elephant in the room — the one wearing a "too busy" t-shirt:

Objection: "Junior reviews slow us down"

Reality: A 10–15 minute review prevents hours or days of incident response, debugging, and rework. When only 2–3 seniors on a 10-person team handle all reviews, those seniors experience overload and burnout, review queues grow, and defect leakage increases due to rushed or fatigued reviews.

Objection: "Juniors don't catch the important stuff"

Reality: Juniors catch different things — ambiguous naming, missing docs, inconsistent patterns. These "surface" issues are exactly what cause maintenance headaches down the road. Plus, they're learning while reviewing, making them better reviewers over time.

Objection: "We can't afford the time investment"

Reality: Investing just 5% of development time in review typically reduces bug-related work by 30–70% (Google, Microsoft, Atlassian benchmarks). Distributing the load prevents bottlenecks and saves far more time overall.

According to Codacy's 2024 study on code review productivity, teams practicing code review see:

10% reduction in time fixing bugs (from 35% to 25% of dev time)
10% improvement in time building new features (from 45% to 55%)
17% reduction in perceived lack of maintenance time (from 76% to 59%)

The math is simple: you can't afford NOT to have everyone reviewing.

How to Review Effectively at Any Level

For everyone:

Start with what you understand confidently — naming, tests, documentation, security basics
Ask questions rather than dictate solutions — "Why did you choose X?" beats "Change this to Y"
Keep comments focused — one clear point beats a wall of text
Target high-impact areas — correctness, design, complexity, standards
Remember the goal — improve the codebase, help the author, and learn

Use Conventional Comments for clarity:

Conventional Comments prefix feedback with labels that clarify intent, making reviews more constructive and less ambiguous:

praise: Highlight excellent work ("praise: Great error handling here!")
nitpick: Minor style suggestions ("nitpick: Consider renaming tmp to tempResult")
suggestion: Propose improvements ("suggestion: We could extract this into a helper function")
issue: Flag problems that must be addressed ("issue: This will fail when input is null")
question: Ask for clarification ("question: Why are we using a timeout here?")
thought: Share considerations without requiring action ("thought: This might impact performance at scale")

This approach helps juniors understand which comments require action vs. are optional, reduces defensive reactions, and creates consistent review language across the team.

For junior reviewers specifically:

Focus on readability: "I don't understand what this function does"
Check test coverage: "Is there a test for the error case?"
Verify documentation: "Could you add a comment explaining why we need this?"
Question assumptions: "What happens if this API call fails?"
Don't apologize for asking questions — they're valuable contributions

For seniors reviewing juniors' reviews:

Praise good catches publicly
Provide context when juniors miss architectural issues
Model constructive feedback tone
Never dismiss or override junior comments without explanation

Making Universal Review Work: Practical Implementation

1. Make it official policy

Explicitly state that everyone reviews code. Add it to onboarding documentation and team agreements.

2. Rotate reviewers

Avoid always pairing the same people.

3. Set clear expectations

Reviews should happen within 4–24 hours (depending on your workflow)
At least one approval required to merge, ideally from someone who didn't write the code
Block merging without review (enforce via branch protection rules)

4. Create review rituals

Daily "review hour" where the team focuses on pending reviews
Celebrate good reviews in standups or retrospectives
Track and visualize review participation

5. Provide training

Run workshops on effective code review
Share examples of good review comments
Create a review guidelines document specific to your codebase

6. Lead by example

Seniors must model the behavior: review promptly, give constructive feedback, accept feedback graciously, and encourage junior participation.

Wrapping up

Universal code review is not a gate. It is one of the highest-ROI engineering practices available.

Every developer — regardless of tenure — must participate. The team's velocity, quality, security, and individual growth depend on it.

The question isn't "Can we afford to have everyone review?" It's "Can we afford not to?"

Illustrations generated locally by Draw Things using Flux.1 [Schnell] model

🔀 Efficient Git Workflow for Web Apps: Advancing Progressively from Scratch to Thriving

Benoit COUETIL 💫 — Thu, 10 Oct 2024 09:29:24 +0000

Initial thoughts
1. First lines of code
2. First development environments
3. First stable environment
4. First production environment
5. First customers
6. Successful application
Wrapping up
Further reading

Initial thoughts

In this article, we will enhance our Git Workflow by gradually adding functionalities for starting web applications. Our main goal is to improve efficiency.

Developers, even experienced ones, have often struggled with the famous Gitflow :

However, the truth is that we don't need this complexity from day one. Many developers even argue that Gitflow is never necessary — it's a symptom of an organizational problem, not a solution.

Let's approach it differently: from team needs to workflow.

1. First lines of code

As we gather the team and prepare to write the first lines of code, the development team members have a few requirements:

Share a common code base
Collaborate on work in progress
Conduct testing

Most web applications consist of a frontend module and a backend module. The question of using a multi-repo or mono-repo arises.

This article won't elaborate on the controversial topic, but for a starting web application project, we choose a mono-repo for the following reasons:

No internal versioning required
Local common module dependencies
One feature, one feature branch
Easy continuous integration

At this phase of the project, we don't need more than the following branches:

Short-lived feature branches that are merged into main when ready, as frequently as possible
A main branch for merged code

During the initial weeks of testing and demonstrations, local computers are sufficient. This allows time for the Platform Engineer to prepare real environments.

For code reviews, any Git platform, such as GitLab or GitHub, will suffice.

2. First development environments

Workflow

After a few weeks, code and features start to accumulate. Tracking bugs and conducting proper testing becomes essential. If the tools allow, these tests should occur before merging. Demonstrations should take place on a shared environment rather than on local computers.

To achieve this, we establish an integration environment that reflects the main branch.

If we have the right CI/CD technologies, we can also create an environment per branch, which greatly enhances team efficiency. Kubernetes and GitLab can help us achieve this goal with minimal effort.

Our Git workflow at this stage looks like this:

Short-lived feature branches and their corresponding environments are merged into main
main branch and environment are used for development team collaboration and client demonstrations

Separate Infrastructure Repository ?

This is controversial and a bit off-topic, but this still a linked matter with efficiency choice at stake.

Many teams choose to separate application and infrastructure repositories. However, I believe that the main reason for this separation is governance and politics, not efficiency.

We choose to have infrastructure scripts in the same repository as the code. Although the lifecycles are different, separating them can lead to complex pipelines if everything is automated, right?

But do we really need full automation for infrastructure? While infrastructure-as-code is desirable, deployment automation may not be necessary. We like to follow the rule: "do not automate before it is boring." Since our product is just starting, nothing is boring regarding infrastructure. We don't need to reinstantiate our Kubernetes cluster every week.

For these reasons, we code and version the infrastructure but do not automate its deployment.

For CI/CD development specifically, tools like gitlab-ci-local can help iterate faster on pipeline configurations without waiting for remote runners—see GitLab CI: YAML Modifications: Tackling the Feedback Loop Problem for a deep dive.

3. First stable environment

Sooner or later, stakeholders will test the application. Frequent redeployment of the main environment leads to downtimes and occasional bugs, which spoil the experience.

It's time to introduce the demo environment. This environment reflects the content of the main environment and is updated before each scheduled demonstration. It remains stable in the interim.

To adapt the flow, the update can be triggered by a Git tag or a Git branch. Here, we choose to create a new demo Git branch.

Our Git workflow now looks like this:

Short-lived feature branches and their corresponding environments are merged into main and then deleted
main branch and environment are merged into demo before demonstrations
demo branch and environment are used for demonstrations

4. First production environment

With the MVP (Minimum Viable Product) just around the corner, a production environment has been built from the ground up and will soon be made public. It's time to deploy our application to the target production infrastructure.

To adapt the flow, the production deployment can be triggered by a Git tag or a Git branch. We choose to use tags since we don't need to handle hot fixes at the moment, and triggering deployment jobs can be done manually.

We don't cascade merges from main to demo to production. The demo and production environments have decoupled lifecycles.

Our Git workflow now looks like this:

Short-lived feature branches and their corresponding environments are merged into main and then deleted
main branch and environment are merged into demo before demonstrations
demo branch and environment are used for demonstrations
Manual production deployment jobs are available on tags (created on main or demo, depending on the team preference)

5. First customers

As we had hoped from the beginning, we now have happy customers.

We want to ensure that we don't spoil their experience. When problems occur in their workflow, we want to be able to reproduce them in an iso-production environment, fix them, and test them in a pre-production environment as well. We will call this environment staging.

Since staging doesn't have real customers, we can deploy automatically using the same tags that trigger manual production deployment.

Our Git workflow now looks like this:

Short-lived feature branches and their corresponding environments are merged into main and then deleted
main branch and environment are merged into demo before demonstrations
demo branch and environment are used for demonstrations
Automatic staging deployment and manual production deployment are triggered by tags

We still don't handle production hot fixes separately. The entire content of main is deployed to production on tags. Unfinished content is not merged into main or deactivated by a commit.

6. Successful application

Now our application is somewhat successful, with many customers using it daily.

Cash is flowing, and the development team is growing. It becomes increasingly difficult to find a satisfying main or demo state to be deployed to production.

Now is the time to add feature toggles (also known as feature flags or feature flipping). Anything can be deployed to production as long as it is protected by a switch to activate it or not.

Our Git workflow now looks like this:

Short-lived feature branches and their corresponding environments are merged into main and then deleted
main branch and environment are merged into demo before demonstrations
demo branch and environment are used for demonstrations
Automatic staging and production deployment are triggered by tags created on main, and feature toggles are activated based on confidence

Wrapping up

We followed the path of a starting project that progressively added only necessary complexity to the git flow.

By following this Git workflow, development teams can simplify their processes, improve collaboration, and deliver high-quality web applications more efficiently.

What is your own best experience of Git workflow ? Please share in the comments below 🤓

Illustrations generated locally by Pinokio using Stable Cascade plugin

🔀🦊 GitLab: Forget GitKraken, Here are the Only Git Commands You Need

Benoit COUETIL 💫 — Sat, 31 Aug 2024 08:11:31 +0000

Initial thoughts
1. Drawbacks of git GUI clients
- Price
- Magic
- Context-specific
- Hard to communicate
2. The GitLab flow
3. Everyday commands when using GitLab flow
- Prepare to make changes to the feature branch
- Commit your work
- Push and merge your work
- Inspect past commits
3. Common situations that you need to be able to manage
- The target branch has been updated
- You need to work on another branch
4. Going further with the command line
- Someone has updated your (remote) feature branch
- Rewrite the history
- Remove old local branches
- Change the user name/email in commits
- Delete all existing files defined in .gitignore
- Fetch changes to a local branch without switching to it
- Remove changes on files against target branche
- Rebase only a specific commit range
5. Git common configuration options and aliases
Wrapping up
Further reading

Initial thoughts

The 2022 stackoverflow developer survey found that more than 90% of developers use git for version control. Despite that, we feel that many developers are intimidated by the git CLI and use it only when their favorite version control GUI is missing a particular feature.

I, too, relied solely on GitKraken after 10+ years as a developer. Fortunately, Benoit Averty taught me everything I needed to know, through a Zenika Git training session. I’m now a happy Git CLI user across various GitLab projects 🤓.

Fast forward 4 years, and together with Benoit, we present this (self sufficient) article for those as intimidated as my past self 🤗.

Yes, the git CLI has many commands, and it can be daunting; but for everyday use you only need to know a few commands, and the git CLI is particularly good at providing guidance on what you may want to do next. In this article, we show that the CLI is not very hard to use, especially when using GitLab to host the git repository and embracing the GitLab flow that is used by many teams using GitLab.

Most of what we will cover in this article should be true if you’re using GitHub and the GitHub flow instead; but small adjustments may be necessary.

1. Drawbacks of git GUI clients

Git GUI clients are often seen as an easier way to use git. However, it is not always the case, and even when it is, it comes with several drawbacks that you need to be aware of.

Price

Some of the best git GUIs out there are paid software. If you’re paying for them, it's probably because you think they’re worth it. Nevertheless, paid software can quickly add up to a hefty price, and we all prefer a free option.

Magic

Using a git GUI is the best way to use git without understanding git. Part of what makes git appear as a hard tool to master is the complexity of its internal structure and the concepts git introduces to manage your source code. GUIs seem easier to use because they abstract away these concepts, but the drawback is that you can use git every day without understanding it. This makes it almost impossible to handle a special case when it arises or to do something that your preferred GUI can't do.

The CLI, on the other hand, stays close to the git underlying concepts and teaches them to you almost without you noticing.

Context-specific

A notable drawback of git graphical interfaces is their context-specific nature. Each GUI represents git features in its unique fashion. The skills you gain from one client may not be directly transferable to another client. This is particularly true if your role involves helping teammates with their git usage or if you aspire to be in such a position.

On the other hand, the commands of the git CLI remain standard, regardless of the specific environment in which you’re operating.

Hard to communicate

It is intrinsically challenging to communicate about graphical interfaces when they allow complex actions (and it is often the case with git graphical interfaces). Helping a teammate perform a git action may involve taking a screenshot or describing buttons and menus, whereas command lines can be copy-pasted without effort, including the result of specific commands that can be used to diagnose the problem.

A corollary of this is that it’s easier to write and maintain documentation or best practices on the specific usage of git in your team if you’re using CLI instead of a graphical interface.

2. The GitLab flow

Many development teams today are using GitLab to collaborate on software projects. The GitLab flow is a branching workflow that takes advantage of the features offered by GitLab’s web interface. It is also a simpler alternative to the famous gitflow, introduced almost 15 years ago.

The GitLab flow can be broken down to this:

In the GitLab web UI
- Create an issue
- Create a MR from the issue, creating a branch
- (Now everyone knows on what you will work)
Locally
- Synchronize code and switch to the new branch
- Do your magic
- commit and push
In the GitLab web UI
- Review with your teammates
- When satisfied, merge the code (automatically deleting the branch)

In this workflow, branches aren’t created locally, and they aren’t even merged locally.

In addition to that, many tasks related to git but not directly related to local code can be performed in the web UI:

List/create/delete branches;
List/create/delete tags;
Show the repository graph.

For the tasks that still need to be done locally, let’s dive into the few commands that you need to know for everyday development.

3. Everyday commands when using GitLab flow

Prepare to make changes to the feature branch

If this is the first time on the project, get the repo locally.

In any case, the first thing that you’ll probably want to do after creating your merge request and your branch is to update your local repository and switch to this new branch.

# Switch to develop/main/master and update your local repository
$ git switch develop
$ git pull

# Switch to the new branch that has been created when you opened your MR
$ git switch your-new-feature-branch

The first two commands are used to synchronize the main branch of the project locally. This is always a good idea because it allows you to check out the state of the main branch locally. But strictly speaking, this is not mandatory because you’re going to work on your feature branch anyway. You could replace both of them with a single git fetch.

The third command is the most interesting. Normally, git switch in this form is used to switch to a branch that exists locally. But git is smart, and if the branch exists on the remote repository, git will create a local branch that tracks the remote branch that you’ve created previously and switch to that newly created branch. All of this is explained in the command output:

branch 'your-new-feature-branch' set up to track 'origin/your-new-feature-branch'.
Switched to a new branch 'your-new-feature-branch'

Commit your work

Once you have something that is worth sharing with your teammates, or if your code is in a state that is worth saving, it is time to commit.

The command that you’ll want to use next is the most important command when using the git CLI:

$ git status

On branch your-new-feature-branch
Your branch is up to date with 'origin/your-new-feature-branch'.

Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
  (use "git restore <file>..." to discard changes in working directory)
    modified:   package.json

no changes added to commit (use "git add" and/or "git commit -a")

Why is it the most important command? Because it tells you everything there is to know before commiting, pushing, or doing anything else really. But also, it tells you what the next command will probably be. Who said command line interfaces were hard to use?

In the example above:

You can see the list of changed files, and know immediately if there is something you don’t want to commit
You have the command that you need to use to add files to the future commit
You have the command that you need to use to discard changes that you’ve made.

Let’s add changes to the future commit.

$ git add package.json
$ git status

On branch your-new-feature-branch
Your branch is up to date with 'origin/your-new-feature-branch'.

Changes to be committed:
  (use "git restore --staged <file>..." to unstage)
    modified:   package.json

Again, git status tells you what you need to do if you want to cancel adding a file. No need to remember many commands.

When you’re ready, commit your work to store it safely inside your local repository.

$ git commit
[your-new-feature-branch 331ef24] Awesome commit message
 1 file changed, 1 insertion(+), 1 deletion(-)

Push and merge your work

After each commit or after finishing your development task, it’s time to push your work to the GitLab server to begin gathering feedback from your teammates.

$ git push

Pushing is not hard when you’re using GitLab flow because there are almost never two developers working on the same branch. You will almost never need to resolve conflicts at this point or have your push rejected because the remote branch moved.

Merging is also as easy as the push of a button in the GitLab webapp.

You probably want to delete your branch once you’ve finished working on it: check the corresponding box. If you’ve made commits that you don’t want to keep while working on your feature, check the "Squash commits" box.

Note that this screenshot is taken on a project that’s configured with the "Merge commit" merge method. If you’re on a project that uses semi-linear history or fast-forward merges only, you’ll be given the option to rebase, also with the click of a button. See GitLab’s documentation about merge methods to learn more.

That’s all there is to know for everyday use when using GitLab flow:

git pull / git fetch
git switch
git status
git add
git commit
git push

The rest is done with the click of a button in GitLab.

Inspect past commits

There's one more thing that’s often needed when working on a git project: inspecting past commits. It may be to get up to speed about recent changes made by your teammates or to refresh your memory about the changes you’ve made on your branch after working on something else or taking a vacation.

Again, you could do this with the GitLab UI (in the menu, go to Code > Repository graph), but if you want to do it locally, here are a couple of solutions

Use git CLI: We suggest the following command, that you can store in an alias if you don’t want to memorize it

  $ git config --global alias.graph "log --decorate --oneline --graph"
  $ git graph [--all]

  * 331ef24 (HEAD -> 1-demo-issue, origin/1-demo-issue) Changing package.json
  *   79c1ae8 (origin/main, origin/HEAD, main) Merge branch 'feat/previous-feature' into 'main'
  |\
  | * 432f691 Implement tests
  | * 946d022 Add an amazing feature
  |/
  * 5c3c723 Hotfix for the broken app incident
  *   aa56183 Merge branch 'awesome-feature' into 'main'
  |\
  | * 10b6622 shorten even more
  * | 0ac37b5 Merge branch 'fixes' into 'main'
  |\|
  | * 8180791 Fix various bugs
  |/
  * Initial commit

Use tig. This TUI (terminal user interface) is halfway between the GUI and the command line. It basically does the same thing as the previous command, but it’s easier on the eyes while allowing you to stay in the terminal. It will also make it easier to see the diff for a particular commit.

3. Common situations that you need to be able to manage

What we’ve discussed in the previous section is the nominal case. It will be enough for most days, but there are situations that will happen often enough that we would be lying if we told you don’t need to know how to handle them.

The target branch has been updated

When merging your branch, it’s likely that someone has updated the target branch (main) since you’ve created your branch. If these changes have no conflict with yours, you have nothing to do: git will be able to merge your branch anyway. But if there are conflicts, you’ll need to solve them before merging.

Resolve conflicts in the GitLab UI

Once again, GitLab allows to do this task right in the web UI. Click on the button, and you’ll be taken to a conflict editor that is probably enough for 99% of the conflicts you’ll have to resolve.

Read more about merge conflicts resolution in GitLab’s documentation about merge conflicts

If you prefer to resolve conflicts locally (or if GitLab can’t do it for some reason), you can do it with the following steps.

If you’re using merge commits:

# Fetch the changes so you'll be able to resolve the conflicts locally
$ git fetch

# If you're using merge commits, merge the target branch (here it is main) into your feature branch
$ git merge origin/main
# See which files are in conflict. Again, git status will tell you what to do
$ git status

On branch feature-branch
You have unmerged paths.
  (fix conflicts and run "git commit")
  (use "git merge --abort" to abort the merge)

Unmerged paths:
  (use "git add <file>..." to mark resolution)

    both modified:   file1.txt
    both modified:   file2.txt

no changes added to commit (use "git add" and/or "git commit -a")

# ...Resolve conflicts with your favourite tool (most IDEs can do it). Don't forget to add files after resolution !
# Finalize conflict resolution and push
$ git commit
$ git push

If you’re using fast-forward merges or semi-linear history, you’ll need to rebase your branch onto the target branch:

# Fetch the changes so you'll be able to resolve the conflicts locally
$ git fetch

# If you're using merge commits, merge the target branch (here it is main) into your feature branch
$ git rebase origin/main
# See which files are in conflict. Again, git status will tell you what to do
$ git status

You are currently rebasing branch 'feature-branch' on '4a56ad3'.

Unmerged paths:
  (use "git reset HEAD <file>..." to unstage)
  (use "git add <file>..." to mark resolution)

    both modified:   file1.txt
    both modified:   file2.txt

no changes added to commit (use "git add" and/or "git commit -a")

# ...Resolve conflicts with your favourite tool (most IDEs can do it). Don't forget to add files after resolution !
# Continue conflict resolution
$ git rebase --continue
# ... Resolve conflicts until rebase is done
# Then push your changes
$ git push

The conflict resolution itself is beyond the scope of this article. Here are some resources you can read:

You need to work on another branch

In a professional development team, it is common to have shifting priorities or production bugs. These situations may force you to switch branches temporarily and to get back to your work later.

There are two ways to store unfinished work: stash them or commit them.

Commit your work and switch to the other branch

There is nothing wrong with making a commit that contains unfinished work. Git allows you to change the commit later and/or squash it with other commits.

# Commit your work to save it in git
$ git commit --all
# Switch to the new branch
$ git switch other_branch
# Work as usual on the other branch (dev, commit, push, merge...)
# get back to the initial branch
$ git switch initial_branch

When you’re done with your work, remember to squash your commits together when merging your merge request so that the commit with partial work doesn’t end up in the main branch history. You can do this by checking the corresponding checkbox in the GitLab UI.

Stashing your changes

If you prefer not to commit temporary work (for example, because you don’t want to squash commits together),
you can also stash your changes. The stash is a local-only area in git designed to store partial work.

# Stash your changes
$ git stash
# at this point, all your changes are removed from the working copy so you can switch branches without trouble.
$ git switch other_branch
# Work on the other branch, then get back to the initial branch
$ git switch initial branch
# apply the last stashed changes and drop the last stash entry
$ git stash pop

Stashing changes is cleaner than commiting it, but the stash entries aren’t named and not tied to any specific branch so it becomes complicated if you work on more than two branches in parallel. See the documentation for more details on using the stash

4. Going further with the command line

If you’ve mastered all the commands up to this point, you can handle almost all situations, at least if you’re using GitLab and the GitLab flow.

As you become more proficient with the command line, you may want to perform more advanced tasks directly in your terminal. Here are some examples of tasks that you can do in the command line that will give you a better understanding of git internals.

Someone has updated your (remote) feature branch

When using the GitLab Flow, feature branches usually belong to only one developer. However, there may be some cases when you want to work on the same branch with two developers.

In this situation, there may be times when you can’t push your commits because the other developer has already pushed changes on the target branch that you haven’t yet integrated into your local copy.

When this happens, you’ll need to integrate the changes into your local branch. This is similar to the case when you reintegrate changes from the main branch, except here you integrate changes from the remote version of the feature branch.

# If you try to push but the branch has changed on the remote, you'll get an error like this. notice how git tells you everything about the situation ?
$ git push
To gitlab.com:your_group/your_repo
 ! [rejected]        1-feature-1 -> 1-feature-1 (fetch first)
error: failed to push some refs to 'gitlab.com:your_group/your_repo'
hint: Updates were rejected because the remote contains work that you do not
hint: have locally. This is usually caused by another repository pushing to
hint: the same ref. If you want to integrate the remote changes, use
hint: 'git pull' before pushing again.
hint: See the 'Note about fast-forwards' in 'git push --help' for details.
# Pull the changes made by the other person on your branch. The --rebase is important here, because otherwise the history will be bloated with merge commits
$ git pull --rebase
remote: Enumerating objects: 5, done.
remote: Counting objects: 100% (5/5), done.
remote: Compressing objects: 100% (2/2), done.
remote: Total 3 (delta 1), reused 0 (delta 0), pack-reused 0 (from 0)
Unpacking objects: 100% (3/3), 286 bytes | 286.00 KiB/s, done.
From gitlab.com:BenoitAverty/sandbox
   95d6184..feefa0b  1-feature-1 -> origin/1-feature-1
Auto-merging README.md
CONFLICT (content): Merge conflict in README.md
error: could not apply 6607d50... Edit README.md locally
hint: Resolve all conflicts manually, mark them as resolved with
hint: "git add/rm <conflicted_files>", then run "git rebase --continue".
hint: You can instead skip this commit: run "git rebase --skip".
hint: To abort and get back to the state before "git rebase", run "git rebase --abort".
hint: Disable this message with "git config advice.mergeConflict false"

The git pull command does two things: fetch the changes on the remote repository, updating the remote branch on your local repo (remote branches are called origin/branch), then rebasing your local commits on top of the fetched changes. Note that there may be conflicts in this step, but you can resolve them just like when you resolved conflicts when reintegrating the main branch. Again, git tells you what to do when you’re done resolving merge conflicts.

After finishing the rebase, you can push your changes, and this time the push should succeed.

As a bonus, here’s a config option that you can set to avoid having to add the --rebase option when pulling changes. This will instruct git to always rebase instead of merging when pulling changes in a branch from the tracked remote branch.

$ git config --global branch.autosetuprebase true
# Now when you switch to a new branch that you've created in the remote repo, git tells you that it will rebase instead of merge when pulling changes
$ git switch 2-my-feature
branch '2-my-feature' set up to track 'origin/2-my-feature' by rebasing.
Switched to a new branch '2-my-feature'

Rewrite the history

No one is perfect. Sometimes you will make changes to your project, commit something, change your mind, and commit something else. Or you will have to work on something else and make a temporary commit.
Or you will want to change a commit message.

You can always squash all your commits together in GitLab when merging, but git allows you to do all of this locally by rewriting the history. There are many ways to do this, but here are two ways that should cover most of your needs.

In both cases, make sure your branch is up to date with the remote branch and your working copy is clean, because rewriting the history can get confusing it the remote has a different history than your branch.

# Update your branch
$ git pull
# Stash uncommited changes.
$ git stash

Make small changes to the history: interactive rebase

The first way to rewrite the history is to use the interactive rebase feature. Interactive rebase lets you replay all the commits that you’ve made while adjusting some of them in the process. When you launch the interactive rebase, an editor will open with a list of all the commits you’ve made on the branch, and each commit will be associated with the pick command, meaning that the commit is kept as is.

You can change the pick command to another to perform various actions on the corresponding commit. The most commonly used commands in interactive rebase are reword (the commit will be kept but an editor will open so you can change the commit message) and fixup (The changes from this commit will be integrated in the previous commit).

All the available commands are details in the editor when starting the interactive rebase.

# start an interactive rebase that contains all the commits that are in your feature branch
$ git rebase -i main

Rewrite the history from scratch

If you prefer to start again, you can also remove all the commits you’ve made in the branch (but keeping the changes to the code themselves) and start again.

$ git reset --soft main

This command will put your branch back to its starting point, the main branch. But if you perform a git status, you will see that all your code changes are still here, staged for commit. At this point you can perform a git commit which will create a single commit containing all your changes (this is like the Gitlab option to squash all changes, but locally), or you can start again to create several commits by adding all your changes in small iterations.

# Remove the changes from the index, this will keep them in the working copy as if you had just made the changes toi the code
$ git restore --staged .
# Re-commit all your changes by adding them in several iterations
git add some-file # add only one file
git commit
git add -p other-file # add only part of a file
git commit
git add other-file # add the rest of the file
git add . # add everything
git commit

Pushing a rewritten history

Once you’re satisfied with the new history, push it to the remote. Since you’ve rewritten the history, git will tell you that your local branch is different from the remote branch (as if someone else had made changes). In this case this is expected, so you can force push your changes. This is why it’s important to be up to date, because force pushing will replace the remote history by the local one you’ve just created. If there are commits in the remote that you didn’t include in the new history, they will be lost.

This doesn’t happen in the classic GitLab flow because branches usually belong to only one developer.

# force-with-lease instead of force will tell you if you forgot to pull and someone has updated the remote branch
$ git push --force-with-lease

Remove old local branches

After some time working on the same project, you will have a lot of branches on your local repo. These branches are often left after the merge request is merged: GitLab will delete the remote branch, but if you do nothing, they’re kept in your local repository.

Here’s a way to remove all branches that aren’t needed anymore.

# Remove local copies of remote branches
$ git fetch --prune
# For each branch that has been deleted by prune, remove the local branch if it exists
$ git branch -D <branch-name>

The second step can be done in batch if there are many branches, but that solution involves a bit of scripting and it can be fragile. The best way is still to do it often, so you don’t have to use this if you don’t fully understand the way it works.

git branch -vv | grep ': gone]' | awk '{print $1}' | xargs git branch -D

Change the user name/email in commits

To change your user name/email only in last commit:

git commit --amend --author="John Doe <jdoe@zenika.fr>"

But you may also want to change them on all commits on current branch.

Start a rebase:

git rebase -i origin/main

Set the targeted commits to "edit"
For each commit until git rebase is done :

git commit --author="John Doe <jdoe@zenika.fr>" --amend --no-edit
git rebase --continue

Delete all existing files defined in .gitignore

git clean -Xfd .

Fetch changes to a local branch without switching to it

git fetch origin feature-branch:feature-branch

Remove changes on files against target branche

You have made a lot of commits on a branch, and notice some files should not have changes. Simply revert the changes on a file or folder :

git restore --source=origin/develop path/to/file/or/folder

And commit any of the changes (back).

Rebase only a specific commit range

Notice the first commit SHA you want to keep, then use the previous one as base :

git rebase -i --onto develop 7050d7bf296^

5. Git common configuration options and aliases

Here are some widely adopted git configuration options:

# set your default user name/email
git config --global user.name "Benoit COUETIL"
git config --global user.email "bcouetil@zenika.fr"
# on fetch, delete all the internal git objects that are not reachable from the remote repository
git config --global fetch.prune true
# re-apply local changes when rebasing
git config --global rebase.autostash true
# when pulling, local new commits are rebased onto distant new commits
git config --global pull.rebase true
# reuse recorded resolution on conflicts
git config --global rerere.enabled true
# display authors in interactive rebase
git config --global rebase.instructionFormat "(%an) %s"

And here are some aliases (multi-platforms):

# graph alias: display git history as a graph
git config --global alias.graph "!git --no-pager log --graph --date-order --date=short -10 --pretty=format:'%C(auto)%h%d %C(reset)%s %C(bold blue)%an %C(reset)%C(green)%cr (%cd)'"
# coall alias: commit all files, including new files
git config --global alias.coall '!git add -A && git commit'
# amend-to alias: amend all changes to a specific commit
git config --global alias.amend-to '!f() { SHA=$(git rev-parse "$1"); git add -A && git commit --fixup "$SHA" && GIT_SEQUENCE_EDITOR=true git rebase --interactive --autosquash "$SHA^"; }; f'
# amend-staged-to alias: amend only staged changes to a specific commit
git config --global alias.amend-staged-to '!f() { SHA=$(git rev-parse "$1"); git stash -k && git commit --fixup "$SHA" && GIT_SEQUENCE_EDITOR=true git rebase --interactive --autosquash "$SHA^" && git stash pop; }; f'
# clean-branches alias: remove local branches deleted from GitLab
git config --global alias.clean-branches '!git branch -vv | grep ":\ gone\]" | awk "{print \$1}" | xargs git branch -D && printf "\n" && git branch -vv'
# search-remote alias: find remote branches containing a given word
git config --global alias.search-remote '!f() { git fetch --all; git branch -r | grep -i "$1"; }; f'
# (optional) content based git diff (versus timestamp based)
# git config core.checkStat minimal

Wrapping up

In this article, we covered how the GitLab UI and the GitLab flow can simplify the tasks that need to be done locally in a git project. Given that there are fewer tasks to do locally, using the GitLab flow can make the git CLI easier to learn and use, while progressively discovering the full potential of the git CLI 🤓

Illustrations generated locally by DiffusionBee using FLUX.1-schnell model

☸️ Kubernetes: A Convenient Variable Substitution Mechanism for Kustomize

Benoit COUETIL 💫 — Sun, 04 Aug 2024 17:56:53 +0000

1. What is Kustomize?
2. Why variable substitution matters
3. Limitations and advantages of kustomize by default
4. Implementing custom variable substitution
- No compromise solution using renvsubst as a transformer
Wrapping up
Further reading

1. What is Kustomize?

Kustomize is one of the prominent tools alongside Helm used for managing and configuring Kubernetes deployments across multiple environments. Unlike Helm, which utilizes templates, Kustomize simplifies management by primarily focusing on plain manifests without direct comparison to Helm’s approach.

2. Why variable substitution matters

Variable substitution becomes crucial when environments are ephemeral, as in the case of feature branches, or when configuration settings like DNS entries need to be determined dynamically at deployment time. Kustomize assumes a fixed environment list (e.g., dev/staging/prod), but real life rarely fits into three neat boxes.

3. Limitations and advantages of kustomize by default

The biggest advantage of Kustomize lies in its straightforward approach to managing manifests. It allows for easy manipulation of fields such as annotations, image names, and replicas across environments. However, Kustomize is strict about variable substitution, preferring a predefined set of options which can be limiting.

To circumvent these limitations, the following official method is often employed within a kustomization.yml file, after creating an environment-specific file:

resources:
  - deployment.yml
  - ingress.yml
  - service.yml
  - sa.yml
  - role.yml
  - role-binding.yml

generatorOptions:
  disableNameSuffixHash: true

configMapGenerator:
- name: environment-variables
  envs: [rendered/api-gateway.env]
  behavior: create

replacements:
- source:
    kind: ConfigMap
    version: v1
    name: environment-variables
    fieldPath: data.API_GATEWAY_URL
  targets:
  - select:
      kind: Ingress
      name: api-gateway
    fieldPaths:
    - spec.rules.0.host
    - spec.tls.0.hosts.0
  - select:
      kind: Deployment
      name: api-gateway
    fieldPaths:
    - spec.template.spec.containers.0.env.[name=API_GATEWAY_URL].value

So much for such a simple need like variable substitution. People will even find Helm appealing at this step. Let's explain why and how to find a simpler solution.

4. Implementing custom variable substitution

While Kustomize apply workflow supports basic variable substitution using tools like envsubst on top of kustomize build, more complex scenarios require custom solutions. One more refined approach involves using renvsubst, to selectively replace variables based on specific criteria (e.g., K_* environment-specific or CI_* CI-specific variables).

We could then use in our pipelines, in ascending order of complexity :

For simple use cases : kustomize build . | envsubst
To avoid replacing some variables, define it in manifests as ${DOLLAR}MY_VARIABLE and then:
- kustomize build . | DOLLAR=$ envsubst
Use renvsubst to choose variables with prefixes:
- kustomize build . | renvsubst --prefix="K_" --prefix="CI_"

But all these solutions have some drawbacks. You have to pipe the kustomize build result before applying manifests; it breaks some convenient wrapping tools features like intelligent local updates using tilt.dev.

No compromise solution using renvsubst as a transformer

For a no compromise solution using a Kustomize transformer exec plugin using renvsubst, follow these steps.

Install renvsubst: Ensure renvsubst is installed from here.
set the KUSTOMIZE_PLUGIN_HOME variable for kustomize to find your plugin, for example KUSTOMIZE_PLUGIN_HOME=$PWD/devops/k8s/scripts/
Set up Transformer Script:

Create a transformer script (renvsubst) at $KUSTOMIZE_PLUGIN_HOME/transformers/renvsubst/renvsubst:

#!/bin/sh

RENVSUBST=${RENVSUBST_PATH:-$(which renvsubst)}

if [ -z "$RENVSUBST" ]; then
    echo "Error: renvsubst command not found. Please install renvsubst from https://github.com/containeroo/renvsubst/releases or set RENVSUBST_PATH." >&2
    exit 1
fi

$RENVSUBST --prefix="K_" --prefix="CI_" -i -

Configure Custom Resource:

Define a custom resource (renvsubst.transformer.yml) next to your kustomization.yml using it:

apiVersion: transformers
kind: renvsubst
metadata:
  name: renvsubst

Integrate with Kustomize:

Update your kustomization.yml to include the custom resource:

resources:
  - ...

transformers:
  - renvsubst.transformer.yml

By following these steps, you can enhance Kustomize's capabilities to handle dynamic environments and complex configuration needs effectively.

Wrapping up

While Kustomize offers simplicity in managing Kubernetes manifests, its default limitations in variable substitution can be overcome through strategic customization. By leveraging custom transformers and renvsubst, Kubernetes deployments with customize can become more adaptable to diverse and evolving environments, ensuring smoother and more efficient operations.

🦊 GitLab: A Python Script Displaying Latest Pipelines in a Group's Projects

Benoit COUETIL 💫 — Sat, 29 Jun 2024 12:33:54 +0000

Initial thoughts
1. Considered alternate solutions
- GitLab's operations dashboard
- GitLab CI pipelines exporter
- Glab ci view
- Tabs grid browser plugins
2. The Python script
- Pre-requisites
- Source code
Further reading

Initial thoughts

As a GitLab user, you may be handling multiple projects at once, triggering pipelines. Wouldn't it be great to see all your pipelines at a glance instead of clicking through 15 project pages? Spoiler: GitLab doesn't offer that out of the box.

That's why we've developed a Python script that uses the power of GitLab API to display the latest pipeline runs for every projects in a group. As simple as :

python display-latest-pipelines.py --group-id 12345 --watch

1. Considered alternate solutions

Some alternate solutions have been explored before making a script from scratch.

GitLab's operations dashboard

For premium and ultimate GitLab users, there is the Operations Dashboard:

This is a nice start, but only the overall pipeline status is available, which is too light for pipeline-focused usages.

GitLab CI pipelines exporter

Mentioned in official documentation, the GitLab CI Pipelines Exporter for Prometheus fetches metrics from the API and pipeline events. It can check branches in projects automatically and get the pipeline status and duration. In combination with a Grafana dashboard, this helps build an actionable view for your operations team. Metric graphs can also be embedded into incidents making problem resolving easier. Additionally, it can also export metrics about jobs and environments.

Very interesting, but the scope is way larger than our usage and does not follow pipelines in progress.

Glab ci view

GLab is an open source GitLab CLI tool. It brings GitLab to your terminal: next to where you are already working with Git and your code, without switching between windows and browser tabs.

There is a particular command displaying a pipeline, glab ci view :

Multiple problems for our usage :

Limited to one project
The result is large and does not allow many pipelines on the same screen
You do not get "the latest" pipeline; you have to choose a branch

Tabs grid browser plugins

Tab grids browser plugins are easy to use, but does not update status in real time, you have to refresh the given tabs

2. The Python script

Here is the script, with some consideration :

GitLab host, token, group-id, excluded projects list, stages width and watch mode are configurable
For one shot mode (versus watch mode), projects pipelines are displayed one at a time to allow more real time display
The least possible vertical space is used as a specific goal; you could obtain nicer (but more verbose) output with some minor script adjustment

Pre-requisites

Some Python packages installed
- pip install pytz
A token having access to all the projects in the group

If you're interested in similar GitLab API scripts, you might also find useful GitLab: A Python Script Calculating DORA Metrics which uses the same architecture to compute deployment frequency and lead time.

Source code

#
# Display, in console, latest pipelines from each project in a given group
#
# python display-latest-pipelines.py --group-id=8784450 [--watch] [--token=$GITLAB_TOKEN] [--host=gitlab.com] [--exclude='TPs Benoit C,whatever'] [--stages-width=30]
#
import argparse
from datetime import datetime
from enum import Enum
import os
import pytz
import requests
import sys
import unicodedata

class Color(Enum):
    GREEN = "\033[92m"
    GREY = "\033[90m"
    CYAN = "\033[96m"
    RED = "\033[91m"
    YELLOW = "\033[93m"
    BLUE = "\033[94m"
    RESET = "\033[0m"
    NO_CHANGE = ""

parser = argparse.ArgumentParser(description='Retrieve GitLab pipeline data for projects in a group')
parser.add_argument('--host', type=str, default="gitlab.com", help='Hostname of the GitLab instance')
parser.add_argument('--token', type=str, default=None, help='GitLab API access token (default: $GITLAB_TOKEN (exported) environment variable)')
parser.add_argument('--group-id', type=int, help='ID of the group to retrieve projects from')
parser.add_argument('--exclude', type=str, default="", help='Comma-separated list of project names to exclude (default: none)')
parser.add_argument('--watch', action='store_true', help='Run indefinitely while refreshing output')
parser.add_argument('--stages-width', type=int, default=42, help='Width for stages display (default: 42)')
args = parser.parse_args()

if args.token is None:
    args.token = os.getenv('GITLAB_TOKEN', 'NONE')

headers = {"Private-Token": args.token}
projects_url = f"https://{args.host}/api/v4/groups/{args.group_id}/projects?include_subgroups=true&simple=true"

def print_or_gather(output, text):
    if args.watch:
        output.append(text)
    else:
        print(text)

def count_emoji(text):
    """Count the number of emojis in the input text."""
    custom_lengths = {
        "\U0001F3D7": 0,  # Construction sign 🏗️
        # Add more special characters as needed.
    }
    count = 0
    for char in text:
        if unicodedata.category(char).startswith('So'):
            if char in custom_lengths:
                count += custom_lengths[char]
            else:
                count += 1
    return count

def fetch_pipelines():
    response = requests.get(projects_url, headers=headers)
    if response.status_code != 200:
        print(f"\n{Color.RED.value}Failed to call GitLab instance: {response.json()}{Color.RESET.value}")
        return
    projects = response.json()

    pipeline_data = {}
    no_pipelines_projects = []
    excluded_projects = set(args.exclude.split(','))

    output = []

    for project in projects:
        if project["name"] in excluded_projects:
            continue
        pipeline_url = f"https://gitlab.com/api/v4/projects/{project['id']}/pipelines?per_page=1&sort=desc&order_by=id"
        response = requests.get(pipeline_url, headers=headers)
        if not response.json():
            no_pipelines_projects.append(project['name'])
            continue
        pipeline = response.json()[0]

        updated_time = datetime.strptime(pipeline["updated_at"], "%Y-%m-%dT%H:%M:%S.%fZ").replace(tzinfo=pytz.utc).astimezone(pytz.timezone('Europe/Paris'))

        updated_at_human_readable = updated_time.strftime("%d %b %Y at %H:%M:%S")
        time_diff = datetime.now(pytz.utc) - updated_time
        delta = time_diff.total_seconds()
        if delta < 120:
            updated_ago = f'{int(delta)} seconds'
        elif delta < 7200: # 2 hours in seconds
            updated_ago = f'{int(delta / 60)} minutes'
        elif delta < 172800: # 2 days in seconds
            updated_ago = f'{int(delta / 3600)} hours'
        else:
            updated_ago = f'{int(delta / 86400)} days'
        match pipeline["status"]:
            case "success":
                color = Color.GREEN
            case "created" | "waiting_for_resource" | "preparing" | "pending" | "canceled" | "skipped" | "manual":
                color = Color.GREY
            case "running":
                color = Color.BLUE
            case "failed":
                color = Color.RED
        print_or_gather(output,f"\n↓ {color.value}{project['name']} for {pipeline['ref']} : {pipeline['status']} (since {updated_at_human_readable}, {updated_ago} ago){Color.RESET.value}")
        job_data = {}
        jobs_url = f"https://gitlab.com/api/v4/projects/{project['id']}/pipelines/{pipeline['id']}/jobs"
        response = requests.get(jobs_url, headers=headers)
        jobs = response.json()

        for job in list(reversed(jobs)):
            job_name = job["name"]
            stage = job["stage"]
            job_status = job["status"]

            match (job_status, pipeline["status"]):
                case ("success", _):
                    emoji = "🟢"
                    job_color = Color.GREEN
                case ("running", _):
                    emoji = "🔵"
                    job_color = Color.BLUE
                case ("pending" | "created", _):
                    emoji = "🔘"
                    job_color = Color.NO_CHANGE
                case ("skipped" | "canceled", _):
                    emoji = "🔘"
                    job_color = Color.GREY
                case ("warning", _):
                    emoji = "🟠"
                    job_color = Color.YELLOW
                case ("manual", _):
                    emoji = "▶️"
                    job_color = Color.NO_CHANGE
                case ("failed", "success"):
                    emoji = "🟠"
                    job_color = Color.YELLOW
                case ("failed", _):
                    emoji = "🔴"
                    job_color = Color.RED
                case (_, _):
                    print(job_status)
            if stage not in job_data:
                job_data[stage] = []
            job_data[stage].append((job_name, job_status, job_color, emoji))

        # Sort jobs within each stage alphabetically by job name
        for stage in job_data:
            job_data[stage].sort(key=lambda x: x[0])

        # Find the maximum number of jobs in any stage for this pipeline
        max_jobs = max(len(jobs) for jobs in job_data.values())

        lines = [" "] * (max_jobs + 1)
        lines[0] = "" # stages start with a border character instead of a space
        # Print out the job data for each stage, padding to make all stages content the same length
        for stage, jobs in job_data.items():
            stage = "[ "+stage+" ]"
            lines[0] = f"{lines[0]}╔{stage.center(args.stages_width - 3 - count_emoji(stage), '═').upper()}╗ "
            for i, (job_name, job_status, job_color, emoji) in enumerate(jobs, start=1):
                # emojis in job names make this exercise a bit more difficult: ljust make them expand the size, so we compensate
                lines[i] = f"{lines[i]}{emoji} {job_color.value}{job_name[:args.stages_width - 6 - count_emoji(job_name)].ljust(args.stages_width - 3 - count_emoji(job_name))}{Color.RESET.value}"
            for j in range(len(jobs) + 1, max_jobs + 1):
                lines[j] = lines[j] + " ".ljust(args.stages_width)

        for line in lines:
            print_or_gather(output,line)

    if no_pipelines_projects:
        print_or_gather(output,f"\n\033[90mProjects without pipeline: {', '.join(no_pipelines_projects)}\033[0m")

    return "\n".join(output)

try:
    if args.watch:
            while True:
                output = fetch_pipelines()
                sys.stdout.write("\x1b[2J\x1b[H")  # Clear the screen
                sys.stdout.flush()
                print(output)
    else:
        fetch_pipelines()
except KeyboardInterrupt:
    pass

Illustrations generated locally by Pinokio using Stable Cascade plugin

☸️ Why Managed Kubernetes is a Viable Solution for Even Modest but Actively Developed Applications

Benoit COUETIL 💫 — Wed, 05 Jun 2024 09:41:12 +0000

Initial thoughts
For development/developers
- Advantage 1: A rich ecosystem of development tools
- Advantage 2: A desired-state system to ease changes
- Advantage 3: advanced networking and subdomain management
- Advantage 4: ready-made solutions for stateful services
For early stages of production
- Advantage 5: a rich ecosystem of production tools
- Advantage 6: self-healing and self-management mechanisms
When the product is successful
- Advantage 7: the features and stability of one of the most maintained open source projects
- Advantage 8: a portable vendor-neutral platform with tight integrations
- Advantage 9: scalability for diverse workloads
- Advantage 10: high availability in continuous delivery context
- Advantage 11: security tools for peace of mind
Wrapping up
Further reading

Initial thoughts

Kubernetes, has emerged as a transformative force in the world of software development and deployment. Born as an open-source project under the guidance of the Cloud Native Computing Foundation (CNCF), Kubernetes has evolved into one of the most influential technologies powering modern infrastructure and application management.

Kubernetes has gained so much traction within the community that people started to use it for small usages, where the need for a container orchestrator is far from necessary. We now see Kubernetes jokes, memes, and even Kubernetes bashing.

I like Kubernetes. but I've bin an even bigger fan of the KISS principle. An efficient monolith is beautiful. An application working well on a single server is beautiful. And yet, I still recommend Kubernetes for modest commercial applications. More precisely, for modest teams (at least 3 full time developers working efficiently). Modest team and not below, because lots of advantages concerns active and collaborative development.

We choose the managed side of Kubernetes, because let's face it, it is hard (and costly) to maintain on your own below a certain tech team size. At least it better be managed from the dev team perspective.

In this article, we will uncover the reasons why Kubernetes still stands as a great choice for a wide array of commercial applications, even small and/or starting ones, for a various reasons, beyond scalability and self-healing.

For development/developers

Way too often, we underestimate the impact of an architecture choice on the team velocity. Yes, these choices do not only impact our production environment. I would dare to say that it is the least of challenges for modest teams. The biggest challenge we face is to build a product at a fast and yet sustainable pace, to hopefully become successful someday.

Some people choose an architecture or another depending on minor cloud cost differences. But the infrastructure cost on an actively maintained modest application is relatively rarely the problem. The biggest cost is (or should be) the team itself. How impactful is a 30$ a day kubernetes service when the team costs more than 2000$ a day ? Trying to save 10$ a day on these servers often leads to more than 100$ a day of team inefficiency. And there is always room for cost optimization for a given architecture without impacting performance, as described for Kubernetes in the article FinOps EKS: 10 Tips to Reduce the Bill up to 90% on AWS Managed Kubernetes Clusters).

Now that we advocated for the cost of Kubernetes clusters to not be a problem for development environments, let's talk about real advantages.

Also, Kubernetes enables efficient development git workflows:

One Environment Per Branch: Kubernetes allows you to create dedicated environments for each Git branch. This isolation aids testing, development, and collaboration.
Rapid Feedback Loops: Developers can quickly iterate by deploying changes to their own environments, ensuring that new features and bug fixes are thoroughly tested.
Collaboration: Kubernetes supports collaborative development by providing, if needed, an environment for each contributor to validate their work before integration, each one in their single namespace.

Advantage 1: A rich ecosystem of development tools

Kubernetes offers an extensive set of development productivity tools. These tools simplify the development process, letting developers focus on writing code rather than dealing with infrastructure. From local development environments to CI/CD pipelines, Kubernetes provides resources that genuinely make a difference — fewer YAML headaches, more shipping.

You can check some maintained lists mentioned in Kubernetes: Awesome Maintained Links You Will Keep Using Next Year.

Advantage 2: A desired-state system to ease changes

Kubernetes, with its declarative constructs and its ops friendly approach, has fundamentally changed deployment methodologies: it allows teams to use GitOps, Thanks to the desired-state principle.

When applying small changes such has configuration or single module code, Kubernetes will perform the necessary minimal actions to meet the new desired state. Once a developer understands the basics of kubernetes manifests (or Helm configuration), he can request changes without knowing how they will be performed by the system.

Advantage 3: advanced networking and subdomain management

Kubernetes provides advanced networking capabilities:

Ingress Controller: Kubernetes offers an Ingress Controller that allows you to manage subdomains, routing, and load balancing effectively.
Subdomain Routing: With Kubernetes, you can route traffic to different services based on subdomains, enabling you to host multiple applications under a single domain.

Once an Ingress Controller is installed and a wildcard domain is affected to a cluster, developers can easily affect sub-domains to sub-environment and/or services, without the need of any additional service or operational teams ((always busy, aren't they ?).

Advantage 4: ready-made solutions for stateful services

Even if we mostly move stateful services outside Kubernetes clusters in production, Kubernetes simplifies the deployment of stateful services for development and testing purposes:

Databases: Kubernetes offers pre-packaged solutions for popular databases like MySQL, PostgreSQL, and MongoDB, making it easy to manage and scale your data stores.
Message Brokers: You can deploy message brokers like Apache Kafka or RabbitMQ as stateful services on Kubernetes, enabling reliable communication between microservices.

For other less common stateful usages, Kubernetes provides StatefulSets, a controller that ensures stable, unique network identities and persistent storage for stateful applications.

These solutions simplify the management of stateful components, allowing you to focus on building your commercial applications.

For early stages of production

Early stages of production life can be summed-up too often as: no time, no money, no product support, lots of bugs.

In this stressful period, various reason make Kubernetes a wise choice.

We still do not need those high-availability features that make funny jokes on small applications, but some others are valuable in this phase.

Advantage 5: a rich ecosystem of production tools

Kubernetes has a rich ecosystem of production tools that contribute to the three pillars of observability. Let's explore these essential components:

Monitoring: Kubernetes provides powerful solutions such as Prometheus and Grafana for real-time monitoring and alerting. These tools ensure you can proactively maintain the health and performance of your applications.
Logging: Efficient log management is crucial for troubleshooting issues in a distributed environment. Kubernetes offers tools like Fluentd and Elasticsearch to centralize and analyze logs, providing insights into the behavior of your applications.
Tracing: Observing the flow of requests through various microservices is vital for understanding the performance of complex systems. Kubernetes supports tracing solutions, such as Jaeger or Zipkin, enabling you to trace requests as they traverse different components.

These production tools collectively enhance the reliability, performance, and overall observability of commercial applications deployed on Kubernetes.

Advantage 6: self-healing and self-management mechanisms

Kubernetes has been built with self-healing and self-management in mind.

Self-Healing: Kubernetes automatically restarts containers that fail, replaces containers, and kills containers that don't respond to your user-defined health check.
Self-Management: Kubernetes automatically manages the lifecycle of your containers, including scaling, upgrading, and rolling back.

Hardware failure is also handled, and failing nodes/servers will trigger automatic workload balance on healthy nodes.

When the product is successful

Advantage 7: the features and stability of one of the most maintained open source projects

In the world of open-source software, Kubernetes is one of the most well-maintained projects. It began as an internal tool at Google called Borg and later became open-source in 2014. Since then, it has grown into a thriving community of developers, users, and contributors who keep it actively maintained.

Community-powered advancements

Kubernetes benefits from a diverse global community. People from various backgrounds, organizations, and cloud providers work together to continuously enhance the platform. This collaborative effort drives innovation, ensuring that Kubernetes stays at the forefront of container orchestration technology.

Swift updates and issue fixes

A well-maintained project delivers updates and fixes problems promptly. Kubernetes exemplifies this by regularly releasing updates and addressing issues efficiently. Users can rely on a stable and evolving platform that keeps up with the fast-paced world of cloud-native technology.

Extensive documentation

Kubernetes places a strong emphasis on providing clear documentation and resources. This commitment to transparency ensures that users, regardless of their expertise, can easily access the information they need. The Kubernetes website, along with community-contributed guides, tutorials, and forums, offers valuable knowledge for users of all skill levels.

Neutral governance

Kubernetes is governed by the Cloud Native Computing Foundation (CNCF), a neutral body that ensures no single entity dominates decision-making. This neutrality encourages a fair and adaptable solution, free from corporate bias. As a result, Kubernetes serves as a versatile and reliable option for various commercial applications.

Advantage 8: a portable vendor-neutral platform with tight integrations

Kubernetes is versatile when it comes to deployment platforms. It can be set up on:

Cloud Providers: Kubernetes is supported on major cloud platforms like AWS, Google Cloud, and Azure, making it easy to deploy and manage your applications in the cloud.
Private Clouds: If you prefer to run Kubernetes in your private data center, it's fully adaptable to private cloud environments, offering control and security.
On-Premises: For complete control over your infrastructure, you can deploy Kubernetes on on-premises servers, ensuring data stays within your network.
Lightweight or Full-Fledged: Whether you need a lightweight setup on a single VM for development or a large-scale deployment spanning multiple VMs for production, Kubernetes can accommodate your needs.

Kubernetes' adaptability across these platforms makes it a viable choice for various commercial applications.

Advantage 9: scalability for diverse workloads

Kubernetes offers architecture scalability that can adapt to diverse workloads:

Lightweight Distribution: You can use a lightweight Kubernetes distribution on a single virtual machine (VM) for smaller workloads, development, and testing.
Full-Fledged Deployment: For heavy production workloads, Kubernetes can scale across multiple VMs or nodes. It can handle a wide range of applications and services efficiently.

Scalability can also be considered at the application level, with a manual update of the number of pods or automatically using pods autoscalers.

Advantage 10: high availability in continuous delivery context

Sure, a single VM is fine for serving a single starting application, and we can accommodate downtimes twice a month for deployment. But the more you lean toward continuous delivery, the more you need to consider the availability of your applications.

Kubernetes offers high-availability solutions:

Deployments, Daemonsets, Statefulsets: Kubernetes offers multiple controllers and deployment strategies, which allows you to create multiple replicas of a single application. This ensures that your application is highly available and can handle any unexpected load.
Canary Deployments: Canary deployments allow you to roll out new versions of your application in a controlled manner, ensuring that users are not impacted by the new features.
Blue/Green Deployments: Blue/Green deployments allow you to roll out new versions of your application alternatively to blue and green production environments, allowing complex upgrades with zero downtime deployments

These solutions ensure that your applications are highly available and can handle any type of deployment anytime.

Advantage 11: security tools for peace of mind

Kubernetes offers solid security features:

mTLS Support: Kubernetes supports mutual TLS (mTLS) authentication, ensuring secure communication between services within your applications.
Role-Based Access Control (RBAC): RBAC in Kubernetes allows you to define fine-grained access control policies, ensuring that only authorized users can perform specific actions.
Integration with secret managers: Kubernetes ecosystem provides secure ways to manage secrets, ensuring sensitive information remains protected.

These security tools allow you to build and operate commercial applications with confidence, safeguarding your data and infrastructure.

Wrapping up

In summary, Kubernetes is a solid choice for commercial applications. It has active maintenance, a rich development toolset, diverse production tools, and works across cloud providers, private clouds, and on-premises environments. Its scalability, advanced networking, security features, and compatibility with various stateful components further reinforce its viability.

Moreover, Kubernetes supports efficient development workflows with Git branching strategies. By using Kubernetes, businesses can improve productivity, ensure scalability, and handle the challenges of modern software development.

When considering architectures for commercial applications, Kubernetes should be a top contender — it earns its complexity budget.

Illustrations generated locally by Automatic1111 using ToonYou model

🤖 Future-Proof Tech Blogging: Understanding AI's Core Traits

Benoit COUETIL 💫 — Fri, 19 Apr 2024 11:14:57 +0000

Initial thoughts
1. The influence of your intentions
2. Understanding reader engagement
3. Leveraging original content to surpass AI
4. Laziness? Achieving greater impact with limited time!
5. Less human content? Prioritizing quality over quantity!
6. Quality content has been concealed? Search tools will evolve!
- Understanding reader experience in tech blogs
- Tech blogging evolution: from scarcity to overabundance
7. Wrapping up
Further reading

This article on the future of tech writing is dynamic and continuously evolving to reflect my changing perspective. Your comments are highly valued and have already influenced significant revisions to the content.

Initial thoughts

"I believe AI is bad, prove me wrong. For over a month now, I have been thinking, what is the value of a post if AI has become this excellent and web scraping AI programs are copying content and shunning them out in large numbers, more people are becoming lazier, and some of us who put in hard work are getting lesser and lesser value."

These words, shared by a fellow tech blogger publishing articles almost every day, encapsulate the growing concerns surrounding the impact of AI (Artificial Intelligence) on the field of tech blogging.

While some may share this sentiment, we can approach this topic from a radically different perspective, shaped by our collective experience in the context of this revolution. As a time-limited tech blogger, I view AI not as a threat, but as a catalyst for innovation and growth. Let me explain why AI is actually your best ally as a tech blogger — not your replacement.

1. The influence of your intentions

if your goal is JUST a broad audience, create listicles on any Javascript subject 😅.

For those wanting to positively impact people with their articles on any other subject, several factors determine a wide readership, including:

Advertisement strategies on popular writing platforms, self-promotion efforts, paid plans, and reaction bots
Reader needs, whether actively searching for content on search engines (traditional or AI-powered)
Social sharing by enthusiastic readers
Accessibility considerations, such as readability, language clarity, and compatibility with assistive technologies

On the other side, for tech bloggers who primarily write for personal reasons, such as showcasing their skills to superiors or close peers, the reach is less important. These writers are less likely to be significantly affected by advancements in AI.

For the rest of this article, we will focus on tech blogging that aims to have a meaningful impact on fellow developers.

2. Understanding reader engagement

How can we make our tech content engaging?

Drawing inspiration from Aristotle's modes of persuasion, readers are influenced by logical reasoning (Logos), the credibility of sources or blogger experiences in the field (Ethos), and emotional connections (Pathos). As a writer, address at least one of these elements to effectively engage your audience.

For example, opinions gain significance when backed by logic (Logos) and presented by a credible writer (Ethos). Writers lacking established reputation may struggle to make a substantial impact, resonating primarily with less discerning readers. Establishing credibility (Ethos) is essential for broadening your audience. However, it is worth noting that emotional appeals (Pathos) can also be highly effective, as most individuals are responsive to them.

Understanding and effectively incorporating the principles of Logos, Ethos, and Pathos in your tech blogging can improve your content, engage a wider audience, and establish your reputation as a knowledgeable and persuasive writer in the field.

3. Leveraging original content to surpass AI

AI has undoubtedly revolutionized content creation by synthesizing existing information in a matter of seconds with impeccable precision. Some argue that it is not, but it is a matter of months before it will be. Blogging posts saying that it is not perfect, like Is Blogging Dead After ChatGPT?, will not age well. Let's consider a state where AI has been perfected to the point that content cannot be distinguished.

Where AI falls short is in providing genuine novelty and unique insights. While it excels at aggregating existing knowledge, it lacks the ability to generate truly original content. AI may soon become the best content synthesizer tool of all time.

We must then acknowledge that not all human content is created equal. Articles that simply regurgitate information or offer generic opinions are quickly becoming superseded by AI. Instead, it is the articles that bring fresh perspectives, personal anecdotes, or profound insights that stand the test of time.

Consider the following content that will not be contested by AI anytime soon:

Original Thought Leadership: Articles that dig into unexplored topics or offer fresh insights are inherently valuable. By being the first to tackle a subject or presenting a unique perspective, writers can capture readers' attention and establish themselves as thought leaders in their respective fields.
Credible Opinion Pieces: Readers place a premium on the opinions of credible individuals. As experienced tech bloggers, we have the opportunity to use our expertise and credibility to offer valuable insights and recommendations.
Emotionally Engaging Narratives: Stories have the power to evoke emotions and create lasting impressions. By infusing personal anecdotes or captivating narratives into our writing, we can forge deeper connections with our audience and leave a lasting impact.
In-depth Analysis: Articles that offer in-depth analysis or explore complex topics in detail are highly sought after. As experienced writers, we possess the knowledge and analytical skills to dissect complex subjects and provide valuable insights to our readers.

4. Laziness? Achieving greater impact with limited time!

Laziness is prompting ChatGPT and publishing the result as-is. But leveraging automation to handle repetitive and time-consuming tasks is a sensible approach. As tech bloggers, we are familiar with tasks that require repetition. It may include keyword research, generating topic ideas, formatting content, translation, optimizing SEO, banner prompt ideas, and even moderating comments.

Additionally, writing efficiently enables us to produce a greater number of articles within the constraints of our limited time. When coupled with high-quality content, this results in an increased volume of original material being shared with the global audience, making knowledge more accessible to all.

5. Less human content? Prioritizing quality over quantity!

When our blog posts revolve around rehashing existing material or, worse, generating content without a thorough understanding of the existing literature, we find ourselves competing with AI in its area of expertise. This is a futile endeavor, regardless of the level of effort exerted.

In the era of AI, we may encounter peers feedback suggesting "ChatGPT could have produced superior content in mere seconds." While this feedback may seem harsh, it raises the bar; it contributes to collective improvement and encourages writers to invest their time efficiently.

6. Quality content has been concealed? Search tools will evolve!

Jon Randy made a well-articulated statement in the comment section:

AI has absolutely trashed all tech blogging websites (that I know of). It's extremely difficult now to find anything at all interesting to read on these sites.

The innate laziness of human beings combined with the social media age's creation of a craving for 'likes', and the weird insistence that creating an online presence/brand is somehow 'essential' has resulted in a tidal wave of generated/assisted rehashed content that has been published a million times before - all in a weird style that makes you feel like you've actually learned nothing from reading the article - about the subject, or the author's actual thoughts about it.

Generative AI has unfortunately drowned out most of the good content that still exists, simply because the majority of people seem to be using it as a pure 'content' producer rather than an assistive tool.

Generative AI has overshadowed quality content as many use it solely as a content generator rather than an aid. This trend is not new or irreversible.

Understanding reader experience in tech blogs

To comprehend the current state of tech blogs, we can analyze the reader experience using Peter Morville's User Experience Honeycomb. Blog content value can be categorized into six facets: Desirable, Accessible, Credible, Findable, Usable, and Useful.

We can draw parallels with Aristotle's concepts: desirable/Pathos, credible/Ethos. The connection to Logos is more abstract.

Tech blogging evolution: from scarcity to overabundance

Let's focus on content that is credible and accessible. Valuable tech blogging content is what users also find usable, useful, and/or desirable. Today, findability is the challenge.

Blogging history can be divided into several eras:

Early Internet: Valuable content was primarily found in books, limiting accessibility. Valuable content was scarce and not easily findable.
Blogging Golden Age: Valuable content was readily available online, with authors sharing unique insights.
Inflated Online Presence: There was a push for creating an online brand, and consequently valuable content was not easily findable. This issue predated the rise of AI.
Generative AI Domination: AI now produces content effortlessly, more and more blurring the line between human and AI-generated content. Original content is increasingly difficult to find.

The future will certainly bring the Post-Google Search Era: Tech websites relying on AI-generated content are fading, and platforms like Medium and Dev.to are combating redundant articles. Tools are emerging to identify and limit low-value ones. AI could assist in evaluating an article's novelty compared to existing knowledge.

Valuable content will soon becoming findable again! Investing in it is surely not a waste of time.

7. Wrapping up

As tech blogging evolves with the rise of AI and ChatGPT, it's crucial to distinguish ourselves by focusing on valuable, original content that offers fresh insights or unique perspectives. By investing our time efficiently and prioritizing quality over quantity, we can create a lasting impact in the era of digital information saturation.

The future holds exciting possibilities: advances in search tools will make high-quality, valuable content more discoverable, and platforms are emerging to combat redundant articles. As tech bloggers, we have an opportunity to invest our time wisely and contribute valuable insights to our ever-curious readers.

Illustrations generated using Stable Cascade on HuggingFace.