Benoit COUETIL 💫 for Zenika

Posted on Jan 30

🦊 GitLab Runners: Which Topology for Fastest Job Execution?

#gitlab #devops #performance #cicd

Initial thoughts
Understanding job execution phases
Comparing runner types
Job speed comparison summary
Recommendation: Shell or Docker executors for fastest jobs
Wrapping up
Further reading

With multiple executor types available—from shared SaaS runners to Kubernetes clusters to single-server setups—understanding how each impacts job speed helps you make informed architectural decisions. Spoiler: the simplest topologies often deliver the fastest jobs.

Initial thoughts

Choosing the right GitLab Runner topology is crucial for fast job execution. With multiple executor types available—from shared SaaS runners to self-hosted solutions—understanding how each impacts job speed helps you make informed architectural decisions.

This article focuses on a single question: which runner topology executes jobs the fastest? We analyze different topologies through the lens of job execution phases, comparing the time overhead each infrastructure type adds before, during, and after script execution. Whether you're starting fresh or optimizing an existing setup, understanding these trade-offs is essential to minimize job duration.

Understanding job execution phases

From runner selection to final cleanup, every job goes through multiple technical phases. Understanding these phases is crucial to optimizing pipeline performance.

Here's a detailed breakdown of all the technical steps involved in running a single GitLab CI job:

Each phase adds latency to job execution. The key to achieving fast pipelines is minimizing time spent in these overhead phases and focusing on actual script execution. Different runner topologies handle these phases very differently.

Comparing runner types

Let's analyze the performance characteristics of each GitLab Runner infrastructure type, examining how they handle the execution phases differently.

📊 Reading the charts: Throughout this article, the durations shown in Gantt diagrams are relative and illustrative—actual times vary based on project size, network conditions, and infrastructure specs. What matters is the color coding:

🟢 Green: Faster than average across runner types

⚪ Grey: Average performance

🔴 Red: Slower than average across runner types

GitLab Shared Runners (SaaS)

GitLab.com provides shared runners available to all users without any setup. While convenient, these runners compete for resources with thousands of other projects, making them the slowest option for job execution.

Topology: Shared SaaS infrastructure - all users share the same pool of runners and cache storage.

Performance characteristics:

Speed advantages:

✅ No wait for infrastructure provisioning

Speed disadvantages:

❌ Slowest overall performance (all resources shared)
❌ Every job pulls images from scratch
❌ Shared network bandwidth slows git operations
❌ Unpredictable performance due to multi-tenancy
❌ Slow artifact uploads/downloads

Best for: When speed is not a priority and zero maintenance is essential.

Kubernetes Executor (Fixed Cluster)

A fixed-size Kubernetes cluster provides consistent resources for CI jobs. While more complex to set up than shared runners, it offers better performance through image caching and dedicated resources—though still limited by network-based cache and pod startup overhead.

Topology: Fixed-size cluster with shared remote cache (S3/MinIO).

Performance characteristics:

Speed advantages:

✅ Fast when warm (images cached on nodes)
✅ Pod creation relatively quick on existing nodes
✅ Consistent performance with dedicated resources

Speed disadvantages:

❌ Jobs queue when capacity is reached
❌ Remote cache adds network latency
❌ Pod startup overhead (even on warm nodes)

Best for: Teams with existing Kubernetes infrastructure where moderate speed is acceptable.

Kubernetes Executor (Autoscaling Cluster)

Autoscaling Kubernetes clusters dynamically add nodes when demand increases. This eliminates queuing issues but introduces significant cold-start delays—new nodes take time to provision, and each job starts with a fresh environment requiring full image pulls and git clones.

Topology: Cluster with autoscaler that adds/removes nodes based on load, pods distributed dynamically.

Performance characteristics:

Speed advantages:

✅ No queue time when scaling up
✅ Dedicated resources per job once running

Speed disadvantages:

❌ Very slow cold starts (node provisioning 30s-2min)
❌ Full image pulls on new nodes
❌ Complete git clones on ephemeral pods
❌ Remote cache network latency

Best for: Variable workloads where cold-start delays are acceptable trade-offs for unlimited capacity.

Docker Executor (Single Server)

A single server running Docker provides the sweet spot between isolation and performance. Containers start quickly when images are cached, git repositories are reused locally, and cache access is lightning-fast through the local filesystem—all while maintaining job isolation.

Topology: Single server with Docker Engine - container isolation but shared local cache via volumes.

Performance characteristics:

Speed advantages:

✅ Very fast local cache access (filesystem)
✅ Image layers cached locally
✅ Quick container startup when warm
✅ Local git repository reuse
✅ No network latency for cache

Speed disadvantages:

❌ Jobs queue when server capacity is reached
❌ Container overhead (minimal but present)

Best for: Teams needing container isolation with near-optimal speed.

Shell Executor (Single Server)

The shell executor is the fastest possible configuration—no containers, no image pulls, no pod scheduling. Everything runs directly on the host system with instant access to local git repos and filesystem cache. The trade-off? No job isolation, requiring trust in your codebase.

Topology: Everything is local on a single server - runner, shell execution, and cache share the same filesystem.

Performance characteristics:

Speed advantages:

✅ Absolute fastest (zero container overhead)
✅ Instant cache access (direct filesystem)
✅ Fastest git operations (local clones reused)
✅ No image pulls ever
✅ Immediate job start (no container creation)
✅ Best optimization potential

Speed disadvantages:

❌ Jobs queue when server capacity is reached

Best for: Maximum speed when job isolation is not required.

Docker Machine (Autoscaling AWS EC2)

Docker Machine spawns fresh EC2 VMs for each job burst, providing perfect isolation and unlimited capacity. However, this comes at a severe performance cost—every VM takes 15+ seconds to provision, and each job starts completely cold with full image pulls and fresh git clones.

Topology: Unlimited autoscaling - VMs created/destroyed on demand, each job on dedicated ephemeral VM.

Performance characteristics:

Speed advantages:

✅ No queue time (infinite capacity)
✅ Dedicated resources once VM is ready

Speed disadvantages:

❌ Slowest cold start (15+ seconds VM provisioning)
❌ First job on new VM pulls all images
❌ Complete git clone on each VM
❌ S3 cache network latency
❌ VM initialization overhead

Best for: Extreme load spikes where individual job speed is secondary to handling burst capacity.

Job speed comparison summary

Here's a comprehensive comparison of job execution speed across all execution phases:

Legend: 🟢 Fast · ⚪ Medium · 🔴 Slow

Runner Type	Wait	VM/Pod	OS	Git	Script	Cache	Artifacts
GitLab Shared	🟢	🟢	🔴	🔴	🔴	🔴	🔴
K8S Fixed	⚪	🟢	⚪	⚪	⚪	⚪	⚪
K8S Autoscaling	🟢	⚪	🔴	🔴	🟢	⚪	⚪
Docker	🔴	🟢	⚪	🟢	🔴	🟢	⚪
Shell	🔴	🟢	🟢	🟢	🔴	🟢	⚪
Docker Autoscaling	🟢	🔴	🔴	🔴	🟢	⚪	⚪

Key insights:

GitLab Shared: Great for zero maintenance but slowest overall
K8S Fixed: Balanced approach with good warm performance
K8S Autoscaling: Best for variable loads but cold starts are slow
Docker Single Server: Fast local operations with isolation
Shell Single Server: Fastest local operations, best optimization potential
Docker Autoscaling: Maximum scalability but slowest cold starts

Recommendation: Shell or Docker executors for fastest jobs

After analyzing all runner types, Shell and Docker executors on single servers offer the fastest job execution for most teams.

Why single-server executors win on speed

1. Local everything = minimal latency

Git repositories cached locally
Dependencies and cache on local filesystem
No network round-trips for cache operations
Instant resource availability (no VM/pod provisioning)

2. Vertical scaling is underrated

Modern servers can handle dozens of concurrent jobs
SSD storage makes local caching extremely fast
RAM caching for frequently accessed data
CPU cores scale linearly for parallel jobs

3. Warm infrastructure advantage

Docker images already pulled and cached
Git repos incrementally fetched (not full clones)
Dependencies preserved between jobs
No cold-start penalties

Shell vs Docker trade-offs

Choose Shell when:

You need maximum speed
Your codebase is trusted
You can maintain consistent tooling
Security isolation is less critical
You want to push performance limits

Choose Docker when:

You need job isolation
Multiple projects with different dependencies
Security/multi-tenancy matters
Slightly slower speed is acceptable trade-off

The performance/price/maintenance sweet spot

📈 Vertical scalability compensates for the lack of horizontal scalability

💪 A properly sized server with local SSD can handle dozens of simultaneous jobs

💰 Predictable costs: No per-job pricing, one server cost

🔧 Low maintenance: Simple architecture, fewer moving parts

📖 Reference: GitLab CI: The Majestic Single Server Runner

Wrapping up

Choosing the right GitLab Runner topology depends on your specific needs, but if minimizing job execution time is your primary concern, Shell or Docker executors on well-provisioned single servers consistently deliver the fastest jobs.

Key takeaways:

Shared runners are great for getting started but have performance limitations due to multi-tenancy
Kubernetes solutions offer good isolation and scalability but add network latency
Single-server executors (Shell/Docker) provide the fastest local operations and best optimization potential
Autoscaling solutions handle variable loads well but suffer from cold-start penalties
Vertical scaling of single servers is often more effective than complex horizontal scaling

The comparison shows that while autoscaling solutions offer flexibility, a properly configured single-server runner often provides the best performance for teams with predictable workloads or those willing to size for peak capacity.

In the next article, we'll dive deep into extreme optimizations you can apply to Shell and Docker runners to push performance even further—potentially achieving job times as low as 3 seconds on multi-million line codebases!

Illustrations generated locally by Draw Things using Flux.1 [Schnell] model

DEV Community

🦊 GitLab Runners: Which Topology for Fastest Job Execution?

Initial thoughts

Understanding job execution phases

Comparing runner types

GitLab Shared Runners (SaaS)

Kubernetes Executor (Fixed Cluster)

Kubernetes Executor (Autoscaling Cluster)

Docker Executor (Single Server)

Shell Executor (Single Server)

Docker Machine (Autoscaling AWS EC2)

Job speed comparison summary

Recommendation: Shell or Docker executors for fastest jobs

Why single-server executors win on speed

Shell vs Docker trade-offs

The performance/price/maintenance sweet spot

Wrapping up

Further reading

Top comments (0)