Last month, while optimizing our CI/CD pipeline for a client project, part of the team wanted to move towards self-hosted runners for more control, while the other part advocated for the ease of integration offered by cloud solutions to avoid maintenance overhead. This discussion reflects a fundamental dilemma that many engineers face in their career journey: is it more valuable to gain in-depth system knowledge in self-hosted CI/CD environments, or to learn to work faster and at a larger scale with cloud-based solutions? In my experience, this choice directly impacts both immediate project efficiency and long-term career development.
There isn't a single "right" answer here, but understanding the opportunities and challenges inherent in both approaches will provide you with a clear roadmap for which skills to invest in for your engineering career. Let's examine in detail both the deep control advantages offered by self-hosted environments and the operational ease and integration power provided by cloud solutions.
Why Are Self-Hosted CI/CD Environments Still Attractive?
Self-hosted CI/CD environments still hold a significant place, especially in large and mature companies or projects with very specific requirements. A few years ago, while working on the development process for a manufacturing ERP, there was a situation where the build and test environments had to remain entirely within the company's internal network; this was critical for sensitive data not to leave. In such scenarios, self-hosted solutions provide us with both data sovereignty and full control over the infrastructure.
This approach also offers a unique learning ground for engineers. When you set up Jenkins, GitLab Runner, or another CI/CD tool on your own servers, you need to understand every layer, from the operating system level (Linux kernel tuning, systemd units) to network configuration (VLAN segmentation, firewall policies) and resource management (cgroup limits, disk I/O optimization). This in-depth knowledge is invaluable for those who want to pursue a career in system architecture and operations. For example, to understand why a build is slow, identifying whether it's I/O bound or CPU bound, and then optimizing the disk subsystem or kernel settings, is a common and very educational situation in a self-hosted environment.
đź’ˇ Learning Opportunity
Self-hosted CI/CD environments not only provide you with practical experience in software development processes but also in areas such as fundamental Linux system administration, network security, and resource optimization. These skills significantly boost your overall engineering competence.
However, this depth also brings a significant maintenance burden. You constantly have to deal with issues like operating system updates, security patches, dependency conflicts, and hardware failures. I frequently encounter problems like full disk space on build servers, bloated Docker images, or crashing runners. While solving these issues is educational, it can take time away from the development team's core work and increase operational costs. In a production environment, such disruptions can directly affect deployment times and lead to business loss.
What Do Cloud-Based CI/CD Solutions Add to Our Careers?
Cloud-based CI/CD solutions (GitHub Actions, GitLab CI/CD Cloud, CircleCI, Travis CI, etc.) play a key role in the rapid adoption of modern software development practices. These platforms relieve you of the infrastructure management burden, allowing developers and DevOps teams to focus directly on pipeline logic. For some of my own side products, especially during rapid prototyping and MVP (Minimum Viable Product) stages, the speed and ease provided by cloud CI/CD have been incredibly valuable.
Cloud solutions typically allow you to create complex CI/CD workflows with a few clicks or simple YAML configurations. This reduces the initial investment cost and operational burden, especially for small and medium-sized teams. From a career perspective, cloud CI/CD experience provides you with modern DevOps culture, Infrastructure as Code (IaC) principles, and the ability to integrate with different cloud services. You can implement scenarios much faster, such as running automated tests when a push is made to a repository with GitHub Actions, compiling container images and sending them to a container registry, or deploying to a cloud environment.
ℹ️ Rapid Adaptation and Integration
Cloud CI/CD platforms integrate seamlessly with other cloud services and third-party tools thanks to their extensive ecosystem integrations. This helps engineers develop skills in bringing different technologies together and achieving end-to-end automation.
However, cloud solutions also have their limitations. Flexibility is less than in a self-hosted environment. Sometimes, when you need a very specific build environment or special hardware, you might have to settle for the limited options offered by the cloud provider. Also, costs can be higher than expected once a certain scale is exceeded. Especially if you have a large number of long-running builds, the per-minute billing models of cloud runners can result in a significant bill. Recently, in the CI/CD process of one of my mobile applications, running all tests on every commit significantly increased costs as the build time extended; in this case, I had to develop an optimized caching strategy.
Which Path is More Valuable in Your Career Journey: Technical Depth or Speed?
This question depends on what kind of specialization you want to develop in your engineering career. Working with self-hosted CI/CD environments gives you unique depth in "how systems work." An engineer pursuing this path typically specializes in areas such as Linux system administration, network architecture, security hardening (fail2ban patterns, kernel module blacklisting), storage solutions (iSCSI, NFS), and database optimization (PostgreSQL connection pool tuning, WAL bloat management). These skills are highly sought after, especially in infrastructure, SRE (Site Reliability Engineering), or system security roles. As someone who can delve from operating system metrics to network packet analysis to find the root cause of performance issues in a production ERP, I know the value of this depth very well.
On the other hand, focusing on cloud-based CI/CD solutions provides you with expertise in "how to make modern software development processes faster and more efficient." An engineer pursuing this path typically becomes proficient in cloud platform APIs, serverless architectures, container orchestration (Docker Compose, preparing for Kubernetes), Infrastructure as Code tools (Terraform, CloudFormation), and observability (metrics, logs, traces). These skills are very valuable, especially for DevOps engineers, cloud architects, or rapid product development teams. On the mobile side, when integrating native packages with Flutter or automating Play Store publishing processes, the ease and speed of integration offered by cloud CI/CD are decisive for me.
⚠️ Risks of Misplaced Focus
Over-focusing on a single area can create blind spots in your career. For example, only mastering cloud solutions might make it difficult to understand the root causes of infrastructure issues, while staying only with self-hosted systems might keep you away from modern cloud practices. A balanced approach is always more beneficial.
In my opinion, the ideal is to be proficient in the fundamental principles of both worlds. You should be able to set up a self-hosted Jenkins for one project while designing a fast deployment pipeline with GitHub Actions for another. This adaptability is one of the most valuable skills in today's rapidly changing technology world.
Cost Factor and Career Decisions: Whose Pocket Does It Come From?
Cost has always been one of the most important factors when choosing a CI/CD solution, and this also affects our career decisions. The initial cost of setting up a self-hosted CI/CD system includes items like server hardware, network equipment, and licenses (if you're not using open source). However, in the long run, for high-volume builds, operational costs (electricity, cooling, maintenance) can often be lower than cloud solutions. In a manufacturing company, using the idle capacity of existing server infrastructure to run CI/CD runners resulted in significant savings compared to cloud bills.
Cloud-based CI/CD solutions, on the other hand, usually operate on a pay-as-you-go model. Initial costs are low, and there are even free tiers for small projects. This is a great opportunity, especially for beginners or individual developers, as they can experience modern CI/CD practices without a high investment. For the backend of one of my side products, I gained a significant cost advantage in the initial stages thanks to the free tiers of cloud CI/CD. However, as scale increases, build times extend, or your number of parallel builds grows, costs can quickly escalate.
The impact of cost on our career decisions is as follows:
- On the self-hosted side: Cost optimization typically involves hardware selection, energy efficiency, efficient use of system resources (cgroup limits, disk storage strategies), and workload planning. Those who specialize in this area can directly save companies operational expenses, which increases their value.
- On the cloud side: Cost optimization is built around writing efficient pipelines (avoiding unnecessary steps), caching strategies, selecting appropriate runner sizes, and understanding the discount models offered by the cloud provider. An engineer who can effectively manage cloud costs has a huge advantage in modern cloud environments.
In both cases, cost awareness and optimization capability are critical skills that increase an engineer's value. Whichever platform you choose, the principles of efficient resource utilization and avoiding waste always apply.
CI/CD Choices from a Security Perspective: Which Environment is More Educational?
CI/CD pipelines are one of the last stops before software reaches a production environment, so security is vital here. A misconfigured CI/CD environment can lead to a security vulnerability or even a complete system compromise. Let's look at what our choices in this area add to our careers from a security perspective.
Self-hosted CI/CD environments give you absolute control over security, but this control also means great responsibility. On one hand, you can implement strict security policies by performing VLAN segmentation in your own network, isolating runners, and defining firewall rules. Issues like blacklisting kernel modules (e.g., disabling modules like algif_aead to prevent a vulnerability like CVE-2026-31431), file integrity monitoring (auditd), and protection against SSH brute-force attacks with fail2ban patterns are directly your responsibility in self-hosted environments. This situation is a great opportunity to gain in-depth knowledge of security and develop practical hardening skills.
🔥 Self-Hosted Security Risks
Managing all security layers in a self-hosted environment requires extensive knowledge and attention. A small mistake can jeopardize the security of the entire system. Therefore, it is an area with a high need for security awareness and continuous updating.
Cloud-based CI/CD solutions, on the other hand, distribute security responsibility according to the shared responsibility model with the cloud provider. The provider is responsible for the security of the underlying infrastructure (physical servers, network), while you are responsible for the security of your own code, configurations, and pipelines. This means you need to learn the cloud platforms' IAM (Identity and Access Management) policies, secret management, security groups, and other platform-specific security features. For example, correctly managing a secret in GitHub Actions or minimizing the scope of authorization forms the basis of cloud security. Specializing in this area gives you the ability to effectively use modern cloud security practices and the tools offered by these platforms.
In my experience, both environments allow you to develop different security skills. Self-hosted environments make you much more proficient in "low-level" system security and network hardening, while cloud environments enable you to specialize in "high-level" platform security, identity management, and automation security. The best career strategy is to understand the fundamental security principles in both areas and develop the ability to choose the right tools based on the security level required by the project.
Challenges and Solutions I Encountered in Self-Hosted CI/CD Setup
A few years ago, I had a major headache setting up self-hosted GitLab Runners for the internal platform of a large Turkish e-commerce site. The project involved numerous microservices, each with its own dependencies. In the initial setup, it was difficult to ensure that the runners worked in an isolated manner for different projects on the same server. Especially in Docker-in-Docker (DinD) scenarios, I encountered resource conflicts and network isolation problems between containers.
The problem was that one project's build was consuming the resources of another project, leading to unpredictable build times and sometimes OOM-killed builds. Initially, I tried to manually adjust cgroup limits, but this became very cumbersome whenever a new project was added or dependencies changed. Additionally, different runners trying to use the same Docker cache also led to deadlocks.
As a solution, I first configured the concurrent limits and limit settings more carefully in the GitLab Runner configuration. Instead of having each runner use its own Docker daemon, I set up docker:dind services separately and had the runners connect to these services. However, the real solution was to define a separate runner tag for each microservice and assign each runner server to a specific microservice group. This way, resource-intensive services could run on their isolated runners, while others used lighter runners.
# /etc/gitlab-runner/config.toml
concurrent = 4 # Number of jobs to run concurrently
check_interval = 0
[[runners]]
name = "backend-runner-01"
url = "https://gitlab.example.com/"
token = "YOUR_GITLAB_RUNNER_TOKEN"
executor = "docker"
[runners.docker]
tls_verify = false
image = "docker:latest"
privileged = false # Tried to keep it false for security
disable_entrypoint_overwrite = false
oom_kill_disable = false
hostname = ""
# Define services for Docker-in-Docker
services = ["docker:dind"]
volumes = ["/var/run/docker.sock:/var/run/docker.sock", "/cache"]
# Resource limits
memory = "4g"
cpus = "2"
[runners.cache]
Type = "s3" # Or another cache backend
Shared = true
[runners.tags]
tags = ["backend", "production"]
With these adjustments, I reduced resource conflicts and made build times more predictable. This process taught me not only the in-depth configuration of GitLab Runner but also Docker's resource management and network isolation mechanisms very well. By learning from my mistakes, I gained the ability to diagnose and solve similar problems faster in the future.
Observed Limitations and Flexibility Constraints in Cloud CI/CD
While cloud-based CI/CD solutions are often touted as "do-it-all," in my experience, I've seen some limitations and flexibility constraints. These limits become more apparent, especially when you need very specific or low-level operations.
For example, when setting up the CI/CD pipeline for my Android spam application on GitHub Actions, I sometimes needed to compile native C/C++ code or run a custom hardware emulator. GitHub Actions' standard runners did not offer sufficient flexibility for such scenarios. It was impossible to load a custom kernel module or directly access a specific hardware interface. In this situation, I either had to completely change the workflow or delegate these steps to a self-hosted runner.
Another limitation is the risk of vendor lock-in. The deeper you integrate with a cloud provider, the harder it becomes to switch to another platform. GitHub Actions' custom actions, Azure DevOps' pipeline templates, or GitLab CI's custom include mechanisms tie you to a specific ecosystem. In one project, migrating our existing CI/CD pipeline from one cloud provider to another required more time and effort than expected because everything was written according to the provider's own custom DSL (Domain Specific Language).
ℹ️ Balance of Flexibility and Customization
Cloud CI/CD offers sufficient flexibility for most scenarios but can be limited for situations requiring extreme customization or low-level hardware/system access. In these cases, hybrid approaches or self-hosted runners come into play.
Cost can also become a constraint above a certain threshold. Cloud runners, initially attractive, can quickly become expensive for large-scale and frequent builds. Especially in projects with long-running integration tests or complex compilation processes, cloud bills can reach unexpected magnitudes. In such cases, I have to consider solutions like optimizing build times, developing smart caching strategies, or deploying cheaper self-hosted runners to reduce costs. So, while cloud solutions offer "just use it" convenience, they still require fine-tuning and strategic planning in real-world scenarios.
Conclusion
The answer to whether self-hosted CI/CD or cloud solutions are better depends on your career goals, the needs of the project you're working on, and your company's resources. If you want to delve into the depths of systems, specializing in areas like infrastructure management, network security, and performance optimization, self-hosted environments offer you a unique learning platform. This path prepares you more for SRE, infrastructure engineer, or security specialist roles.
On the other hand, if you want to focus on rapid delivery, modern DevOps practices, cloud platform APIs, and automation skills, cloud-based CI/CD solutions offer you a wider range. This path strengthens you more in DevOps engineer, cloud architect, or software developer roles.
In my experience, the most valuable engineers are those who are proficient in the fundamental principles of both worlds. Being able to set up a self-hosted Jenkins for one project while designing a fast deployment pipeline with GitHub Actions for another makes you versatile and adaptable. This adaptability is the most solid way to secure your career in this rapidly changing technological era. Remember, what's important is not blindly adhering to one technology, but having the wisdom to choose the right tool at the right time.
Top comments (2)
Fantastic, I can relate to this - except our systems are GitHub actions, GitHub runners and we have now a mix of GitHub hosted runners, AWS dedicated hosts (Mac) and EC2 for Linux, Windows and Roku devices. Our biggest expenses were for Mac - Mac is billed for 24hrs and Mac and Linux on GitHub runners were more expensive and slower. We decided to move bulk of our CI runners to AWS but this has pushed TCO (total cost of ownership higher) which was not as apparent earlier. We saw immediate monthly reduction in CI with AWS reserved pricing, getting beefier machines to run CI and some other workloads but AMI management, security patching, building our own load, business / weekday / holiday based scale up down etc. ended up costing for close to 8+ months during this time we had to build a fair bit of automation and this was ~3years ago pre-AI. As we have already got 2 big platforms setup in AWS for the company and more we inherited lots there with IAM, networking etc. It resonates with me as earlier in the cycle cost / month for CI and CI performance drove a decision with hidden TCO that slowly emerged over time. We learned a lot in last 3 years through this where in last 2-3 we would have to spend 1 engineer week / quarter for various updates, fixes and so on. Now with various AI backed automation this is now at ~1 engineer day / quarter and a large portion of it is mostly security patching and Apple Xcode updates.
Thanks for sharing!
Thanks for sharing your experience! I really liked your point about the hidden TCO. Monthly CI costs are easy to measure, but the operational costs—AMI maintenance, patching, scaling logic, and platform automation—often emerge much later. It’s also interesting to see how AI has shifted that balance by reducing routine operational effort. That’s a great example of why CI/CD decisions should be evaluated over years, not just the first month’s bill.