DEV Community: Hossein Zolfi

Your "DevOps Team" Is the Silo DevOps Was Supposed to Kill

Hossein Zolfi — Thu, 02 Jul 2026 09:55:28 +0000

If your org chart has a "DevOps team" sitting between the people who write the code and the people who run it, stop and look at what you actually built. You didn't add DevOps. You relabeled the old Ops silo with a trendier name and moved the same wall one desk over. The developers still throw work over it; it just has a nicer sign now.

That's not a nitpick about job titles. It's the single most common way organizations think they've adopted DevOps while structurally guaranteeing they haven't. So it's worth being precise about what the term actually means, what it doesn't, and why the mistakes cluster around the same handful of patterns.

The short answer

DevOps is the set of principles, practices, and cultural norms that treat building software and running it in production as one continuous responsibility, instead of two departments that hand work to each other and then argue about who broke what. The name is a portmanteau of Development and Operations, and the entire point of the term is to erase the seam between them.

That's the one-sentence version. Here's what it actually means day to day, why it works, and where it gets misapplied.

What DevOps is not

It helps to start with the myths, because they're where most organizations go wrong.

It's not a job title. Hiring a "DevOps engineer" and putting them between the developers and the infrastructure is usually just relabeling the old Ops silo with a trendier name. If one person or one team now owns "the DevOps stuff," you've reconstructed the wall you were trying to tear down. The goal isn't a new specialist role; it's that the people who build a thing also carry responsibility for how it behaves in production.

It's not a tool or a tool stack. Kubernetes, Terraform, Jenkins, Docker: none of these make an organization "do DevOps." Large-scale research comparing delivery performance across companies has found almost no correlation between the type of system (legacy mainframe vs. greenfield microservices) and how well teams deliver software. The newest, trendiest architecture is no guarantee of anything if the underlying coupling and process problems aren't solved. What predicts performance is whether a team can test and deploy its own service independently, without waiting on another team's calendar; not which vendor's logo is on the yaml file.

It's not the absence of change control. A common misreading is that DevOps means "move fast, skip the approvals." In practice, the highest-performing teams still gate every change: they just replace a slow, low-context external review board with fast, high-context peer review plus an automated pipeline that runs the same checks on every single change, every time. That combination has been shown to outperform traditional approval boards on both speed and stability. Removing a bottleneck isn't the same as removing a check.

It's not confined to engineering. The deepest version of the idea is that technology stopped being a department a business can delegate and forget. It's more like literacy or basic math, a competency every part of an organization now needs, because when the systems fail, the business fails with them, regardless of which org chart box the systems lived in.

The core idea, in one framework: flow, feedback, learning

Most of what gets called "DevOps practice" traces back to three underlying principles. They were originally articulated for factory production lines and manufacturing flow, then translated into knowledge work, and they hold up as a mental model no matter which specific tools or ceremonies an organization uses.

1. Optimize the whole flow, not the local parts. A team that maximizes its own throughput while creating a pile of unfinished, unintegrated work for the next team downstream hasn't made anything faster. It's just moved the waiting somewhere less visible. The unit of optimization has to be the entire path from an idea to a customer actually using it, not any one stage of that path. In practice this means making all work visible, keeping batch sizes small so nothing sits half-finished for weeks, and relentlessly finding whatever single step is truly constraining the whole system's throughput, because improving anything except that bottleneck is, provably, wasted effort. If your database migrations take three weeks to provision an environment, shipping code twice as fast doesn't help; the environment is the bottleneck, and it needs to be fixed first.

2. Make problems visible immediately, and stop to fix them. In a complex system, failure is not a possibility to be prevented once and forgotten: it's a constant, ambient condition to be managed. The healthy response, borrowed from manufacturing floors, is a literal or figurative cord anyone can pull: when something breaks, the team stops taking on new work and swarms the problem until it's understood and contained, rather than patching around it and moving on. This feels expensive in the moment and is cheap, because the alternative is letting small, easily-diagnosed problems compound into large, hard-to-diagnose ones. A broken build that nobody fixes for a day doesn't stay a one-line bug; it becomes three unrelated changes tangled together, each hiding the others' effects.

3. Build a culture where learning beats blame. The first two principles only function if people are willing to surface problems honestly, and that requires a culture where reporting a mistake doesn't end a career. This isn't a soft, optional add-on: it's the load-bearing precondition for the other two. A team that hides failures to protect itself politically cannot make problems visible, and a team that gets punished for pulling the stop cord will stop pulling it. The counterintuitive part is how you get this culture: not by hanging a poster of values in the break room or running a workshop, but by changing what people do. Adopt blameless postmortems, protect time for fixing root causes instead of just symptoms, and the trust follows the behavior. You don't need to fix people's beliefs first.

What this looks like in practice

The principles above cash out into a fairly concrete set of practices. None of them is DevOps by itself; together, they're what makes the principles real instead of aspirational.

Continuous delivery. The ability to get any change, such as a feature, a config tweak, or a one-line bug fix, into production safely and quickly enough that releasing becomes routine rather than a quarterly event everyone dreads. This rests on automated testing that runs on every change, developers integrating their work into a shared mainline at least daily instead of working in long-lived branches, and treating a red build as an emergency, not background noise.
Loosely coupled architecture. However a system is decomposed, whether services, modules, or a well-organized monolith, the property that actually matters is whether a team can change and ship its piece without needing permission or fine-grained coordination from every other team. This is consistently the single largest lever on delivery speed, larger than any specific automation tooling. Using "microservices" doesn't automatically buy this: services that all read and write the same shared data are just a monolith with extra network calls and extra latency.
Infrastructure as code. Defining servers, networks, and configuration as version-controlled, reviewable, testable code instead of manual console clicks or one-off scripts. This turns infrastructure changes into the same kind of reviewable, revertible, auditable artifact that application code already is.
Deployment pipelines. An automated, multi-stage path that every change travels from commit to production, running increasingly rigorous checks at each stage. This is where governance actually lives in a mature setup: not in a meeting where a committee eyeballs a spreadsheet, but in code that runs identically, every time, on every change.
Visible work and limited work-in-progress. Dashboards and boards that show what's actually in flight, combined with hard limits on how much can be in flight at once. Counterintuitively, limiting work-in-progress increases throughput, because a system running at 100% utilization has infinite queueing time in front of it.
Security built in, not bolted on. Security review and automated scanning integrated into the same pipeline as everything else, rather than a separate gate that shows up right before launch with a list of findings nobody has time to fix. Teams that do this consistently show both faster delivery and fewer security incidents: it isn't a speed-versus-safety trade-off, it's the same "catch it early, catch it cheap" logic applied to a different category of defect.

It's measurable, and the measurements are counterintuitive

One of the things that separates this body of practice from a lot of management fashion is that it has been rigorously studied at scale, across thousands of organizations, using validated survey instruments rather than anecdote. Four metrics have proven to reliably distinguish high- from low-performing organizations:

Lead time: how long from a change being committed to it running in production.
Deployment frequency: how often changes reach production.
Time to restore service: how quickly the team recovers when something breaks.
Change failure rate: what fraction of changes cause a problem requiring a fix.

The first two describe tempo; the last two describe stability. The finding that keeps getting rediscovered and keeps surprising people is that these are not in tension. Organizations that are fast are also stable, and organizations that are slow are also unstable: there's no meaningful cluster of "fast but reckless" or "slow but rock-solid" teams. The old assumption that you must trade velocity for safety, and that a change-approval board and a slow release cadence buy you reliability, doesn't hold up against the data. What actually buys reliability is small changes, fast feedback, and automated verification, the same things that buy speed.

A related finding worth internalizing: don't try to measure your way to a "mature" end state. A maturity model implies a fixed ladder everyone climbs in the same order to reach a finish line called "done." A capability model instead treats improvement as continuous and context-dependent, where different teams need different levers depending on where their constraint actually is, and the bar for what counts as "good" keeps rising as the whole industry improves. There is no finish line, and pretending there is one is itself a source of complacency.

How it relates to nearby ideas

A few terms get used interchangeably with DevOps that are actually more specific applications of it, worth distinguishing:

Site Reliability Engineering is best understood as one particular, unusually rigorous implementation of the same underlying principles, with its own vocabulary: an error budget (the acceptable amount of unreliability over a time window, spent deliberately to buy release velocity) turns "how reliable should this be" from a political argument into arithmetic.
Platform engineering is the organizational answer to the question "what happens once DevOps needs to scale past a single team?" If every team owns everything end to end, every team eventually re-solves the same infrastructure problems badly. A platform team's job is to absorb that shared complexity behind a self-service interface, run as a product, with usability, a roadmap, and real users, not as a pile of internal tools nobody asked for. Done badly, a platform team becomes exactly the bottleneck DevOps was trying to remove; the discipline that keeps it from happening is building the smallest platform that actually helps, and no bigger.
DevSecOps extends the same "build it in, don't bolt it on" logic to a third party at the table besides Dev and Ops: security. Same argument, same evidence pattern, one more silo dissolved.
Agile is a close cousin but not a synonym. Agile's classic feedback loop mostly stops at "code complete": did we build the right thing, are we iterating on the backlog. DevOps picks up exactly where that loop stops and extends it through deployment and into production: does the thing actually work when real users touch it, and can we find out fast enough to matter.

Common failure modes

A few patterns show up often enough to be worth naming directly.

Renaming the silo instead of removing it. This is the one we opened with: creating a "DevOps team" that developers throw work over the wall to is the antipattern the whole movement exists to prevent, wearing the movement's own name as camouflage. It's the single most common way an organization can point at a team, a title, and a Slack channel and genuinely believe it has "done DevOps" while the actual handoff problem is untouched.

The rest of the failure modes tend to follow from the same root cause, treating DevOps as something you install rather than something you change:

Buying tools before fixing coupling. An organization with a tightly coupled architecture that adopts container orchestration and calls it done will find its deployment cadence barely moves, because the actual constraint, teams that can't ship without coordinating with three other teams, was never touched.
Treating the four metrics as a stick. The moment delivery metrics get used to rank individuals or teams punitively, people start gaming them, and the numbers stop reflecting reality. They're diagnostic instruments for finding where to invest, not a scoreboard for performance reviews.
Over-building the platform. Left unchecked, engineers building an internal platform will keep bolting on shelves nobody asked for, because building infrastructure scratches an itch that shipping features never quite does. The discipline is to build the thinnest thing that actually removes friction, and grow it only when a real, felt need shows up.
Skipping the culture work because it's the hardest to measure. It's tempting to adopt the tooling and the pipeline and skip the harder, fuzzier culture piece. But the pipeline only works if people trust it enough to let a red build actually stop them, and that trust is exactly what a blame-first culture destroys.

The steel-manned objection: isn't specialization just how orgs scale?

Here's the strongest version of the counter-argument, and it deserves a real answer rather than a strawman: don't organizations specialize as they grow precisely because generalists don't scale? Nobody expects every engineer to also be a security expert or a database administrator: those are dedicated teams, and nobody calls that a silo problem. Why should "runs the infrastructure" be any different? A dedicated platform or DevOps team could just be sane division of labor, not a regression.

That argument is genuinely right: under one specific condition. It wins when the specialist team operates as a self-service platform that other teams pull from on their own schedule, the same way a team pulls in a well-documented open-source library: no ticket, no queue, no waiting for someone else's sprint. It loses the moment that team becomes a gate other teams must line up behind to ship anything: a mandatory review, a provisioning request, a "file a ticket and we'll get to it." The difference isn't the org chart box the specialists sit in; it's whether using their work requires asking permission. A security team that ships a self-serve scanning library is fine. A security team that must personally approve every deploy is the silo again, just wearing a different department's name.

The underlying bet

Strip away the vocabulary and DevOps is a bet about where value gets created: not in the moment code is written, but in the moment it's running, serving real users, generating real feedback. Everything else, the pipelines, the metrics, the culture work, the org design, exists in service of shortening the distance between "we had an idea" and "we know whether the idea was any good," as many times a day as the organization can stand. Speed and safety turn out not to be opposite ends of a dial you have to choose a position on; they're both downstream of the same thing, which is how tight that feedback loop actually is.

So if you're looking at your own org chart right now: is your "DevOps team" a self-service platform other teams pull from freely, or is it a queue they wait in? That answer tells you more about whether you've actually adopted DevOps than any tool in your stack will.

Look at your own org chart: is your platform team something other teams pull from freely, or something they queue behind? Drop a comment. I especially want to hear the queue stories, those are the ones worth fixing.

The Hidden Tax You Are Paying Every Day: A Dev's Journey into Ops Automation

Hossein Zolfi — Sun, 23 Nov 2025 05:09:02 +0000

Starting a new career is kinda like walking into a workshop full of tools you've never seen before. You know they're useful, maybe even powerful, but honestly... you're not totally sure what they do or whether you'll break something the moment you touch them. That's me right now. Everything feels new. And confusing. And fun. Sometimes all at once.

This article is part of a series where I'm basically thinking out loud about the stuff I'm learning. I want my future self to read these later and go like: "Oh wow, that's where I started." Or maybe he'll laugh at me. Either way, it's worth documenting.

Along the way I ran into a bunch of questions. Some I answered. Some I couldn't. Some probably don't even have a real answer. I'm gonna put them here anyway because they're part of the journey.

Step 1: Identify the Problem (aka "Why am I doing this manually??")

One of the things I'm struggling with is: how do I share operations with my future self? Suppose I want to set up a client on a machine that wants to ship logs to external servers, how should I do it? How should I remember what I've done before? Reading documentation or command history to find out what I did before? Seriously. What about urgent situations: What happens if we lose a VM? How do I know what's even on these machines?

💰 The CEO/Manager ROI Corner

If you are a manager reading this, here is the math: Manual ops is a hidden tax. Every hour I spend SSH-ing into a box to fix a config is an hour I am not building features. Plus, manual changes mean human error, which means downtime risk. Automation isn't just "cool", it is risk management.

Coming back to the ops world after doing dev work, my first thought when I saw these GCP instances was: how do I get rid of manual work.

Throughout my entire life as a software engineer, I've never been able to persuade myself to do repetitive work manually. I just... I can't. I hate it. If I have to do the same clicks and commands more than twice, I literally start to feel dizziness. It just feels wrong.

This is basically the DRY Principle (Don't Repeat Yourself) applied to infrastructure.

Every piece of knowledge must have a single, unambiguous, authoritative representation within a system

To save time in these kinds of situations the obvious answer is docs. Notion, Confluence, Google Docs, whatever. But... why? Just to tell other people (technical or not) what's installed? And how I set it up? Especially since not everyone can just ssh in and check.

That just doesn't work for me. We're software developers. We're supposed to like code, right? We live in it. And for me, running a command from a script is always safer and faster than typing it manually. If someone else is doing this manual work, it isn't a reason I should do it. So in the middle of my work and tasks, I explore my options.

Step 2: Exploring Options

The real answer, for me, has to involve Git. A repo.

My mind immediately went to Ansible. But I did pause for a second to think about the other players in the room.

Ansible is open-source, command-line IT automation software written in Python. It can configure systems, deploy software, and orchestrate advanced workflows to support application deployment, system updates, and more.

That's exactly what I need.

Here’s a quick mental matrix I went through:

Tool	The Vibe	Why I picked (or didn't pick) it
Terraform	The "Industry Standard" for infra. Great for creating VMs, less great for configuring inside them.	Overkill for right now. I have the VMs; I need to configure the OS. The learning curve is steep when I just need to install Docker.
Pulumi	"Infrastructure as Code" but actual code (TS/Python).	Super cool, but adds complexity I don't need yet. I want simple config files, not a compile step.
Ansible	The "Old Reliable". Agentless, runs over SSH.	Winner. It uses YAML, handles server configuration perfectly, and I already know the basics.

Decision Point: Why Ansible?

To be honest, it's just what I know. I used it years ago, so I'm familiar with it. My focus right now wasn’t to learn a brand-new provisioning tool. That just wasn't aligned with my career goals at this moment.

Note: When you are a solo dev or a small team, Velocity > Perfection. Always.

So, good or not, I chose Ansible. It's my baseline. It'll tell me what I set up, and it'll help me install services in minutes instead of hours (like webhooks or configuring logs for Loki — should that take hours? No. But the context-switching kills you). And we can place the configuration in code and commit it in a Git repo.

Implementation: OK, let's do this

So, I chose Ansible. Now. How do I use it to provision and configure machines in GCP?

This is where it got weird. The GCP infra is really different from what I'm used to. The network access is super abstract and hidden. The OS runs differently (it has agents! that configure things from the web console!).

And everything is about regions and zones. You've seen the docs:

Regions are independent geographic areas that consist of zones. A zone is a deployment area within a region.

Yada yada. Why do I care? Cost.

This stuff isn't free. If you're self-hosting, you just stack your servers in one datacenter. In GCP, if you have a machine in Europe and one in North America, you're paying for that data transfer (like $0.05 per GB).

Even external IPs cost money! A static IP in Belgium is $3.65/month. It's not much, but for 10 VMs? That adds up. Even if the machine has an external IP to connect, its IP address can be ephemeral! It means if a machine stops and starts (not restarts) GCP will assign a new external IP address. Just wanting to have an external IP doesn't mean I should reserve one.

This leads to a simple conclusion:

If you don't need an external IP, DON'T ASSIGN ONE. It's just wasted money.

Step 3: The Actual Problem

Okay, so if I follow my own advice and don't assign an external IP how the hell does Ansible connect to it? Ansible needs an IP in its inventory file. Where can I find the IP address? Am I wrong to choose Ansible as my provisioning tool? So I have Ansible — how can I configure machines with it?

This is where the gcloud command comes in. GCP gives you this tool to manage everything. You can connect to a machine with it even without a public IP:

gcloud compute ssh demo --zone=europe-north1-a --project=my-project

It's basically just magic. It tunnels through or something. I don't care how right now, just that it works.

So, if gcloud can ssh maybe Ansible can use gcloud instead of ssh?

The Solution: A Wrapper Script

The idea: we tell Ansible:

"When you want to ssh, run this script instead."

ansible.cfg:

[ssh_connection]
ssh_executable = gcp-ssh-wrapper.sh

Now we need to build gcp-ssh-wrapper.sh.

The script receives all the arguments Ansible passes to SSH and instead passes them to gcloud compute ssh.

The core looks like:

exec gcloud compute ssh "${opts[@]}" "${host}" \
  --tunnel-through-iap \
  --no-user-output-enabled \
  --zone="${zone}" \
  --project="${project}" \
  -- -C "${cmd}"

To get the host and command:

host="${@: -2:1}"
cmd="${@: -1:1}"

To collect SSH options:

declare -a opts
for ssh_arg in "${@:1:$#-3}"; do
  if [[ "${ssh_arg}" == --* ]]; then
    opts+=("${ssh_arg}")
  fi
done

But... wait. gcloud needs zone and project. Ansible doesn't know about them. So the script has to ask Ansible:

zone=$(ansible-inventory -i "${inventory_path}" --host "${host}" |
    jq -r '.zone')
project=$(ansible-inventory -i "${inventory_path}" --host "${host}" |
    jq -r '.project')

💾 Grab the Code
Instead of copy-pasting the snippets above and hoping for the best, I've uploaded the final, working gcp-ssh-wrapper.sh script to this GitHub Gist. Download it, chmod +x it, and you are good to go.

The Inventory

# inventory/inventory.ini
[project1]
demo zone=europe-north1-a

[project1:vars]
project=my-project

Now it works.

Let's test it

Playbook:

---
- name: Configure Docker logging
  hosts: all
  become: true
  gather_facts: true

  tasks:
    - name: Template Docker daemon configuration
      template:
        src: templates/daemon.json.j2
        dest: /etc/docker/daemon.json
        owner: root
        group: root
        mode: '0644'
      notify: Restart Docker
      tags: docker

  handlers:
    - name: Restart Docker
      service:
        name: docker
        state: restarted

Template:

{
{% if loki_enabled %}
  "log-driver": "loki",
  "log-opts": {
    "loki-url": "{{ loki_url }}",
    "loki-batch-size": "400"
  }
{% else %}
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "300m",
    "max-file": "2"
  }
{% endif %}
}

Vars:

---
loki_enabled: true
loki_url: "http://10.166.0.20:3100/loki/api/v1/push"

Run:

ansible-playbook -i inventory/ --diff -l demo playbook_docker.yml

Important Note on Refinement

Before you copy-paste this into production and yell at me: I know this is a Proof of Concept.

I'm aware that querying ansible-inventory twice per connection is a performance bottleneck. I also know a true "Enterprise" setup would use the official google.cloud.gcp_compute plugin for dynamic inventory and leverage Service Accounts for CI/CD.

However, this approach trades 'Best Practice' complexity for immediate velocity. When you are a solo dev just trying to ship features, that is a trade worth making.

That said, here is my Technical Debt list to tackle later:

Speed: Caching the inventory lookups to stop spawning processes.
Scale: Handling multi-region/project setups without hardcoded zones.
Resilience: Better error handling for when gcloud hiccups.

Final Thoughts

So, the job is... done? Kinda.

I'm left with a few more questions to explore in the next part of this series:

The Magic Tunnel: How exactly does IAP (Identity-Aware Proxy) work under the hood?
Name Collisions: What if two machines in different projects share the same name?
The "Cloudflare" Layer: There are some blackbox areas regarding how this interacts with Cloudflare that I need to document.

Anyway, that's where I am. It's a hack, but it's my hack, and it works.

This article took nearly a month to write, test, and clean up. I'm really glad I started using Ansible. it's solving problems I didn't even know I had. If you've struggled with similar manual-ops dizziness, let me know in the comments.

Too Much Talent: Why Building a Great Team Isn't Just About Seniority

Hossein Zolfi — Tue, 13 May 2025 11:35:01 +0000

From Stars to Team

Before I became an engineering manganager and when I was just reading software engineering books, I believed the surest path to success was hiring as many top-tier engineers as possible. My logic was simple: great software design comes from great engineers. A colleague once told me, "Hire stars—they'll build game-changing features!" And often, they did. These engineers introduced powerful designs, elegant abstractions, and impressive capabilities that elevated our products.

But over time, I saw the limits of that approach. Debates dragged on. Unpopular tasks went unowned. Engineers left because they felt underutilized. After reflecting on what I saw in my own teams—and what I learned from others—I realized:

Focusing solely on individual brilliance—even when it delivers great features—often sacrifices teamwork, consistency, and long-term growth.

The real magic happens when the team works as a cohesive unit. The goal isn't to collect stars—it's to build a constellation.

When building teams—whether in sports or software—the instinct is to stack the roster with as much talent as possible. More senior engineers, more superstars, more wins, right? Not always. Research and real-world experience suggest a more nuanced truth: too much talent can actually hinder performance. Even when individual engineers introduce brilliant features, the lack of balance and collaboration can slow the team down. Here's why—and how to build a team that thrives.

The Research: Talent Has Diminishing Returns

A 2014 study by Swaab, Schaerer, Anicich, and Ronay analyzed highly interdependent team sports like soccer and basketball. They found that adding talent improves performance—up to a point. Beyond that, performance suffers. Too many stars can lead to coordination breakdowns, ego clashes, and poor collective results.

This challenges the assumption that talent and output scale linearly. In interdependent teams—like most software engineering groups—excessive star power can actually disrupt the very collaboration needed to succeed.

Key takeaway: Talent drives performance, but only if it doesn't undermine teamwork.

Why This Matters for Software Teams

Software engineering is a team sport. Engineers must align on architecture, review each other's work, and ship features together. A team made up solely of senior engineers or high-performers can run into serious problems:

Decision Gridlock: Strong opinions may clash over tech choices or system design, delaying progress.
Neglected Execution: Critical but unglamorous tasks—like CI/CD setup, writing documentation, or fixing edge-case bugs—may get ignored.
Leadership Friction: Too many leaders without clear roles often compete instead of collaborate.
Blocked Growth: Without mid-level or junior engineers, there's no natural path for mentorship or long-term team development.

Even when talented engineers ship great features, a lack of cohesion or clarity in roles can erode team efficiency and morale.

Real-World Examples: Talent Alone Isn't Enough

Google's Early Struggles

In the early 2010s, Google's engineering teams were packed with elite talent. Yet, some teams struggled to ship:

Endless architecture debates delayed progress.
Glue work like documentation and infrastructure lagged behind.
Junior engineers felt sidelined, which hurt morale and retention.

Google eventually recognized that technical brilliance wasn't enough. They began emphasizing soft skills—collaboration, humility, team orientation—in hiring and promotions. They also formalized the role of the Tech Lead to create alignment and speed up decisions.

Node.js and Open Source Governance

Node.js, in its early days, suffered from too many strong voices pushing in different directions. Conflicts and lack of governance stalled progress. The project rebounded only after implementing clearer roles and decision-making processes. This empowered contributors at all levels and helped the project scale.

These examples show that raw talent is not a silver bullet. Without structure and shared purpose, even the best engineers can struggle to work effectively together.

The Role of Tech Leads in Balancing Talent

A well-balanced team still needs guidance. That's where Tech Leads come in. The Tech Lead isn't the smartest person in the room—they're the one who keeps the room aligned.

Before I became a team lead, I didn't fully understand this role. In my past companies, the Tech Lead was treated as the boss—someone who went to meetings, made decisions alone, and became disconnected from the team. Often, these leads lacked strong software instincts and sometimes introduced friction with senior engineers. This was far from the kind of leadership that fosters a healthy engineering culture.

Now I realize:

A Tech Lead is not a boss. A good Tech Lead is a coordinator and a listener.

The role is about creating alignment—not control. It's about ensuring that everyone contributes, conflicts are resolved constructively, and decisions move forward without dragging the team down.

An effective Tech Lead:

Guides Without Dominating: They set direction while making space for team input.
Resolves Conflict: They facilitate decisions so the team doesn't stall.
Leverages All Levels: They ensure seniors mentor, mid-levels execute, and juniors grow.
Focuses on Outcomes: They prioritize what's best for the product—not their own technical legacy.

Spotify's Squad Leads operate in this mold, helping cross-functional teams stay aligned while maintaining speed and cohesion.

Without a Tech Lead—or with one driven by ego—teams of superstars often descend into dysfunction.

How to Build a Balanced Team: Practical Tips

If talent alone isn't the answer, what is? Here are some actionable ways to build a high-performing team:

Hire T-Shaped Engineers: Seek people with deep expertise in one area, but enough breadth to collaborate across functions.
Assess Gaps Before Hiring: Need innovation? Hire a senior designer or architect. Need delivery? Mid-levels are crucial. Need culture? Add a mentor.
Prioritize Soft Skills: Interview for humility, communication, and collaboration—not just algorithms and system design.
Foster Mentorship: Pair seniors with juniors. This builds culture and spreads knowledge.
Empower a Tech Lead: Choose someone who values team success over individual heroics. Support their growth with training in facilitation and leadership.

A Fun Metaphor: The LeBron James Problem

You might call this the LeBron James Problem:

You'll wait forever trying to hire three LeBron Jameses—they're rare and expensive.
Even if you succeed, they may clash instead of cooperate.

Software teams aren't basketball fantasy drafts. Obsessing over rockstar hires wastes time and breeds dysfunction. A team of role players with clear leadership often outperforms a group of misaligned all-stars.

Final Thoughts: Build a Constellation, Not a Cluster of Stars

Talent absolutely matters. Superstar engineers can introduce groundbreaking features. But once you reach a certain threshold, adding more talent without balancing the team dynamic can backfire.

To build a truly great team:

Seek balance: Mix senior, mid-level, and junior engineers.
Prioritize teamwork: Hire for compatibility as well as capability.
Support leadership: Use Tech Leads to align the team and resolve friction.
Value the whole: Celebrate team outcomes over individual accomplishments.

Don't build a cluster of lone stars. Build a constellation—where everyone shines together.

Remote debugging Go App

Hossein Zolfi — Sat, 12 Oct 2024 13:59:06 +0000

For the longest time, I wasn’t a fan of debuggers. Coming from a background in Spring Framework, Java, Python, and PHP (Symfony/Laravel), I always found logs and traces more reliable for debugging. I had even dabbled with GDB in the early days, but it didn’t stick. Instead, I relied on logging to figure out what my applications were doing.

However, my perspective changed when I started working with Go. While developing in Kubernetes and working on microservices, I faced a situation where the complexity of the service made logging and tests insufficient. This was a large service with multiple dependencies, serving both users and operators, making bug fixes particularly challenging.

Let me share a recent experience where using a debugger saved me a significant amount of time.

Debugging in Kubernetes: A Real-World Example

A few weeks ago, I was developing a feature that affected multiple parts of a microservice running in Kubernetes. This service was being called by several other components (like the app and admin panels), and I needed to trace what was happening inside the service, step by step. Tests couldn’t cover everything, especially when I needed to ensure that certain flags were set correctly to prevent sending multiple SMS notifications to users.

I could have used logs to trace the service, but every time I missed an entry, I would have had to add a new one, push the code, and wait for the CI pipeline to complete—wasting 3-5 minutes each time just to deploy to staging.

Instead, I decided to try using a debugger. Here’s how I set it up.

Setting Up Delve Debugger in a Kubernetes Pod

After a lot of trial and error, I found that adding the following lines to the Dockerfile allowed me to run Delve (a debugger for Go) inside the Kubernetes Pod:

ENTRYPOINT ["/bin/sh", "-c", "/go/bin/dlv --listen=127.0.0.1:8001 --headless=true --api-version=2 --only-same-user=false exec /path/to/exec"]

To install Delve, I added this line to the Dockerfile:

RUN go install github.com/go-delve/delve/cmd/dlv@v1.23.0

Next, I compiled the Go application with specific flags to disable optimizations and inlining:

go build -gcflags "all=-N -l"

Connecting to the Debugger

Once the service was deployed, I used port forwarding to connect to the debugger. I added the following port configuration to the Kubernetes manifest:

ports:
  - name: dlv
    containerPort: 8081

Then, I forwarded the port to my local machine:

oc port-forward $(oc get pods -l app=LABEL -o name | head -n 1) 8001:8001

Now I could connect to the debugger from my local machine using:

dlv connect :8001

Setting Breakpoints and Configuring Paths

With the debugger connected, I set breakpoints, such as:

(dlv) break main.main

Because the service was built in a different environment (GitLab’s pipeline), the paths didn’t match my local setup. To solve this, I used substitute-path to map the paths:

(dlv) config substitute-path /usr/local/go/ /Users/hossein/sdk/go1.23.1/

Running these commands every time was tedious, so I created a script (dlv.init) to automate the setup:

dlv connect :8001 --init dlv.init

Explanation of Key Commands

Delve Debugger Command:

dlv --listen=127.0.0.1:8001 --headless=true --api-version=2 --only-same-user=false exec /path/to/exec

--listen: Opens the debugger on 127.0.0.1:8001 for remote connections.
--headless=true: Runs Delve without an interactive UI, suitable for remote debugging.
--api-version=2: Uses API version 2 for better tool compatibility.
--only-same-user=false: Allows users other than the process owner to connect.
exec /path/to/exec: Starts or attaches to the Go executable for debugging.

This setup allows remote debugging of a Go service in a Kubernetes environment.

Go Build Command:

go build -gcflags "all=-N -l"

-N: Disables optimizations.
-l: Prevents inlining of functions.

These flags ensure the code stays closer to the source, making it easier to debug by allowing you to step through every function.

Example Usage

To demonstrate how to debug a remote service, I use a local Docker container for simplicity. However, in practice, this service would be deployed on Kubernetes.

To build the Docker image, run the following command:

make build

This command will build a Docker image. Once built, you can start the service by running:

make start

Output:

$ make start
docker run -it -p 8001:8001 -p 8080:8080 --rm my-app
API server listening at: [::]:8001
2024-10-12T13:48:31Z warning layer=rpc Listening for remote connections (connections are not authenticated nor encrypted)

The prompt shows that the debugger is running. To attach the debugger to the remote service, run:

make debug

Output:

$ make debug
bash -c "dlv connect :8001 --init <(sed 's|PWD|'`pwd`'|g; s|HOME|'$HOME'|g' dlv.init)"
Type 'help' for list of commands.
Breakpoint 1 set at 0x2188cc for main.main.func1() ./main.go:9
(dlv) c

Once attached, send the c command to continue execution. Then, send a request to the service using the following:

make send-request

The debugger will show output like this:

> [Breakpoint 1] main.main.func1() ./main.go:9 (hits goroutine(17):1 total:1) (PC: 0x2188cc)
Warning: debugging optimized function
Warning: listing may not match stale executable
     4:     "fmt"
     5:     "net/http"
     6: )
     7:
     8: func main() {
=>   9:     http.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
    10:         fmt.Fprintf(w, "Hello, World!")
    11:     })
    12:
    13:     http.ListenAndServe(":8080", nil)
    14: }

At this point, you can manually debug the service. For example, to inspect the HTTP method, use:

(dlv) p r.Method
"GET"

Once troubleshooting is complete, use the c command to continue and the client will receive the output.

Conclusion

Using a debugger in this case allowed me to step through the code and understand the service's behavior without constantly pushing new logging code. It was a game-changer, especially when working with complex microservices in Kubernetes. Debugging Go in production environments can be daunting, but with the right setup, it can save you a lot of time and frustration.

You can check out the full example of this setup on my GitHub.