Silos and How They Impede Productivity

Have you ever started a new job and there were tickets waiting for you? I'm talking eight month old issues in the backlog that have multiple comments that increase in frustration, and then end with, "On hold until we hire a SME (Subject Matter Expert) to figure this stupid shit out."

I've walked into a pile of these, but this is one of my favourites. I'd been hired to manage some relationships, work on new initiatives, and do some infrastructure migrations. One particular ticket was for PHP and it throwing a WSOD (White Screen of Death). The engineers couldn't figure it out despite multiple attempts. They're smart folk, but they felt it was outside of their expertise. So this issue sat around and became stagnant.

Until I came along. I looked at the issue and replicated the steps. I got the same WSOD that they had been encountering. I was expecting that. The issue wasn't with PHP though. Given the locked down nature of the box, I could not access all of the logs that I needed to diagnose... but I had a suspicion.

I did the unthinkable: I talked to the folks in DevOps and Infrastructure. I asked them my nagging question, "Those environments you set up... are they also running a newer version of Apache?" (For my newcomers, Apache is a type of web server.)

"Yes."

Bingo bongo.

Access control for Apache changed between version 2.2 and 2.4.

• Apache 2.2 uses the authz_host_module to control access using directives like Deny, Allow, and Order.
• Apache 2.4 also uses the authz_host_module for access control, but adds the authz_core_module, which gave us the Require directive.

To summarize the issue: if you're trying to use Apache 2.2 access rules for an Apache 2.4+ server, you might have a bad time.

The solution was editing the .htaccess file so that it was in 2.4 format instead of 2.2. Multiple projects were stalled for over eight months because of an .htaccess file and no one thought to bother talking to folks outside of their team. What the hell.

My fix looked a little like this:

# Old: Apache 2.2
Order allow, deny
Allow from all

# New: Apache 2.4+
Require all granted

Allow all requests for Apache. Though if you're not certain what version you have in production, I recommend adding conditionals. <IfModule authz_core_module></IfModule> can save your tail.

A one-liner.

A fucking one-liner.

How much time had been lost because no one bothered to talk to one another? It's great to become a specialist in your chosen domain/language(s), but often the problems you encounter will lead you outside of your comfort zone. It's so easy to shrug it off as "not my problem."

That was one of the most satisfying issues I've ever closed. Not because it was technically complex, but because it was so obvious after one simple conversation. I do have a lot of server administration knowledge, so I don't want to discount that. I knew what to ask. But the thing is: the DevOps team had all of that knowledge. Not to mention that it would have been trivial to get them to look at the error logs.

Ask questions. Share your wealth of knowledge. Add more to your problem solving toolbelt. You might be holding you and your team back.

Comments

[MertSenel]

Hi Jodie, nice points I believe the the resistance comes from lack of transparency within the organisation.

Im sure your DevOps team also deploys the infrastructure in an automated fashion hence have the config as code stored in a repo.

If you had visilibility to the repository better yet their story boards or wikis etc someone from the other silo could look into.

Sometimes a Dev who is enthusiastic to talk servers are looked down as they are speaking out of their turn or sth. I love it when devs take full ownership and devops works better if everyone works for it.

I love the fact that all my changes can be trailed via either code history or pipeline logs, or last resort cloud peoviders logs. Takes away the lets blame the infra/environment element out.

Communication is key but I also wonder why application's infrastructure is maintained and provosioned by one team but configuration is provided by another? Seems like a bad practice to begin with.

How is this real DevOps besides from a name change?

[Jodie Struthers]

How is this real DevOps besides from a name change?

It wasn't, which was only one part of the problem. There was an issue with sysadmins/ops/SRE getting stretched thin, with DevOps itself splintered and not at all prioritized by the organization. I ended up filling part of that gap (mostly CI/CD), but was still under-resourced.

A lot of things could have been solved via more transparency, and by others sharing ownership. Eg. deconstructing responsibility silos; this entire issue could have been resolved sooner if the first developer had access to the right logs. Instead they had arbitrary logs and 1/4 of the picture. So it can be summarized as a lack/poor implementation of (real) DevOps.

I could have titled this "Why We Need Dedicated DevOps," haha.

[MertSenel]

I currently work in a company where they are practicing something called "embedded" DevOps where each product team has a dedicated DevOps resource (Im one of those) and there is also a central DevOps/ProdOps team that handles practice level stuff and help out the embbeded engineers as needed.

It has been working pretty well so far.