DEV Community

Jonathan Eccker for DealerOn Dev

Posted on

Why Agile Frameworks Fail - A Psychological Analysis

I would like to paint a picture. If it's familiar, keep reading. If it sounds outlandish and not relatable, this is not the article for you.

You have a small company.

You are winging it from day to day, adapting and iterating with no, or minimal, process.

Everyone talks out loud of process, of predictability, of structure they will eventually implement. But your small team pushes forward with late hours, big over-the-weekend features that come with incremental wins, and while everyone is exhausted, the sentimentality is "at least we don't have big company culture". The entire team is vastly intelligent, capable to find patterns and angles on problem spaces that many others can't see.

Innovation is built in at every step. It's a core value, as is similar mantras like "have fun" and "employees first" (even though not a single person in the team knows how to maintain "work-life" balance).

You become successful.

You either get a big client or you reach a tipping point on income that allows you to grow ambitious and plan further than one month ahead.

So you grow.

You adopt a framework, the structure you always dreamed of. The word "Agile" got used somewhere in the description, so you call the framework "Agile". "We are doing Agile now."

And you slow.

Some of the former powerhouses shift into leadership. Some of the powerhouses remain attached to fast-moving projects that "must happen". And one or two others move on, attached to "small company culture". The most vocal quit, either attached to "small company culture" or feeling generally not enabled to provide the value they used to. It's a huge loss as they were specialists on technology or a specific vertical.

But that's part of the cost of growth, right?

You keep growing.

You go through a V2 of your "Agile" process that really pins down your process to make sure "quality is baked in at every step".

To some extent, the "Agile" framework helps. You are able to sustain multiple teams, and as you start measuring velocity, you get an understanding of effort-to-value ratios.

But most of the engineers and some of the management feel like they are focusing on metrics rather than deliverables.

Why are they so caught up with the numbers? They are meant to just be validation, not the product.

Everyone in leadership says that, but when it comes to getting things done, it's always about the metrics.

Do the numbers themselves look right? You aren't sure.

You most certainly don't feel like you're getting as much done as you had before.

What the Hell Happened?

You switched the motivational framework.

In an unstructured environment, especially in Engineering, motivations come from unsolved, complex problems, novelty, urgency, and personal interest. Responsibility is a byproduct or guiding factor, not a driver.

Throw those motivation factors into a Google search.

I'll wait.

Additionally, in small companies, there is a lot of agency and freedom to tackle problems that require unrefined requirements to be reduced to logical, repeatable patterns that are represented in a vastly complex system.

Throw "individuals with advanced pattern recognition and rules extrapolation and a pathological need for agency" into that search.

Wait, my Team Isn't ADHD or Autistic

It's entirely possible.

But the line for the two diagnoses revolves around "disability", which, by the social model, is scoped to the relationship between the individual and the environment, not the individual itself. In the right environment, this means symptoms of the disability can (depending on the severity or type of disability) become completely transparent or seem like "just part of the job".

It's basically accidental masking by being successful, and it is fairly common.

I have, personally, become fairly convinced that there exists an alarmingly high amount of undiagnosed ADHD and Autistic Leadership in Engineering that are unintentionally masking simply by having found the right environment to fit their motivational framework. This is partly informed by many tech giants speculating that they would likely have been diagnosed as Autistic when younger (see: Bill Gates).

On that point, it's worth taking a moment to note that this article exists in the space of "Lived Experience" (community discussions/personal experience) and "Emerging Theory" (professional speculation). "Institutional Knowledge" (DSM-5) is far behind the active discussion space (by design).

To give an idea of just how far behind institutional knowledge is, it was just a bit more than a decade ago that the DSM recognized that Autism and ADHD could both be diagnosable in the same individual, and there's now an estimated ~50% co-diagnosis rate.

The point being that discussions on neurodivergence are moving fast.

What I can confidently say is:
1) Most everyone already knew the Engineering world had a high ratio of neurodivergent representation. There exist studies on this already.
2) Society is going through an unprecedented wave of demasking and diagnosis. (If you're successful in Engineering, I suggest taking this test https://raads-r.net/ if you have some time to kill)
3) Even for those not severe enough to be diagnosed, Autistic/ADHD traits are present across "neurotypical" society. Those traits are going to be very common in engineering, which requires non-linear and highly cognitive processing. (search: Autistic traits continuous in general population).

How Does this Relate to Agile?

Great guiding question, me!

I would like to call attention to the Agile Manifesto. Not an Agile framework. The core philosophy.

https://agilemanifesto.org/principles.html

In particular:

Build projects around motivated individuals.
Give them the environment and support they need,
and trust them to get the job done.

This is, coincidentally, in line with the current directive for most neurodivergent therapy. Integration, not assimilation. In HR terms, we call it equity.

When you had low process and high agency, with constant stimulation driven by urgency, novelty, and personal passion, you had an environment that appealed to a neurotype that is very powerful when motivated.

This is a neurotype that will often put in 60 hours a week and like it (assuming they are actually motivated). This is a neurotype that will solve complex problems over a weekend that might take others months to even understand.

The misguided lessons society has taught itself about "work-life balance" have led us to adopt the 9-5 workweek, based on the assertion that it would work for everyone and that it's the box you must fit into to be successful.

Many companies then built a bunch of fancy metrics around this philosophy in a misguided attempt to show just how productive this 9-5 workweek was at driving motivation.

And neurodivergents, in response, invented the mouse wiggler.

In some jobs, time worked is the measurable deliverable.

Engineering is not one of those jobs. You deliver software, not time.

You're Suggesting 60-hour Weeks are GOOD?

Sustainably? Every week? Absolutely not.

I've personally had 60-hour weeks and 10-hour weeks.

I've solved months of value in 2 hours, and gone down the wrong rabbit hole for weeks.

AuDHD (Autism + ADHD, that's me) in particular often comes with a drive to solve problems. I'm either solving a problem or looking for a new problem to solve. Responsibility is a guiding factor, not the engine. Burnout stems from either working on long-term projects out of obligation (and not motivation) or from too little going on ("boreout").

There's a bit of nature vs. nurture in the discussion of where that "eternal antsyness" originates. I'd highly recommend doing some research on that if the question "should AuDHD be that way" popped into your head. It's a fascinating rabbit hole that will show you the bigger picture of just how much society shoots itself in the foot with misleading messaging and expectations.

You Just Implied Velocity Isn't Constant

I'm not implying.

I'm stating.

The illusion of constant velocity is one of the most harmful lies society tells itself.

Across a team and over a sufficiently long timespan, velocity can approach consistency. But not at a granular, individual level.

There's an important entry in the manifesto.

Agile processes promote sustainable development.
The sponsors, developers, and users should be able
to maintain a constant pace indefinitely.

And a second one.

Deliver working software frequently, from a
couple of weeks to a couple of months, with a
preference to the shorter timescale.

Preference is for shorter time periods, but the extreme of "day to day velocity" becomes micromanaging (and risks encroaching on the neurodivergent need for agency, which has a high risk for causing burnout and turnover. Search: Pathological Demand Avoidance).

Two-week sprints are generally a good time frame for most teams to average out to a "constant pace", so those tend to be what get used. I do believe it's important for a team to have an established contract on their iteration team, especially since time-boxing is such an important concept for ADHD (which often comes with perfectionism).

But ultimately, at an individual level, the smaller the time frame you consider, the harder it is to say "spontaneous bursts of innovation and complex problem solving" and "predictable delivery speed" in the same breath.

There's a balance.

As a side note, search for "ADHD time blindness"; it will likely explain why you struggle with overcommitting or undercommitting in almost every sprint. Many neurodivergent employees try to plan around their highest velocity rather than their expected velocity. Combine that with a bunch of biology around dopamine dysregulation that I'm not going to get into here, and you get powerhouses that are incapable of predicting just how powerful they will be.

Wouldn't that Show up in the Metrics?

Enter Autism.

Pattern and rules extraction are our entire thing. (That's a reductionist statement for the sake of making a point, we have a lot of things, don't @ me)

Intuition is not intuitive for us. Big-picture thinking (holistic intuition) is often low, while expertise and wisdom (pattern-based intuition) are often higher, but take time to build.

This is by design in Engineering.

You want employees who are cognitive-forward; employees who are missing the caching layer of intuition. It means the problems get properly analyzed (cognition) with very few assumptions made (intuition), and a sustainable solution is implemented. Engineers don't just write code; they pattern-match and extract rules from requirements.

Intuition would tell you, "Don't re-invent the wheel."
Cognition would tell you, "But the wheels we have aren't built for this terrain. In a few weeks, we can invent the plane."
(And Intuition might then question, "Why are we even talking about wheels? We build houses.")

But when we haven't had time to iterate on the same problem space for long periods (building expertise), we default to repeatable processes and numbers to reduce the problem space to bite-sized, actionable information.

More often than not, this leads to gamifying the numbers. Mouse wigglers, not pulling in extra work when work is done faster, over-pointing efforts, under-pointing efforts, skipping process completely for certain efforts, etc.

In the absence of more concrete directives on deliverables, making sure those numbers look good, min/maxing them, is inevitable. It's not generally an active decision; it's just a piece of the puzzle we incorporate into the planning process when everything else is open-ended and doesn't match a pattern we are familiar with.

This is Still about Agile, Right?

Yes!

The core moment you went wrong, the single "what should we have done differently?" decision point, was picking a singular Agile Framework to prescribe to a full department.

These frameworks often prescribe, or are interpreted as prescribing:

  • Low, or no, rollover
  • No mid-sprint changes
  • Predictable velocity at a granular scale
  • All work to be completely scoped before committing
  • Time commits on work to be predictable and exact
  • Turning Engineers into code monkeys, rather than problem solvers, moving decision-making entirely into other teams (Architects, Product, UX, QA, etc.)

If you look at the core 12 principles of the Agile manifesto, every single one of those bullet points comes back to a misinterpretation of the Agile philosophy.

And most importantly, every single one of those bullet points is incompatible with the neurotypes you have built your team out of.

You have not created an environment for motivated individuals; you have created one for the "neurotypical" mentality, which so few people thrive in. (If you want another fun rabbit hole, search "does the 'neurotypical' person exist at the biological level". Spoilers: It's a sociopolitical term, not medical. Depending on how you draw a few lines in definitions in 'normal', it's not even a majority.)

Assuming You're Right, What Do We Do About it?

Be Agile, don't do Agile.

Provide the right environment.

The manifesto is the important part, not the frameworks.

You need different processes for different types of teams, different types of projects, and different budgets.

I want to call attention to an important entry in the manifesto:

Simplicity--the art of maximizing the amount
of work not done--is essential.

And another one:

The best architectures, requirements, and designs
emerge from self-organizing teams.

One-size-fits-all process is the problem, not the existence of process. Process is important, process enables, process protects. But doing less, quite literally, translates to doing more.

I can personally tell you that when I am mid flow-state on a big project, even transitioning a Jira ticket that carries no perceived value can be enough to break momentum. It's unbearable. I can physically feel the frustration from it in my veins.

Those without ADHD are likely thinking this is melodramatic. If you know someone with ADHD, ask them. This is a major reason why "context switching" is viewed as a cardinal sin among engineers. Got into the flow in the morning, but was forced into a meeting on a completely unrelated topic? It's a literal crap shoot on whether I get into that flow again.

Reduce process to only the process that enables development or protects a system.

Highly critical services in maintenance mode require high process. Lesser critical systems require less.

"Must do", non open-ended work (if you have buckets of work catered to specific important clients, for instance) is good in high process to self-regulate on commits.

If something is greenfield and doesn't affect production, does it need a full regression test suite and a Jira step for Product approval in a lower environment? Can you just demo on your local machine?

For the other Software Architects out there: Design systems to enable greenfield development to occur out of band from those critical services. This is one of the subtle but most valuable benefits of microservices. It lets you right-size the process.

Another quote:

Working software is the primary measure of progress.

Reduce metrics to those that drive SMART goals. SMART goals are the most empowering weapon for neurodivergent employees, who often crave explicitly defined deliverables (just don't overprescribe how they get TO those deliverables, see: Pathalogical Demand Avoidance again).

A Couple Other Principles Viewed through the Neurodivergent Lens

Welcome changing requirements, even late in
development. Agile processes harness change for
the customer's competitive advantage.

Pattern-based and cognitive-forward decision-making takes time. Couple this with the low Implicit Social Cognition you get with Autism (difficulty to intuitively keep up with an ongoing discussion), you get employees who need longer than your 1-hour meeting to think or process.

Sure, you did sprint planning. That was important. You did due diligence on attempting to plan.

Plans are nothing; planning is everything.

An Engineer realizing a flaw in a plan a few days into a sprint is part of the process. Encourage text-based communication for asynchronous planning, which is very neurodivergent-friendly (giving time to right-size communication and process).

Business people and developers must work
together daily throughout the project.

I mentioned Pathalogical Demand Avoidance a few times.

This is the best way to avoid the problem. Make engineers feel involved in decisions. They are specialists on the system you are building, use that, don't silo it. Studies exist to prove performance gain from engineer engagement, if you need proof. See: https://link.springer.com/article/10.1007/s10664-023-10293-z

The most efficient and effective method of
conveying information to and within a development
team is face-to-face conversation.

This might sound counterintuitive, because many people think of "anti-social" when they hear Autism. If you tried to initiate a discussion about the weather, I might give an awkward smile and do my best to extract myself from the physical pain that is "small talk".

But when it comes to an area of interest, you can't shut us up.

The literalness of "face to face" leaves much open to interpretation, but at the end of the day, vocal communication, at the very least (even if just Zoom), is significantly more efficient and effective time-wise. Just define your deliverables going into the meeting and organize the conversations with live note-taking to avoid endless tangents and rabbit-holing.

And finally:

At regular intervals, the team reflects on how
to become more effective, then tunes and adjusts
its behavior accordingly.

This is, coincidentally, how many people with AuDHD simply live their lives. Iterate -> Analyze -> Consult (ideally) -> Iterate. An endless series of "focus hard, then reflect".

That, at its core, is what you're aiming for with being Agile.

Well, I'm Neurodivergent, and I figured it out, they Can Learn to Push Through it Just like I did

I want to call this out since I've actually heard it said, specifically in the context of prescriptive Agile frameworks.

This is the equivalent of telling someone with severe depression, "just smile like the rest of us". It's a form of generational trauma, has ableist undertones, and risks violations of ADA directives on handling ADHD and Autism as a disability.

The world's take on "undue hardship" and equity in ADHD/Autism is shifting. There are already successful lawsuits on underplaying the severity of mental disabilities in the workplace (including the usage of derogatory words like "lazy" or "disorganized").

Do some searching. Educate yourself. Become powered by neurodiversity, not disabled by it.

Top comments (0)