NOTE: I had an opportunity to discuss this article in more depth on the MousePaw'dCast Episode 5. Jump to 30:45 for that segment.
I first picked up programming in my junior year of high school -- perhaps better understood as my second junior year, as a traumatic brain injury [TBI] two years prior had set me lightyears back academically, from 4.0 Sophomore and college-level reading to failing pre-K material. I had crawled, hand-over-claw, back to the same level of coursework I had once been able to work with. In the process, I had lost my ability to fully grasp the biological sciences I'd once loved, but I had gained a tremendous understanding of math. Along with it, I now had the ability to visualize algorithms in higher dimensions. I've had it compared to a superhero origin story, albeit without the tangible powers.
It was during this "second junior year" of high school, during which time I was playing around with Visual Basic, that I picked up a copy of Dreaming in Code by Scott Rosenberg.
In the very first chapter, Rosenberg describes something he calls "Software Time"...
Time really does seem to behave different around the act of making software. When things go well, you can lose track of passing hours in the state psychologists call "flow." When things go badly, you get stuck, frozen between dimensions, unable to move or see a way forward. Either way, you've left the clock far behind. You're on software time.
My brain, which already perceived time as a fluid fabric instead of a rigid line, immediately jumped to attention. In my fleeting few months of programming, I'd experienced both ends of this. I knew what Software Time felt like.
Dreaming in Code goes on to describe the plethora of strange phenomena that appears around software development and the project management thereof.
It was at this moment I knew: I wanted not only to become a programmer, but also to be a software project manager.
Fast-forward eight years, and here I am, running an open source software startup. I have personally encountered dozens of the bizarre phenomenons described by hacker scribes such as Rosenberg, Brooks, and Raymond...and I love it.
My staff will tell you, as a Lead Developer, I'm a bit like Doctor Who: eccentric, possibly crazy, operating from some inexplicable perspective, and sometimes given to bizarre solutions that either work out impossibly well or fail spectacularly. Yet it works -- we actually do produce functional software, out of a team composed entirely of remote six-hours-a-week programmers (except for myself), all of them interns or graduates thereof.
I can't imagine the "Gallifreyan" approach to project management is unique to me, but I have observed that it's rare. It seems that most management techniques either attempt to cram the wibbly-wobbly-timey-wimey nature of software development into rigid structures, or else to have no plan at all.
I might be crazy (although it's a brand of crazy that works for our projects), but I hope that some of my techniques may be useful elsewhere. Sometimes it's hard getting my non-linear thinking into an expressible form, but here's my best shot.
Whovians may want to get out their sonic screwdrivers. Everyone else, just buckle up and try to hang on. This topic is bigger on the inside.
Allon-sy!
Flux: "It's a Timey-Wimey Thing"
I've learned that I can never explain the entire project roadmap to anyone else; every attempt ends in the same glazed, slightly panicked look. Basically, I've learned to keep my yap shut about the long game, simply because no one needs to know.
Yet, so far, there's rarely been a tremendous deviation between my projected roadmap and the project's actual course. This isn't due to some superhuman foresight on my part (I'm not that good), but rather to my ability to comprehend and plan for a lot of flux.
Pulling analogy from Doctor Who, you know how there are "fixed points" in time that should never be altered, and everything else is a whole lot of "maybe"? That's literally what I see in project management.
To put it another way, I've learned there are three things that are never predictable in software:
Problem complexity
Production time
The viability of third-party software
In my roadmap, these are huge areas of flux. I just don't know what's going to happen there. My visualization of it within a project timeline is a little like a blacked-out bridge over a chasm. It might only be a few inches or hundreds of feet long, but we won't know until we get there. It's important to note that you can predict, with considerably accuracy, how many flux-bridges lie along a path. You merely don't know how long they are.
Basically, the only thing you can predict about flux itself is that it's unpredictable. This is the part that scares most project managers. I'd be lying if I said it never scared me, but it's less terrifying because of two simple rules I follow.
Again, it's hard to get my non-linear thinking into words, so let's call upon the "flux-bridge" analogy again.
Set a hard limit on how many flux-bridges a single path can have before you abandon it.
Set a hard limit on how long you walk along a flux-bridge before you abandon a path.
To put this another way, project management isn't unlike the old shortest-route algorithm problem. It's NP. There's no way to solve it optimally, so you have to brute-force it. Start down one path until it gets too long, and then try another path.
Relativity: The Cloister Bell
Travelling with the Doctor is dangerous. If you're going to survive, you've got to get some semblance of a grasp on the basics of time and relative space, or else you're toast.
Project management isn't just one person's responsibility. In a healthy team, everyone has a role in it. For me, this meant I had to find a way to enable my entire team to spot those areas of flux.
At MousePaw Media, we have a standard called Quantified Task Management, which boils the essential information about a programming task (issue, bug, or feature) down to four 0-5 scales: Gravity (importance), Priority (urgency), Friction (available help resources), and Relativity.
That last one is perhaps the most important. Relativity is the probability that an issue is a Black Hole. For you Whovians, it's kind of like the TARDIS's cloister bell, sounding whenever there is significant temporal danger.
Basically, a Black Hole is a flux-bridge with no endpoint. You don't ever reach the other side. It will consume infinite time and resources. Black Holes exist in software. There is no recorded case of a project ever pulling out of a Black Hole. You must go back to square one and start over on a different path.
Tasks have a lot of flux on account of their problem complexity. You don't really know how much work a task is going to require, or how hard it will be to complete, until you start working on it.
Our Relativity rating ranks the probability that the task is a Black Hole:
-
r0: No chance of black hole. No flux. -
r1: Low black hole probability. Probably safe. -
r2: Moderate black hole probability. Some flux, but looks possible to complete. -
r3: High black hole probability. Still possible to complete, but take caution. -
r4: Almost-definite black hole. Completion possible, but unlikely. -
r5: Collapsing. Bail out NOW.
Black Hole tasks are described in Dreaming in Code (pg. 13-14). In the book, the development team had a less nuanced approach. Every Black Hole bug was tagged "scary".
It's actually not as hard to determine a task's relativity as one might guess, as the book describes...
"Within the first hour of working on the bug," Burgess volunteers, "you know which it's going to be."
Time: "We're Going Always"
As I mentioned earlier, I basically learned out of the gate that linear time goes out the window when you're dealing with software development.
To put it another way, software has its own timeline. You can try to set all the deadlines you want, but in the end, those are just aspirations without any possible substance behind them. There are only two ways to potentially meet a deadline:
Forget "feature-complete" even being a thing, and ship with whatever works, or,
Fling quality out the window altogether.
Now, if you have any shred of decency as a programmer, you won't even view #2 as an option. Thus, you can aim for ONLY one of two things: your feature set or your deadline. It won't be both.
Sometimes projects pleasantly surprise you, and you ship on time with your full feature-set, and then some! It's always great when this happens, but don't ever make book on it.
By way of example, 2017 was a weird year at MousePaw Media. We completed two projects: PawLIB 1.0 and Omission.
PawLIB, by all standards, should be considered overdue by nearly a year. We hit a lot of flux in the development process, but we finally shipped the first version. In the end, we only managed this by marking several major features as "experimental" and shuffling their release off to PawLIB 1.1 (currently in development). We might have actually met the theoretical deadline on PawLIB, but we valued the release of certain key features.
Omission, on the other hand, came together much faster than expected, and with more features than originally planned. The irony about this project, however, is that the release was delayed by five months (and counting) due to an r4 Black Hole in packaging of all places. Bypassing the Black Hole would have required a complete rewrite with a different GUI, but we pushed through with some outside help, and now have a fairly certain release goal of February 2018.
I learned not to mess with production time flux in a now infamous incident a few years back. We needed a fully Unicode-compatible string class in C++ with certain features, and I underestimated the amount of flux involved. I assigned it to one of our developers with the statement that "this should only take a weekend."
Said string class is expected to be stable and ready for initial release later this winter, two years and three developers later. It's a truly magnificent thing, but it had a scary amount of flux.
Third-Party Software: "When I Say Run...RUN!"
The third, and perhaps biggest, source of flux is third-party software. Any time you choose to rely on a tool, a language, a library, or a framework, you are introducing a HUGE unknown quantity into your project. Any number of things can go wrong:
The software may be too complicated for developers to learn or use.
There may be a bug or missing feature that you don't yet realize is blocking your progress.
There may be a unique incompatibility between the software and any other piece of your development stack.
As I mentioned earlier, we were prevented from releasing Omission in 2017 because of a Black Hole in packaging. We had chosen to use Kivy, a Python GUI library, to build the game. While the library did everything we wanted in the code, it had a fiddly bunch of dependencies (and a setup.py hack) that made it virtually impossible to package for Linux, one of our major target platforms. We only got unstuck when one of the Kivy developers managed to compile Omission with pyinstaller, and the end result doesn't fit the Debian packaging standards (requiring some non-standard .deb packaging). As I said before, we could have bypassed this Black Hole, but it would have required a complete rewrite using a different GUI library.
Now another example: earlier in 2017, a major part of our project was held up by our dependency on a C++ XML library called Xerces. Two developers spent the better part of 2016 learning how to use the library, with little progress on their respective code sectors. Finally, we decided as a team to pull the plug on using Xerces - it was too hard to learn, and too difficult to use - and to use pugixml instead. We literally had to ditch the problematic software to bypass the flux.
Limiting Flux
So far, I probably haven't told you much that you don't already know. Many software projects are tangibly delayed and bogged down by lost deadlines, infinitely-deep tasks, and hard-to-use third-party dependencies.
But remember, we aren't entirely without ways to fight back! Recall those flux-bridges. In my mental project roadmap, I can usually figure out where most of these flux-bridges are, and use them to plan!
Let's review those two rules:
1. Set a hard limit on how many flux-bridges a single path can have before you abandon it.
I really don't have a particular number to give you here for the limit. Every project is a little different, and with time, you get a feeling for how many flux-bridges are too many.
For example, we were recently debating how to send data between two processes. One path was to build two solutions in a macro, switching on platform: UNIX systems would use sockets, and Windows systems would use that native approach. The other path was to use Google Protocol Buffers.
With Approach #1, we knew we had some flux in regards to writing our own protocol. Approach #2 involved a third-party dependency, and a dynamically-linked library no less. We had already run into significant issues with dynamically-linked library. Thus, while the second approach was clearly portable, we had two major points of flux, versus only one for Approach #1.
See how that works? You figure out how many flux-bridges are on each possible path, and pick the one with the least.
2. Set a hard limit on how long you walk along a flux-bridge before you abandon a path.
Once you start down the path, and you realize there's more flux there than you can handle, bail out and try another way. The time spent starting over is almost always less than the time wasted continuing into a Black Hole.
Revisiting the Xerces incident from earlier, we recognized that we were spending way too much time learning the library. Learning is non-transferable, meaning that every new developer on the project would have to walk that same flux-bridge of grokking Xerces and our use of it. We finally set a hard limit for libraries: if it takes more than 20 hours to do something meaningful, dump it.
There's one more consideration I make with flux:
3. When you're trying to get two paths to converge, keep the number of upcoming flux-bridges roughly equal.
Making paths converge is quite the hat trick. We had to do it once, when we needed three major features done at the same time for the release of PawLIB 1.0, without holding up other projects.
We abandoned another attempt, trying to get two dependency libraries written at the same time so the project they blocked could commence with the entire team behind it. Honestly, when you can't keep the flux-bridge count lined up, you just ditch the attempt and refactor.
Last Thoughts
If I'm honest, I look a lot more in control than I actually am. I don't have much control over flux. I try to steer around it as best as possible, but sometimes we still wind up in black holes.
That leads me to the single most important principle of Gallifreyan Software Project Management...don't pretend to have it all together! Whether you've got a Time Lord's view of the project or not, you can only handle problems as they come.
My team is used to me. They know I'm steering this TARDIS, and they are usually confident in me, but sometimes things go wrong. That's when I have to admit that we're in Software Time.
Or in the words of the Eleventh Doctor: โDo what I do. Hold tight and pretend itโs a plan!โ
Top comments (4)
Thanks for sharing, Jason. This is a super interesting perspective and system. I love the idea of identifying and understanding the different areas of potential "flux."
I often think about the "known knowns," "known unkowns," and the "unknown unknowns." It seems like the idea of "flux" helps sketch out some of the shape surrounding the various unknowns.
One result that isn't mentioned here, but is important to this argument, is that of J.P. Lewis' seminal paper on large limits to software estimation. This paper should be required reading in every class on software project management as tempering of the optimism shown by well-meaning but unwary managerial (project or otherwise) types (including textbook authors) who believe that software development can be rendered purely a process, not understanding that we are really in the business of R&D every single day.
You had me hooked the moment I saw the word "Gallifreyan" in the title. Big fan of Tennant, Smith, Capaldi and the whole Whovian universe. I've bookmarked the article, will read it a bit later.
Project time estimation reminds me of weeping angels; the more you look at them, the closer you get to that moment of imminent doom. The same analogy could be applied to coding; the more time you spend looking clueless and hoping the code will somehow materialize itself, the closer you get to getting zapped back to the unemployment line ๐.
I just had this thought bouncing around in my head and had to let it out in a form of this comment.
The reality is different of course, if people got fired for missing the deadlines, there wouldn't be any developers left. Maybe procrastination is the main character trait of every developer ๐
As a developer and a whovian, I really really loved the post and your focus. Great work.