One of the things I wished I knew earlier in my career is how important project management is for a software engineer. Especially early in our career, the main focus is usually on learning a lot about technology, frameworks and patterns. It might take a couple of engineering roles in companies with different styles and processes to figure out what project management is (and why it is important).
Why do I think it's good to understand the basics of project management early on?
As an engineer, even though you are not likely to be responsible for these processes, knowing the related terminology allows you to communicate effectively with different disciplines, usually product or project managers, quality assurance folks, or other stakeholders. Understanding how a project is managed end-to-end might increase your opportunities to participate in meetings where decisions are made. It can ensure that your projects have a higher success ratio because estimates are more accurate or risks are raised early, building better trust in your team. Moreover, if you decide to experiment with freelancing, it will be easier to estimate the time and the cost of your offering and communicate with your customers.
In a nutshell: writing code is only part of the process of creating valuable products and business value. But how you do this effectively, especially across multi-disciplinary teams and many stakeholders, is the magic of project management. There are many methodologies and schools when it comes to it, but I'm not going to discuss their pros and cons in this post. There are already many resources online that do so much more successfully. In reality, many organisations use a hybrid approach, combining some of the principles of each methodology, as there is no "one size fits all" when it comes to software projects.
Instead, what I plan discuss is some key concepts involved in software project management and some practical tips and ideas that I discovered throughout my career, which helped me manage and deliver projects more successfully.
Terminology
Let’s start with some terminology:
- A project is a piece of work that needs to be completed to reach a specific outcome, which adds business value.* It has a temporary character. Unlike operations, for example, which are activities that need to occur to maintain and support a product indefinitely (or until it is sunsetted), a project has starting and finishing points. Project management is typically defined within the boundaries of a team implementing the work. For work spanning among multiple teams and projects, a program management process is usually needed.
There is also the term product management, which refers to the process of managing a product, its vision and growth. A product does not have the ephemeral character that a project has. Even though some of the concerns mentioned overlap with some product management considerations (especially in organisations in which these processes are mixed), I am focusing here mostly on managing the lifecycle of software projects from an engineer’s point of view.
Methodologies
Project management techniques and tools were first used in the 1950s for managing complex engineering projects, and have evolved since then to cater to the needs of modern projects. Depending on the size of an organisation and the type of projects involved, different methodologies are a better fit. Some common approaches used in software projects and their respective implementations are:
- Phased approach (Waterfall)
- Iterative approach (Agile, Scrum, XP)
- Lean (Kanban)
- Product-based planning (Prince2), etc.
No matter how your organisation implements methodologies, there are typically some phases involved when running a project. It's useful to communicate the stage of the process effectively to everyone involved to manage expectations.
A software project life cycle typically involves certain phases, which are:
When going through each phase, some recurring themes and considerations come up. Even though the stages do not apply in the same way to every project type or organisation, I find it useful to at least pose the question when taking on a new project. Whether the project is my weekly task or a 6-month project involving multiple team members, thinking through its lifecycle in a deliberate way has helped me ask better questions and take more appropriate actions.
Is our idea valid?
For example, a project might be in the idea validation phase, where we have an idea and want to validate it before dedicating resources to planning and implementation. This phase might involve market research, interviews, data analysis, etc. Depending on the organisation’s structure, different stakeholders take part in this stage. Collaboration with research, analytics, sales, and customer support teams are useful in gathering information and backing a decision. Typically in larger organisations, the idea validation phase is completed before the software team is involved by teams whose focus is business analysis or market research. In smaller organisations, engineers might be more involved in doing data analysis on usage metrics around their product area or shipping a minimum viable product (MVP) to get early feedback.
If we have an idea, but we are unsure about the technical feasibility of what is requested, we might need to do a proof of concept (POC) or a spike. These are useful when the team has not implemented something similar before and want to validate an approach by building a simple version. A spike usually refers to answering a specific technical question around the project like “can we integrate with this service for X functionality or do we need to build our own implementation?”, while a POC is typically a bit bigger: it refers to the feasibility of the project in its entirety and may use the outcomes of spikes to come up with conclusions. Both can help us identify blockers early on and confirm the feasibility of what we want to deliver. During the early stages of a project, it might be useful for engineers to consider if a spike or POC is needed and propose dedicating some time for it.
Requirements & Scope
The requirements gathering and, subsequently, the scope definition phases are critical - likely among the most crucial ones to get right. If precise requirements are not collected from the customers, we do not have a way to measure success at the conclusion of the project. Whether our customers are external clients or another department in the organisation, like the user research or the sales team, or somebody in the leadership team suggesting a new idea, it’s important to understand and capture those needs well.
Requirements can be functional or non-functional. Functional requirements describe specific user flows and interactions that we want to enable. For example, “we need to implement a more intuitive way for the user to check-out.” Non-functional requirements refer to considerations about the quality of the systems, like security, performance, accessibility, etc. A non-functional requirement could be “we need our application to process X requests per minute with Y latency.”
Once we have our requirements, agreeing on the scope involves inputs from the implementation team on technical feasibility and cost estimates (be it time, performance or monetary cost). Data from research, spikes and POCs, in combination with historical context, can be used. The goal is to come up with a consensus on what is technically feasible and how much it will cost to build/operate versus the business value product offering and its margins/return on investment (ROI). Negotiating the project scope and the timelines or suggesting alternatives are useful techniques for the team, but it's essential to fully comprehend the real user needs behind the request to do so successfully.
After the scope is agreed upon, we should be wary of new requests or extra work added to the project. It might be tempting to “just do this little tiny extra bit of work too since we are at it,” but introducing scope creep not only puts our estimates at risk but also derails focus from the original goal of the project. Every change request should be assessed and can potentially involve renegotiating on scoping or timelines. If we choose not to take on these new requests now, we can keep track of them using a feature request backlog for future iterations.
Time estimates
Estimation is hard - humans are said to be bad at it. I have seen many ways of doing estimations throughout the years, and hardly any of them were perfect. My personal opinion is that software estimations are particularly tricky due to the fast-paced and dynamic character of the profession. It is not uncommon for software teams to change a lot or move to new unexplored areas and technologies. Projects are usually not repeatable because if we had a way of doing something already, we likely wouldn’t be doing it again. There might be technical debt associated with an area of the project that will slow things down. There are almost always unknowns during the implementation phase that might impact our timelines.
However, this is not a valid reason to ignore time estimates. Each project and team is different, and it will take a few iterations to improve. Dedicating some time to break down the work and assess how long it will take to complete is useful to have a baseline. As teams get more experienced, it is easier. Different types of organisations use different techniques, from analogous estimates (like t-shirt size or story points) to parametric or bottom-up estimates. When I work on a project, I usually do something along the lines of:
1. Break down the work in small pieces that are well defined.
This process can be done in many ways, depending on your application architecture and team structure. The tasks should be well-understood pieces of work, but not too small that only add overhead in the process and no value in the estimations. Some examples can be:
- by user flow or operation
- front-end vs back-end work
- by different systems involved (for example databases, workers, UI components, use of internal or external APIs)
- by operations that need to be performed (for example data backfills, data transformation, or migrations to new systems)
2. Assign a best-case and a worst-case estimate to each piece of the work.
The best-case estimate reflects the situation where everything I know is sufficient, and no unknowns will come up. It takes into consideration writing automated tests for my work, performing manual tests and delays in deployment. The worst-case estimate takes into account known risks or factors that might delay the project (like technical debt, external dependencies, integration considerations, etc.). It might initially be a "guestimate" and is usually more straightforward to do after having some experience with the codebase and the systems. In case of an ambiguous task that has a flexible scope, timeboxing how much time we should dedicate to it can be a good idea.
Something to consider here is that the amount of time we spend on this exercise should be proportionate to the level of accuracy we need in our estimates. As estimates cannot be 100% accurate, we should strive for a level of confidence that justifies working on the project altogether. For example, if we are 70% confident that we need more than 6 months to complete the work, it might not be worth taking this project on at all based on other work that we could do instead. If our process doesn't require much accuracy upfront, we can spend less time and give higher level estimates. Then, it can be an iterative approach, in which we revisit our estimates during the implementation, as we learn more. Adaptive environments are usually more flexible in estimating the work and re-scoping or adjusting timelines as the project progresses.
3. Aggregate above durations for all the tasks, and calculate the best-case, worst-case and average estimates.
Then communicate those with the team and other relevant stakeholders. We might want to add some extra time to cater for unknown risks that have not been identified yet.
Estimating and scoping should go hand to hand, as we negotiate scope and details of the implementation based on tradeoffs and how long things take to build. It might be more important to get something out to users early on, rather than polish a specific interaction. Effective communication is crucial to find the best possible outcome before we start planning and building.
Availability of the team
So now that we have the high-level time estimates of the work, when are our project team members available to action them? Some questions that need to be answered at this stage are:
- Are any of your team members taking vacations during the duration of the project? Is there any public holiday during the duration of the project?
- Will they need to do other work in parallel, like operations or support?
- Are team members going to work full-time on this project, or are there more projects running that need their involvement? How can we ensure that context switching will not negatively impact their performance?
- What is the breakdown of the experience levels of engineers in the team?
- What is the expertise of the team members in the area of this project?
- Are there other teams involved, and if so, what is their capacity?
Cost estimates
Estimating costs is a process that I have not seen happening too much in software teams. For customer-facing product teams, which has been mostly my experience, I assume that this has to do with not always being able to link revenue back to a specific feature, or that the market analysis showed that we lack some offerings compared to our competitors, so the cost is justified to keep up with the market, especially in start-up environments. In infrastructure teams, it is usually more common to monitor the costs of running our systems. In larger organisations, usually other departments (like business analysts) have already assessed the market value, and this is not exposed to teams who implement the project.
However, I think it's interesting to have an understanding of how much a project takes to complete or a new feature to operate after completion. On a high level, some things that can affect the project cost are:
- Computational resources, like the servers that we need to run our software, depending on our expected traffic and performance requirements.
- The salaries of the team members involved for the duration of the project.
- The cost of operational resources needed for smooth project delivery at go-live.
- Software licenses used during the implementation or to operate the feature after completion.
- Educational material or training that is needed for the team members.
- Cost of external vendors that we engage with for parts of the work.
- Cost of pursuing certifications of compliance with specific standards if necessary, etc.
Alright, we agreed on the deliverables, we justified the business value of the project and the timelines, so are we ready to start building now? Nope, not yet! One last thing!
Risks & Assumptions
Unless the project is something that we’ve implemented in the past – which is not usually the case in software development – there are certain risks and assumptions that we need to keep in mind. Risks are factors that might put our project in danger, for example, "there is a risk that this new operation will increase the load on our database, and we will need to do extra work to scale it better or pay more to operate it."
Assumptions are short-cuts that we take during the validation phase to save time from trying to validate every hypothesis for the success of the project. An example of an assumption can be, "we assume that this new functionality will be discoverable and adopted by users intuitively." Those might be identified during the initiation phase of the project, when we do spikes, gather historical context, do user research or analyse the current state of the systems.
It is useful to keep a risk register and an assumption log for our project, while also assessing how much of it we are willing to accept versus how much time we want to invest in de-risking further. As the project moves forward, we should update those to reflect the current state to keep relevant stakeholders informed.
Now - you can go ahead and start building!
Building & Quality
And after all this work, we are ready to start the development process. Managing the list of tasks, assigning them to team members, and monitoring their completion is an iterative process and can almost warrant a new blog post on its own. Through every step of the way, reassessing what we know and updating our risk register, our assumptions log, our project backlog, and potentially our estimates help ensure that we progress as expected, and our project is in good shape.
Visualising the progress for people not involved in the day to day operations is important to keep everyone informed and draw attention if things don’t go as expected. This can be done by using project boards and establishing communication channels designed for keeping stakeholders informed. As stakeholders might be dealing with multiple projects at the same time, posting regular updates with the status of the project (“on-track”, “delayed”, or “at-risk”) and raising issues that need further attention can be used to make the information available but not overwhelming.
While we develop our software, we want to make sure that we build the right thing correctly. Quality assurance or manual and automated testing – name it how you want – is the process of ensuring that customer needs are met with our deliverables after the implementation finishes. For this, we need to assess against the initial scope and ensure that we delivered what was asked. We also need to make sure that there are no defects that will reach the customer. Depending on the way the team is structured, quality assurance is organised in different ways. The developers should typically do automated testing for their work and do code-reviews for code readability and correctness. In some organisations, there are dedicated automation testers that add additional tests.
In the same way, if there are dedicated quality assurance team members in the team, they will sign off the successful completion of specific tasks based on the initial requirements. Before the project delivery, usually, a group testing session is a good idea, where everyone tries to break the new feature. Ultimately, quality is everyone's responsibility in a group.
As mentioned earlier, not all of the above apply to every project. The project management process needs to be tailored to the needs of the project and the team.
Corrective actions
Something to always keep in mind, regardless of the process followed, is the need to course correct when things don't go as expected – which happens more often than not. Communicating with the team and stakeholders early enough and raising concerns to relevant people is among the most crucial parts of the work. If you are blocked when approaching a problem, try to reach out early enough to somebody who can help. If you see somebody struggling, reach out to offer your help. This is what regular catch up meetings are best for: seeing how everyone is doing and identifying problems. Maybe the best solution is to ask for more time, or perhaps it is to descope a feature. In case an expert is available, we can engage them to help the project move forward. Whatever the solution, it has to be a collaborative decision between different stakeholders, so the earlier we bring up the problem, the better. Ultimately, effective collaboration is the most vital part of a successful project and a happy team.
Completion
Finally, when our project is happily delivered, we should close it off before moving to our next work. Something we‘d like to do is reflect on what we learned from working on it. Should we update our knowledge management system for future reference or improve any of the team processes based on its performance? We would also like to make sure that we put proper metrics in place to measure its success. And what about clean-ups? Should we clean up any resources or feature flags that we don’t need anymore? Finally, did we track all potential improvements and feature requests for the future? All of this information is fresh on our minds at the moment, but we will need to search through a black hole to recall in a few months!
Project management is a huge subject – I’ve only scratched the surface here. As an engineer, you will probably not need to drive the above, but it’s beneficial that you know what’s involved and how to best participate. From contributing to the initial phases of the planning with your input and expertise to ensuring that progress is smooth and blockers are raised and handled, your input is valuable to make the process successful.
Do you have any feedback on the above or would you like to suggest any techniques and tips for managing successful projects? Would you like to hear more about any of the areas above? Let me know on twitter or in the comments :)
Top comments (1)
Love this. "Break down the work in small pieces that are well defined." <- one of the best guidelines I ever learned, both for engineering and for product & project management.