Working remotely is becoming the new norm with more companies adopting long term, yet little analysis has been completed to measure its benefits. Before scaling their team Echobox measured the impact on remote working on performance. Marc Fletcher the CTO of Echobox explains how they analyzed remote working in their team by using data collected on productivity over a 2 year period.
Marc Fletcher graduated with a PhD in Physics (Quantum Computing) from the University of Cambridge and has been the CTO at Echobox since 2014. Whilst not jumping out of planes or competing for GB as a professional skydiver, he's particularly passionate about maximising productivity in high performance cross-functional technology teams.
These are some useful resources that related to the episode:
- Nicole Forsgren - Author of Accelerate: The Science of Lean Software and DevOps, and is best known for her work measuring the technology process and as the lead investigator on the largest DevOps studies to date.
- Echobox are hiring - https://careers.echobox.com
- Github Organization - https://github.com/ebx
- AWS resources for remote working - https://aws.amazon.com/blogs/aws/working-from-home-heres-how-aws-can-help/
- Halo effect - https://en.wikipedia.org/wiki/Halo_effect
Listen
Learn from some of the smartest people in the business by subscribing to:
- Spotify
- Apple Podcasts
- Google Podcasts
- Breaker
- Stitcher
- RSS feed to listen on your platform of choice
- More details can be found on the episode page
Transcript
Rob: Welcome back to Startup Engineering. A podcast goes behind the scenes at Startups. I'm Rob De Feo startup Advocate at AWS. Together we'll hear from the engineers, CTOs and founders, that built the technology and products at some of the world's leading startups. From launch through to achieving mass scale and everything in between. Experts share their experiences, lessons learned and best practices. Our guest has a PhD in quantum computing and competes for Great Britain as a skydiver. Mark the CTO Echobox since 2014 is particularly passionate about maximizing productivity across technology teams. In this episode, Mark explains how they use data to maximize cross-functional team performance. We know that making data driven decisions is the best approach. But what do you do when you need to make a decision with far reaching consequences? And data is hard to come by. Mark will help explain how his team at Echobox approach this problem. Could you start by giving us some background and describe the problem that Echobox is solving.
Marc: We're a six year old startup. We build products to help automate and optimize digital publishing. The vision of the company is to help automate as much of the publishing space as possible. To allow companies to reinvest all of the extra time and quality research and journalism.
Rob: One of the things that I found was quick to realize about Echobox is that you use data everywhere, especially when it comes to making decisions.
Marc: Absolutely. I think trying to take data driven decisions is very much part of our culture. It's one that encourages us to always go back to the data to keep learning to change our decisions when the data changes. That's something we've done from the very beginning.
Rob: You make it sound quite easy, but it's really difficult to do early stage in the startup. Often there isn't enough data. And then as things are growing, things are growing almost too quickly. You don't have time to get the look at the data. How do you strike that balance?
Marc: That is a challenge. I think, to every startup will recognize you have very limited resources not only to deliver a business value, but also to evaluate and optimize the processes that your implementing in order to deliver that business value. There's never a right answer. You can always look back in hindsight and say, we wish we'd spent more time focusing on this particular process because now we've learned a better way. And that's a balance that every startup is constantly trying to find.
Rob: Many of the startups I speak to use data to make decisions, particularly on product or where to make investments. You're also using it internally with your processes. You talk us through the example when your team came to asking to work more remotely and how you approached this?
Marc: We try and take a data driven decision. We obviously want people's working environments and lives to be as fun as and as productive as as possible.
Rob: Productivity and fun may or may not be connected. Can you talk us through your thought process and how you approached figuring this out?
Marc: For any decision that's we're trying to make that impacts the way in which we work or the processes involved, is to go out and try and learn from other people's mistakes. Most of the time, it'll involve a lot of Google searches to ask the same questions of other companies and see how they go about solving this particular problem or what optimizations have they found. Once we've collated all of the different perspectives that are out there, we try and apply it to ask particular use case and say "which of these processes do we want to try?", "Which do we want to experiment with?" Generally, we do that through a very quick show of hands. The ideas that have the most interest will try first. That's a process that we are doing continuously. It's a never ending stream of trying things differently. And over time, the hope is that we end up, leaner more efficient process that everybody feels that they've been able to contribute to.
Rob: Yeah, that's great. It keeps people involved. It sounds iterative. I don't know about you, but when I'm looking around on the Internet, I'm always stumbling across these great new ideas of these are sort of things that you're going to try out?
Marc: The newest and latest ideas are great for inspiration. But one of the things that we're very careful of is referred to as the halo effect. Which is the tendency people have to focus on the high financial performance of a successful company, and then they assume that that performance has been achieved as a consequence of all of its attributes, even the negative ones. I've spoken to people to of what, to Google for over 20 years. They will generally argue that Google does because it works for them. Google has obviously created a fantastic culture. It's a structure and process that works for them. But at the same time, it's it's far from perfect. When people are looking to these new ideas and concepts, I think it's always important to evaluate them for your industry, for your stage of company, for your old team size, these are very important factors.
Rob: Remote working been around for a while in different forms. What would the concerns that you had, if any, when your team wanted to scale out working remotely?
Marc: For us There hasn't been any think with potentially higher impact than that of remote working. As a company we have always tried to maximize the flexibility people have when executing their work. We want to be very outcome driven rather than how would you necessarily get to that outcome. From the very beginning of the company we've allowed very small amounts of remote working. Around two years ago we started having people in the team asking if they could increase the amount of remote working. You cross a threshold where once someone is remote more than they are in the office. That creates some potentially very large questions as to what the impacts of that going to be on the company. The costs of experimentation there might be very high because once someone is working fully remotely. It may be that they don't want to come back into the office. If you find out that actually it's having negative consequences. There's huge considerations when deciding how do you test that, how do you proceed with that, and how do you address those kinds of requests?
Rob: You spoke about it being difficult to experiment because of cost of experimentation is high. What were the metrics that you were most concerned about and how did you adapt your approach to be able to measure and mitigate these?
Marc: I think what's unique about the approach to Echobox is taken when thinking about remote working in particular is the productivity impact or the speed of business value delivery from a company perspective. The vast majority of cases when people talk about remote working people are considering the employees perspectives and the benefits to them and the impacts on team dynamics. For us, it was very much a case of we want to start from a position of what's the company impact, then sort of make sure that all of the other considerations were also, part of that process.
Rob: This is where I think your approach is particularly interesting as you've looked at it from the perspective of a company. And one of the key concerns you have is how would productivity be affected? Were there any other benefits outside of this that you were hoping to realize if this was successful?
Marc: The ability to hire fantastic developers. We-re a very sort of cutting edge company that works with the latest A.I. and machine learning technologies. Being able to hire great people makes a huge difference in our ability to deliver those kind of technologies in products that people care about. When it comes to hiring, particularly in a city as large as London, the amount of competition is fierce. If as a company we can confidently hire people that are fully remote. That gives us a huge or much, much larger pool of potential candidates to pick from. It's not just that these candidates might be technically more capable. They also come with different experiences and different cultures. These kind of things also enrich a company beyond just the products that we build.
Rob: We've got a good idea of the benefits, what's the ideal way to run an experiment, collect the data and then roll this out?
Marc: I think most people in this kind of situation would love to run a controlled AB test where they get half of their workforce to work fully remotely. Then compare the results between team members that have been working fully, remotely and the ones that have been in the office. Arguably that's one way to get a reasonably definitive answer, assuming of course your sample size is large enough. The problem, and this is particularly acute, for small companies like startups is one, they don't have a large sample size. And two, the risk to them, if all of a sudden it does blow up in their face is ginormous. All of a sudden they've committed the company down this path and then they find out the results have been negative. Now they need to try and recover any of the productivity that has been lost. What generally ends up happening is that people only report on the anecdotal or personal experiences of people working remotely. What we've tried to do at Echobox and I know that there's other studies that have done similar things, is they've just done a retrospective analysis on the data they already have available to say, "Okay. Over time, we've had different levels of remote working. How has that impacted various metrics that relate to productivity? And what can that tell us about the risks or the costs of trying this at a larger scale or on an ongoing basis?"
Rob: You mentioned about the experiences from the perspective of an employee, almost the anecdotal experiences. How do you take these into account when you're running this experiment?
Marc: This is where things start to get really interesting when it comes to remote working, because I think the anecdotal experiences of individuals very rarely lines up with the reality of what the data tells you. For the vast majority of people that work remotely, they will argue, "I felt more productive" or "I was more productive". That's a question. We could potentially look at the data and ask, if that's correct. I think far more subjectively where data is isn't going to be as helpful or feelings of stress, of motivation of the ability to feel creative, ability of not having to commute and travel, the proximity to customers, potentially different time zones are involved. These are just some of the multitude of factors that someone will personally experience when when they experiment with remote working.
Rob: Let's leave aside one moment, the immeasurable or at least difficult to measure a focus on what you were able to measure and how you measured it.
Marc: We will have a longer form article available that people could look at. There's a couple of things. We have measured the echo box to try and give us insights into the productivity impacts of remote working. The first one is obviously the proportion of team members time is spent remote vs in the office. Generally, we would have people self select a given number of days a week when they would be in the office and vice versa. What we can then also measure and compare against that particular baseline number is the number of deployments that we have to the products. And there's some additional research by Nicole Forsgren, they have a great book that talks about engineering productivity. One of the predictive relationships they establish is the number of deployments and productivity, that's sort of been our primary metric for measuring productivity. We also measures are team velocity, that's the number of story points that engineers are competing on average in any given month.
Rob: You were using the number of deployments and the number of story points completed and presumably some sort of flag indicating whether they're working remotely or not. What did you have to do to collect this information?
Marc: This is the data we were already collecting as part of our normal drive to try and measure the things that are measurable that, you know, may or may not be useful for current and future use cases. In this particular case, we were just very lucky that we were measuring the number of remote days. When we actually were asking questions about the company impact of remote working, we were able to go back and pull out measures of deployments and story points.
Rob: After you pulled this data what were the findings? How were story points and deployments affected?
Marc: I would provisor this by saying our sample size is small. And I think the predictive relationship between these metrics is still something that is being established in the scientific literature. And also just because we see these results doesn't mean that other people will see different results. I think that's again, one of the conflating factors when it comes to understanding the impacts of remote working. But in our particular case, we saw no correlation between the level of remote working and the productivity of our engineers. The relationships are flat. We can't say whether it was up or down. In our case, it looks like there was no impact.
Rob: This finding is huge. What you're able to establish with some degree of certainty is that someone working remotely and someone worked in the office. There's no significant difference in productivity, as you measured it, by story points and deployments.
Marc: Yes, that that's what our data seems to be sharing, which very much goes against the anecdotal reports of these same team members that felt they were much more productive when working remotely. There's obviously an interesting thing going on there where people perhaps feel happier and they feel more productive. But in reality, they are achieving the same amount of work. They might be happier while they're doing it.
Rob: There's something interesting going on here you've got people's perceptions thinking that there are the more productive or happier. But the data shows exactly the same performance, when you present this to a team how do they respond to it?
Marc: I think when people look at the data themselves, they can almost integrate that into their own perspectives and understand that from a company's perspective, there may be no advantage or disadvantage to people working remotely. It very much becomes a personal question, of personally how do I feel about working remotely? I think for the most part, we found that people's preference lies somewhere between always being in the office and always being fully remote. At least for our engineers, because there is a balance required us to get those feelings of being part of a team and the ability to collaborate and learn creatively is as a group. These kinds of processes are much easier to do in person than over a video call or something like that. At the same time, you have some amount of work that requires focus and a destruction free environment. There's generally no better place to achieve that than by working out of the office where all of these distractions might be. Take time away from your work.
Rob: You mentioned the distraction for the environment so often away from the office, at least in my personal experience. I think that depends. For example, if I'm able to choose when I'm working remotely, I can schedule my time in and around things I know about school holidays or building work that's being scheduled. If this is something that I know do all the time, I'll lose some of that flexibility. Is this something that you observed?
Marc: That is the key thing to take away when it comes from talking about remote working, is that it all depends. There's so many different underlying factors that ultimately impact somebodies productivity. It's not the remote working that's directly impacting this. It's the contribution and the interplay between all of these underlying factors that really impacts the end result.
Rob: If you're doing this for the first time it's like anything, it's new to you, its going to take you some time to learn how to work remotely effectively. Did you notice any productivity difference between when someone first started working remotely or after a while?
Marc: Absolutely. I think when people start working remotely for the first time, there's almost an anticipated dip in productivity, while people work out "okay, what is the correct environment?", "Where in my home should I set up my office?", "Do I need a better office chair?", "What software should I be using?", "What headset and microphone is going to give me the clearest and most engaging communication video calls with your colleagues?" These things takes time to optimize and it's over time that you improve and iterate those until you end up with the environment that is very much customized to your personal preferences. I think maybe that's where people subjectively now feel more productive. They may well certainly be and in other environments. But at least in our case, once people have achieved that sort of long term stabilization of this is my setup. This is what works for me. We haven't seen an increasing productivity.
Rob: It makes sense. It would take someone time to get that setup as functional as an office that's evolved over time to be good to collaborate with colleagues. One of the things I'm thinking over time is how does the relationship change? If I was used to working with my colleagues in front of me and now I don't, how do they change over time? Is that something you're able to measure?
Marc: That would be something great if we could measure it right now. Other than our quarterly employee happiness surveys that most companies will be running. There's no measurement for that and I think the scale of our company at the moment, that doesn't really give us enough resolution to make any statistically significant decisions.
Rob: This is one metric that you're not yet tracking. But on top of deployment's, employee happiness and story points are there other metrics that you'd want to track as you scale out remote working.
Marc: We'd love to being a data driven company. We would love to have very high accuracy on all of the data that we collect. We're still relatively a small company. We can't lose sight of the fact that we should be trying to deliver business value as quickly as possible. It is definitely very easy to end up in a position where you're constantly trying to improve your processes to the point you're not actually delivering any business value. I think people will see this a lot from a technical perspective. People want to spend time removing technical debt from a platform so that it's easier to make future changes to reduce the cost of the platform. You can spend a huge amount of time on these things and you haven't really moved the product forward for your customers in that time. It's always about finding the balance between the right amount of measurements, the right amount of retrospectives, the right amount of building new things. That's the hard part, is deciding how to balance those different things out.
Rob: Yeah, that's hard. So how do you do it? How do you balance between moving quickly, building new things and making improvements?
Marc: So the way in which we balance these requirements at Echobox is to measure everything that we can in the time that we have available in order to track the things that we think we care most about. We tend to limit to a couple of hours a month per person to track. Generally speaking, on a monthly basis, a couple of high level metrics. I would absolutely agree that having too much data can be a negative thing because it might encourage you to start abnormally hunting and asking questions about why certain patterns exist that might not actually be that relevant to your high level business metric. It's better to track a small number of things well that you can have a high degree of confidence in so that when you do start asking longer-term questions or retrospective questions, as as we have tried to do with our analysis of remote working, you can similarly have a good degree of confidence in your ultimate conclusions.
Rob: If there's something that could be done to improve your measurement, what would that be?
Marc: I think it would be great if there were more well-established measures of productivity, particularly within a technology organization that allowed you to be more confident in the metrics you are tracking, ideally down to an individual level. In all of the work we have done to try and assess the impact of remote working, we've really only been looking at team level metrics. The reason I think being able to do it. It's an individual level is the ideal case is that you might find some people within the team are more productive when they're working remotely and other people are perhaps less productive when they're working remotely, which on average will obviously balance out to zero. That would be very good evidence suggests it is important to allow people to make optimizations at the individual level. And really as a company that's the situation we've more or less ended up with. We want to be as flexible as possible in how people get the work done. There's many choices that people have to make for themselves. Some people would rather be in the office more, others want to be in the office less. One of the nice things that's the work that we have done so far allows us to do is to have discussions with people about supporting those different use cases.
Rob: What you've mainly been describing is remote work and rather than distributed teams, as in there's a central office, instead of people spread around the world. Is it something that you're looking to do to create a more distributed working environment?
Marc: Right now, as a company all of our technology focused team members are in the same time zone, or at least team members that need to jump on a video call regularly are in the same time zone. I imagine that as the company scales and you start having teams across time zones. This is yet just another complicating factor that you need to manage and try and find efficient ways of dealing with it.
Rob: We've mainly been talking about how to measure people's productivity from home working and inside the office. But how do you look to improve people's productivity when people are working in different locations?
Marc: We certainly do the standard retrospectives when it comes to delivering business fairly every couple of weeks. When you plan the next iteration of work, everyone is asking themselves, "how did the last couple of weeks go?", "What can we do to improve?" We also have dedicated retrospective discussions once every six or seven weeks that allows people to take a more long term look at the last month or quarter and say, what changes would we like to propose? What experiments would we like to run in terms of the processes that we execute? That's something that's just very much ongoing.
Rob: Based on some of these learnings that you've gathered on your team, if you are helping someone to scale out remote working, what would you advise them to do or how would you advise them to approach it?
Marc: I like to think that if you have a particular goal in mind, whether that's a process or a technical objective, you minimize your risk by trying to iterate towards that. If we look at remote working as a particular example, what that would mean is it's probably most difficult to go from being in an office full time to being fully remote full time because it's the most discontinuous in terms of experience and process. My recommendation would be do one day remote initially, use that time to optimize your setup. And then if you find to actually that one day is beneficial, then you can try two days and you just keep going until you feel you found a personal optimum or whichever company you work for believes theres particular times you need to be in the office for important meeting and you just keep iterating. The hope is that over time you make these small, low risk, low cost changes that you're always moving towards your company's optimum for these processes.
Rob: Now that you've established that there's no gain or improvement from working remotely for your company. Are you looking to scale out the right working? And are you looking to hire from different pools of talent?
Marc: As a small company, we want to make sure we have an environment that allows for maximum collaboration and creativity, which inevitably means we have a slight preference to people being in London and can come into the office for some proportion of the week.
Rob: Does that mean that you're actively looking at hiring people and potentially even looking at hiring a distributed team?
Marc: Yes, absolutely. Because of the work we have done, we also just as happy to consider fantastic people from anywhere in the world because we accept that there might be people that are better outside of London. And even though that might be a small negative impact due to not being able to collaborate as freely. The other attributes of these candidates can make up the difference. For us as a company that's been one of the main conclusions of all of this work, is it just allows us to be more flexible, to take advantage of greater opportunity out there without having to worry about it being a significant negative impact on the company.
Rob: Aside from hiring, are there any key learnings or large benefits that you've discovered?
Marc: We very much like the ability to be able to make new on its decisions, live in a very complicated world that's fast changing and new technology comes along that completely changes the landscape and the way in which people work. We like being able to take these data driven decisions that we review and people can comment on provides suggestions all the time. I think that becomes a strength in the long run because we're always in a position where we can take advantage of opportunities as they come along.
Rob: I think theres a the temptation when you change one thing in this case, the location where you're working to keep everything else as close to as was the same before. I think this misses a big opportunity to change the way people work. An easy change to mention is working asynchronously isn't having a gap between communication. Are there other things that you tried out?
Marc: People shouldn't expect to work the same when their fully remote, than when they're in an office. I think one of the huge benefits of being able to work remotely is the extra flexibility that might mean for a lot of people. They start their day a little bit earlier. They might take more regular breaks because they have the opportunity for going outside, going to the shops, and then they finish a little bit later. From a company's perspectives, that's obviously a good thing because you have people who can help resolve problems outside of normal business hours a bit more easily. Even if at the end of the day the number of hours their working is still the same. It's just perhaps not as compressed.
Rob: This is a big change to implement in a very different way than implementing a new key feature or even re-architecting a large piece of technology. This is a change that impacts people's lives. Individuals will have different ability to do this depending on the personal circumstances. So how do you work with your team to make this a success?
Marc: The key thing that everybody in the technology teams at Echobox is used to is experimentation. Nobody who joins our company is ever afraid of trying something differently for fear of it not working out or it doesn't work. Because these are the discussions we will have with everybody after we've run an experiment to find out how they feel personally, professionally. When it comes to processes like remote working, it's not something we've ever forced on people. People will experiment a little bit. If they like it, they'll keep doing it. And that in and of itself can be a good sign the experiment is a success from one perspective. If people want to continue doing something a certain way, then it means it's easier for them or they feel it's more efficient. In that kind of environment nobody ever feels like something is being forced on them just because someone wants it to be a certain way. Everybody's been involved in the experiment and understands the considerations that went into it, their aware of the results. We've always tried to optimize for people being in different locations, whether that's on the opposite side of the office or perhaps someone has just finished a meeting with a customer in the side of London. A lot of our processes were already set up to support video calls or working on documents collaboratively over the Internet that has made sense to do more or less from day one.
Rob: Mark has looked in-depth at the impact of working remotely, particularly what impact there is on the business. This gives a perspective that's often missing. He identified that scaling out remote working could quickly become a one way door, meaning a decision that's difficult to reverse. When confronted with a choice like this collecting data, analyzing a moving slowly is important. In this case analysis was made more difficult because there were no established methods for collection and measurement of productivity data when working remotely. Echobox had good existing practices including a heuristic on how much time to spend collecting data. I really like the idea of spending a few hours per month to measure the most important pieces of data. That acts as a guideline but also as a constraint. Mark and his team had to find the best metrics from the available data they had already been collecting. They used the number of story points, deployments and a flag indicating if someone's working remotely, plus employee happiness scores. Their methodology was sound. Too often people collect data to prove what they already believe in, leading to confirmation bias. This is where you prefer data that confirms your existing beliefs. The robust approach is attempt to disprove a hypothesis, in this case show there was no effect on productivity when a team works remotely. The result of disproving that working remotely affects productivity might sound underwhelming, but it's actually powerful. It means they can be sure that productivity will not change based on their team working remotely. Being confident in this means they can scale out remote working with no expected impact on productivity. Let's get back to Marc for his learning best practices and advice on how to successfully implement remote working in your startup or as an individual how to be successful in starting to work remotely. You've gone through this process in depth, a monitored it closely with data. What are the biggest learnings that you can ship based on this?
Marc: Be aware of the different underlying factors that might ultimately influence the changes in productivity. You need to consider things like distractions and focus of your team's. The ability for them to work creatively and collaborate, learnings that are achieved through group discussions. You need to consider people's levels of motivation and the impact of different feelings of isolation, somebody stress, costs of training replacements potentially if you have higher turnover. You need to consider the additional time that someone might be able to contribute because they no longer have to commute. Your office space costs, proximity of your customers, particularly if timezones are relevant. You also need to consider the beliefs and attitudes of your employees. There's some great research that's out there that suggests someone's success or failure at remote working might be entirely down to their belief that it will work. At that level it becomes almost a self-fulfilling prophecy. When it comes to starting remote work I would always recommend people over communicate as much as possible. Once you're a couple of weeks in, get some feedback from your colleagues, ask them if they feel you have generally been over communicating what they feel the level is about, right. Communication I think is one of the key things to try and get right when working remotely.
Rob: Ok, great. Do you have any advice for engineers that are about to start remote working in their current roles or even for junior engineers perhaps the first role being a remote job?
Marc: Things like remote working are particularly challenging for junior team members not necessarily new team members, but junior team members in particular. Because when someone is starting out in their professional career, they haven't yet fully developed the right approaches to focus and motivation and how to solve problems or how to get help solving certain problems. My recommendation to anyone that is junior and is starting to experiment with remote working is make sure you get yourself a mentor, someone in your team that's perhaps more familiar with the processes involved and have regular unofficial catch ups with them. It might be your manager in which case there'll be a bit more structure to it. But ask lots of questions there's never a bad question you can ask. Find out what has worked for other people and what hasn't. And experiment run those personal experiments to find out what is the best way for you personally to get the most out of remote working.
Rob: Thank you Mark for sharing this inside story on how you approached and validated there was no significant productivity impact for remote working for your team, the lessons learned, and best practices. If you're excited about building the next big thing or you want to learn from the engineers that have been there, done that. Subscribe to Startup Engineering wherever you get your podcasts.
Remember to check out the show notes for useful resources related to this episode.
And until the next time. Keep on building.
AWS Activate founder tier providers $1,000 in AWS Credits, access to experts and the resources needed to build, test & deploy. Activate your startup today.
Top comments (0)