DEV Community: Chris Lewis

What It’s Like To Work Through A Security Incident

Chris Lewis — Sun, 09 Dec 2018 13:04:12 +0000

Earlier this year, the company I work for as a software developer suffered a security incident. What followed was a very unique few months that I’ll remember for the rest of my career.

Prior to the incident, I’d had no real clue what to expect if one were to happen. There seem to be few first-hand accounts of what it’s like to be in that kind of situation published online. I want to share what my experience was like during this time, so that you might have at least some idea of what it would be like if your company were to experience the same.

Disclaimer: The following is a story about my personal experience of working through a security incident as a developer. Security is not my specialty; I was not at the “front lines” of the event, and I won’t be divulging any technical details of it here. Depending on your role and the company you work for, circumstances could be entirely different for you, but I hope you’ll find value in it all the same!

Initial Discovery

Around late May, I was working from home and saw a company email go around stating that some suspicious activity had been discovered in one area of the system, and it had been quarantined while investigation was ongoing. I was pretty intrigued by what exactly had happened, so on my next day in the office, I asked around about it. The impression I got was of a fairly minor infraction; something that had been dealt with before there was even the potential for any harm. It sounded like everything was done and dusted, so after chatting about it for a little, I went straight back to work like normal.

Just an hour or two later, I got a tap on the shoulder from one of our leads and was asked to hop into a meeting that was in progress. I wasn’t sure what he wanted but I was happy to join. When I did, I walked into a room containing several other leads, a few developers, and our entire security team. I was quickly briefed that everything being discussed was of a “need to know” nature and not to be casually spread around the office. Ultimately, it was said that the potential impact of the incident was more widespread than initially thought. Although there was no longer an active threat, we were in a position where we had to audit much of our system for any anomalies or signs of intrusion. Additionally, in light of the incident we had to re-assess any current security risks and plan out solutions to ensure this couldn’t happen again.

I was taken aback by what I’d been told — when just earlier I’d thought that there was nothing at all to be worried about — but I must admit that it also felt good to be included in something confidential because my skills would be useful. In particular, I volunteered to be part of a group of people that would perform the auditing tasks, since my past projects had given me a good understanding of the software infrastructure that we would be inspecting.

Once the meeting was over, I essentially dropped all of my other tasks and joined the ad-hoc auditing team that we had just formed.

In The Thick Of It

The next week or two was an interesting, confusing and somewhat frustrating time for several reasons.

Although our auditing work was very important, it was not exactly the most engaging. Auditing such a large amount of our infrastructure was a new task for us, and although we were seeking out tools to help us with it, we undertook a lot of it manually. Essentially, we would spend hours at a time clicking through resources to look for anything suspicious, and confirm there’s nothing wrong. Finding nothing is a good thing, but also becomes very boring and draining quite quickly. It’s also difficult to feel like you’re adding value when doing repetitive tasks like these that don’t have much of an outcome. Maybe it was just me, but morale in the team felt a bit low as we made our way through this task.

Being mindful of word of mouth throughout the office was a little awkward too. While I was working away in the auditing team, the amount of work that was coming out of our response to the incident continued to grow. As the work grew, more developers needed to be brought in to work on it, and so the “need to know” information slowly spread further and further. Eventually, nearly every single developer in the company was working on something related to the incident. At this point, it becomes difficult to know how much any one person knows about everything that is going on. If I hear some new information from Developer A, should I be telling that to Developer B? Is the information that I’m privy to still as confidential as it was when I first found out? Although I fully trusted my coworkers, dealing with confidential information remains a little uncomfortable when it’s not clear who should be knowing what.

Additionally, working with the auditing team meant that I was isolated from my normal development team. In fact, just a week before the incident, I had moved into a new team which had all the other members stationed internationally. This made the situation even more difficult because, for data sovereignty and contractual reasons, our international team did not have the same level of access to our systems, meaning that they could not be involved in the response efforts and hence were not informed of all the details of what was going on. This had the upside of meaning that they could continue with their planned work, with the downside of them being a little left in the dark during our response. It felt very detrimental to the forming of my new team to have me pulled away into some project that I couldn’t really talk too much about and that required all of my attention. As the only local member of that team, I actually felt somewhat guilty that I didn’t do more to try and keep them in the loop of what was happening.

Stabilising Operations

After the first couple of weeks, our development teams had addressed most of the immediate risks and concerns, and the urgency around the situation was slowly decreasing. At this point, we were definitely shifting out of “fire fighting” mode to be starting work on some newly planned projects that were allocated a high priority after the incident. The auditing team that I was part of was running out of things to do, so its members were starting to move off into other teams to work on these new projects.

The project team that I moved into was given the opportunity to replace a fairly old but stable piece of our system — the kind that you would probably never touch under normal business circumstances because it “just works”. Our task was to improve security of the system by replacing it with a brand new implementation of our choosing. Compared to the auditing work I was previously doing, this was a total breath of fresh air. We were able to rapidly experiment with new tech, narrow it down to the solution we thought best, and then refine it through to delivery in production — and the results we saw were really, really satisfying.

Honestly, working on this new project brought me to consider some of the “upsides” of what we were experiencing as a company. Suddenly, projects that would normally never get prioritised due to a lack of urgency were being worked on all over the place. As a developer, it was quite exciting to have the chance to work on these, even if the circumstances behind them were less than fortunate.

Around this point, the company was also looking to complete a third party verification to provide reassurance that there was no longer any active threat inside our system. Up until this is complete, your perspective on your software infrastructure and resources changes subtly. When you see something you don’t recognise, or something that isn’t clearly named with a clear purpose, you start to wonder if it could not be some innocuous resource left lying around and instead be a security risk. Once or twice I’d see a group of coworkers talking hurriedly amongst themselves about the discovery of a potentially suspicious resource, only to discover after a tense hour or two that it was something completely harmless. In this way, suffering a security incident can really make you start to jump at shadows and treat your system with paranoia.

News & Social Media

After the incident was publicly announced, it didn’t take long for news outlets to start reporting on it. Several people in the company suggested not reading any of the coverage, but I — and I wager many of my colleagues — read them anyway.

The news stories went exactly as I expected: sensationalist headlines cherry-picking alarming quotes from our public announcement were the norm. Even though I wasn’t surprised by their content, I couldn’t help but take some level of personal offence at the way the stories were written. Every day I could see the situation slowly improving as my coworkers gave their best efforts to set things right again; meanwhile, articles would try to spell out doom for the company or exaggerate the impact of the incident itself. After reading a number of these articles, I did start reading less and less of them, as there wasn’t much point to me reading inflammatory articles anymore.

Social media was a similar story. Out of some morbid curiosity, I would search for mentions of the company on Twitter to see what people were saying. When you work at a company which is typically not in the public eye, it feels very strange seeing people talk about it. There were a fair few angry and upset tweets about the incident, and when I saw people touting inaccurate information, it was tempting to chime in and correct them. Obviously, doing that would have been a terrible idea — getting defensive on a public forum could invite all sorts of unwanted attention and achieve nothing useful for myself or the company. Besides, I could definitely understand why people would be angry about what happened, especially at a time when there was limited information for people to really understand what impact the incident could have on them.

Industry Peers

The response from media outlets made me a little apprehensive about how the incident would be viewed by other developers in local industry circles. If I attended or spoke at a meetup and told people where I work, what kind of response would I get? Would others’ perception of me be coloured by what they’d heard about the company I work for?

I didn’t want to let this faze me too much, so I continued to introduce myself and the company that I work for just like normal— which had some interesting results! I quickly learned that whenever I told someone where I worked, it would only take a moment to work out whether or not they had heard about the incident. Those who had heard would exhibit a certain wide-eyed expression (for just a brief moment) as they linked the name of the company back to their memory of whatever they had read about us. To be perfectly honest, it was quite entertaining! Whenever I noticed this, I couldn’t help but acknowledge it with a laugh.

More importantly however, the conversations that ensued were always encouraging. I immediately got the impression that my peers in the industry were very understanding of the situation that we were in, and were often conscious that with enough bad luck, it could just as easily have happened to them instead. I think our incident is a cautionary tale to a lot of companies, reminding them that they need to take their security very seriously, and it was very heartening to see that everyone was so supportive about what we were going through. On occasion, some of my professional network would reach out to me asking how things were going and wishing us well, which was much appreciated.

Ongoing Impact

All in all, it took a few months for things to really start feeling like they were going back to normal. Several months after the incident, it feels like we’re back into a fairly normal rhythm of “business as usual”, but you can still see its impact in almost everything we do.

There’s still a lot of projects (both large and small) that need to be undertaken as a result of what happened, which has a huge impact on development roadmaps. Projects that were stopped in their tracks when the incident hit might not be able to be resumed, or do so only with considerable revisions to their scope and priority. There’s still lots of exciting work to be done, but it can be a little sad seeing a project that you thought was going really well get upturned by the change in priorities.

One must also consider the financial impact to the company. Although I don’t know any exact details, it’s quite clear that both the upfront and ongoing costs of an event like this are incredible. Despite this, I haven’t actually noticed anything different in the company from a financial perspective myself, which is comforting — though I definitely feel for those responsible for crunching the numbers and organising budgets for the foreseeable future.

Final Reflections

Experiencing a security incident first-hand was equal measures stressful, confusing, and exciting. Although it was tough going for everyone in the company, it also allowed us to showcase some of the qualities that I’m really proud of in our workplace:

No-blame culture. At no point was there even any question of who was “responsible” for the incident or the events that lead to it. Every single developer felt a shared ownership of the problem and what was required to fix it.
Team agility. Dealing with the security incident entailed a breakdown of the normal lines of communication, with new teams rapidly forming around the problems to be solved, and developers from all areas joining in to assist where their skills were most needed. Although it was hectic at times, everyone was able to take it in their stride and work well together.

To me this all exemplifies the high level of trust we place in all of our developers. It sounds a little selfish to say, but in a way, I’m glad that this all happened where I could see it unfold first-hand and witness how well we could work together under such trying circumstances.

Prior to writing this, I was not able to find many similar stories about such developer experiences online, presumably because any information about security incidents is typically quite confidential and difficult to share. I hope that moving forward we can be more open about experiences like these as a community, as security incidents will continue to happen to more companies and they will all have a considerable impact on their employees.

Finally, I hope that reading about my experience can be of some help to you in your career. Although I wouldn’t wish an event like this upon anyone, know that it can still hold a lot of positives for you as a developer, and can be a great learning experience in and of itself.

Many thanks to unDraw for the awesome free images used in this post.

Prettyplan: an open-source Terraform formatting tool for devs

Chris Lewis — Thu, 06 Dec 2018 22:46:58 +0000

tl;dr: prettyplan is an online tool where you can paste in your output from terraform plan and see it presented in an easy-to-read tabulated format. It is available here!

I'm not sure how many people here use Terraform - a popular infrastructure-as-code tool - but if you have, you may have run into trouble before when trying to read its command-line output when working with large projects.

The default output that Terraform provides to let you know what changes it will make is pretty good, but in some cases it can be quite difficult to read... for example, consider trying to read what has changed here:

To help with this, I created prettyplan: a small web app that parses your Terraform output and presents it in a much easier to read tabulated format. It's been a fun project to work on and one that I hope can help other devs too.

With prettyplan, the previous example looks like this instead:

Which (at least to me) looks a lot nicer and easier to manage! So if you've found yourself reading through that kind of console output before, give it a shot. :)

Contributions

If prettyplan is useful to you and you're interested in contributing, getting started and making changes is quite simple as the project is currently all vanilla JS with no dependencies or build (with the exception of some unit tests that run on commit). I welcome any improvements that people think could be made!

What's your sense of "duty" to your team?

Chris Lewis — Mon, 19 Nov 2018 10:07:16 +0000

In my career so far, I've only been part of a few different software teams, but I find it hard to leave each one because I usually grow attached and have a sense of duty to the team that makes me feel bad for leaving.

As an example, I'm currently faced with an opportunity to switch teams again in a couple months (after six months in this one), and although I feel it's best for me to take it, I can already see myself feeling bad for leaving my current team - like I'm letting them down by moving to something that's more interesting to me, and leaving my teammates to adapt to the new team makeup in my absence.

Do you feel a sense of duty to your team at work? What does that duty mean to you? How do you balance what's best for your career versus what's best for the team?

I'd love to know your thoughts, and whether or not you've dealt with similar feelings in your career!

My Hacktoberfest 2018 Experience

Chris Lewis — Sun, 18 Nov 2018 22:55:48 +0000

This year I participated in my first ever Hacktoberfest. Hacktoberfest is a yearly event run in October by DigitalOcean that presents developers with a simple challenge: create 5 pull requests to open-source projects on GitHub, and get a free t-shirt!

As a developer with a few solid years of experience but barely any open-source contributions under my belt, I decided that this year I would finally have a crack at the challenge and force myself to get deeper into the world of open-source.

Why? In my time as a developer, I’ve already taken advantage of a great many open-source projects to help with my work or professional development. It feels right to give something back to open-source when I’ve gotten so much out of it already. However, making first-time contributions can be quite intimidating, which is why I hadn’t really done any before now — and Hacktoberfest gave me the perfect excuse to finally break that barrier.

My Goals

Before October began, I decided that if I was to make the most of the opportunity that Hacktoberfest presents, I should set some goals for myself beyond just the basics.

Hacktoberfest’s audience spans across all skill levels of developers, and as such they place very few requirements on what you need to do to complete the challenge. For people who are new to software development, learning about pull request workflows by making 5 small documentation updates is a totally valid and rewarding way to participate. At my level of experience, however, I knew that I should push myself further to make sure I am actually learning new things during Hacktoberfest, rather than just participating in order to get a free shirt. For me, the shirt is just enough incentive to reward me for getting out of my comfort zone and meeting most of the goals that I set out for the month.

The goals that I ended up with for my Hacktoberfest were to complete:

one documentation update
one on a Microsoft project (which would net me another free shirt)
one on a language I don’t normally work in (i.e. not C#)
one to address an existing GitHub issue
one on a development tool that I use

I figured this would be enough to encourage me to make a variety of contributions throughout the month and maximise my learning.

Finding Projects

When October 1st came, I signed up for Hacktoberfest and immediately started looking for projects to contribute to. I knew that my time to actually work on my pull requests alongside my day job would be limited, so I figured I could note down a variety of projects for me to come back to later on.

I found this process actually fairly challenging. At first I was browsing the long list of GitHub Issues with the ‘hacktoberfest’ tag, but I found that many of these were either a bit too simple, were in languages that I wasn’t interested in practicing, or were just projects that I wasn’t interested in contributing to. Instead, the most effective way for me to find projects to contribute to was a combination of GitHub’s Discover page (which gives you a list of suggested repos based on your history) and looking at the dev tools that I normally use to see which I could possibly contribute to. By the time I was done, I had at least a couple of projects listed that looked good to me, but I did keep looking throughout the month.

My Contributions

By the time Hacktoberfest was over, I had made exactly five pull requests and completed the challenge. In order, those were on the following projects:

#1: awesome-vscode

A list of cool/useful extensions for Visual Studio Code that I found through GitHub’s Discover page. An extension I really liked wasn’t yet listed on the page, so I whipped up a PR to add it in. This was a nice, easy starter contribution, and (technically) ticked off my goal for a documentation update!

#2: terraform-provider-aws

I use Terraform quite a bit, so the idea of contributing back to it sounded perfect for my Hacktoberfest plans. Thankfully, the Terraform AWS had lots of ‘good first issues’ tagged that were suitable for me to pick up. Since all the code is in Go, it was also a good chance to reinforce the small amount of Go knowledge I already had.

Interestingly enough, this contribution really ticked off multiple goals in one: I addressed a GitHub Issue on a tool that I use in a language that I don’t normally use! It almost felt like cheating…

#3 and #4: vscode-hybrid-next

Hybrid Next for VS Code has been my favourite editor theme for a while now, but there were some cases where the syntax highlighting for C# wasn’t fully correct. It wasn’t until Hacktoberfest that I actually realised that I could just contribute back to fix those edge cases that I discovered!

#5: vscode-go

As the month was nearing its end, I still hadn’t yet made a contribution to a Microsoft repo. I really wanted to contribute to the core VS Code repo, but it was so popular that every good Issue I found had already been snapped up by someone! So instead, I had a look at the Microsoft-developed Go extension for VS Code, which was still quite active but not too active for me to find something to work on.

What I Learned

Although not all of my pull requests were large, each one offered at least some kind of valuable learning, and there are a couple of parts that stick out to me from the experience.

With each contribution I made, it honestly felt really rewarding to be giving something back to the projects in question. Not only did I feel the accomplishment of having put in the time to put the change together, but I had the satisfaction of knowing that this could benefit many many more people than just myself.

Working on a variety of projects also gave me some really welcome exposure to new development experiences. My Terraform contribution taught me a little bit more about its inner workings, as well as teaching me more about developing in Go. Working on the Hybrid Next theme showed me what it’s like to develop a custom theme for VS Code, and working on the Go extension gave me even more insight into VS Code’s extension development while exposing me to a little bit more TypeScript.

Most importantly, I realised that contributing to open-source projects really doesn’t have to be as scary as it might seem. The maintainers that I interacted with were friendly and helpful. Even the process of jumping into a totally new codebase was not as difficult as I had imagined, especially when you can look at other people’s pull requests as a rough guide for what you might need to do in yours.

All in all, I’m really glad I decided to participate in Hacktoberfest this year. Although I’m not sure when I’ll next make any open-source contributions, I can move forward knowing that if I see any more projects that I would like to work on, I can do so with a lot more confidence than I had before (and with two new shirts!).

Thanks for reading! How was your Hacktoberfest?