Managing QA in Production

Kat Markava — Mon, 20 Jul 2020 14:01:24 +0000

One can judge the maturity of the product by how stabile its production system is. But bugs happen, and having an efficient issue tracking and resolution flow is critical.

As a product manager, I see the need to be able to work with the QA closely and efficiently. This collaboration might be in part definitive of the the whole team's dynamic (I want to write more on this sometime).

Here's in short how we have it organized:

Logs - real-time monitoring tools may show that there is an issue. I like the level of usability on Rollbar or Kibana. Especially if their notifications integrate into Slack.

Customer Success - requests come from customers directly to the success team. They try to reproduce it, collect as many details as possible before informing the QA or handing it over to the engineering.

Daily Testing - QA team monitors automated daily tests that run in Selenium or does spot-check user tests and discovery.

Customer Surveys- alert about unknown problems, mostly with the user experience.

So once the issue is reported, the QA needs to follow these steps:

1. Verify:

reproduce the issue to make sure it was really an issue

2. Categorize:

Is it a newly 'Discovered' issue?
Is it an old one that's 'Happened' again?
Is it something that was 'Designed' this way, but turned out to be a clunky experience?

3. Prioritize:

after step 1 and 2, get together with the product manager and/or engineer to discuss the severity and the need to escalate.

IF the issue is escalated then it's assigned to the engineer who is most familiar with this particular logic in the form of a ticket (we use Jira).
The hotfix is pushed as soon as possible to PROD and the users who experienced the issue are informed about the resolution.

IF the issue is NOT escalated then the additional data goes into the QA tracking board where it lives until it's escalated or prioritized later in the development cycles.

In the early days of a young product, the role of the QA is often overlooked. Everyone does testing and debugging. Which is ok to a certain point.

As the product scales and the team grows, however, what I found to be the most important part of the process, is having the dedicated QA person. The person who has this strong sense of ownership and is passionate about the system doing its best to support the customers through their journey. Without that passion, things become much more complicated and even dangerous.

Deploying weekly and... remotely

Kat Markava — Fri, 26 Jun 2020 03:17:54 +0000

Some facts:

My team is used to deploying to production every week.
My team is not used to being remote.
My team now deploys to PROD remotely.

We have always been paying a lot of attention to the process of the release cycles, constantly chipping away at the steps, slimming and de-risking it. So, we believe we have a pretty reliable and efficient flow by now. 💪

Yet, when the first COVID-19 lockdown's release day arrived, all of us involved -- engineering, QA and product-- were sweating a lot. I mean, without having everyone within arm's reach at the office, how do you make sure that:

a. no one misses any migration steps, if they are needed?
b. all environmental variables are in place?
c. you did not reach some mysterious limits on Jenkins?

(Right, we are using Jenkins for deploys.)

These are just some of the things that can go wrong. There can be many more, of course, because our product is very complex with a ton of interdependencies and jobs that have to run in production.

That's why I wanted to share with y'all and also selfishly reflect on how we managed to not screw it all up.😅

What seemed to have worked:

A clear checklist that the lead engineer and I agreed upon and documented;
A separate MS Teams channel dedicated to the release communication;
Locking in the release window -- just as with space launches -- if we do not make it by the agreed time, we abort;
And, my personal favorite, Blue-Green Deployment that uses Blue Ocean interface and makes the new PROD build a simple click of a button (well, a few series of buttons).

To summarize, the combination of solid team communication, operational preparation, and the work DevOps had put into visualizing the deploy pipeline, made it possible for us to continuously deliver even after this sudden shift in work style.

What still needs improving:

Communication -- yeah, there can never be enough of it. Now and then, there are threads that get lost in translation or simply overlooked when someone's less than strategically placed comment pushes out critical content;

Ok, so to be fair this stuff is not easy.

I see it as a constant work in progress.

What've your experiences been like? 🤓

What advice do you have for remote CICD during a pandemic?

DEV Community: Kat Markava