DEV Community: Gaurav Singh

How to setup report portal dashboards using attributes for test observability

Gaurav Singh — Fri, 22 Nov 2024 00:00:00 +0000

Created by microsoft copilot using DALL.E 3

What is a report portal?

Successful QA engineering teams use data and test observability (o11y) to their advantage to identify areas or tests that need attention and have scope for optimization and drive corrective measures

Suppose you don’t have a good way to measure and visualize how reliable your test suites are.

It is easy to be overwhelmed by your suite’s many broken or flaky tests.

Engineers often struggle with:

Resorting to ad-hoc local spreadsheets to track bugs for a failing test
Maintain a record of which cases are broken/flaky

ReportPortal is a free and open-source solution that elegantly solves this exact problem.

I’ve personally evangelized it at scale-ups like Gojek and CRED, and have been impressed by its power and flexibility with engineering teams deriving a ton of value.

Quite often, engineering teams spend a lot of time setting up Extent reports, and Allure reports stored in S3 or GCP buckets which while good for static and presentable reports lack a lot of modern real-time analysis features that the report portal provides out-of-the-box

There is a better way; why not try it? 🤷

⚙️ Setup

Install report portal

I had previously written a few blogs on how to set ReportPortal on a single machine using docker and set logging integration using log back for a Java-based project or with Pytest for a Python-based project

You can read them below:

Here, I’ll just recap the main steps quickly so that this blog is a good starting point and self-enclosed, but will encourage you to read Deploy with Docker for more complete intuition.

You can skip point 2 since we have a usable docker-compose.yml file already in the Git repo that we’ll work with for this tutorial

Install Docker Desktop on your operating system (OS) from the Docker website. Instructions for Mac can be found on Install Docker Desktop on Mac
Download the latest docker-compose.yml file

curl -LO https://raw.githubusercontent.com/reportportal/reportportal/master/docker-compose.yml

Bring up docker containers by running:

docker-compose -p reportportal up -d --force-recreate

Create a directory for OpenSearch and give permissions

mkdir -p data/opensearch
chmod 775 data/opensearch
chgrp 1000 data/opensearch

Verify ReportPortal containers are up by running below and checking that all containers are in a healthy state

docker ps -a

Next access ReportPortal using http://localhost:8080 and use the default username and password superadmin\erebus to login

In the bottom left, click on the Superman logo and then on Administrate

This will take you to All Projects, click on Add New Project and type the project name as test-infra

Next in the bottom left, again click on the Superman logo and then on Profile

This will take you to the User profile section, click on API keys and then on Generate API Key

Enter a key name like test-infra and then make a copy of the generated key. Please note: This key will not be accessible after this screen so it’s important to note it down somewhere safe

Clone git repo

We’ll use https://github.com/automationhacks/test-infra repo on my Github account.

You can clone the repo using either HTTPS/SSH or Github CLI and then open the build.gradle file using IntelliJ IDEA

You’ll notice this project already has a docker-compose.yml file in the root directory, if you want to use the same for your setup, you could run the command to set up docker containers.

docker-compose -p reportportal up -d --force-recreate

Once Gradle downloads all the dependencies

Search for reportportal.properties

and replace the rp.api.key=<your_api_value> copied in step 10 of previous section

Required gradle dependencies are already set up in the build.gradle

testImplementation 'org.testng:testng:7.10.2'
testImplementation 'com.epam.reportportal:agent-java-testng:5.4.2'
testImplementation 'com.epam.reportportal:logger-java-logback:5.2.2'

And logback.xml with configuration for logging is present in src/test/resources/logback.xml

<?xml version="1.0" encoding="UTF-8"?>
<configuration>
   <appender name="ReportPortalAppender" class="com.epam.reportportal.logback.appender.ReportPortalAppender">
       <encoder>
           <pattern>%d{HH:mm:ss.SSS} [%t] %-5level - %msg%n</pattern>
       </encoder>
   </appender>

   <appender name="ASYNC_RP" class="ch.qos.logback.classic.AsyncAppender">
       <appender-ref ref="ReportPortalAppender"/>
       <!-- Sets the maximum number of log events in the queue -->
       <queueSize>512</queueSize>
       <discardingThreshold>0</discardingThreshold>
       <!-- Set to true to include information about the caller of the logging method. -->
       <includeCallerData>true</includeCallerData>
   </appender>

   <appender name="CONSOLE" class="ch.qos.logback.core.ConsoleAppender">
       <encoder>
           <pattern>%d{HH:mm:ss.SSS} [%t] %-5level - %msg%n</pattern>
       </encoder>
   </appender>

   <root level="INFO">
       <appender-ref ref="CONSOLE"/>
       <appender-ref ref="ASYNC_RP"/>
   </root>
</configuration>

The gradle test job looks like below which registers the report portal listener

test {
   useTestNG() {
       useDefaultListeners = true
       listeners << 'io.automationhacks.testinfra.alerting.TestListener'
       listeners << 'com.epam.reportportal.testng.ReportPortalTestNGListener'
       parallel = 'methods'
       threadCount = 6

       includeGroups System.getProperty('includedGroups', '')
       excludeGroups System.getProperty('excludedGroups', '')
   }

   // pass system properties from commandline
   systemProperties = System.properties

   // show stdout and stderr of test JVM on console
   testLogging.showStandardStreams = true

   finalizedBy jacocoTestReport
}

You can recap the blogs mentioned at the top for more details on these setups however this is sufficient to get started with pumping data into ReportPortal

Let’s run some tests

If you navigate to src/test/java/io/automationhacks/testinfra/reqres package

You’ll see that I’ve written some rest-assured tests to test the ReqRes API, in general, either get/put/post/delete certain resources and assert their outcomes.

We can run the test to push the results in the report portal and see how the launches look like by running the below commands

These are also mentioned in the README.md

./gradlew test -DincludedGroups=identity -Drp.launch=identity_tests

./gradlew test -DincludedGroups=onboarding -Drp.launch=onboarding_tests

./gradlew test -DincludedGroups=performance -Drp.launch=performance_tests

Notice here I’m filtering tests that are tagged with a certain TestNG group like identity, onboarding, and performance using -DincludedGroups, and also using the -Drp.launch param to give the launch (or test run) a meaningful name

Attributes as metadata

ReportPortal supports adding additional metadata as key-value pairs either via the command line or as test annotations

At launch

If you open Launches, you’ll see the 3 launches show up for identity_tests, onboarding_tests, and performance_tests and they also have some default attributes or tags like those below which are specified in reportportal.properties for rp.attributes

group:test_infra;test_type:backend

These attributes and tags are very useful to filter launches and create informative dashboards

In a real project, you may want to add even more attributes to a given test suite

You can override rp.attributes while running the tests via the Gradle command by appending the below

-Drp.attributes="group:test_infra;test_type:backend;team:identity"

Each key: value pair is separated by a colon and delimited by a semi-colon

Here we have added another attribute called team so that we can derive some team-level metrics using this

Our test commands look like the below now

./gradlew test -DincludedGroups=identity -Drp.launch=identity_tests -Drp.attributes="group:test_infra;test_type:backend;team:identity" --info

./gradlew test -DincludedGroups=onboarding -Drp.launch=onboarding_tests -Drp.attributes="group:test_infra;test_type:backend;team:onboarding" --info

./gradlew test -DincludedGroups=performance -Drp.launch=performance_tests -Drp.attributes="group:test_infra;test_type:backend;team:performance" --info

You can observe that now we have another attribute at the Launch level to filter results by.

At test

We can also add these attributes at a test level

Say, you want to store who is the on-call or maintainer of a given test so that seeing a failing test indicates the current owner for context

We can annotate the test with below

@Attributes(attributes = {@Attribute(key = "team", value = "identity")})

The complete test looks like below:

public class ReqResCreateUserTest {

   @BeforeClass(alwaysRun = true)
   public void setup() {
       RestAssured.baseURI = "https://reqres.in/api";
   }

   @Test(groups = {Team.IDENTITY, Groups.SMOKE})
   @Attributes(attributes = {@Attribute(key = "team", value = "identity")})
   public void testCreate() {
       String requestBody = "{\"name\": \"morpheus\", \"job\": \"leader\"}";

       given().contentType(ContentType.JSON)
               .body(requestBody)
               .when()
               .post("/users")
               .then()
               // TODO: Broken test example, change to 201 to fix
               .statusCode(200)
               .body("name", equalTo("morpheus"))
               .body("job", equalTo("leader"))
               .body("id", notNullValue())
               .body("createdAt", notNullValue());
   }
}

Now if we look at the test in launches, you’ll notice the attribute is also stored at a test level.

It is possible to add multiple attributes at a test level or have multiple keys with the same value or one key with multiple values. You can read the details in test item attributes wiki from ReportPortal

Below are quick snippets taken fromthe above wiki for reference

Many keys to one value

@Test
@Attributes(multiKeyAttributes = { @MultiKeyAttribute(keys = { "k1", "k2" }, value = "v") })
public void third() {
 Assert.assertEquals(1, 1);
}

One key to many values

@Test
@Attributes(multiValueAttributes = { @MultiValueAttribute(key = "k", values = { "v1", "v2" }) })
public void fourth() {
 Assert.assertEquals(1, 1);
}

Multiple attributes

@Test
@Attributes(attributes = {@Attribute(key = "key1", value = "value1"), @Attribute(key = "key2", value = "value2")})
public void sixth() {
 Assert.assertEquals(1, 1);
}

Dashboards

Let’s unpack the power of visualization using dashboards

There are a couple of distinct personas report portal supports

🤹 As a leader

I may want to see broad metrics for the 3 teams that I’m leading and identify hotspots that need attention.

A few widgets make a lot of sense such as:

Component health check (table): See component level health by splitting them using attribute team and observe overall pass/failed metrics in a tabular format
Component health check: See the split of the last test suite at a team level into visual cards with the option to open actually failed tests
Launch statistics chart - bar view: Bar chart with a summary of how many cases passed/failed/skipped
Overall statistics - donut view for latest launches: Donut chart with a summary of failed/passed cases and how many cases are left to investigate

You can also see additional charts like

Test-cases growth trend chart: to see how many tests were added or removed from the suites over time
Failed cases trend chart: to analyze the status of your broken tests

How can you configure these?

The magic lies in creating filters with a combination of certain attributes and then using them to create the visualization you desire.

I promise it’s pretty intuitive once you play with it.

For instance, below is how you can set up the component health check table

Select a filter that filters all launches having group equal to test_infra
Select the latest launches to see the last launch
Select team attribute as a way to split the data

And say you want to configure the Overall statistics widget

Select filter as “group as test_infra”
Select Donut view
Select Latest launches

And you are done.

You can also edit the filter by adding more conditions, like below

🧑🏻‍💻 As an engineer

The previous visualizations are great for getting a bird’s eye view of how a given group with multiple teams or a product is doing.

Often you as an engineer would rather focus more on how are your tests behaving and may have a team-specific or flow-specific dashboard.

There are a few more dashboards that are more useful for analysis.

These are focussed mostly at a launch (or one test suite level) and can be used as lower-level dashboards to drive analysis and fixes in a focussed area.

Component health check: sliced for services to find the health of fine-grained services
Passing rate per launch: to find the overall status for the last launch
Flaky test cases table (top 50): identity the top flaky cases to look at
Most failed test cases table (top 50): identify all the broken tests

Launches duration chart: to see which launches take how much time in order to find slow-running suites
Most time-consuming test cases widget (top 20): to find slow-running tests to be optimised

Recap

To summarize

Report portal is an excellent test observability solution adhering to true FOSS (free and open source software) spirit that can be installed over Docker containers on your private network
Setup is quick and frictionless with running docker-compose, creating a project, getting the API key, and setting up logging integration
Launches provide you last test run status with support to analyse cases
Use attributes at the launch or test case level to add rich filtering capabilities
Create dashboards with either aggregated metrics or low-level analysis metrics suitable both for a leader or engineer persona

So what are you waiting for?

Go ahead and have a much better picture of your test suites and keep them in good health 🫶

| Thanks for the time you spent reading this 🙌. If you found this post helpful, please subscribe to the substack newsletter and follow my YouTube channel ( @automationhacks ) for more such insights in Software Testing and Automation. Until next time 👋, Happy Testing 🕵🏻 and Learning! 🌱 | Newsletter | YouTube | Blog | LinkedIn | Twitter. |

🎤 Good communication makes great software engineers

Gaurav Singh — Tue, 29 Oct 2024 00:00:00 +0000

Created by microsoft copilot using DALL.E 3

How can software engineers be more confident communicators in their written, spoken and non verbal communication. This blog provides practical tips and insights from my decade plus career as a software engineer.

You may have heard this anecdotal advice earlier

“Improving your communication skills will help you land senior roles and unlock career growth beyond technical skills”

What does good communication skills mean in practice?

Let’s dive in.

Disclaimer: I’m not a communication expert; i’m just going to reiterate some practical advice and insights that i’ve picked up over my career as a software engineer.

To judge what good looks like, observe; and just look around you.

At the office, conferences, meetups or the internet and notice what makes someone stand out in their communication?

You may notice the following traits

Clear and concise thought process
Excellent written communication - Ability to write clear messages on chat messaging, 1 pager or design docs, tech blogs
Confident spoken communication - Ability to express the ideas and thoughts confidently. They do not appear confused and don’t use a lot of filler words like um, ahem, etc
Positive Body language - A smile on their face, and calm demeanour even in arbitrarily challenging situations

Some of these may be innate personality traits or a function of their environment from home, school, college or initial workplaces but most of them can also be learned, just like any other skill

How do you become a better communicator?

IMHO, There is no secret sauce.

Good communicators are not born with it.

They like many others develop and hone their craft over a no of years. Each day improving just a little bit. A large part of it is doing the work and practice. This also involves being open and willing to put yourself on shaky ground or in uncomfortable situations to learn from.

As a senior engineer. I would expect you to have a solid grasp on good communication skills in 3 key areas:

Written
Spoken
Body language

Let’s break it down shall we?

👩🏻‍💻 Written: Great writing is great thinking

What can you do to improve your writing skills?

Here comes the obvious advice. 😓

Take the time out and write a lot.

It may mean spending a little extra effort in writing the report for your stakeholders and make them more engaging without adding a ton of fluff. You may find value in reading Technical Writing - Google for Developers

A small tip is to always start a doc with a summary or a TL;DR (too long don’t read) and then break down your content into What, why, when, who, where, and how (5W1H) or the standard problem space, solutions, metrics, and next steps format.

The format is not that important.

You need to form a habit to write and write well.

Also, feedback is a gift. Try to get your design docs reviewed from senior engineers and seek their feedback on how to improve

Communication at work will only take you so far

To develop as a writer you should try and write either blogs, newsletters, or short notes. These could be internal or better still a public blog where you write about whatever you are curious and passionate about.

Even if one person (that’s you) reads it, it’s enough

You’ll notice your written communication skill will compound over time.

To build a good vocabulary, reading could be a supplement. Observe how others who write well compose their paragraphs, break down, and elicit thoughts. Initially this could help you maybe mimic their style and then develop your unique voice or writing style.

If you don’t take anything else; just keep at it.

If you could build a habit of writing regularly and derive some sort of satisfaction from it, you’ll pretty soon move on from mediocre write-ups to more polished ones

Writing well is a super power. Especially, in the days of LLM (large language models) and Gen AI being all the buzz. Having the ability to communicate in natural english in a nuanced manner goes someway in getting better results.

Also, Writing helps you slow down your thoughts and make a lot of concepts more concrete in your mind

Great writing could also incorporate storytelling but it is easy to get carried away in the beginning. It’s better to focus on concise and clear writing first and then when you get a hang of the basics, move on to more advanced steps

Here are some actionable tips for you to consider:

Start a personal tech blog
Write on the internal company blog
Develop the habit of writing 1 pager or design docs for your projects and improve and iterate with feedback
Learn touch typing to improve typing speed and accuracy, it will help you compound like nothing else
Use a grammar and spelling checker like Grammarly or Google Docs Spell Check and internalize the writing principles and corrections they suggest.

🔊 Spoken: Finding your voice

What about being a good confident spoken communicator?

This may be especially hard to develop as an introvert since you may be afraid of what others will think about you if you say something weird. Trust me. Nobody cares! Engineers’ brain space is full of problems they are solving and they often focus on their thoughts. No one notices the small stuff.

Remember your peers and managers are not mind readers.

If you cannot find a way to communicate your thoughts in spoken words, it is very hard to be an effective software engineer

Please don’t take it the wrong way though; I’m not suggesting you become a loud-mouthed obnoxious person on the team with fake confidence

I’m talking about becoming someone who really know their stuff and can also express their thoughts as well. A deadly combination if you ask me.

Here is a practical tip \

Don’t be a passive attendee in meetings. You may feel like you can multi-task on that coding task and get something else done, but if you think about it, it’s quite hard to do so with split attention.

If you find a meeting where you are a contributor. Do a bit of research before the meeting and create some small bullet points on what points you want to discuss and then actually bring them up.

This may be scary for the initial few times, but the more you do it. The more you find your voice automatically affords you a seat at the table. Your manager would be very happy with you for sure as they are often in silent rooms with less participation from team members

Another way of improving spoken communication is to give talks

Prepare a topic and then submit a CFP (Call for proposal) at conferences. If you are accepted as a speaker, it may be quite a jarring experience to prepare and deliver the talk but trust me you’ll love the experience afterward as folks usually come up to you after and discuss their ideas and ask their questions. The networking from these events does not hurt as well. You never know what doors open for you with each new connection.

Some more actionable tips:

Prepare for your meetings and then speak up during them
Present at internal tech talks within the team, company
Present externally in conferences and meetups

Want to sound more confident in your spoken communication?

Below are few examples that illustrate how during spoken communication we do a lot of self sabotage.

Remember: You can always choose a more positive and confident tone.

I’ve found short reels by jefferson fisher on instagram quite helpful for this purpose.

Not Confident: “Sorry, I’m not sure if this is right, but maybe we could…”

Confident: “I propose we consider this approach…”

Not Confident: “This might be a silly question, but…”

Confident: “I have a question about…”

Not Confident: “I’m just a junior developer, but I think…”

Confident: “I believe that…”

Not Confident: “Um, I think maybe we could try using a different approach here, but I’m not sure if it would be better.”

Confident: “I recommend we explore using a different approach here. I’ve done some preliminary research, and I believe it could improve performance.”

Not Confident: “Sorry to interrupt, but I have a question about the database schema.”

Confident: “I’d like to raise a question about the database schema.”

Not Confident: “I don’t know if I’ll be able to finish this task on time. I’m still learning the new framework.”

Confident: “I’m working hard to learn the new framework, and I’m committed to completing this task. However, I may need some additional support or an adjusted timeline.”

What if you really don’t know?

Hey, we are all there! No shame in admitting that but you don’t have to sound overly apologetic or worse try to cover it up with some half assed answer

You could instead say:

“I don’t have the information on that right now, but I can look into it and get back to you.”

“That’s a great question. I’ll need to do some research and come back on this.”

“I’m not familiar with that specific aspect, but I’m happy to learn more about it.”

Also, Learning good conversational patterns is also a skill. You can get better at it.

🦿Body language: Non verbal actions matter

This one is especially hard unless you train yourself

Here are some small tips:

Make eye contact and nod when you agree as you talk to someone (either in person or even on webcam in a meeting). It shows you are listening actively goes a lot towards building trust.

Having an upright open posture with a smile goes a long way in establishing you as a confident individual.

If you always tend to have your arms crossed or slouch a lot, you may come across as someone who is not comfortable

I found a nice read in this article → How to Understand Body Language and Facial Expressions, and i’m sure you’ll find a ton of resources on this on the internet. Please feel free to do your own research and practice having a more open and confident body language

It helps!

Summary

So let’s bring it all back now.

I hope you leave with these key takeaways.

Written Communication:

Practice writing regularly through reports, documentation, blogs, or personal notes.
Seek feedback from senior engineers to improve.
Develop a clear and concise writing style.
Use tools like Grammarly to improve grammar and spelling.
Learn touch typing for speed and accuracy.

Spoken Communication:

Actively participate in meetings and prepare talking points beforehand.
Present at internal and external tech talks to build confidence and share knowledge.
Use a confident tone and avoid filler words.
Be willing to research and learn to answer questions effectively.

Body Language:

Maintain eye contact to build trust.
Practice open posture and a friendly demeanor to project confidence.

Now what are you waiting for. Go out there and practice. You got this! 💪

Is there growth in testing?

Gaurav Singh — Sat, 12 Oct 2024 00:00:00 +0000

Created by microsoft copilot using DALL.E 3

Over the past month, I have mentored 3 engineers directly on topmate. It’s been a fun experiment, and I love the experience of talking to new engineers at different stages in their jobs, hearing about their challenges and perspectives, and hopefully nudging them in the right direction. If you or someone you know wants to get in touch with me, here is my topmate page: https://topmate.io/automationhacks/

I’ve been getting a recurring question from them and over the years many other engineers have reached out to me on LinkedIn, and Twitter DMs, asking me what is the growth ladder for a testing professional

Some variants of this question are:

“I’ve heard testing maxes out at 10 years of experience. Should I move to dev or management?”
“My manager is suggesting that becoming a manager is the way you can progress in your testing career”
“I feel stuck and don’t know what else to explore to transition from QA automation to SDET”
“Is testing a dead-end job, is there a career to be made in this?”

These are critical questions and each engineer is right to ask them

They also paint quite a gloomy picture of the role

Let me help you paint a better picture

🪜 Job Ladder

There are tons of different testing setups in the industry.

If you want to get a feel of it you can read Deep dive into evolution of testing organizations in which I’ve explored some of these.

There are 3 major roles that people play in the testing space

Test Engineer
Software engineer in Test (SDET)
Engineering manager (EM)

In many companies there are well-defined career ladders for each of them, some it is a very flat org and a whole buffet of different setups in between

🧑🏻‍💻 Individual contributor (IC)

I’ve seen companies have a level distribution generally like the one below for individual contributors

Intern, fresher, entry-level (0 - 2 years)
Mid-level (2 - 6)
Senior level (6 - 9)
Staff/Lead level (8 - 12) (may have direct reports)
Architect/Principal (12+) (may have direct reports)
… some further levels like senior principal/architect, distinguished (this is quite rare)

A small caveat:

Please don’t take the years of experience thing so seriously, they are general approximations

I don’t believe in the years of experience level filtering because while indicative of breadth of experience it may not convey the level of depth your company needs for a given role.

The notion that years of experience automatically entitles you to a given level is a bit absurd. You may not have exposure to that level of responsibility, impact, and scale. Past performance, strong aptitude for the role, and transferable skills are probably better indicators, but hey, who am I to question everything that’s broken with recruiting and compensation 🤷

In some companies, Test engineer roles are capped at a certain level, while SDET role generally has these levels.

One clear signal to someone reading this is that developing software engineering skills is crucial to your long-term success as an IC

👩🏻‍💼 Engineering manager (EM)

Typically there are below levels

Engineering manager (5 - 8)
Senior engineering manager (8 - 12)
Director (12 - 15)
VP (Vice president) (15+)
SVP (Senior vice president) (15+)
GM (General manager) (15+)

VP+ levels are usually present in larger companies, but levels up to Director are quite common

Many companies may cap the levels at a certain point depending on whether that role is required and if the company is big enough

I would not go in-depth on what are the roles and expectations from each level in this blog.

Any solid leadership group worth their salt would have enumerated this as an internal rubric, but let me know in the comments if you would like to hear my take on it and I can expand on this as well

🧗🏻 Climbing the ladder

Your level on the engineering ladder is a strong indicator of growth.

It has implications for your title, and compensation and directly influences the level of agency and responsibility you can have in the org as well as your impact

What does the path look like to climb this ladder?

This is a pretty deep question, it may even demand a book read like The Staff Engineer’s Path: A Guide for Individual Contributors Navigating Growth and Changeor The Manager’s Path: A Guide for Tech Leaders Navigating Growth and Change or even a new one just for professionals in testing

I’ll attempt to give you an ELI5 (explain like I’m five) answer without going into deep nuances

Loosely speaking,

Learn: Became a deep expert in your domain, make sure you have the technical proficiency needed to do your job, which may involve learning a new technology, stack, programming language, or people skills
Execute well: You should clearly understand what role you are playing in your org, talk, and ensure you and your manager are on the same page on this, then go out and execute, execute, and execute. Keep on knocking off project after project and do the damn best job you can do on them with the customer in mind. Make sure you are in the top 5% of your team and try your best to create value everywhere and be a force multiplier on the team.
Perform at the next level: Then look at your level + 1 role and expectations and ensure you take some of them on. Keep on knocking them off and present a demonstrated history of strong performance. Performance appraisals and reviews are usually lagging and if you are already performing at the next level and your manager is good, chances are you can expect a promotion soon.
Seek and mentor: Find out the best people in your company, local region, or globally, and follow them. Learn from them and soak their wisdom. Seek out mentors discuss ideas with them and go back and implement and show hands-on value. You should also pass on what you’ve learned to help others around you get better

This advice and general framework has personally served me quite well in my career but like I said earlier, there is tons of nuance here. Maybe I’ll write some more about these in future blogs.

🦖 Develop the Breath

Tech

Is climbing the ladder all there is to it?

If you’ve reached the Principal/Architect level, have you made it in your testing career?

What’s next?

You’ll find that only seeking compensation, title, and level as proxies for growth can turn out quite shallow in the end.

Please don’t get me wrong, it is super important and you need to do a good job at it.

✅ Your life and your family depend on you for this. I will encourage you to keep on working on it to make sure you are rewarded and paid well for your efforts

❌ But it’s not the only thing.

When people say testing maxes out at X years of experience, they are taking a quite shallow view of using the job ladder as the only growth indicator

But testing is not just your years of experience

It’s a crucial part of SDLC without which no software should ever go to production.

You’ll be a fool to ignore this at your peril.

There is a ton of breath to be covered and if you are a person on your way to becoming a master, you have much to learn and experience young padawan 🥷

Let me attempt to give you a peek at how I think about it

Areas to explore

We can divide the software engineering world into 6 broad areas

Mobile
Web
Backend
Infra
Data
ML/AI

Each of these areas has opportunities to choose which layer you write tests at:

Test Pyramid

Unit
Integration
E2E

And broader areas to apply to the above

Engineering productivity
Code coverage
Exploratory testing
Building community with like-minded peers
Test frameworks
Test infrastructure
Test tooling
CI/CD
Security
Performance
Accessibility
Chaos/Resilience

You can take a cartesian product of these three and that is one unique combination for you to explore deeper

E.g. Mobile test infra, data security, data performance, backend chaos/resilience, etc.

It’s a whole wide world out there for your explore

I have 13+ years of professional experience in testing and I can safely say I’ve just scratched the surface

I also don’t think I’ll be able to explore them deeply in this lifetime but I’m still curious and choose to make incremental progress towards some of them.

Each of these areas in itself could be a project for you to explore and become an expert in over a few quarters

The good news is there is a ton of overlap and you can build a lot of deep software engineering skills; such as

Reading code
Writing code
Understanding programming principles
Understanding system design: How do DB, Queue, cache, CDN, and Gateway work and are scaled, how do you test with them
Building solutions
And so on …

🍦 Soft skills

You’ll notice that above is a list of technical things

To grow into a mature leader you also need polished people skills.

They are table stakes if you are in a manager role or in an IC leader role like an architect

Building trust with peers
Driving cross-functional projects
Project management
Strong written and verbal communication skills
Empathy
Servant leadership
Engage and develop others with 1:1 mentoring
Sponsoring someone’s career growth
Public speaking in internal presentations, all hands, meetups, or conferences
Negotiation
Conflict resolution
Succession planning and removing single points of failure in teams
Setting direction and strategy
So on and so forth

We can keep on adding to this list as well

What’s next

Say you are someone who has worked on most of these and still feels stuck.

How do you grow then?

Firstly congratulations 🎊

I’m sure you are in a very small company of elite people who have proven experience with all or most of these.

Well,

Software engineering is huge.

If vertical growth is not possible, you may consider even shifting to a horizontal ladder in a different set of skills like a developer, product manager, data scientist, AI/ML engineer, etc.

As you can see there is no endgame in this world and endless challenges are waiting for you with great rewards.

All you can do is:

Work hard, have fun, and make history

That is Amazon’s motto and a pretty good way to summarise this blog

Keep on delivering exceptional value for your customers, because that is what truly matters

If your company does well, you will also do well and leaders around you will be happy to share more responsibilities.

Growth will happen organically.

Go out there, kill it!

Hope this helps. Let me know in the comments if you want to share a personal journey around how you grew in software testing or what I’ve missed. I’m looking forward to learning from your thoughts

📛 Common terminology misses in software testing

Gaurav Singh — Thu, 26 Sep 2024 00:00:00 +0000

Created by microsoft copilot using DALL.E 3

I posted about this topic on LinkedIn and saw a lot of engagement. Somehow I have this eerie feeling that I’ll be pointing people to this blog in the future as well 🔮

It’s not automation testing, it’s automated testing - We are not testing automation, we are automating tests
It’s not QA Automation, it’s just Test automation or better Automation In Testing (AIT) - we cannot fully automate quality assurance, it is a broad term and some activities will require human intuition, we can automate tests to give faster feedback at scale.
It’s not a QA Automation engineer , call them SDET or Test engineer please, and yes people with these titles both write automated tests but there are differences, read this
QA Engineer is not the same as SDET , again there is a difference 😉 - read blog in point 3 for more details
Flaky tests are not the same as broken tests, there is a difference 😉 - Broken tests are nice as they fail all the time, but flaky tests? not so much, sometimes they pass, sometimes they fail, but they always point to a nice problem underneath to debug, don’t ignore them, and don’t retry them 5 times.
Assertions and checks are mostly the same! I rest my case!
Quality engineering is loosely the same as QA Engineering as a term to refer to the organization - dropping assurance does not make much difference. Leadership does implicit conversion anyway and applies their old biases. Calling it Test engineering is more explicit
QA is a process, it’s not an engineer - Would you refer a person with something abstract? Probably not. Calling them testers is much better and more explicit. 🙏
100% automation , really? - your energy is much better focussed on ROI (return on investment) driven automation. The Pareto principle exists for a reason. Chasing vanity metrics just leads to unnecessary suboptimal waste

Your mileage may vary. We can refer to the same thing with different labels and that’s completely okay

Did I just stir up a hornet’s nest? 🫣

I’ve learned to not care about these much but occasionally push some buttons and I prefer some over the other. I thought I’d just put it out there.

🫨 Should an SDET move over to SDE?

Gaurav Singh — Sun, 22 Sep 2024 00:00:00 +0000

Created by microsoft copilot using DALL.E 3

This question often crosses the mind of people working as an SDET one or more times during their careers. In this blog, I offer a personal story from my own career and hopefully give you a mental model around thinking about this.

Let me start with a personal story

Back story

In 2018, I joined Gojek as a Product Engineer QA with around 7 years of experience. I joined as the 3rd full-time QA engineer amidst a large consultant team.

During my initial month, I met another bright engineer with around 4-5 years of experience who was super passionate about Ruby programming.

In Gojek, we had a Java-based mobile and API testing framework used by teams already, he wanted to build his own version in Ruby. That seemed like an audacious goal that surely would not be able to land well without some solid technical justification, leadership, and influence

Oddly, I had just come from a Python background and was head over heels in love with the language, and wanted to use Python to rewrite our framework into something more simpler.

Ah, the naivety and simpler times.

We debated over which language was better and drew and architecture but could not come to terms with language choice among other things, and that eventually led to this idea not really materializing after a couple of weeks of effort and we both went to focus on our own teams

After couple of months into my new role, I heard that person is switching over to being a backend engineer.

🧑‍💻Should I switch to dev?

I wondered curiously if switching to a developer role makes sense.

At the time, I had personally not come across a lot of examples of people who had made this transition

I also did not feel strongly about making the move myself.

Why?

It was quite simple for me

I loved software testing ❤️

I also loved coding ❤️

I felt I was good at both

This love for the craft of software testing did not come originally but I developed it organically as I grew in my career as a tester

I started to appreciate learning its nuances. I loved peeling one more layer of the onion at a time, every new project or challenge helped me become better by adding another tool or approach to my toolset. I was a journeyman on the way to explore different depths of testing.

I also felt that there were lots of things that I had not yet explored yet

I did some personal reflection at the time and tried to imagine how my role would be different if I was to transition to a developer role

I would have to get trained on one specialization like backend, mobile, web, or data and self-learn most of it myself or on the job.

I would write a lot of code and get better at it

Probably take care of on-calls and any production incidents and earn my keep as a mid-level developer

But, It would probably mean not getting enough time to test and I would mostly be in one vertical to have any reasonable chance of being any good at it.

I might switch teams and domains but the core skill would remain this at least for sometime

I did not like the notion of being with one technical domain.

I wanted to be able to switch between web, mobile, backend, performance, accessibility, tooling, etc, and see how I could bring my testing mindset to add value in these diverse domains

Each of these areas appeared super interesting to me and I liked that as a tester and programmer in general I could learn and switch in these domains in service of testing

At the time, I also personally felt that the experience I’ve accumulated to this point in testing would go wasted. That thought was probably not true. I’ve come to realize that solid testing skills are transferable and super valuable in the life of a developer as well

Heck, I would not be far off to presume that software testing is more than 50% of the job of any decent developer anyway.

If it’s well tested, its not really developed is it?

💰Hey, what about the money?

Would I make more money as a developer?

Probably!

that was certainly attractive to consider.

However, my 1st hike at Gojek proved this wrong as I was paid on par with developers with similar experience and levels. I also personally did not index a lot on pay but more on learning and developing myself as a professional.

I was lucky that I was in a company with solid leaders who took care of compensation without me having to worry about it.

So in the end it was an easy decision for me.

I decided to stay in testing

Within 3 years, I grew from an individual contributor (IC) as a Product engineer QA to a Lead SDET in a couple of years and then on to an Engineering manager SDET within another year managing a team of leaders and ICs

It was fantastic growth. I worked with a diverse team of engineers, some into exploratory testing, and some deep into coding and building frameworks and tools. I enjoyed building a backend community with other SDETs and also forayed into public speaking at conferences, writing a blog, etc.

I met talented, humble people who were all passionate about the craft of software testing and doing it well to ensure our company can succeed in its mission on providing a quality experience to our customers.

It was a ton of fun

🤷 Who is an SDET anyway?

I also realised that as a Test engineer turned into SDET

I am not really a tester, nor a developer, and not a product manager

I’m someone in between who oddly can empathize with each function and possess some of the skills each people in above roles have

I have my legs in multiple worlds.

I’m a developer who thinks, breathes, and eats testing for breakfast and loves it

I obsess over coverage, over getting fast and reliable testing signals so that teams can make decisions to ship or not

I love building scalable tooling, frameworks, and infrastructure to help everyone test better

I also love understanding how the actual applications or systems work under the hood not just functionally or non-functionally but what they mean to an end user.

I can think in user flows and journey touch points and also step into the details of the code and also zoom out at a user level

This is a pretty amazing and very challenging role if you think about it.

I wrote about all the things an SDET is expected to do in this blog, Who the heck is an SDET? Feel free to give it a read to form a better intuition.

SDET vs SDE?

In my view, we are both programmers at heart

The difference lies in that we as software engineers are just focused on different things, working together to bring customer value sooner and safer

But really, who is an SDE?

An SDE has lots of things on their plate

They are very much focused on taking requirements from products, and businesses and using their knowledge and experience of the systems, to translate it into changes in the application.

They may write new APIs, make changes to existing ones, write a new app screen, or sometimes build a completely new one from scratch. They may sometimes work on general tooling as well at CI/CD, deployment level

SDE also focus on testing, they write unit tests and integration tests with mocks and may do some manual testing or occasionally write an E2E test as well.

I have personally seen many SDEs who are very good testing minds as well and such individuals experience high growth in their careers since they are usually appreciated by their leaders for shipping services or apps with high-quality

They also fix any bugs in the system either self-discovered or by someone other than them

SDEs are experts in their own systems and sometimes get tunnel vision by focusing too much on their code where they are not able to zoom out and see the feature as a whole.

It’s not really by choice, the amount of work and pace is often demanding enough and when put under the burner to deliver faster, a dev may opt for doing just good enough testing leading to missing bugs which are a factor of simulating different conditions

SDE’s also take care of deployments and ensure production systems are running smoothly, occasionally resolving any production outages

They focus on elegant software architectures and learn about many different technologies based on their domain.

What about an SDET?

You should read the above blog to get the full description, but let me give a short summary

An SDET is a developer who focuses on testing problems.

This person can write application code (APIs, apps, tools, etc) if required but spends the majority of their time building frameworks, tools, and testing infrastructure to enable everyone on the team to do better efficient testing.

The line is really quite thin between the two roles with a lot of overlapping skills

This also means that an SDET is expected to develop generalist software engineering skills based on the domain or team they serve.

If they are in a backend platform team, they should understand the backend architecture, tooling, programming language, and CI setup so that they can reason about how to test them better

If they work on a mobile team, they should understand the app codebase, what testing frameworks and solutions exist, and be able to work with and enhance them

In some teams they may author automated tests at unit, integration and E2E level, in some, they may not

An SDET could work on different problems depending on their company size, stage, and setup

Some examples of problems they usually solve:

Build code coverage tooling
Build test observability tools and solutions
Bring tooling to do better testing at unit test levels like mutation, pairwise,
Build distributed systems testing solutions around reliability, resilience
Test boundaries of a system at scaled load
Work on testing mobile, web, and backend systems: Each has its own set of nuances and challenges
Tooling for security
Work on efficient CI/CD systems

I could go on and on and on … it’s a really long list

It is also hard for one person to be knowledgeable in all of these and usually requires a whole career and lots of bandwidth to develop breath as well as depth in a few

Also, SDET != QA?

A slight tangent

Calling an SDET a QA engineer is very confusing, and honestly changes the narrative about what they are expected to do.

The testing industry has come with many different titles to describe different activities that people do and for the benefit of colleagues in general, we roll up everything to QA, more on this topic in Testers identity crisis - QA/QE/AE/SDET/SET/TA? and The problem with titles for testers

QA (Quality assurance) is not a dirty word

It has been bashed and confused a lot

Often managers hear QA and they would be quick to dumb down and reduce people with this title as simple button pushers which is not true to reality.

Some people find a hack to rename it to QE (Quality engineering) but that is also quite ambiguous

I would just prefer if people call it what it is

Test engineering

We are focussing on efficient testing at scale, nothing more, nothing less.

So, should you switch?

I can’t and won’t make that decision for you.

It’s personal.

You have to decide for yourself as you know your own context the best

Hopefully, this blog gives you some sense of what are the differences between SDE and SDET so that you can make an educated guess about what role would you prefer

Also if you decide to move to a dev role, it is not a one-way door.

I would rather think of it as a two-way door and if you don’t like what you see, you can always switch back as an SDET or to some other vertical.

Software engineering is a wide enough field to support multiple lateral transitions and we should not see our careers as being rigid or sticking to one thing only

Go on, explore. Try out different roles and see what you really fancy. It’s a whole wide world out there.

Why don’t you take a leap of faith?

⚖️ Balancing product vs platform engineering as an SDET️

Gaurav Singh — Thu, 12 Sep 2024 00:00:00 +0000

Created by microsoft copilot using DALL.E 3

Any SDET working on a product team faces this

Does any of the below sound familiar?

“We want to focus on building tooling/frameworks and infra to enable everyone on the team to test better but we don’t have the bandwidth to build this out”

“Our current setup is sub-optimal, we need things to improve, but also need to deliver on projects X, Y, and Z which are critical and time-sensitive priorities for the org”

“This is part of your core competency and expectation as a senior engineer and you must dedicate X% of your time to this”

“I feel overwhelmed and burned out and just don’t have the time for this, can we set realistic expectations”

“There are tons of meetings every day, how can I ever have time to work on this?”

“We cannot use BAU as a reason for not working on this”

Sounds familiar?

Some of these may be strong statements, but you get the drift.

Is this a problem? 🤔

Yes and no! 😏

If you are an SDET embedded in a product team where there are always next projects or priorities lined up for you and your company also has expectations around product engineers doing platform work as well, you are surely going to be in this situation at a certain point in your career

An interesting side effect of this is that SDET on these teams almost start to feel guilty for not being able to do platform engineering work, leading to a false sense of degraded job satisfaction, lack of motivation, and unnecessary friction toward product testing work

Many engineers feel a strong pull and must work on more cross-cutting problems.

Some do seem to find the time to do it as well. What are they doing differently? 😖

Before we peel more layers of the onion, let’s define what we are talking about here first

What is platform engineering?

It has many names within the software testing space that I’ve come across like test infra, engineering productivity, Testing center of excellence (CoE), etc…, but the idea is quite simple at its core

Engineers build testing tooling, frameworks, and infrastructure to help everyone test better thereby increasing shipping velocity without sacrificing quality.

This is interesting work and is also highly valued in many big companies depending on whether leaders recognize the value of infra and tools. They know even a small productivity gain for engineers can compound massive savings and speed boosts.

It’s sexy because it is complex and challenging with opportunities to learn and develop deep skills as a generalist software engineer. Who would not want to work in this space?

Also, What could this work look like in practice?

One month you may be building bespoke tooling for a specific org problem or another month solutions that are more cross-cutting spanning code coverage, mobile device management, Test observability, CI/CD pipelines optimizations, intuitive frameworks on the backend, frontend (mobile, web), general test tooling for data setup, test environment isolation, etc

The list is long.

The role demands dedicated bandwidth , which unfortunately our engineer in this situation does not have

Many companies have realized the simple fact that this is a different kind of work to product engineering work and build dedicated specialized teams to work on these and then evangelize solutions to other teams.

Such teams gather feedback and then keep on iterating, hopefully making things seamless over some time. Since the nature of the work is quite ambiguous without clear ROI many times, leadership may find it difficult to justify investments in this space as well.

But, I digress. That is a tangent for another time.

Let’s come back to our original issue.

What can you do? 🫢

Now that we have ample context about the problem

Let’s explore some solution spaces

Product work is not all bad

How you think about a problem matters

I think we need to change the narrative a bit.

As an SDET, we are accountable for the quality of a given feature/group/service, etc. It is our core job role to ensure anything that is shipped is of high quality and delights the customer.

Efficient testing at scale is one clear lever you have but there could be other ways as well like getting devs to help out and own a larger chunk of test authoring, doing mob testing, building automated solutions that can run continuously and provide rapid feedback as you tackle increasingly complex problems

At the end of the day, You should remind yourself that your role is really in the service of the customer.

You are here to act as a force multiplier and get everyone on the team excited about testing.

You will need to be on top of changes, immerse yourself in context, attend meetings, influence the product roadmap, and of course handle the entire QA cycle of test planning, design, execution, automation, reporting, and bug life cycle and occasion prod issues

Before graduating to platform engineering, you need to be excellent at all of the above.

Please remind yourself that this work is what product, business and leadership care about.

You have to make peace with this situation and dedicate a large part of your day to ensuring that this objective is served. This is full-time work and you should never feel guilty for doing this

But …

We need to look further as well right?

What about platform engineering work? 🧰

If you think about it

You practically have three options infront of you

Move to a dedicated platform engineering team 🧑‍🤝‍🧑
Make focus time in your day-to-day job to work on platform projects 🧑‍💻
Join/form a community and do things bottom-up 🙋

Move

The 1st option is a no-brainer, in the short term, it may be difficult to materialize but works out better in the longer term.

If you can move to a team whose sole charter is to focus on platform engineering work within testing

You will have all the bandwidth you need to pick the right problem and work on it

The funded team will have a clear charter and roadmap, a set of problems they are solving, and automatic leadership buy-in.

It’s also a little scary once you shift your mindset from product engineering to platform engineering. You’ll realize the patterns and mental models you have in one do not translate immediately to the other.

You will need to acquire and build broader and more generalist software engineering skills. Also, develop awareness about different tools and technologies and the right problems to solve. This role requires judgement, communication, and external awareness to identify industry solutions and approaches to problems as well

These like any other skills can be learned to a degree, but you need the right approach, growth mindset, and the willingness to put in the time hopefully with some experienced mentors

Make the time 🐸

If option 1 is not feasible, because your team does not support internal mobility or such a group or team does not exist itself.

What can you do then?

You should first align with your manager that you want to spend X% of your time on platform engineering work. In my experience asking for 20% is fair. If you think about it, it is 1 day in a week or 2 hours every day for a 40-hour work week and usually a small enough period that other stakeholders in your team will not even notice.

If your manager does not care or is too much in a firefighting setup all the time. You may have an environment that is not conducive to this and then it is time to evaluate other options

But assuming you have alignment

When was the last time, you blocked your calendar and sat heads down to work on solving a platform problem?

Think about it for a moment, before you read on.

In my experience, time blocks on your calendar that you protect almost like a hawk are helpful. If you ensure to work without any distractions during this time, chances are you will make some progress for sure. People refer to this as “Eat the frog” i.e. work on the toughest part of the problem first thing in the morning when your mind is fresh.

This of course requires discipline and changing a few other habits as well like mindless doom-scrolling slack messages without really getting much signal among endless noise and chatter

I usually turn off notifications, kill Slack or other messaging apps, put on DND on my phone, and then get right down to business. Time block helps a little in mitigating Parkinson’s law since time-bound tasks tend to get finished than unbounded tasks.

Another tip would be choosing your personal time of the day when it’s less chaotic and people are not demanding of your time will work better, so either earlier in the morning or later in the afternoon, or early evening. It depends on when you are more focused.

Go on, Try it, trust me it works wonders!

Community 🤝

Do you have to be the hero who saves the day alone and gets all the credit?

Well, No. ⛔

That is not how modern teams work

You may still choose to do it, but you’ll have a tougher road ahead getting people to adopt whatever you choose to build.

Instead, one alternative could be to form a community of like-minded peers

Talk to your peers and get a few other engineers excited about solving a problem, and then figure out times in your schedule to work on it and solve it.

A small 2 pizza bottoms-up team can work wonders. You don’t need a lot of folks, just people who are really motivated.

You get bonus experience in leadership, community building, and building deeper relationships with peers either in the same team or beyond your team as well. It’s a win-win!

This kind of organic initiative is something leadership usually likes a lot and it’s also good for your career growth within the company since you build the org and make it stronger by opening up mentoring opportunities.

Oftentimes more than the work, investing in building meaningful relationships may have an outsized long-term impact on your careers

You just may make a workplace friend as well as another bonus. 🤷

Fin 🐠

I know I did not solve your problem

But hopefully, it gave you some ideas on how to think about this in your day-to-day and perhaps some approaches you can try.

Go ahead and try your own experiments, some may just work out. 😉

Also, if you have faced this, how have you solved this in the past. Let me know your thoughts in the comments.

BYOL: bring your own learning ⚡️

Gaurav Singh — Tue, 13 Aug 2024 00:00:00 +0000

Created by microsoft copilot using DALL.E

If you work in a dynamic fast-paced company, you will learn a lot

Of course, having experienced talented peers who are invested in your growth or willing to share their knowledge can also teach you a lot.

Situations and challenges that you are exposed to and your personal experience trying out different approaches in different contexts also help you grow

All these are good, but are they enough?

Your employer does not own your learning

Now this may come as a surprise to some

The company does not owe it to you to develop you completely as a professional.

You’ll see ad-hoc learning and development programs that seem like an afterthought and don’t really have quality content

Sometimes engineers in the community start voluntary communities like Lunch and Learn, weekly book clubs, etc but these quickly fizzle out when the original organizer and their supporters lose interest or just move on

You’ll also be wise to realize that you are ultimately there to do a job

Do what you are being paid to do, and do it well.

Hopefully, you work on something that is intellectually interesting and challenging as well and that the company needs to meet its own goals but in many cases it may be business-as-usual tasks

If you think about it, You are ultimately responsible for creating business outcomes that make the company profit and that is completely fair for a business owner to expect from an employee

All doom and gloom?

In your career, some managers will prioritize your growth and learning by helping find mentors, suggesting conferences, and training, and brainstorming and following up on progress during 1:1s and some really won’t and may index more on delivery and other things that the company cares about

You are lucky if you have a leader like the former.

In such work environments, it is quite easy to feel stuck get demotivated, and feel like you are not growing fast enough and others are racing ahead.

Social media allows for easy comparison with your peers and past colleagues and while you just see the end shiny result, you are ultimately comparing your career with the highlight reels of theirs

Given all this, What would you do?

You could change teams or groups
You can take ownership and try to find mentors within the company yourself
You could quit and move on and try to seek greener pastures elsewhere. Most of the advice on the internet suggests something along the lines of: “If you can’t change your manager; change your manager”

Are they scalable?

Probably not.

A way out?

What perhaps would work better is a deliberate consistent self-learning habit.

You may counter immediately and say you don’t have time for this.

Yes, you may be super busy in your off hours, have social and family obligations, etc…

But if you pause and reflect, is taking out 30-45 mins a day or a few hours over the weekend that difficult? You may be wasting this time doing N other activities

Building this muscle can bring you compounding effects that years of doing your routine job may not necessarily bring.

Have you heard the quote:

“Do you have 10 years experience or just 1 year repeated 10 times?”

I don’t know who said this first, but this was how I used to approach the initial years of my career. Living the good life, going mostly with the flow at work, etc, and complaining and sometimes wondering how others are achieving more.

Until one day stopped and took control of my own learning…

I changed directions toward making this change and focussed on building a learning habit often at a fixed time and tracked this in Habitica as well.

I’ll say while most of the events in my career have been incremental, this habit has worked quite well for me over the years.

Why spend more time learning?

If you still need a bit more context

Aside from the technology landscape constantly evolving and this being just the need of the hour, why would you want to do this?

Mostly because It’s super fun since you are not constrained by the tech choices, tools, environment, or language that your employer needs. You can build skills in other vertical or horizontal areas.

Another side bonus is that you’ll be more confident about establishing yourself and speaking as a person who knows their stuff

You may build new and unique skills by working on side projects

Learning about things more deeply and hacking on side projects can be oddly very self-satisfying

And who knows, you may end up using this knowledge at work sometimes in obvious or nonobvious ways as well. Growth is not always linear

What next?

Also, don’t just stop at self-learning, put the Feynman technique to work

“You keep on learning and learning, and pretty soon you learn something no one has learned before.” ― Richard Feynman

Bring this learning to your workplace and unique context.
Discuss it with peers and see how you can implement it to make things faster, better, and more efficient
Teach it to folks you are mentoring (if they are genuinely interested)
Talk about it in tech talks either internally
Write a blog about it
Give a talk about it at a public conference

Try some of these, whatever feels comfortable for you, and see how it can unlock levels of value that are not immediately obvious but you’ll see the difference slowly compounding over time

The best time to start this habit was 5 years ago, but the second best time is today.

So don’t procrastinate, start now and begin anywhere!

Go on, you got this!

I’m rooting for you! 🤝

TribeQonf 2024 Debrief

Gaurav Singh — Sun, 14 Jul 2024 00:00:00 +0000

The test tribe, TribeQonf 2024

Why read this?

I had the privilege of presenting a talk on Mastering gRPC testing at the TribeQonf 2024 conference on 12-13th July conducted at Radisson blu in the silicon corridor ORR bengaluru.

It was a fun 2 days of high energy, insights, and networking. I had the chance to hit some of the sponsor booths and chat with peer speakers and attendees, It was a pretty solid conference experience put up by The Test Tribe team

In this blog, I wanted to digest and decompress some of the advice and insights that I took away. Some of these validated thoughts that I have already had while others were novel ways of looking at problems.

I hope it is useful, and it would be fantastic also to hear your thoughts in the comments and if you were an attendee, what were your top takeaways.

I’ll break it down into a few sections to keep it contextual

Talks
Sponsor booths and what problems they are solving
Networking with peers

Talks

Inside-out look at a Tester’s success - Pradeep Soundararajan

Pradeep called out he faced challenging conditions but was able to thrive regardless as a tester by using them to his advantage

He explained how he was let go from different jobs before he was able to turn it all around by starting his own company. His journey into fitness, spirituality, and prioritizing his health with a holistic lifestyle while focusing on the 1% of things that are in your control vs all the other noise and chatter we tend to focus on really stood out to me.

Whatever we achieve in our careers is not solely due to our own efforts but also due to people around us.

We should focus more on living a holistic life than chasing the next promotion or title bump and deal with all the self-induced stress that comes from it.

Moolya and Pradeep were also kind to provide his new book Growth Driven Testing as part of the attendee kit and I’m looking forward to reading and learning more from it.

AIoT is here testers. Are you ready for the challenge? - Vipin Jain

Vipin broke down challenges in the IOT testing space and how there are multiple challenges in different layers of the OSI model. He explained how data aggregation is a key process and also provided an example of how AI could play a role in HealthTech for medical report analysis, alerting and providing patients care.

AI-Augmented Testing: How Generative AI and Prompt Engineering Turn Testers into Superheroes, Not Replace Them - Jonathon Wright

This talk was super high energy

Jonathan played some AI-generated Hindi songs while graciously showing off his dancing skills on stage as well. No small feat if you ask me.

Jonathan works at Keysight which provides a platform for test automation on hardware and sensor devices. Keysight also acquired Eggplant which provides the ability to automate UI flows using computer vision technology and provides a low code solution to create automated tests using a UI

Jonathan suggested running GenAI models on local servers (or even a surface tab 😉) as a way to get leverage out of these tools without sharing your confidential data with big AI companies

He also reshared Tariq King’s method of using markdown to provide a more procedural prompt to GenAI to get better results.

An example of this is shared below which was also built upon by Rahul Verma in this LinkedIn post

# MISSION
Generate data in multiple languages for a given context.

# INPUT

1. A user registration page in a web app takes the following inputs:

* `First Name`
* `Last Name`
* `Date of Birth in dd-MonthName-year format.`
* `City`
* `Country`
* `Full Address`

2. The web site accepts data in multiple languages.

# STEPS

1. `Choose 10 natural languages which can typically have challenges in a multi-lingual websites.`
2. `Create a Language Table with Language and Challenges.`
3. `Create one example of user data for each language by incorporating the challenges identified in the context of user registration data.`

# RESPONSE FORMAT

1. `Language Challenges Table`
2. `Tests Table with Test ID, Language, First Name, Last Name and so on columns for each data item.`
3. `The response should include only the above 2 tables.`

Building Harmony in Chaos: Orchestrating Agile Delivery in Startup Realms - Nishi GG

“Pace is continuous; stress should not be” - Nishi GG

This quote stood out to me

Nishi provided practical advice on how to structure agile teams to deliver while not letting stress levels rise in startups

Some of her advice revolved around:

Template for tracking sprint items with Dev, QA estimates and 2W daily Gantt to call out when Dev/QA and other activities would take place and reduce ambiguity
Identifying a clear POD structure with a short or long-term charter and continuing or dissolving them when the business outcomes are met to focus on the next problem
She mentioned why reserving bandwidth of senior engineers and architects is important for multiple reasons and how it could be used for engineering productivity goals
Importance of having clear focus time for teams while not neglecting play and creative time as well and how a growth pod could be set to ensure engineers get opportunities to work on horizontal items instead of their vertical initiatives
She talked about pair/buddy testing as an important ingredient of a successful agile team for all the obvious benefits it could provide
How Sprint - 1 automation model is followed in teams at Pictory

Revolutionizing Quality Engineering: Breaking Orthodoxy, Shaping the Future - Sahil Puri

Sahil’s talk was packed with insights into the challenges in quality engineering space from the lens of a realist and had suggestions on what could be some pragmatic approaches to solve:

Quality is everyone’s responsibility
- The key point was to not be defensive about the “Why did we miss this bug?” question but instead lead the RCA and truth seek on the gaps in your testing and delivery process and plug that leaking hole
- Quality is an abstract term for some, it is better to index on ensuring testing happens at all phases of the cycle by stakeholders with test professionals helping to set adequate guardrails
Look beyond just testing
- Focus on the pre-development stage - understand the go to market (GTM) strategy, business metrics, and outcomes and feed that into your test strategy
- Understand system design, implementation details, and the expected scalability to do better testing around functional and non-functional requirements
- Keep an eye on post-launch metrics to learn whether solutions actually solved the problem and moved the metrics in the right direction, take those insights and feed them back into your delivery process
Coding dilemma
- Learn to code; it would be immensely helpful
Speed vs quality
- It’s not always a tradeoff
- Teams should index towards increasing release velocity while not sacrificing product quality by the use of priority and data to identify the right scope and automated processes
Priority
- Test engineers should develop their advocacy and sales skills to ensure they are able to convey value in language that business and leadership teams understand
Focus on breath as well as depth
100% tests should not be automated
- Focus on should not instead of could not
- Focus on areas that can help you release faster with confidence instead of chasing a vanity metric
- Identify a clear CI/CD strategy and also understand DevOps bits to have better control over the release and quality processes

Pattern Thinking for Prompt Engineers: A Case Study in Test Design - Rahul Verma

Apart from the markdown method that Jonathan shared earlier, Rahul explained how even providing additional context by using markdown bold could lead to better outcomes, he also showed how learning plain text diagraming tools syntax like Mermaid, and PlantUML could be used to generate state transition tests

Changing the language to passive voice could lead to better outcomes while creating custom GPTs

Navigating Your Career Journey: Strategies for Growth and Success - Ajay Balamurugadas

Ajay provided actionable advice on how career growth for engineers could look like in a very engaging talk

He touched upon

The need to build a network of peers who could act as accountability partners and help you learn
- You should ask “When was the last time you tried something new for the 1st time?”
- A few book recommendations - Atomic habits, Deep Work, Extreme Ownership - Be the CEO of Your Life, So Good They Can’t Ignore You
Use of 5W1H (What, Why, when, where, Who, How) and 5 why’s as a method of thinking deeply about problems and solutions and additionally add else and not to be more critical in your thinking
Use the disposable time during your workday to do some experiments about what activity you can do things differently.
Resources:

Sponsor booths

It was amazing to see different companies trying to leverage some form of AI, GenAI solution in order take care of generating test plans and even code using existing UI frameworks. I hope to learn more about this space in near future and hopefully build some of these tools as well.

DevAssure

DevAssure provides interesting oppurtunity to generate tests using GenAI taking web app screenshots and figma mocks as an input. While they are focussing on web to start with this definitely seems like a start up to watch out for.

BrowserStack

Browser stack apart from their device and web farm capabilities is building tooling in test observability space with a product already out called BrowserStack Test Observability. The demo was quite interesting and could provide better insights into test automation metrics

Keysight

Eggplant Test Automation Intelligence solution offers the user of computer vision to provide a smarter low code solution to automated web, desktop, and mobile apps

BlinqIO

BlinqIO offers an AI test engineer to provide capabilities for GenAI test generation using multiple underlying LLMs for web apps and API

Catchpoint

Catchpoint offers observability on network metrics like latency, response times, reliability by intrumenting mobile and backend systems. While they compete with other trace and monitoring tools like datadog and new relic. The ability to get feedback before builds are shipped to prod at different POP (points of presence) was interesting.

Devzery

Devzery solves fast autonomous API testing by leveraging GenAI to generate API tests with CI/CD integration

TestSigma

Testsigma provides low code solution to take care of tests on web, mobile and even have a cloud device/web farm to boast about.

FinerCircle

Finer Circle provides a niche community of testing professionals with courses, interest based cohorts and much more.

Networking

One of the main benefits of a conference is that it brings smart like minded people under one roof and allows the merging of ideas

I was able to connect with some peers and industry experts and get insights from their rich and diverse experiences.

Capturing these for posterity and hopefully serve as a reminder for folks reading this to go out and talk and network with folks. You never know what these connections may manifest for you in the future.

Geosley Andrades, Director AccelQ - we chatted about his experience of transiting from a testing professional to engineering manager to a director in product marketing space. How his hands on experience helps him in his new role.
Pradeep soundarajan , CEO moolya about his personal life and early married life
Sahil puri EM Zupee I enjoyed learning about his experience managing an engineering team at zupee, challenges in test automation space and QA, org dysfunctions. His talk was of course well delivered and really insightful
Shrini Kulkarni, ex Director - we chatted about unit, integration testing and specifically use cases of consumer driven contract testing
Chakravati Singh Automation Architect PhonePe about problems PhonePe is solving at mobile testing layer, pre submit stages and how PhonePe testing org has evolved. Understanding how a dedicated platform team solves automation problems was very interesting
Prem Kumar Nagaraj, staff engineer Razorpay about his experience building tooling at CI/CD, security tools, test selection and his prior experience at amazon
Tushar sharma , growth and marketing, The Test Tribe - we talked about community building, how testing communities are evolving and what are the pain points and challenges that finer circle is solving
Sayanthan Roy, Architect at Amadeus about how a plan to transition testers into automation engineers could look like, the approach and challenges that come with working with legacy frameworks
Gunesh Patil , Test architect, Ushur - I learned about technical challenges he’s trying to solve at his company in an interesting space on customer experience and engagement

And thats a wrap.

Pretty action packed conference.

I think it will take some time to digest the dense information overload and I have nothing but gratitude about getting a chance to experience this conference and would definitely encourage folks to attend this next time.

Please let me know if you have questions or thoughts in the comments.

Testing gRPC #5: Load test a gRPC service

Gaurav Singh — Sun, 02 Jun 2024 00:00:00 +0000

Image powered by DALL-E 3

This is the final blog in this series where we explored gRPC testing and automation.

A quick recap, In the previous blogs in this series, we explored testing different layers of the test pyramid within gRPC context:

What gRPC is, using web UI to explore the API
How to unit test the server
How to unit test the client code.
How to E2E test a server method

Please feel free to catch up on those before reading this one.

What you’ll learn? 🌱

In this blog, I’ll explore how to setup a load test written in Java leveraging locust4j library to setup a locust slave.

Our slave worker will push aggregated results to a python based locust server.

I hope you are also excited like I am to explore this topic.

Let’s go 🏃

Why write load tests?

Any modern successful business is bound to hit scale at some point in their journey. When it does, you don’t want to caught with a bad surprise wherein your systems are not able to scale, handle faults gracefully.

Load testing prepares engineering teams for such events.

Identifying how the system holds up with next level of scale is extremely important.

A system that is not scalable, reliable or fault tolerant when put under stress would let down its users and often that means direct revenue loss or even worse losing a customers trust.

Preventing such bugs to leak to production should be an acceptance criteria for any service seeing the light of the day in production.

⚓ Setup locust master

I hope we are on the same page that writing load tests are super important.

Let’s focus on understanding how to do this for our route guide service

We will use popular load testing framework called locust for this example. I had previously explored how to write a locust python test on a mock API. Please feel free to give ita read if interested

How to setup a load test via Locust - automation hacks

We will need python3 to be installed on our machine.

Most modern mac comes with python3 pre-installed, if not, you can follow this wonderful guide to Properly Installing Python

We will also setup virtualenv to install required dependency, in this case locust and will use Pipenv. You can read How to manage your python virtualenvs with Pipenv - Automation Hacks to get familiar with Pipenv

Below is a short summary:

Ensure you have below setup in your .zshrc, .bash_profile or in your windows system environment variables

# One time setup
# Set below in .zshrc or .bash_profile depending upon the type of shell you are using
export WORKON_HOME=~/virtualenvs
export VIRTUALENVWRAPPER_PYTHON=/usr/local/bin/python3
export LANG=en_US.UTF-8
export LC_ALL=en_US.UTF-8

Next install pipenv using python3

pip3 install pipenv

You can then clone or take a pull from this repository GitHub - automationhacks/grasp-grpc: Ports routeguide example service from grpc-java repository and adds functional and non functional API tests and ensure you cd to folder with Pipfile

Run below to start virtualenv

pipenv shell

Finally, we’ll start our locust master server using:

locust -f locust_master.py --master --master-bind-host=127.0.0.1 --master-bind-port=5557

You’ll see something like this

[2024-05-31 08:29:15,073] Gauravs-Laptop/INFO/locust.main: Starting web interface at http://0.0.0.0:8089
[2024-05-31 08:29:15,078] Gauravs-Laptop/INFO/locust.main: Starting Locust 2.28.0

What does this command do?

Here we are running a dummy locust server in master mode and binding the server to localhost and a random port 5557. In a production setup, this script could be running on a remote server which we would use in our tests.

Below is how locust_master.py looks like:

from locust import User, TaskSet, task

class DummyTask(TaskSet):
   @task(1)
   def dummy(self):
       pass

class Dummy(User):
   tasks = [DummyTask]

Here we init DummyTask as a sub class of TaskSet and have a method dummy with a weight of 1 using the @task() annotation.

Weight is a way in locust to increase the probability of a certain task to be picked ahead of others. In this case, we have a single task so it does not really matter.

class DummyTask(TaskSet):
   @task(1)
   def dummy(self):
       pass

We also have the Dummy class of type User with our tasks list with this DummyTask

class Dummy(User):
   tasks = [DummyTask]

Generate required files using proto compiler

Ensure all relevant files are generated using gRPC proto compiler by executing below.

./gradlew installDist

You only need to run this if you’ve made some changes to the proto message or RPC. This step could be skipped if there are no changes. In a prod setup, you may have a automation process that updates dependencies as part of a CI/CD pipeline

Start gRPC server

Next, let’s ensure our server and API is up by executing below

./build/install/grasp-grpc/bin/route-guide-server

You’ll see something like this

SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
May 31, 2024 8:34:47 AM io.automationhacks.routeguide.RouteGuideServer start
INFO: Server started, listening on 8980

Writing a locust slave

The key component of this load test is a locust slave written in Java that can be used by locust4j to start load test.

I’ve added an example slave in RouteGuideLoadGen.java

And it looks like below:

package io.automationhacks.routeguide.perf;

import com.github.myzhan.locust4j.Locust;
import io.automationhacks.routeguide.perf.tasks.GetFeatureTask;

import static io.automationhacks.routeguide.constants.Constants.*;

public class RouteGuideLoadGen {

 public void getFeatureLoad() {
   var locust = configureLocustMaster();

   locust.run(new GetFeatureTask(1));
 }

 private Locust configureLocustMaster() {
   var locust = Locust.getInstance();

   locust.setMasterHost(LOCUST_MASTER_HOST);
   locust.setMasterPort(LOCUST_MASTER_PORT);
   locust.setMaxRPS(LOCUST_MAX_RPS);

   return locust;
 }

 public static void main(String[] args) {
   new RouteGuideLoadGen().getFeatureLoad();
 }
}

Let’s unpack this

We have a method to configure our slave worker with the locust master instance

We then invoke the run method and provide it a task instance

public void getFeatureLoad() {
   var locust = configureLocustMaster();

   locust.run(new GetFeatureTask(1));
 }

We also use the below method to get a Locust instance and set the master host, port as well as the peak requests per second (RPS) for the slave.

private Locust configureLocustMaster() {
   var locust = Locust.getInstance();

   locust.setMasterHost(LOCUST_MASTER_HOST);
   locust.setMasterPort(LOCUST_MASTER_PORT);
   locust.setMaxRPS(LOCUST_MAX_RPS);

   return locust;
 }

Locust load task

We also have to write a Locust task that has the actual gRPC API that we want to invoke.

public class GetFeatureTask extends AbstractTask {
 private final int weight;

 RouteGuideTestClient client =
     new RouteGuideTestClient(ROUTE_GUIDE_SERVER_HOST, ROUTE_GUIDE_SERVER_PORT);

 public GetFeatureTask(int weight) {
   this.weight = weight;
 }

 @Override
 public int getWeight() {
   return weight;
 }

 public String getName() {
   return "RouteGuide.GetFeature";
 }

 public void execute() {
   int latitude = 407838351;
   int longitude = -746143763;

   Point point = Point.newBuilder().setLatitude(latitude).setLongitude(longitude).build();

   // Record the startTime for the API call
   var startTime = getTime();
   // Add thread safe logic here to make the API call
   Feature response = client.getFeature(point);
   // Record the endTime for the API call
   var endTime = getTime();

   // Do not swallow any exceptions from the API, we should propagate any exceptions to locust
   assertWithMessage(
           "Could not find the feature at lat: %s long: %s".formatted(latitude, longitude))
       .that(response.getName())
       .isEqualTo("Patriots Path, Mendham, NJ 07945, USA");

   // Record success or failure to locust
   long responseTime = endTime - startTime;
   int contentLength = response.getName().length();

   Locust.getInstance()
       .recordSuccess("GetFeature", response.getName(), responseTime, contentLength);
 }
}

Above is the complete method, let’s understand how this works

We create a task GetFeatureTask that extends AbstractTask from locust4j

We configure the local Grpc client

RouteGuideTestClient client =
     new RouteGuideTestClient(ROUTE_GUIDE_SERVER_HOST, ROUTE_GUIDE_SERVER_PORT);

And also configure a constructor and getter to set custom weight if we want from the caller

public GetFeatureTask(int weight) {
   this.weight = weight;
 }

 @Override
 public int getWeight() {
   return weight;
 }

We set the name to be used during the load test

public String getName() {
   return "RouteGuide.GetFeature";
 }

Finally the execute method has the actual logic wherein we invoke the locally running service

public void execute() {
   int latitude = 407838351;
   int longitude = -746143763;

   Point point = Point.newBuilder().setLatitude(latitude).setLongitude(longitude).build();

   // Record the startTime for the API call
   var startTime = getTime();
   // Add thread safe logic here to make the API call
   Feature response = client.getFeature(point);
   // Record the endTime for the API call
   var endTime = getTime();

   // Do not swallow any exceptions from the API, we should propagate any exceptions to locust
   assertWithMessage(
           "Could not find the feature at lat: %s long: %s".formatted(latitude, longitude))
       .that(response.getName())
       .isEqualTo("Patriots Path, Mendham, NJ 07945, USA");

   // Record success or failure to locust
   long responseTime = endTime - startTime;
   int contentLength = response.getName().length();

   Locust.getInstance()
       .recordSuccess("GetFeature", response.getName(), responseTime, contentLength);
 }

An interesting thing to note is that we capture the total time taken for the API call and response to come back and then later on persist it in the Locust instance using below:

Locust.getInstance()
       .recordSuccess("GetFeature", response.getName(), responseTime, contentLength);

Start the load test

We can run RouteGuideLoadGen.java to start the slave and register it with the master.

If we navigate to the locust master host running at http://0.0.0.0:8089, we can see below UI

Here we’ll be kind and just use 5 users with a ramp up of 1 user per second.

We can see the charts with real time performance RPS and failure stats and other diagnostic data.

The workers tab also shows the current no of workers registered

And we can also see stats about the overall test

Since this is a local test running on my machine, I also used the Activity monitor app and could see spike in CPU and memory usage stats while the test was running. Within a short time, I bombarded 40K RPS on my locally hosted service. Surely a pretty sizable scale for a single service. 🤩

References

Standing on the shoulder of giants.

I referred to below resources to come up with this write up. Hopefully this is useful for your further exploration.

Locust python notes on Grpc testing
Grpc Test Infra
Distributed Load Testing of REST/gRPC APIs using Locust
Blog from blazemeter
Locust4j
Example Main.java file to start a locust test
Locust Http load example project
Locust Http load another example project
wrk an example HTTP benchmarking tool

Conclusion

This was a short introduction to the world of load testing with locust and demos how you are not really limited to knowing python to leverage this wonderful tool. I hope you’ll give it a try and explore the nitty gritties of this tool

To summarize, we learned:

We can write locust load test for a gRPC service using Java with a slave to Locust master setup
How to setup locust master in python
How to generate required build file for Java gRPC service
Start a local gRPC server
Write a utility to setup a Locust Java slave worker
Start execution and observe metrics on Locust web UI

Please let me know if you have questions or thoughts in the comments.

Testing gRPC #4: How to E2E test a gRPC service

Gaurav Singh — Wed, 20 Mar 2024 00:00:00 +0000

Image powered by DALL-E 3

In the previous blogs in this series, we learned what gRPC is, using web UI to explore the API and then how to unit test the server and client code. Please feel free to catch up on those before reading this one.

What you’ll learn? 🌱

In this article, I’m gonna walk you through how to write a functional API test for a gRPC API using the live service. It’s super important to write these end-to-end tests to make sure your API works the way it’s supposed to. We’ll start with setting up the codebase and then create a basic test client to talk to the gRPC API.

After that, I’ll show you an example of a functional API end-to-end test using TestNG and how to check if the API is responding properly. Let’s go! ⚡

Why write E2E tests?

Writing unit tests is probably the fastest way to get quick feedback about your services and codebase health. They are the lowest fidelity and fastest among different types of tests that you can write.

However, would you be confident in shipping a unit-tested API directly to production?

Probably not!

You still would want confidence that the API works functionally.

Let’s assume your API is consumed by an upstream API, in such a case, you would want to make sure your API accepts valid input as per the contract that you’ve agreed with the dependant team and returns expected responses to the upstream service in positive, negative and edge test cases

You can get some of this isolated feedback with an integration test as well, however, if the workflow is complicated and you have multiple dependencies between APIs and databases, then writing integration tests could be tedious at best.

That’s where a bunch of E2E tests come pretty handy.

While they run slower than a unit or integration test, in return you get realistic feedback about the state of your API, workflows, etc.

In this post, we’ll build on top of what we’ve learnt so far and explore how we can write functional API tests for a gRPC service.

Hint ⚡ : It’s not so much different 😉

Set up repo and codebase ⚙️

For this next section of the blog, I’ve copied the relevant files from grpc-java and created a new standalone project in my Github account that only covers routeguide service.

This should help make it easier for someone new to not be overwhelmed by the huge project structure and seeing a lot of unrelated complex code.

Repo: https://github.com/automationhacks/grasp-grpc

We will clone the repository and set up a new project in IntelliJ and ensure the build the successful by running these commands:

Using SSH

git@github.com:automationhacks/grasp-grpc.git

Using Github CLI

gh repo clone automationhacks/grasp-grpc

Let’s check we can build the project without running tests for now

./gradlew clean build -x test

Let’s ensure that we have all the classes generated by building the project

./gradlew installDist

Let’s start our test server by running below

./build/install/grasp-grpc/bin/route-guide-server

We can see that our server is running at localhost:8980 port

Mar 19, 2024 9:52:42 PM io.automationhacks.routeguide.RouteGuideServer start
INFO: Server started, listening on 8980

Write an API test client

When we test a live API, it is usually hosted at some specific host and port on infrastructure that could be either our company local data center or cloud platforms like Google cloud, AWS, Azure, etc.

We’ll write a test client for our E2E API tests so that we can invoke our gRPC APIs and also maintain this as we add more functions

Below is how we can create a simple client with one method for getFeature() service method

RouteGuideTestClient.java

package io.automationhacks.routeguide;

import io.grpc.ManagedChannel;
import io.grpc.ManagedChannelBuilder;
import java.util.logging.Logger;

public class RouteGuideTestClient {

 private final Logger logger = Logger.getLogger(RouteGuideTestClient.class.getName());
 private final RouteGuideGrpc.RouteGuideBlockingStub blockingStub;

 public RouteGuideTestClient(String host, int port) {
   ManagedChannel channel = ManagedChannelBuilder.forAddress(host, port).usePlaintext().build();
   blockingStub = RouteGuideGrpc.newBlockingStub(channel);
 }

 public Feature getFeature(Point point) {
   return blockingStub.getFeature(point);
 }
}

Let’s unpack this,

We initialize RouteGuideBlockingStub to allow us to make API calls

private final RouteGuideGrpc.RouteGuideBlockingStub blockingStub;

Within the constructor, we accept the host and port where the server is running and then use ManagedChannelBuilder.forAddress() method to get a ManagedChannel instance

public RouteGuideTestClient(String host, int port) {
   ManagedChannel channel = ManagedChannelBuilder.forAddress(host, port).usePlaintext().build();
}

We then use this channel to initialize our blockingStub

blockingStub = RouteGuideGrpc.newBlockingStub(channel);

And then write a method for our getFeature() API that accepts a Point and then returns the Feature at that specific point

public Feature getFeature(Point point) {
   return blockingStub.getFeature(point);
 }

While we can go ahead and implement other service methods, this single fine for now and we’ll implement them in the future while understanding other testing concepts

Writing an API E2E test

Let’s write a functional API test to verify the getFeature() service method is working fine.

Below is the complete test which you can also find in examples/src/test/java/io/grpc/examples/routeguide directory in the codebase

RouteGuideE2ETest.java

package io.automationhacks.routeguide;

import static com.google.common.truth.Truth.assertWithMessage;

import org.testng.annotations.Test;

public class RouteGuideE2ETest {

 private static final String HOST = "localhost";
 private static final int PORT = 8980;

 @Test
 public void testGetFeature() {
   System.out.println("Executed testGetFeature()");
   RouteGuideTestClient client = new RouteGuideTestClient(HOST, PORT);

   int latitude = 407838351;
   int longitude = -746143763;

   Point point = Point.newBuilder().setLatitude(latitude).setLongitude(longitude).build();
   Feature response = client.getFeature(point);

   assertWithMessage(
           "Could not find the feature at lat: %s long: %s".formatted(latitude, longitude))
       .that(response.getName())
       .isEqualTo("Patriots Path, Mendham, NJ 07945, USA");
 }
}

Let’s understand our test

Within our test, we have specified our host and port where our server is running

⚡ In a real framework, it’s a good idea to store these in an environment properties file and read it using Java classes to manage system properties. This would ensure if the host/port changes, we only have to change properties file, or build in flexibility to make the test environment agnostic

private static final String HOST = "localhost";
private static final int PORT = 8980;

We’ll use TestNG test runner to run our E2E tests and we specify the test method by annotating our function with @test

@Test
 public void testGetFeature() {}

To make the API call, we need to initialize the test client that we had earlier created

RouteGuideTestClient client = new RouteGuideTestClient(HOST, PORT);

We use the builder provided by protocol buffers to create a point object and set latitude and longitude

int latitude = 407838351;
int longitude = -746143763;

Point point = Point.newBuilder().setLatitude(latitude).setLongitude(longitude).build();

We then make the API call via the client like below:

Feature response = client.getFeature(point);

Finally, we assert if the response name provided matches our expectation, we use fluent assertions from Google truth library

   assertWithMessage(
           "Could not find the feature at lat: %s long: %s".formatted(latitude, longitude))
       .that(response.getName())
       .isEqualTo("Patriots Path, Mendham, NJ 07945, USA");

How do we know this latitude/longitude would return this feature name?

When the server starts, we read from a JSON file specified called route_guide_db.json and store it in the in-memory collection.

RouteGuideServer.java

public RouteGuideServer(int port) throws IOException {
 this(port, RouteGuideUtil.getDefaultFeaturesFile());
}

Let’s run our tests using below command

./gradlew clean runTests

We can see our test was executed since it printed our logger entry Executed testGetFeature() and our test passed 🎀

> Task :runTests

Gradle Test Executor 15 STANDARD_ERROR
    SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
    SLF4J: Defaulting to no-operation (NOP) logger implementation
    SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.

Gradle suite > Gradle test > io.automationhacks.routeguide.RouteGuideE2ETest > testGetFeature STANDARD_OUT
    Executed testGetFeature()

BUILD SUCCESSFUL in 2s
10 actionable tasks: 10 executed

Conclusion

Now you are well equipped to go ahead and write many E2E API tests for your gRPC APIs, so what are you waiting for? Go ahead and test some gRPC APIs 😀

Please let me know if you have questions or thoughts in the comments.

In the next post, we will grasp how to write a Non functional load test on our API, we’ll attempt to gain more confidence that the service works when dealing with live load.

| Thanks for the time you spent reading this 🙌. If you found this post helpful, please share it and follow me ( @automationhacks ) for more such insights in Software Testing and Automation. Until next time 👋, Happy Testing 🕵🏻 and Learning! 🌱 | Newsletter | YouTube | Blog | LinkedIn | Twitter. |

Testing gRPC #3: How to unit test a gRPC client

Gaurav Singh — Thu, 07 Mar 2024 00:00:00 +0000

Image powered by DALL-E 3

⏪ Recap:

In the previous blog, we understood how to write a unit test for a gRPC server using InProcessChannel, and InProcessServer and how to set automatic cleanup using GrpcCleanupRule. In case you missed it, please feel free to read it here

⏩ What we’ll learn:

In this blog, we’ll understand how a client could be written for our Grpc service and also how can we unit test that.

Let’s go ⚡

System under test

Before we understand how the client is written, let’s first grasp how the service is implemented for this method since we have to write a client method for essentially this.

You can see the complete listFeatures() method in RouteGuideServer.java below:

RouteGuideServer.java

@Override
public void listFeatures(Rectangle request, StreamObserver<Feature> responseObserver) {
 int left = min(request.getLo().getLongitude(), request.getHi().getLongitude());
 int right = max(request.getLo().getLongitude(), request.getHi().getLongitude());
 int top = max(request.getLo().getLatitude(), request.getHi().getLatitude());
 int bottom = min(request.getLo().getLatitude(), request.getHi().getLatitude());

 for (Feature feature : features) {
   if (!RouteGuideUtil.exists(feature)) {
     continue;
   }

   int lat = feature.getLocation().getLatitude();
   int lon = feature.getLocation().getLongitude();
   if (lon >= left && lon <= right && lat >= bottom && lat <= top) {
     responseObserver.onNext(feature);
   }
 }
 responseObserver.onCompleted();
}

Let’s walk through the code and grasp how this server method is implemented

If you observe the method signature, We pass in a Rectangle request and StreamObserver<Feature> responseObserver

We know from route_guide.proto , that Rectangle is a collection of 2 Point (lo and hi), such that each point has a latitude and longitude. This represents a bounding rectangle in a map. Think of a rectangle that covers the central part of Bangalore (of course, there could be much better representation to create a geofence but for the sake of simplicity let’s go with this 😏)

Below is the proto in case you need a recap.

route_guide.proto

// Points are represented as latitude-longitude pairs in the E7 representation
// (degrees multiplied by 10**7 and rounded to the nearest integer).
// Latitudes should be in the range +/- 90 degrees and longitude should be in
// the range +/- 180 degrees (inclusive).
message Point {
 int32 latitude = 1;
 int32 longitude = 2;
}

// A latitude-longitude rectangle, represented as two diagonally opposite
// points "lo" and "hi".
message Rectangle {
 // One corner of the rectangle.
 Point lo = 1;

 // The other corner of the rectangle.
 Point hi = 2;
}

We also pass in StreamObserver<Feature> responseObserver which is used to return a stream of Feature objects to the client. The stream could loosely be understood as an array that the client can read either all at once or one object at a time.

public void listFeatures(Rectangle request, StreamObserver<Feature> responseObserver)

Let’s look at the service methods body, We then extract 4 corners of the rectangle by finding min and max values from lo and hi longitude and latitudes to get an integer that represents each corner.

 int left = min(request.getLo().getLongitude(), request.getHi().getLongitude());
 int right = max(request.getLo().getLongitude(), request.getHi().getLongitude());
 int top = max(request.getLo().getLatitude(), request.getHi().getLatitude());
 int bottom = min(request.getLo().getLatitude(), request.getHi().getLatitude());

Nice, We then iterate in the features already stored in our database (in this example, the features are stored in an in-memory Collection but in a real application this could be any other data store) and check if a feature exists

for (Feature feature : features) {
   if (!RouteGuideUtil.exists(feature)) {
     continue;
   }

If you see the exists() it just checks if the feature has a valid name

RouteGuideUtil.java

/**
* Indicates whether the given feature exists (i.e. has a valid name).
*/
public static boolean exists(Feature feature) {
 return feature != null && !feature.getName().isEmpty();
}

If the feature exists then we check

if that feature longitude lies between left and right ranges
and latitude lies between the bottom and top ranges

if we find such a feature then we add it to the responseObserver’s onNext method

int lat = feature.getLocation().getLatitude();
   int lon = feature.getLocation().getLongitude();
   if (lon >= left && lon <= right && lat >= bottom && lat <= top) {
     responseObserver.onNext(feature);
   }

Finally, we call the responseObserver.onCompleted(); method to indicate that the rpc call is complete.

How to Implement a Client

So now we understand what our service is actually doing.

Let’s understand how a client method could be written for listFeatures() method

Any client for a service method/operation usually performs 3 functions:

Construct the request for the service
Call the service method using either a blockingStub or an asyncStub (in other words either sync or async call)
Logs or return the response back for further processing

With above context, We can write a gRPC client for this listFeatures() method like below

RouteGuideClient.java

public class RouteGuideClient {
 private static final Logger logger = Logger.getLogger(RouteGuideClient.class.getName());

 private final RouteGuideBlockingStub blockingStub;
 private final RouteGuideStub asyncStub;

 private Random random = new Random();
 private TestHelper testHelper;

 /** Construct client for accessing RouteGuide server using the existing channel. */
 public RouteGuideClient(Channel channel) {
   blockingStub = RouteGuideGrpc.newBlockingStub(channel);
   asyncStub = RouteGuideGrpc.newStub(channel);
 }

/**
* Blocking server-streaming example. Calls listFeatures with a rectangle of interest. Prints each
* response feature as it arrives.
*/
public void listFeatures(int lowLat, int lowLon, int hiLat, int hiLon) {
 info("*** ListFeatures: lowLat={0} lowLon={1} hiLat={2} hiLon={3}", lowLat, lowLon, hiLat,
     hiLon);

 Rectangle request =
     Rectangle.newBuilder()
         .setLo(Point.newBuilder().setLatitude(lowLat).setLongitude(lowLon).build())
         .setHi(Point.newBuilder().setLatitude(hiLat).setLongitude(hiLon).build()).build();

 Iterator<Feature> features;
 try {
   features = blockingStub.listFeatures(request);

   for (int i = 1; features.hasNext(); i++) {
     Feature feature = features.next();
     info("Result #" + i + ": {0}", feature);
     if (testHelper != null) {
       testHelper.onMessage(feature);
     }
   }
 } catch (StatusRuntimeException e) {
   warning("RPC failed: {0}", e.getStatus());
   if (testHelper != null) {
     testHelper.onRpcError(e);
   }
 }
}}

Initialize client

Let’s unpack the client and understand how it works

We start with Initialising our client class RouteGuideClient and setup our logger using standardjava.util.logging.Logger;

public class RouteGuideClient {
 private static final Logger logger = Logger.getLogger(RouteGuideClient.class.getName());

Next, We declare a RouteGuideBlockingStub and a RouteGuideStub to make desired sync or async calls to the server and also initialize few other utilities like Random and TestHelper which may be used

 private final RouteGuideBlockingStub blockingStub;
 private final RouteGuideStub asyncStub;
 private Random random = new Random();
 private TestHelper testHelper;

We then pass in the channel to the client constructor and then initialize our blocking and async stubs using method from RouteGuideGrpc

/** Construct client for accessing RouteGuide server using the existing channel. */
 public RouteGuideClient(Channel channel) {
   blockingStub = RouteGuideGrpc.newBlockingStub(channel);
   asyncStub = RouteGuideGrpc.newStub(channel);
 }

listFeatures

We implement our listFeatures client method that accepts 4 int parameters, as we saw before they represent the co-ordinates for bottom left and upper right corners of the bounding rectangle

lowLat → Latitude for a lower point of the rectangle
lowLon → Longitude for a lower point of the rectangle
hiLat → Latitude for the upper point of the rectangle
hiLon → Longitude for the upper point of the rectangle

public void listFeatures(int lowLat, int lowLon, int hiLat, int hiLon) {
 info("*** ListFeatures: lowLat={0} lowLon={1} hiLat={2} hiLon={3}", lowLat, lowLon, hiLat,
     hiLon);

For our client, we prepare our request by constructing the Rectangle object that the service expects. We can directly use the newBuilder() exposed for each proto and use builder pattern to construct our request

 Rectangle request =
     Rectangle.newBuilder()
         .setLo(Point.newBuilder().setLatitude(lowLat).setLongitude(lowLon).build())
         .setHi(Point.newBuilder().setLatitude(hiLat).setLongitude(hiLon).build()).build();

Next, we make the actual request to get our list of features using blockingStub and store it in a collection called Iterator<Feature> that we can iterate upon

Iterator<Feature> features;
 try {
   features = blockingStub.listFeatures(request);

We then iterate and log each feature found within the specified coordinates. If we catch a StatusRuntimeException then we log this as a warning

In a real application, the client might do further processing based on these features but for now, we just log the features received

try {
   features = blockingStub.listFeatures(request);
   for (int i = 1; features.hasNext(); i++) {
     Feature feature = features.next();
     info("Result #" + i + ": {0}", feature);
     if (testHelper != null) {
       testHelper.onMessage(feature);
     }
   }
 } catch (StatusRuntimeException e) {
   warning("RPC failed: {0}", e.getStatus());
   if (testHelper != null) {
     testHelper.onRpcError(e);
   }
 }

How to unit test a client

So now, we understand the basics

How the service method under test is implemented
How is the client for such a service written

Let’s come to the fun part.

How will we ensure our client works fine?

Of course, we write a test

You can find the complete test examples/src/test/java/io/grpc/examples/routeguide/RouteGuideClientTest.java but we’ll focus on how to write a test for listFeatures()

RouteGuideClientTest.java

@RunWith(JUnit4.class)
public class RouteGuideClientTest {
 /**
  * This rule manages automatic graceful shutdown for the registered server at the end of test.
  */
 @Rule
 public final GrpcCleanupRule grpcCleanup = new GrpcCleanupRule();

 private final MutableHandlerRegistry serviceRegistry = new MutableHandlerRegistry();
 private final TestHelper testHelper = mock(TestHelper.class);
 private final Random noRandomness =
     new Random() {
       int index;
       boolean isForSleep;

       /**
        * Returns a number deterministically. If the random number is for sleep time, then return
        * -500 so that {@code Thread.sleep(random.nextInt(1000) + 500)} sleeps 0 ms. Otherwise, it
        * is for list index, then return incrementally (and cyclically).
        */
       @Override
       public int nextInt(int bound) {
         int retVal = isForSleep ? -500 : (index++ % bound);
         isForSleep = ! isForSleep;
         return retVal;
       }
     };
 private RouteGuideClient client;

 @Before
 public void setUp() throws Exception {
   // Generate a unique in-process server name.
   String serverName = InProcessServerBuilder.generateName();
   // Use a mutable service registry for later registering the service impl for each test case.
   grpcCleanup.register(InProcessServerBuilder.forName(serverName)
       .fallbackHandlerRegistry(serviceRegistry).directExecutor().build().start());
   client = new RouteGuideClient(grpcCleanup.register(
       InProcessChannelBuilder.forName(serverName).directExecutor().build()));
   client.setTestHelper(testHelper);
 }

/**
* Example for testing blocking server-streaming.
*/
@Test
public void listFeatures() {
 final Feature responseFeature1 = Feature.newBuilder().setName("feature 1").build();
 final Feature responseFeature2 = Feature.newBuilder().setName("feature 2").build();
 final AtomicReference<Rectangle> rectangleDelivered = new AtomicReference<Rectangle>();

 // implement the fake service
 RouteGuideImplBase listFeaturesImpl =
     new RouteGuideImplBase() {
       @Override
       public void listFeatures(Rectangle rectangle, StreamObserver<Feature> responseObserver) {
         rectangleDelivered.set(rectangle);

         // send two response messages
         responseObserver.onNext(responseFeature1);
         responseObserver.onNext(responseFeature2);

         // complete the response
         responseObserver.onCompleted();
       }
     };
 serviceRegistry.addService(listFeaturesImpl);

 client.listFeatures(1, 2, 3, 4);

 assertEquals(Rectangle.newBuilder()
                  .setLo(Point.newBuilder().setLatitude(1).setLongitude(2).build())
                  .setHi(Point.newBuilder().setLatitude(3).setLongitude(4).build())
                  .build(),
              rectangleDelivered.get());
 verify(testHelper).onMessage(responseFeature1);
 verify(testHelper).onMessage(responseFeature2);
 verify(testHelper, never()).onRpcError(any(Throwable.class));
}

/**
* Example for testing blocking server-streaming.
*/
@Test
public void listFeatures_error() {
 final Feature responseFeature1 =
     Feature.newBuilder().setName("feature 1").build();
 final AtomicReference<Rectangle> rectangleDelivered = new AtomicReference<Rectangle>();
 final StatusRuntimeException fakeError = new StatusRuntimeException(Status.INVALID_ARGUMENT);

 // implement the fake service
 RouteGuideImplBase listFeaturesImpl =
     new RouteGuideImplBase() {
       @Override
       public void listFeatures(Rectangle rectangle, StreamObserver<Feature> responseObserver) {
         rectangleDelivered.set(rectangle);

         // send one response message
         responseObserver.onNext(responseFeature1);

         // let the rpc fail
         responseObserver.onError(fakeError);
       }
     };
 serviceRegistry.addService(listFeaturesImpl);

 client.listFeatures(1, 2, 3, 4);

 assertEquals(Rectangle.newBuilder()
                  .setLo(Point.newBuilder().setLatitude(1).setLongitude(2).build())
                  .setHi(Point.newBuilder().setLatitude(3).setLongitude(4).build())
                  .build(),
              rectangleDelivered.get());
 ArgumentCaptor<Throwable> errorCaptor = ArgumentCaptor.forClass(Throwable.class);
 verify(testHelper).onMessage(responseFeature1);
 verify(testHelper).onRpcError(errorCaptor.capture());
 assertEquals(fakeError.getStatus(), Status.fromThrowable(errorCaptor.getValue()));
}

}

I know, its a lot. 🤟

Let’s break it down step by step and grasp how this works

Test Structure

The code is a test class named RouteGuideClientTest designed to test a client for communicating with a service called RouteGuide. It uses JUnit4 as we saw before

@RunWith(JUnit4.class)
public class RouteGuideClientTest {}

Automatic Cleanup

The @Rule annotation tells JUnit to manage an instance of GrpcCleanupRule.
This rule ensures that test servers are automatically shut down after each test, keeping the test environment clean.

/**
  * This rule manages automatic graceful shutdown for the registered server at the end of test.
  */
 @Rule
 public final GrpcCleanupRule grpcCleanup = new GrpcCleanupRule();

Setup

@Before
 public void setUp() throws Exception {
   // Generate a unique in-process server name.
   String serverName = InProcessServerBuilder.generateName();
   // Use a mutable service registry for later registering the service impl for each test case.
   grpcCleanup.register(InProcessServerBuilder.forName(serverName)
       .fallbackHandlerRegistry(serviceRegistry).directExecutor().build().start());
   client = new RouteGuideClient(grpcCleanup.register(
       InProcessChannelBuilder.forName(serverName).directExecutor().build()));
   client.setTestHelper(testHelper);
 }

We write @Before annotated setUp method to ensure each test method starts with a unique instance of the client

We create a unique in-process server (running within the same process)

// Generate a unique in-process server name.
   String serverName = InProcessServerBuilder.generateName();

We also use a registry to allow different services to be registered for different test cases.

// Use a mutable service registry for later registering the service impl for each test case.
   grpcCleanup.register(InProcessServerBuilder.forName(serverName) .fallbackHandlerRegistry(serviceRegistry).directExecutor().build().start());

We then initialize the client and pass it the channel which is also registered for auto cleanup

client = new RouteGuideClient(grpcCleanup.register(
       InProcessChannelBuilder.forName(serverName).directExecutor().build()));

We inject a mock object called testHelper for observing calls and verifying behavior. We’ll see that this is an interface that exposes

client.setTestHelper(testHelper);

Tests

We then write 2 unit tests that do below on a high level:

listFeatures : This tests successful communication with the server simulating a positive case
- Creates a fake service implementation to control server behavior.
- Calls the client’s listFeatures method to initiate communication.
- Asserts that the correct request was sent and the expected responses were received.
listFeatures_error : Tests how the client handles errors.
- Set up a fake service that throws an error.
- Verifies that the client properly propagates the error to the test helper.

✅ Positive case

Let’s unpack the positive case first to understand this a bit better

Below is the complete test at a glance. We will do a walkthrough on it below:

/**
* Example for testing blocking server-streaming.
*/
@Test
public void listFeatures() {
 final Feature responseFeature1 = Feature.newBuilder().setName("feature 1").build();
 final Feature responseFeature2 = Feature.newBuilder().setName("feature 2").build();
 final AtomicReference<Rectangle> rectangleDelivered = new AtomicReference<Rectangle>();

 // implement the fake service
 RouteGuideImplBase listFeaturesImpl =
     new RouteGuideImplBase() {
       @Override
       public void listFeatures(Rectangle rectangle, StreamObserver<Feature> responseObserver) {
         rectangleDelivered.set(rectangle);

         // send two response messages
         responseObserver.onNext(responseFeature1);
         responseObserver.onNext(responseFeature2);

         // complete the response
         responseObserver.onCompleted();
       }
     };
 serviceRegistry.addService(listFeaturesImpl);

 client.listFeatures(1, 2, 3, 4);

 assertEquals(Rectangle.newBuilder()
                  .setLo(Point.newBuilder().setLatitude(1).setLongitude(2).build())
                  .setHi(Point.newBuilder().setLatitude(3).setLongitude(4).build())
                  .build(),
              rectangleDelivered.get());
 verify(testHelper).onMessage(responseFeature1);
 verify(testHelper).onMessage(responseFeature2);
 verify(testHelper, never()).onRpcError(any(Throwable.class));
}

Alright, let’s break this down.

We first annotate our method with @test and then build two feature objects by using the builder as before by passing in the name as “feature 1” and “feature 2”

@Test
public void listFeatures() {
 final Feature responseFeature1 = Feature.newBuilder().setName("feature 1").build();
 final Feature responseFeature2 = Feature.newBuilder().setName("feature 2").build();

We create a thread-safe reference to the Rectangle object using Java AtomicReference

final AtomicReference<Rectangle> rectangleDelivered = new AtomicReference<Rectangle>();

In this test, we are only interested in testing our client , thus we create a fake service implementation for listFeatures method. This ensures isolation for this unit test.

To do so, we construct new RouteGuideImplBase()and define our anonymous subclass by overriding the listFeatures method like below

// implement the fake service
 RouteGuideImplBase listFeaturesImpl =
     new RouteGuideImplBase() {
       @Override
       public void listFeatures(Rectangle rectangle, StreamObserver<Feature> responseObserver) {

Inside the body, we set the rectangle passed in from the request into the AtomicReference<Rectangle> rectangleDelivered we had defined earlier

rectangleDelivered.set(rectangle);

We also want the server to stream and return the two features we had earlier created to the client, thus we use the onNext() method in responseObserver to do so. This is a way for gRPC to provide server-side streaming

// send two response messages
responseObserver.onNext(responseFeature1);
responseObserver.onNext(responseFeature2);

Finally, we complete the RPC by calling onCompleted() method

// complete the response
responseObserver.onCompleted();

We can see the complete fake service implementation below.

// implement the fake service
 RouteGuideImplBase listFeaturesImpl =
     new RouteGuideImplBase() {
       @Override
       public void listFeatures(Rectangle rectangle, StreamObserver<Feature> responseObserver) {
         rectangleDelivered.set(rectangle);

         // send two response messages
         responseObserver.onNext(responseFeature1);
         responseObserver.onNext(responseFeature2);

         // complete the response
         responseObserver.onCompleted();
       }
     };

Now, we will add this fake service to our service registry as below

serviceRegistry.addService(listFeaturesImpl);

Awesome, Let’s make our service call via the client. Here 1, 2, 3, and 4 are the lat and longs for the lower left and upper right corner of the rectangle as we saw before.

client.listFeatures(1, 2, 3, 4);

After making the call, we should check whether the rectangle object returned from the response matches what we expect with the below

assertEquals(Rectangle.newBuilder()
                  .setLo(Point.newBuilder().setLatitude(1).setLongitude(2).build())
                  .setHi(Point.newBuilder().setLatitude(3).setLongitude(4).build())
                  .build(),
              rectangleDelivered.get());

Lastly, we also need to verify if our client actually called our fake service. We can ensure that using verify() method from Mockito library

We use the verify() method from Mockito library to verify interactions with the mocked object testHelper as below

verify(testHelper).onMessage(responseFeature1);
verify(testHelper).onMessage(responseFeature2);

This interface is defined in RouteGuideClient.java as below and if we go over its definition, it exposes two methods onMessage() and onRpcError. We can use this to ensure the client calls the onMessage() method with responseFeature1 and then with responseFeature2

/**
* Only used for helping unit test.
*/
@VisibleForTesting
interface TestHelper {
 /**
  * Used for verify/inspect message received from server.
  */
 void onMessage(Message message);

 /**
  * Used for verify/inspect error received from server.
  */
 void onRpcError(Throwable exception);
}

We also ensure that the onRpcError() method was never called since this is a positive case.

verify(testHelper, never()).onRpcError(any(Throwable.class));

You can see other examples of verify() and never() method calls and their usages on mockito docs

⛔ Negative test

How would we write a negative test for our client?

In this test, We want to check what happens when our service throws an RPC error

The initial setup is identical, except in this case we only prepare one responseFeature1

@Test
public void listFeatures_error() {
 final Feature responseFeature1 =
     Feature.newBuilder().setName("feature 1").build();
 final AtomicReference<Rectangle> rectangleDelivered = new AtomicReference<Rectangle>();

We initialize a fakeError of StatusRuntimeException type and then initialize it with an INVALID_ARGUMENT value

 final StatusRuntimeException fakeError = new StatusRuntimeException(Status.INVALID_ARGUMENT);

We again implement our fake service and repeat the same steps as above:

// implement the fake service
RouteGuideImplBase listFeaturesImpl =
   new RouteGuideImplBase() {
     @Override
     public void listFeatures(Rectangle rectangle, StreamObserver<Feature> responseObserver) {
       rectangleDelivered.set(rectangle);

       // send one response message
       responseObserver.onNext(responseFeature1);

       // let the rpc fail
       responseObserver.onError(fakeError);
     }
   };

Notice, the last step. 🔵

We now use the onError method to return the fake error that we had created earlier. In a production app, the service may use this to communicate to the client that something went wrong in processing.

responseObserver.onError(fakeError);

We then register our service, call our client with the same input, and assert that the API returns us the similar output by asserting on the rectangle object

serviceRegistry.addService(listFeaturesImpl);

client.listFeatures(1, 2, 3, 4);

assertEquals(Rectangle.newBuilder()
                .setLo(Point.newBuilder().setLatitude(1).setLongitude(2).build())
                .setHi(Point.newBuilder().setLatitude(3).setLongitude(4).build())
                .build(),
            rectangleDelivered.get());

To verify the service works for our negative scenario

We perform below 4 steps:

Set up a capture mechanism for errors.
Verify that a message was received by the service (likely before the error).
Verify that an RPC error occurred and capture the specific error.
Assert that the captured error’s status matches the expected error status.

Let’s see how this could be implemented using the Mockito library

We check that the onMessage() method of the mocked object testHelper was invoked with the specific argument responseFeature1

verify(testHelper).onMessage(responseFeature1);

We define the ArgumentCaptor class from Mockito to capture any exception of type Throwable to our mocked method

ArgumentCaptor<Throwable> errorCaptor = ArgumentCaptor.forClass(Throwable.class);

Then, we ensure that onRpcError() method of mocked testHelper is called and by using capture() method of ArgumentCaptor, we capture the error returned by the GRPC service

verify(testHelper).onRpcError(errorCaptor.capture());

We also assert that the captured error status matches expected fakeError.getStatus()by retrieving the captured Throwable using errorCaptor.getValue() and convert it into a GRPC Status object using Status.fromThrowable()

assertEquals(fakeError.getStatus(), Status.fromThrowable(errorCaptor.getValue()));

Conclusion

Let’s recap what we learned in this post:

We understood how listFeature() method is implemented and how can a gRPC server return a streaming response
We understood how the client method is written for this service
Later, we dove into how can we unit test the client
- We looked into how to write a fake service
- We implemented a positive case and checked if calls were made using Mockito verify()
- We implemented a negative case and then checked if error was returned using Mockito ArgumentCaptor

That was lot of ground 🌆, and this wraps up what I wanted to cover with unit testing. There are other aspects to it but I’ll leave that to you to explore with other methods.

Please let me know if you have questions or thoughts in the comments.

In the next post, we will grasp how to write a functional API test for this to get confidence that the service works when dealing with live data.

What I wish I knew about Software Testing and growth at the start of my career 🌱

Gaurav Singh — Sun, 25 Feb 2024 00:00:00 +0000

Image powered by DALL-E 3

Hi everyone,

I was reflecting on many conversations I’ve had around growth in testing with some peers in the community and up-and-coming testers and engineers. While it would take many blogs and talks to unpack these, I thought it would be fun to come up with a random unordered list of skills, approaches, habits, and growth ideas that I wish I had known at the start of my testing career to mature.

How do I think about testing now? 🤔

Testing is a deep and multifaceted discipline with many things to learn and incredible power to deliver delightful outcomes for your customers. If done well, it can enable the whole company to move fast and ship with confidence. The ultimate goal of testing is to enable the building of a quality product and service that brings delight.

Skills 🤹

Learn Touch typing
Learn mind mapping
Learn how to gauge risk and increase test coverage at different layers systematically
Learn the fundamentals of testing well and then don’t stop, keep going.
Speak up in meetings and discussions (don’t be shy)
Ask those dumb questions, and always be curious
Be polyglot and learn a static and dynamic programming language and then some more
Learn to code and read lots of code
Learn to design efficient CI/CD pipelines
The test pyramid is also an indication of what you can learn about different layers of testing
Don’t focus on only UI or Backend. Look at the system as a whole.
Focus on the customer and their UI/UX touchpoints. These should never break
Learn the basic API of the tool or framework and then keep going deeper
Learn to problem solve in leet code and understand data structures and system design
Learn how to manage up
Learn how to build your network up

Approach and attitude 🙂

Begin anywhere
Be a tinkerer
Be humble and grounded. Recognize that there is always room to grow
Be a friend to someone new and help them grow along with you.
Encourage psychological safety in your team.
Find a mentor and learn from them on a set of topics. Then find another. Keep growing.
Don’t restrict yourself to one area (web, mobile, backend, data, performance, security, CI) - Mix things up!
Quality is everyone’s responsibility but someone needs to champion it
If you are not learning in an environment, either change yourself or change your environment.
Leverage the power of the internet and the wide and open testing community. You’ll learn better and faster
Don’t give up when you see a problem; be comfortable saying “I don’t know this …… YET”
Learn something, teach it to others, and move on to other and better things to focus on.
Don’t be a silo and a single point of failure
Reading docs is your best friend
You don’t need to wait for a course or a class to teach you something. Be self-taught as much as possible
Clear writing is clear thinking
Don’t focus on the title but on developing skills. You are not your title
Always be a net positive contributor on your team, focus on outcomes
Don’t limit yourself to only testing but also learn from engineering (Dev, DevOps, AI/ML, and PM) and nonengineering communities and their practices.
Don’t restrict yourself to one persona of a tester but be a generalist software engineer

Habits ⌨️

Build a note-taking system. It will compound you faster.
Schedule consistent and deliberate learning time
Maintain a summary and narrative of your work regularly
Build the habit of reading books and summarising them
Listen to podcasts while doing life chores. Make your chores fun!
Read blogs and newsletters to learn from your peers
Propose talks at conferences as a way of learning
Adopt open source, be part of the community, and contribute however you can.

Growth 🌱

All of the above and …

Build, learn, and share in the open as much as possible
Develop a personal roadmap and OKRs and evaluate your progress
Maintain a personal tech brand
Explore working in diverse work environments like services, products, startups, scale-ups, and big tech to develop a broad perspective

Did I miss any? Please let me know in the comments

| Thanks for the time you spent reading this 🙌. If you found this post helpful, please share it with your friends and follow me ( @automationhacks ) for more such insights in Software Testing and Automation. Until next time, Happy Testing 🕵🏻 and Learning! 🌱 | Newsletter | YouTube | Blog | LinkedIn | Twitter. |