DEV Community: Vicky Nogueira

Mobile Performance Metrics You Should Know – Part 1

Vicky Nogueira — Fri, 30 Jul 2021 18:37:39 +0000

Consumers and enterprises alike expect more from software. Speed, UX, availability, and frequency of updates are increasingly important with mobile apps. The responsibility of developers keeps growing, and as mobile apps get more complex, new tools for mobile performance monitoring and testing are being born.

Successful APM tools can provide teams with visibility into what’s happening to users in the real world and address issues in the next app update. But this process usually takes a couple of weeks. During that time, users can get frustrated with performance issues making them more likely to leave a bad review in the app store.

When it comes to having this visibility during development and testing (and avoiding frustrated users), we’re still in the early innings. If you can measure and improve app performance before releasing a new version of your app, it will not only make your users happier but also save your team a lot of headaches, improve developers’ productivity, and allow more time to focus on new features.

This article will help you understand the main mobile performance metrics that are measurable in your mobile native app (Android or iOS) before releasing a new version. That way you can start tracking performance early on and prevent issues from affecting the end user.

Device performance metrics
There are different factors that affect mobile app performance. When it comes to the client-side performance of a mobile app, there are two big areas that you can measure: the device and the app itself. In this first post, we’ll focus on device performance and the metrics related to it.

CPU usage
The CPU (Central Processing Unit) is the unit responsible for carrying out all the instructions of an application. This includes how to run different subsystems that keep the OS running, such as multimedia, audio, rendering, and more. When CPU usage is high, the user may experience sluggishness or higher battery usage (among other symptoms). Since CPU usage is a shared resource, abusing it may prevent other services that are running at the same time to work correctly. This can affect the user experience and proper functioning of the OS (plus the applications that run in parallel). Moreover, with a higher number of instructions, the CPU increases its speed with a consequent increase in voltage, which can cause the device battery to drain faster.

As a good rule of thumb, you want to monitor the CPU usage of your app and always keep it below 80% on all testing devices. If you see peaks in CPU usage, you might want to take a deeper look at the root cause and determine whether it’s expected or not.

Memory
Random-Access Memory (RAM) is one of the most valuable resources in any software development environment. It’s even more valuable in mobile environments where physical memory is constrained. Retaining memory that the app doesn’t need can cause out of memory (OOM) exceptions or constraints on the system’s overall performance.

Specifically for Android apps, there are three areas of memory you can analyze:

Proportional Set Size (PSS): PSS is the total memory footprint of a process calculated by the Android System. A process will always have a private amount of memory to use plus an extra amount of memory that is shared with other processes. PSS takes these into consideration to provide a fair quota of the app’s memory usage.

PSS = private memory + (shared memory/amount of processes sharing it)

Native Heap: Native Heap is handled directly on the Android kernel and typically stores assets like bitmaps and other native libraries. Native memory handling depends on the frameworks that your app uses and requires more caution when handling and releasing memory as there is no Garbage Collector.

Dalvik Heap: Dalvik Heap is the memory space handled by the Android System (Dalvik or ART). This will typically store all the app’s objects that are created using the Android SDK.

To maintain a functional multi-tasking environment, Android sets a fixed limit on the Dalvik heap size for each app. The exact Dalvik heap size limit varies across devices based on how much RAM the device has available overall. If your app has reached the heap capacity and tries to allocate more memory, it will receive an OutOfMemoryError.

Virtual machines typically count on a Garbage Collector (GC), which is part of most modern programming languages. It’s a mechanism that automatically frees up memory that’s not being used by any thread. Although GC performance is usually improved in almost every new Android version, it can still cause some performance issues. Try monitoring the GC time (the amount of time that this process takes to complete). Issues like excessive memory allocation in the app could lead to higher GC times.

Storage I/O
I/O throughput and latency can vary a lot depending on the device hardware. Many apps use storage or databases such as SQLite for local assets. SQLite is highly optimized for efficient access. It’s a good idea to rely on it when writing and updating existing content.

In any case, measuring the amount of data read and specially written is another important aspect of good performance. The basic measurement is the amount of logical and physical memory used per second by your app.

In our next post, we’ll focus on app performance and the most important metrics to measure.

At Apptim, we want to help companies consistently deliver fast and reliable mobile experiences. It doesn’t matter whether you’re just getting started with mobile performance or you’ve been involved in this practice for some time, Apptim will allow you to easily integrate mobile performance metrics in your current mobile development process. To start measuring the performance of your Android or iOS apps, you can sign up for Apptim here for free.

Mobile Commerce Testing: How Global Leaders Optimize User Experience

Vicky Nogueira — Wed, 14 Jul 2021 23:32:36 +0000

Mobile accounts for over 67% of all e-commerce sales worldwide and it’s expected to keep growing as more people are getting used to shopping on their phones. Nearly four out of five smartphone users have made at least one purchase in the last six months.

Since mobile performance directly affects user experience, retention rates, conversions, and ultimately, revenue, improving it has become top priority for mobile commerce companies globally.

With this in mind, we invited one of our customer advocates, Nithin Satheesh, QA Automation Team Lead at Fave as well as Joshua Alvarado, an iOS engineer at DoorDash, and Juan Felippo, IT Expert at Mercado Libre, to participate in a live panel discussion led by our CEO, Sofia Palamarchuk.

Below is a recap of the discussion, including questions provided by the audience:

What’s One of Your Biggest Challenges with Mobile App Performance?
The three panelists essentially agreed that the biggest challenge they face with mobile app performance involves having the right data, knowing where it’s being tracked and knowing what to do with it. It’s not only important to have a way to capture the performance data, but also understand the story behind it. What requires more attention and what doesn’t?

For Joshua Alvarado, one of the biggest challenges he faces at DoorDash with their mobile app performance project involves data visualization:

“There’s a lot of data, and you can make a lot of pretty charts, but make sure that those charts and that visualization actually tell the right story.”

He added, “Especially as a developer, you’re probably not working on creating charts a lot. But it’s just very important to start to understand that data because if you can log that data, and then make it flow into a visualization, that’s really powerful.”

Juan shared that he and his team at Mercado Libre face a similar challenge. Today, with the growing competition, it’s not enough to check crash stability, he claimed, but nowadays, teams have to monitor a lot more KPIs that directly affect the user experience. He said that the key for this is to have visibility into each metric and how they impact the user. For that, you have to generate a lot of data, so you can plan what to do with each metric. Some of the metrics he mentioned include startup time, device resource consumption, app size, frozen frames, etcetera.

He stated there are several tools that you can use (like Apptim) or if you work in a large company, your team may build a custom solution. He emphasized that it’s important to make a plan for each specific metric once you have that visibility. Also, strive to have visibility over any changes in performance that may occur from one version to another.

Bottom line: Make sure to gain visibility on the most important performance metrics that impact the end user.

In line with Juan and Joshua’s answers, Nithin described delivering a seamless user experience as his greatest challenge at Fave:

“Understanding our customers is always the key factor for success; what they want, what they are looking for… What performance metrics should we capture to ensure we’ve got the user’s attention? How big and brilliant a new feature is doesn’t really matter if we don’t deliver a seamless user experience.”
What Steps Have You Taken to Confront this Challenge?
Joshua found Google’s Data Studio to be instrumental for data visualization. He explained, “We use Firebase Performance Monitoring for app performance tracing and Data Studio is another product offering from Google. It’s an integration that you can just instantly add on to Firebase and BigQuery.”

He noted these tools’ usefulness in being able to see what other companies share on their dashboards regarding their setups. You can see what they highlight in their dashboards and that’s very helpful. He said when they got started, they did so holistically at a high level. And then with each page in the app, they went into more detail, so they could see a story and make sure that they’re just tracking the correct metrics that make sense for them.

Some of the top KPIs they’re tracking today that they’re seeing in his graphs include:

Response time
Payload size
App startup time
Rendering for certain views
Just as concerned with providing excellent user experiences, Nithin shared some of the things his team is working on at Fave:

“Initially, we were fully focused on the server side. Our developers used profiling tools in Android Studio and Xcode. So it was full dev-centric performance testing to ensure user experience, initially. And then we got an opportunity to inject QA into performance testing aspects.”

“We tried Apptim for capturing the performance-related data and metrics and it was pretty awesome with the comprehensive reports. We eventually dropped all the performance capturing done from the dev side and we do functional testing together with Apptim. We us it to capture all the data and share with our team.”
Nithin shared they not only use Apptim, but also have some tracking in CleverTap and Firebase, as Joshua mentioned.

How Will 5G Change the Mobile Commerce Landscape?
The panel agreed that 5G is already here and the effects can be seen, especially since it is adopted in the iPhone 12, which means it will become the standard moving forward. Joshua had this to say about 5G:

“It will be very, very important to double down on performance with 5G, especially on networking. It’ll be an end-to-end solution here, because on the back end, you’ll be able to communicate a lot faster, but you need to make sure that the client can also take that speed as well and interpret it and render it onto the view very quickly. It will be a whole entity that you need to double down on.”
Sofia chimed in, “It feels like the bar is going to be even higher than today, because we’re going to be able to provide faster experiences through the network, which is sometimes the bottleneck and in many countries, it’s less. But then you need to also meet those expectations from the server and client side as well.”

Nithin believes that 5G will drastically change everything as it will eventually bring more customers online. And that will in turn, bring more revenue to companies. In addition, he shared, from the user experience perspective, 5G will have a lower latency so it will deliver faster results for users and ensure that a seamless interaction happens between the client and server.

Other benefits of 5G, he continued, are that more companies in the e-commerce landscape will start leveraging AI/ML capabilities in their mobile apps. Fave recently adopted AI/ML capabilities into its app to deliver more personalized experiences. It is said that 5G will also be able to provide more accurate location-specific features for apps that use geo-tagging features. Therefore, 5G will be more helpful for teams to scale up and bring better results to their customers.

Sofia shared that she’s interested in learning more about how teams are including AI capabilities in their apps since most of Apple’s machine learning algorithms are being run on-device. This means that more processing is happening on the device which could potentially affect performance and the end user experience.

Juan also agreed that 5G is going to be a game changer, especially in emerging markets such as Latin America, where his company, Mercado Libre, operates.

He shared that the Mobile Economy Latin America Report 2020 forecasts almost 62 million connections across the region by 2025, with almost a 10% 5G adoption rate and an 80% smartphone adoption rate.

5G will also help increase the amount of apps that people download, as many users consider app size and download speed before trying a new app. Juan shared that he read an article which stated that 70% of people in emerging markets consider app size before making a download.

Juan continued, “I think that from the app size to also the network performance, with this new technology, you will have lower latency which will translate to a better product and better user experience. I expect a lot from 5G. I think that in Latin America, we are a bit behind other regions. But I think it will greatly benefit Latin America who has traditionally seen very poor connections.”

How Do You Choose the Minimum Device Set to Test the Performance of Your App?
Joshua was the first to answer this question, sharing, “There’s not exactly a hard rule but I will say when an issue arises, we definitely do our best to prioritize. Based on the severity of the issue, we look through logs, we look to performance. We actually call “dashers” (DoorDash drivers) as well and talk to them to see how widespread this issue is to make sure that they can keep going and doing deliveries and dealing with feedback on the Dash.”

Juan returned to the importance of having the right data to make these decisions. He said his team looks at the device data from their users to know which devices are the most important. He shared, “We have the visibility on the performance of each device that we have on our users. And with that data, we can see if the performance improves or worsens in each version. Also, we take the amount of devices or the majority of devices that the user has and we run our test taking in consideration that data.

Nithin echoed Juan about using data to drive this decision at Fave, “Ours is also a data-driven testing approach. It is practically impossible to test on all devices and OS combinations, so we capture the data from our customers… What is the majority of the devices they are using and what are the majority of OS solutions they’re having? And then we prioritize based on that. All our tests are designed based on that specific data.”

To find this information, Nithin’s team turns to the APM tool Sentry when there is an issue on the client side. Sentry specifies the device fragmentation and OS combination. He added they capture other data from CleverTap so all the details of the devices and the user-specific things are available to them.

Does Your Test Automation Use Machine Learning and Has it Been Effective?
Nithin was happy to take this question first, responding that he hasn’t yet implemented AI/ML capabilities into Fave’s test automation framework, but he evaluated Testim and Test Project. Both of these tools have a self-healing mechanism whenever the most common flakiness occurs as objects change. This helps ensure tests are more stable and minimizes time needed for test maintenance. One of his biggest struggles with UI automation is tests failing after objects change.

Like Nithin, Juan shared his team has not yet begun to implement AI in test automation, but it is a very interesting topic that’s worth taking a deeper dive into.

Will Apps Get Heavier with the Implementation of Capabilities like AR/VR? Will They Change Where They Do the Processing?
The next question for the panelists had to do with whether or not apps will become heavier due to new AR/VR capabilities. Will teams still try to avoid processing a lot on the client side and get most of the computation to happen on the back end? Will anything change because of 5G?

Joshua stepped up to answer from his personal experiences as an app consumer: “With 5G, what I have been seeing is that things have been moving to the server side a little bit more. The cloud is something that’s definitely heavily leveraged. Now with 5G, you can have even faster interaction with that. As far as processing the client, I think it definitely depends on the platform. I’ve seen Apple is taking the approach of doing a lot more client side processing. Even now recently, Siri is being processed on the client side, versus that being done on the cloud.”

“I think it’s just going to be half and half. There’s some apps that are very much heavily reliant on server side processing versus client side, so that makes a lot more sense. But then there’s other apps like Signal, which is very privacy-centric, and I believe that they do everything on the client side. There’s really not much that touches the server side.”
He continued that at DoorDash, he hopes their app doesn’t become heavier or bigger. Something about which they are very cautious is the binary size. But, he stated, with 5G, he doesn’t see there being a big change.

Juan shared an optimistic view of the benefits of 5G and anticipates that app creators will want to take full advantage of the added benefits it offers.

He commented, “Let’s say, for example, a few years ago, you had some devices with less than one gigabyte of RAM. Now you have a cell phone that has almost the same capability as a notebook. So I think that the more resources you have, the more you want to make benefit of. So I think that for some years, we will see a lot of benefit. But at some point in the future, we will implement a lot of technologies to make the best use of 5G.”

Nithin agreed with Juan about users in Southeast Asia also being concerned over app size, “So if it is an e-commerce app, and we’re saying that it is 100 MB in size, people won’t download it at all because we’re not delivering as many capabilities as a gaming app, let’s say. So what matters is what our app delivers. And then mainly depends on the connectivity.”

He commented that Fave tries to keep their app to a smaller size, because people often complain about how long it takes to download or update the app. He’s hopeful that connectivity issues will be rectified with 5G technology and reinforced that an app’s size really depends on what features it offers.

How far do you think we are from adopting AR and VR in mobile shopping experiences?
Nithin pointed out how we are pretty much there already. Companies like IKEA and Airbnb are already using AR to bring the offline experience online, which has especially gained importance for these companies as a consequence of COVID. Some industries are faster to adopt these technologies than others, mostly due to external pressures and the need to innovate to stay relevant.

Without spilling any secrets, Joshua shared that DoorDash is very interested in AR and VR, especially to use them in the Dasher app. They are trying to think of out-of-the-box uses for these technologies beyond the obvious in order to make a dasher’s experience better.

Do You Have Any Practical Advice for Someone Looking to Start a Performance Testing Practice?
For Juan, the best thing to do when starting out is to start small. Don’t try to make a custom solution for performance. Use a tool or platform already available like Apptim to start getting visibility into the metrics.

Once you have that, you can start understanding what the pains are for the users, the general user experience of the application, and then you can get more advanced. He warned,

“Don’t try to tackle the entirety of the metrics that you have; startup, app size, networking, all at first. When you have a basis for one metric, you can move on to the next.”
Joshua liked Juan’s advice and offered more advice to build on top of it. He said, “Before even touching the keyboard and developing anything or integrating any solutions, take a step back and look at it at a high level, and diagram it out. Involve other teams that might benefit from performance of the application and the entire platform.”

He added that accuracy is key here, so go slow, and don’t forget to plan it out.

For those getting started, Nithin believes it’s helpful to I would say, pair up with your developers and collaborate more with the infra side to get a holistic view of the entire application. Break it down and narrow the scope.

Quality Sense Podcast: Sumit Agarwal - DevOps and Testing

Vicky Nogueira — Fri, 26 Mar 2021 13:42:37 +0000

In today’s Quality Sense episode, Federico has a conversation all about DevOps and testing with Sumit Agarwal, the Lead Cloud Architect for a global fin-tech leader with over $4.5 billion in revenues that helps clients get ahead of today’s challenges. Listen to today’s episode where they touch upon the origin of DevOps, testing and dealing with legacy code, and making the necessary culture shifts to successfully implement modern software delivery practices.

Episode Highlights:
The origins of DevOps and how testing relates to it
Challenges of implementing Scrum in larger enterprises and testing legacy systems
How testing can help with the mindset shift and cultural changes that need to take place for DevOps to thrive
How to generate team spirit and proper collaboration not only with developers but also with colleagues in other areas while working remotely
Relevant Links:

Follow Sumit on Twitter – @aga_sumit
Check out the All Day DevOps Conference: https://www.alldaydevops.com/

Quality Sense, a Software Testing Podcast · S3E6 – Sumit Agarwal – DevOps and testing
Listen on Soundcloud
Listen on Spotify
Listen on Apple Podcasts

Episode Transcript:
Federico:
Hello, Sumit, how are you doing today?

Sumit:
I’m very good. Hi, Federico. Nice to be here.

Federico:
Thank you so much for joining. I saw that you are also an organizer of different conferences around the topic of DevOps. Can you tell me a little bit about that?

Sumit:
Sure. So I’m involved with All Day DevOps from a culture transformation track perspective. Derek Weeks, who is the co-founder for All Day DevOps, he actually credits the track to me. We were out at a dinner while he was visiting London and we got talking and I presented at the first, All Day DevOps conference, and I just mentioned to him that I believe what was missing was a cultural transformation track given DevOps has such a heavy culture component. And he said to me, if I wanted to run it, I’ve got it, so I then hosted from the next year on the culture track. I think this year was the first time that I actually moderated other tracks instead of the culture track. But it’s 24 and it started long before we were hit by COVID and everything went virtual.

So it was the first 24 hour live virtual conference to get DevOps conferences, to people who didn’t have travel budgets or were in remote parts that didn’t have DevOps conferences. And then when I moved to New York after attending my first devopsdays New York, I actually got in touch with the organizing people and started working with that committee. So that was 2019, 2020 and COVID hit us right as we ended devopsdays New York, 2020. March 5th was when the conference ended and everything pretty much locked down. If it had been towards the end of that week, we would have probably had to cancel the conference.

Federico:
Oh, just in time.

Sumit:
Just in time, yeah. So, yeah, my love for the DevOps community started with me attending devopsdays in London. And then Andi Mann, who’s with Splunk, we got talking and he introduced me to All Day DevOps. So yeah, the community overall is fantastic at sharing and promoting good ideas.

Federico:
Yeah. One of the few positive aspects of this pandemic is that we have the possibility of attending conferences all over the world without extra budget for that, right?

Sumit:
Yeah, indeed.

Federico:
Yeah. And talking about DevOps and the culture around DevOps, what’s your view on how testing is related to DevOps?

Sumit:
So, I think the name just happened, DevOps the name just came about because of the history of it with a Birds of a Feather session, and then Patrick Debois setting up DevOps days Kent, and the conversation from there going into, how do we… And I think, agile was already a thing. And so scrum, agile, were already things and so when they use the word dev, they kind of assumed we were talking about dev teams, which included testing and QA as a function within dev. And the conversation was about how we get agility to infrastructure and the operations aspect.

“So the name, DevOps, didn’t exclude testing, we’ve heard DevQAOps, DevSecOps, but that wasn’t the idea of it all. It assumed that QA was already part of that dev wording.“

SUMIT AGARWAL
And so, yeah. And then there’s the continuous delivery aspect that comes into it and I think you can’t do continuous delivery without that staying incorporated into it. The whole idea of DevOps as Gene Kim has talked about and everywhere else. Pipelines or automation on pipelines is an integral part of the DevOps culture, the DevOps mindset.

The whole idea is increasing feedback loops and shortening those feedback loops, the cycle time. And by speed of delivery, you actually want to make it less risky and risk equals testing in some respect. So I believe in all of this, testing and risk reduction is at the heart of DevOps.

We want to have more confidence in what we’re deploying into any environment. So we want to move left on everything. So whether that’s unit testing or any other aspect, it should be incorporated into.

Federico:
I really like talking about the culture around DevOps, because I have seen that many people confuse the term DevOps with continuous delivery or automated pipelines or things like that, but what’s happened with the so-called manual testing or exploratory testing. How is that connected with DevOps?

Sumit:
The thing is, and as I was talking about being more comfortable with software releasing or releasing software, and we then start talking about, and Martin Fowler spoke about the testing pyramid, there’s so much that automated testing can do that there is still space for manual and exploratory testing. So what DevOps and automation doesn’t mean that you can replace that human aspect of it when something that automation can only cover so much, if what needs to be in a certain place, you could make it transparent on a screen and still have the automation pass.

You could probably check the colors, but how many permutations and combinations and then there’s the whole accessibility aspect of what if I’m color blind? And does everything match up? And yes, you could automate all of that, but the amount of time you would spend to automate that versus actually having a human test that manually or exploratory best thing, it doesn’t take away from that fact. And you still need somebody who has a testing mindset to come up with what needs to be tested and what should be tested automatically every time we’re putting it through the system versus, or through the pipeline versus what should be done once. Like I built it, I’m not going to be changing some of that, and that’s more of an exploratory thing. Yeah, I mean, again, UX and design, etcetera, you can’t automate tests. That is something usable that has to be very much a manual thing.

Federico:
Yeah. The mindset, I think this is the key here, is like having the mindset and then with this testing mindset, you can decide which things you should automate in order to be more efficient and less error prone or improving your processes. But also there are many things and aspects that you should continue doing enough in a manual way.

Sumit:
I mean, with the agile delivery, what are you changing more frequently and what is likely to break, you automate that piece, but what you don’t change and what isn’t adding value. If you end up spending 50 days writing an automated test for something that never changes you’d really be not adding value to your testing process. So there’s the space for manual testing on that front.

Federico:
You have a background working in a big financial company, I think when talking about agile, many people tend to think of small scrum teams or maybe start-ups. How is it different in a big corporation?

Sumit:
So I think with a big company there’s multiple challenges. One, you’ve got a huge amount of scale and there’s the whole legacy aspect as well. We have systems that were written in the ’80s and ’90s, well before TDD was a thing or test automation was a thing or waterfall ways of delivery of those systems, and they continue to make a lot of money for the organization. So there’s been partial rewrites, there’s components that are added with more modern technology, ways of delivery have changed on those things from waterfall to agile, but still the legacy aspect as, and there’s various different definitions of legacy, some people call it legacy equals money-making, legacy equals non-tested code, so from that perspective, I’d call it a combination of all of those, that there may be bits and pieces that are tested from a regression testing perspective.

So we’ve got regression tests that have been written out, and those have been offshore to continue doing manually there’s even things around trying to automate that in an end to end fashion. So there’s aspects and we were talking about agility, so there’s aspects of agility around, yes, I need to be able to deliver in two, three, four weeks sprints, but testing and the whole aspect of a dev equals dev and test… that might not be true in some big organizations because of the legacy aspect of the code.

Yeah, it just takes far too long, the code base is so big that it wouldn’t be possible to automate fully. And so there’s obviously lots of different initiatives in a big organization and with so many systems that govern how to get better at covering that regression. And I think the only way to do that is with an iterative approach as you write new codes and I think, one of the great books on that is Working Effectively with Legacy Code, which talks about how to introduce more automated testing into code that is non-tested.

Federico:
Yeah. And maybe by doing so you can adopt more agile practices.

Sumit:
Yeah. And another thing is sometimes with compliance and how your audit controls are written, et cetera, that you have it written in your compliance that you will have segregation of roles and duties, that you need that QC separate unit that certifies that something has been tested. And then you start getting those silos, which then introduce some of that repetition. So yes, I’ve got things covered with TDD or unit tests, but you still need to cover the whole thing because in your controls you’ve got it written that the whole working software is going to be tested by the QA team, because they can now trust what’s been done in code already with TDD. So, coming back to the culture aspect and what some of the greats have talked about in the DevOps field is how do you change these processes to make use of the automation?

So how do we talk to audit and rewrite those controls to say, we trust our pipeline and the dev team wouldn’t be to change or disable tests, because that’s what you’re trying to guarantee with some of these controls is yes, there’s a body that knows how to test. We’re going to take that, build that into the pipeline now. And as a result, we can now say, okay, it isn’t the function of a separate silo to verify the software because the pipeline itself ensures that once a test is introduced, it can’t be taken off. And then the pipeline’s insuring it and so you start building in that level of trust with the pipeline and all of that. So yeah, again, some of this comes from the added knowledge that all of the spaces add to DevOps, I think.

Federico:
So, as I understood, you’re proposing to automate some of the scenarios that maybe an audit could include these automated scenarios in that pipeline, so you can run them more frequently and have an early feedback, and also working there in the communication between the different silos. So you can maybe avoid duplications in that course to verify these types of things.

Sumit:
Absolutely. And when I talked about the test pyramid that has been spoken about previously, you want to have lots of very quick running unit tests so that you get a lot of confidence in the code as it’s coming out. Early feedback.

So the developer checks in code, it runs tons of small little tests that run in memory without say, for example, using a relational database. And then as you move up the stack, the tests start becoming slower and probably more fragile. So you want to do less of those, and as a result you don’t want that duplication, if something’s already been verified at the unit test level, then why do them at the integration test. You don’t need to be doing all the different scenarios at the higher layers, you want to do very minimum of those. And that’s all kind of cultural as well as how do you change that in a big enterprise where you’ve got these controls? So those controls need to be rewritten with the automation and everything in play.

Federico:
Yeah. And many people involved, I think that this makes the challenge even harder because there are many people involved in all the processes. I find it fascinating that you can try to modify the culture of a company through the technology or through the technical aspects of how you’re working with automation.

Sumit:
Yeah. And there’s been some debates on Twitter that I’ve followed, which is all about, is DevOps about tools? Or is it about culture? And we don’t need a tool, we need a culture or the other way around. And I think a couple of those conversations, people were just debating for the sake of debating, they never thought it was one or the other. And people taking stands on the side of those just to make a point and that’s why it has to be the trifecta of people, process and technology. So technology being the tool.

So you have to change mindsets in big enterprises, and when I say code living on for 30, 40 years, there’s different mindsets. There’s people who’ve grown up doing it a certain way and you are asking, and they’re probably on the verge of retirement at some time and you’re asking them to now do things very, very differently to what they know and they’ve grown old doing that. So you need to change mindsets, you need to change processes because they’re written in, you need to comply with what you’ve set. That’s an audit requirement, the government comes after you, the clients come after you, the authorities come after you if you don’t follow what you put down, so there’s a lot of work on that side. And then yes, the technology has moved on to help make those changes possible, like we couldn’t have done that before, but we can now because the technology makes it possible.

Federico:
We have to work in both aspects at the same time, in the technology and the mindset of the people involved in all the process.

Sumit:
Yeah. And so, one of the things that we talk about DevOps engineers or automation engineers, and that’s one thing. But one of the definitions, I think I picked it up from Puppet, was:

“A DevOps engineer is somebody who has the ability to actually affect change.”

SUMIT ARGAWAL (Inspired by Puppet)
And one of the first DevOps meetups that I went to, I think it was a Java meetup, but the talk was about DevOps. And when the presenter was showing all the books that he recommends, I think there was a whole slide dedicated to change management related books, because DevOps from that perspective is also about change, and testing is about change.

So it is all about, we want to be constantly bringing change into our software, into our organization, and one of the other things there is, how do you continuously bring about change? Because that’s what we’re trying to do with software delivery, it’s continuous delivery of new features, bug fixes, etcetera. There’s continuous testing to make that possible, but we also then need to keep looking at our processes to keep adjusting and improving from that point.

Federico:
Yeah, you know that at Abstracta we offer software testing services. But, we’ve been working a lot of times with companies, trying to help them adopt an agile culture or a DevOps culture from our activities in software testing. Because I really believe that from this perspective, you can have some influence in different areas to help with the feedback loops or so many different aspects that can contribute to this culture and mindset aspect.

Sumit:
Yeah. And while we’re talking about that, for example, without the cloud, it would be very difficult to spin up test environments at will. Can you do that on a private cloud? Do you have the capacity to just be able to spin up? Let’s say 20 environments, 50 environments to just very, very quickly run your tests. But then adopting the cloud can be a very process oriented change for an organization because they’re not ok with putting their data onto the cloud. And then things like, can I do copy backs from production into my lower environments? Do I have to do masking? Again, we could have technology help with that because again, cloud could help spin that masking workload very, very quickly versus previously you just didn’t have the ability to one, bring up so many environments or be able to have that compute available to mask, because if you’ve got really large databases, you can’t.

So I think it’s all of that putting it together, but you also then have training effort because how many of your people truly, if they’ve been working on legacy stuff and in the data centers, how do you get them on up to speed? So that’s one of the very big challenges. I mean, even training people in the testing mindset. So if I’ve always been a manual tester that basically executes these scripts that have been written out manually on those systems, now asking them to do automated regression tests or become an SDET, software developer in testing.

There’s a big learning curve because you’re completely changing the skillset. How to test a system interactively is very different to testing a system programmatically, and you have to start making architectural changes to the software to allow for that testing to happen as well.

Federico:
Yeah. Many changes in the tools that you use, in the activities that you do, but also in the way you interact with your peers and areas of your organization, right?

Sumit:
Yeah. I mean, we spoke for business before, and one of the challenges is I’ve been approached by enough companies that are offering testing as a service. And what I liked from getting in touch with you was that you were actually affecting change in how to think about testing from a modern agile perspective.

So if I just want manual testers in an offshore location, there’s plenty of body shops out there that’ll do that work, but really what’s needed to advance software delivery with a DevOps mindset has to do with adopting the change and the culture around. How do I improve the feedback loops?

And I think in the State of DevOps report, Nicole Forsgren and others called out that large scale functional outsourcing causes problems, or is correlated with lower delivery speeds. So, should you be wholesale outsourcing QA, or should you be bringing in more from a consulting perspective, bringing in people to help improve your ways of doing that?

So, it’s interesting. Large scale, again, from a large company’s perspective, you’ve got plenty of things to think about from how much of this is going to get modernized, or do I need to start saving costs and kind of more expensive locations by shifting network across the globe, the world’s flat. So how do you?

Federico:
So for me, this is a very interesting topic because mainly nowadays with the pandemic we are all working and collaborating from different parts of the world, or we don’t need to be seated in the same room to foster collaboration. So the thing here is how to generate that team spirit and proper collaboration, not only with developers, also with testers, with people working in support, with people working in different areas of the organization, this is a great challenge and solving this, I think we can make a huge impact.

Sumit:
Yeah, that’s another topic that’s very close to my heart is around organizations and the social technical systems that make up an organization and collaboration for global teams and getting to know people at a personal level, because in a small team, I mean, are we using the chat and collaboration systems like Slack and teams… just for work, but are you really bringing your whole self to work? And then if we’re going to be part of a team that is remote and virtual, for example, how do you make that personal connection? Because, there’s so much more to a person than the working life. Like if they’re having a challenge at home, and they’re not fully present, as a team, it’s impacting the team’s productivity. And that’s another side of it that we got to think about is how to bring the team closer together while still being virtual.

There’s some great stuff from, I was actually recently revisiting Gitlabs’ remote working culture documents there that published them openly across. They actually talk about making informal communication formal. So the whole idea being that if you’re in an office and you go grab a coffee, you’ll bump into somebody, the conversation’s just going to happen. You don’t need to be very formal about setting up that informal communication because you’ll bump into people, you’ll meet people, you’ll meet new people within an organization.

But, when you’re fully virtual, those opportunities don’t automatically come up. So you have to be very intentional about creating that space for that informal communication. Like do you set up meetings for a happy hour, or coffee time and organizing a lunch so that you can have informal communication, but making it more intentional. For example, we spent months trying to coordinate this, but we intentionally did, so yes.

Federico:
So we already have the topic for our next conversation! But I agree that this is a very interesting topic and we could discuss different related aspects. And it’s very connected with the DevOps culture because collaboration is key and motivation is key. Trying to wrap up this conversation today, I have a couple of questions.

One, you already mentioned some books and some interesting material that I will add to the episode notes, but do you have any other book to recommend to our listeners?

Sumit:
So my latest favorite that I’ve done one very quick read of, but I need to read over and over again, is Sooner, Safer, Happier, and we’ve talked about it today, a lot about agility. And the book tries to focus on how to be nimble and agile or agility without using agile as a big, A, Agile and a big T, Transformation. So it summarizes a lot of DevOps ideas into the book around agility. So Sooner Safer Happier.

Federico:
Perfect. Great. Thank you. What about habits? Do you have any habit that you will suggest to form in order to improve productivity or your happiness at work?

Sumit:
So I think it’s a journey for me, but one of the things for me within the DevOps space has been reading. I end up reading and listening to podcasts, etcetera. So when I used to drive to work, because I didn’t get a chance to read as much, so I would listen to stuff more. And with the pandemic, all of it has ground to a hault a little bit, but I still try to read as much as possible.

Federico:
Yeah, there are many things to learn from other people, always.

Sumit:
Yeah. The thing is I do Twitter a bit and there’s people I follow from the DevOps community, there’s always new content and new thoughts. Also, one of the things I’d recommend is if you are in a testing, for example, or an agile scrum master type person, try to broaden the reach there, because I feel… When I look at conferences, DevOps, so people assume automation engineers, and Ops, but even from an organizing perspective is what I don’t see is as many people coming from the testing community or looking at the agile community, and it’s very important to widen that scope to look at other communities and what’s happening in those. So bringing in testing people, or looking at content from those other fields is quite important.

Federico:
Yeah, totally. And do you want to invite our listeners to do something, reach out or whatever?

Sumit:
You can reach out to me on Twitter it’s @aga_sumit , on Twitter and that’s probably the most, from a social media perspective where I’d like to talk. LinkedIn becomes too formal.

Federico:
Okay, perfect. Sumit, thank you so much for all the conversation and all the knowledge and ideas that you share. I really appreciate it.

Sumit:
Thank you very much Federico. It’s been a pleasure.

Federico:
See you. Bye-bye.

Sumit:
Take care.