DEV Community: Caito_200_OK

Data Driven Development for Complex Systems, Part 3

Caito_200_OK — Wed, 07 Apr 2021 17:36:46 +0000

Note: This is the third post in a series- check out Part 1: Intro and Part 2: “Metrics-Driven Metrics” for more context.

This post continues a series on data-driven development best practices. These are specifically made for software systems with highly complicated integration points, but are applicable to many different situations. This series uses stream processing with Apache Flink for the examples.

The initial post provided an introduction and relevant terms, and the second post covered another set of best practices for observability. This post is the next step in leveraging those observability principles to better streamline the interpersonal components that often come with these use cases.

Increased Investment in Internal Communications

Even before the pandemic, there has been an upward trend of companies investing in improving their inter-team communication. From 2017 to 2019, there was a large uptick in corporate content on the importance of team trust and the crucial role of inter-team communication for system resilience and disaster recovery.

However, at the same time, there has also been a rise in articles and commentary on “communication overload:” employees are attending 13% more meetings and polls report there has been such an increase in internal messaging that a majority of employees (particularly in leadership roles) are unable to respond to all new messages in a day.

And Then, Pandemic Happened

In a recent peer survey I conducted, the majority of respondents said that the jump to full-company remote work in 2020 has improved their and/or their company’s communication skills. However, the same respondents also noted that the lingering effects of pandemic continue to take a serious toll on inter-team communication and trust. Although “Zoom fatigue” is now a well known phenomenon, there has been a rise in the public recognition of what really is involved in remote worker burnout during the pandemic.

Communication Breakdown in Complex Systems

Building a software system with multiple (or highly complex) integration points will always have additional communication challenges.

With stream processing, a powerful streaming engine’s strength is in how fast, and how much data it can process, naturally adding complexity to existing integration points. Moreover, a sophisticated streaming system is likely to use shared resources (like storage and deployment tools) differently than non-streaming applications. This means that other teams who share these resources should also be counted as integration points, just as upstream and downstream teams would. With this enlarged circle of interconnected teams, this now also has a more significant impact on stakeholders and leadership who oversee these groups as well.

So — cross-team communication is an inherent challenge when building software systems that have multiple, and/or complex integration points. Additionally, leading up to, and particularly because of the pandemic, internal communication has become more difficult, and more important than ever. And yet somehow, there is also an overwhelming amount of it.

“Internal communication has become more difficult, more important than ever, and yet there’s also too much of it.”

This is where having good data-driven communication comes in, and more specifically, where “metrics as a shared language” can help mitigate and automate these challenges.

“Metrics as a shared language” is a set of best practices for using metrics about your system as a base for efficient, accurate communication between impacted teams and stakeholders. The high level goals and implementation are very straightforward:

Goals

Use system metrics to:

Mitigate (or eliminate) miscommunication.
Streamlining communications in high overhead situations (i.e. where you have complex integration points with other teams).
Build trust with impacted teams and stakeholders.

Implementation

Pre-requisite: make sure your metrics are meaningful and in small, modular units.

Identify your “audience” (impacted groups).
Identify the most effective tools for these groups.
Enable automation.

Identifying Your “Audience”

Identifying your impacted groups is as simple as going over your architectural diagram and listing which teams are involved in each integration point that your system has with the surrounding ecosystem. The only “tricky” part is being proactive about who will really be impacted by your software, and answering questions like “what shared resources will this application use, and will we use it differently from how the other applications use it?”.

Pro Tip: the most common groups I see left out in this type of planning are the operations/DevOps teams, and leadership for adjacent teams.

The next step is to sort these groups based on what kind of information they will need. This is where some of the concepts from the first post come in, specifically, determining who will most benefit from purely quantitative data, which group(s) will need more abstracted, “data-aware” information, and which groups need a hybrid.

Identifying Your Tools

Companies and employees are reporting that communication breakdowns occur most often due to a lack of proper communication channels. Identifying the appropriate tools to use for each group can eliminate a lot of unnecessary wastes of time and resources.

On a high level, the heuristic here is to find a way to integrate your metrics into the preferred tool or platform of your audience, and to abstract this for the groups that need the data to be filtered.

For instance, for engineering and DevOps teams along your pipeline, this could mean having a shared Slack channel just for the PagerDuty alerts that impact that team, and enabling an integrated observability tool (DataDog, Prometheus, etc) to automatically post relevant charts on a regular cadence.

Pro tip: make sure the alerts and metrics you share with these teams are labeled in a way that actually makes sense to engineers outside of your team.

For your customer-facing group, this could mean having a blog post with embedded, auto-updating charts or summarized numerical outputs. This is a good place for “vanity metrics” and other information that is useful to marketing and sales.

Pro-tip: the most common challenge I see here is keeping it simple for these groups and sufficiently abstracting the data. This is as basic as asking them first what 1–3 outputs matter most to them and supplying just those numbers.

For groups who need a hybrid, like engineering managers, other technical leaders and stakeholders, a URL endpoint they can query can go a long way. Or, just including them in both the “granular” channels and “abstracted” posts can be equally effective.

Pro Tip: another “group” that is in this hybrid category is the whole company, at the moment where your system is launched. Having a “multilingual” presentation and resources available for the rest of the company when the system is announced and goes live, can eliminate a lot of extra communication overhead at an already busy time.

Enabling Automation

At a time where workers around the world are reporting more of a need for social connection, automation can seem impersonal. However, the benefits of implementing these practices can actually ease this tension.

Firstly, employees can get answers faster. Workers are reporting that with the sudden jump to full-company remote work in 2020, it has been harder to get answers to questions quickly enough — having to wait hours for the answer to a basic “yes or no” question. Second, as the majority of human communication is nonverbal, reports show that employees are having to expend more energy to communicate with coworkers to compensate for the cumulative effect of missing these important cues for over a year. Streamlining and simplifying any areas of communication possible provides a significant relief and energy back to employees, who can now spend that energy on the areas of communication that do require more nuance.

Lastly, this particular set of best practices naturally sets clearer expectations for communication style. For instance, some people treat Slack as a tool for asynchronous, casual interactions, while others expect a message over Slack to receive an immediate response. In my experience, following these guidelines has almost eliminated friction caused by mismatched expectations. This is possible by making data genuinely accessible and digestible, which enables other teams and stakeholders to be much more autonomous.

Automation Guidelines

Proactive

Ask each group ahead what information matters most to them.
Make sure there are metrics covering these items, and that they are readily accessible.

Reactive

Stay flexible: there will still be unpredictable questions.
What questions do the engineers building this system get asked the most? (a good question to ask both the engineers, and those asking the engineers).

Combining static and dynamic resources

Populate static resources (like internal blog posts) with relevant auto-updating charts (or other outputs).
Adding answers for new questions to static resources, as well as creating additional alerts or monitoring around them.

Results

Continuing the example from the previous post of an engineering team I was on that built the company’s first Apache Flink pipeline: this was a complex system that needed to be integrated into an even larger, more complicated, non-streaming technical ecosystem.

For us, properly implementing “metrics as a shared language” meant a thorough review of our integration points, creating that list of our impacted groups and sorting this into where they fell on the spectrum of most granular, to hybrid, to most abstracted. We then met with all of our upstream and downstream engineering teams. By just reviewing our alerts and observability dashboard and selecting and re-labeling some metrics to integrate into a Slack channel, we were able to streamline weeks of back-and-forth into a ~45 minute conversation.

As for our customer-facing stakeholders: after some trial and error (mainly due to us wanting to constantly share an unnecessary amount of detail), we created 3 new panels on our observability dashboard and embedded them into a carefully-worded blog post. This post was made as accessible as possible: pinned in several channels, and linked to from internal Wiki pages. This post also included a glossary of terms and frequently asked questions, which grew along with the project. This could have been an easy task to forget and allow to get stale, so we added regular tickets in our Jira board to update it. Additionally, our manager started asking us engineers at weekly planning meetings about our communications with other teams: if we were getting repeat questions, or any common misunderstandings, etc.

For “Hybrid” groups like technical leadership and architects, we provided Slack and blog post options, as well as an endpoint they could query, filter and embed into their own systems.

This is not to say that we did this perfectly: we didn’t even know to do this at all, until we had to, partway through the project when we found ourselves in the middle of a cross-team communications breakdown. However, even a last minute, inexperienced approach was still a drastic and immediate improvement.

Overall, implementing these principles (even partially and last minute), provided net gains for our team and the company. On the business and development side, we caught a significantly higher number of technical issues faster. This was enabled through automated sharing of appropriate metrics via Slack integrations with our impacted engineering teams. On a personal level, providing the hybrid options for teams who wanted to query our data but within the appropriate context, helped us regain the trust of several teams who had been suffering from inadequate and poorly-delivered data. As for our communication overhead: we noticed a drastic decrease in the amount of meetings and time responding to messages from stakeholders, architects, designers and managers each week after publishing our internal, metrics-embedded blog posts.

In this series

Part 1: Introduction + Foundational Concepts
Part 2: Metrics-Driven Metrics Cycle
Part 3: Metrics as a Shared Language
Part 4: Hands-On- Monitoring for Stream Processing — arriving in April, 2021

Related talks

“Sweet Streams are Made of These: Applying Metrics- Driven Development to Apache Flink” | Flink Forward EU/Virtual | 10.19–22.2020
“Sweet Streams are Made of These: Lightning Version” | DDD EU | 02/05/2021 [recording in progress]

Find me

Twitter: @caito_200_ok
Web: http://caito-200-ok.com/

Data Driven Development for Complex Systems, Part 2

Caito_200_OK — Thu, 18 Mar 2021 23:14:21 +0000

Part 2 of 4: Anti-Patterns and The “Metrics-Driven Metrics” Cycle

Note: This is the second post in a series- check out Part 1: Intro for more context.

This post is a review of what I jokingly (and now seriously, since it stuck) call the “Metrics-Driven Metrics cycle.” This is essentially a set of best practices around the role that a healthy observability cycle plays in a data-driven development process.

This post will cover these practices, using real world examples for each component. The examples use stream processing (specifically, Apache Flink) but the concepts are applicable to any scenario where you are building or combining systems that have complex integration points.

Firstly, using stream processing — especially when you’re really, truly leveraging its strengths — means you’re probably pushing mass amounts of data through your system, very fast, with complex data transforms. You’re likely ingesting multiple data streams, maybe splitting them or joining them together. This of course is where a powerful, flexible stream processing framework really shines.

But, it also means you now have a complex addition that the rest of your technical ecosystem may not be prepared to handle; whether that’s your downstream applications or sinks, or the teams who own them. Or, it could be the impact on shared resources like storage or internal deploy tools that just aren’t configured for something that powerful or complex. Maybe reconfiguring them isn’t complicated technically, but it may require buy-in and negotiation between other impacted teams, all of which can add to development time and complexity.

One of the most common mistakes I see people make when building a complex stream processing system — especially one that needs to be integrated into an existing, non-streaming environment — is treating it like any other new “traditional” or batch framework. Additionally, anything involving this level of complexity in its integration points, also requires much more detailed communications between other teams and even stakeholders.

For all of these reasons, best practices that could previously be loosely adhered to or ignored can suddenly require a lot more formalizing in order to avoid common pitfalls.

Step 1: Solidify “dashboard hygiene”. Some people love this sort of task, but for many of us, it can be hard to sit down and do this when there’s all of the main programming work to do. So, when we start organizing a dashboard, it is easy to fall into certain anti-patterns.

To counteract that, I like to start with this reminder:

“You can measure almost anything, but you cannot pay attention to everything”
- Anonymous

I like this quote, because — I also hate it.

Personally, this is the trap I fall into the most frequently, because I just want to measure ALL THE THINGS, ALL THE TIME.

And, sure — there’s nothing wrong with wanting a lot of data about your system. But if you want to really leverage your metrics, some good dashboard hygiene becomes a necessary baseline.

And that’s why I appreciated where a colleague of mine went with this, when he said:

“Any situation where people create their own dashboards, without structure, quickly starts to look like the cockpit of a 747”

https://www.reddit.com/r/pics/comments/5vv8qt/the_pilots_seat_and_cockpit_of_a_boeing_747/

I appreciate this particular take as it provides such a great image of just what an anti-pattern this is.

Unfortunately, it’s rare that I talk to someone who has NOT experienced this at one job or another. As validating as that is to hear for people like me who so often fall into this trap, it really should not be the norm.

Which bring us to, how do you organize a truly “clean” dashboard for stream processing (among other types of complex) systems without having to spend a lot of energy upfront, or on maintenance? This is where the “Metrics-driven Metrics” cycle comes in.

High Level Goals

The dashboard that contains metrics on your application or system should always:

Prioritize meaningful data
Be easy to iterate on
The data should be very accessible.

Implementation

The dashboard prioritization should reflect:

Your team’s roadmap
Your application’s highest risk components
Your application’s most uncertain components.

Example

The following dashboard is from a team I was on at a previous company. It was for this company’s first Apache Flink application in production, which was also one of the first stream processing ones there. Additionally, it involved integration into a large and highly complicated non-streaming technical ecosystem.

This example has remained highly relevant as it really clearly embraces the implementation of these goals.

Reflecting the Roadmap

This screenshot was taken early in the development process. This application had multiple incoming data streams, each from a separate, autonomous teams’ application. Our application was ingesting each of these via a Kafka topic. Our roadmap priority at this time was to make sure each stream was being ingested properly and had sufficient alerts and other safeguards.

This is a great example of where “knowing your normal” comes in. As I mentioned in the previous post, this is the next step once your metrics are sufficiently meaningful, iterative and accessible. This is where your dashboards are so clear and familiar to your team, that abnormalities are as noticeable to your developers as they are to your alerting system.

Much of my use of this practice I owe to a previous team mate of mine. This team mate went to great lengths to ensure that the whole group adopted this properly into our routines and individual habits. He added this dashboard to every available screen in our team space and would make sure we reviewed and iterated on it during every sprint planning. I mention these examples because they were such a straightforward and effective way to instill these practices.

Since this level of familiarity with our metrics was now so ingrained into the team, an “oddly shaped” chart would quickly motivate us to investigate further. In this example, everything looked normal as per our alerts (which had been thoroughly set up through meetings with upstream teams and architects). However, half of our Kafka lag charts took on one shape and half took on another. This led us to take additional measurements of data flow. In looking at the structure of the output, we were able to determine that some of our upstream teams aggregated and transformed their data very differently than we expected.

This is where our metrics really became a cycle: this information led us to continuously adjust our monitoring to better reflect the behavior of the data coming in as we proceeded to investigate all discrepancies. Additionally, this particular metric influenced our roadmap and design: it caused us to completely redefine responsibility for integration points in a way that was based more accurately on the behavior of the whole system itself.

Note: these practices can catch a lot of these sort of discrepancies. However, they’re meant to be used as a compliment to, and not as a replacement for things like having formal schemas that are well defined, regularly tested, and accessible for teams all along the pipeline.

Highest Risk

Next, we have our highest risk metrics. In this example, it’s watermarks. Normally, that’s not a particularly high risk element. However, it was for this team during this stage of development.

At this time, our watermark configuration was based on virtually no information about our brand new incoming data. And, due to the nature of watermarks — being so essential to event-time operators, late record handling, etc — keeping them front and center at this time was integral for us being able to build a robust and efficient stream processing system. That is, it was important at that moment, until our team could be confident that it was sufficiently attuned to the normal and expected behavior of our incoming data.

This part of the dashboard should reflect what is the highest risk for you, at this time, knowing this will likely change and evolve with the development of your streaming application.

Most Uncertain

As I mentioned, in this example, we had just started joining many different Kafka topics from various unrelated upstream applications. So, at this time, our Kafka consumption held a lot of uncertainty and was something we needed to keep visible until its behavior could become sufficiently familiar. This meant that Kafka consumption would become the next most visible panel on the dashboard for this time period.

Summary

In this example, we have a dashboard that reflects the current priority on the engineering team’s roadmap, the highest risk component to the team at the current time, and the most uncertain element at this stage of development. Anything following this would be a repeat of these categories, for the next level of priorities.

More importantly, we have a dashboard whose panels are meaningful, iterative, and accessible. This dashboard is also an active part of an observability cycle, and of a cycle of influence with the roadmap.

All of this is intended to aid in avoiding many of the hurdles inherent in adopting a new technical system which adds complexity to, or additional integration points. These practices are of course also helpful for other systems or applications, as they provide a checklist for anyone who wants to create a more sustainable and reliable data-driven development process.

That being said, these practices still need to be paired with other healthy team and inter-team baselines like a solid architectural plan, accessible schemas, and well organized communication channels across the whole pipeline. There are also some additional steps to leverage what was covered here for streamlining the non-programming challenges, which will be covered in the next blog post.

In this series

Previously — Part 1: Introduction + Foundational Concepts
Part 3: Metrics as a Shared Language — arriving on March 18, 2021
Part 4: Hands-On- Monitoring for Stream Processing — arriving in April, 2021

Related talks

“Sweet Streams are Made of These: Applying Metrics- Driven Development to Apache Flink” | Flink Forward EU/Virtual | 10.19–22.2020
“Sweet Streams are Made of These: Lightning Version” | DDD EU | 02/05/2021 [recording in progress]

Find me

Twitter: @caito_200_ok
Web: http://caito-200-ok.com/

Virtual Events: Speaker & Organizer Guide, Part 4

Caito_200_OK — Sat, 06 Mar 2021 02:15:11 +0000

Part 4: Conclusion & Resources

Previously, this series covered accessibility and inclusion, the speaker experience, and a knowledge share of best tactics and innovations in the form of some cheesy-looking but well-earned awards. This post adds the conclusion, and a consolidated list of the resources and tools that were discussed.

Conclusion

Whether or not virtual conferences will be a permanent option, they’re the option we have now, and will likely stay in some form or another. There are a lot of benefits to leveraging the opportunities they bring, and in getting creative with the challenges.

This year I have been impressed with methods that leveraged the virtual platform to better accommodate global reach, wider accessibility options, and different learning and social styles. Even more so, it’s been refreshing to see solutions spread across a wide spectrum of manual thought-work to automated tools that are constantly improving in this new virtual-focused world. I am genuinely looking forward to continuing to see the limits pushed of these tools and techniques, and seeing what that can get us in terms of attendee inclusion, content and connection.

Tools

Icebreaker video
Virtual photo booth
Happy hour app — DigiBaari
Happy hour app — Spatial.chat
Happy hour app — Gather.town
Customizable Marketing — Canva
Talk recording tool — OBS
Social media graphics — Canva

Credits

In This Series

Virtual Events: Speaker & Organizer Guide, Part 3

Caito_200_OK — Sat, 06 Mar 2021 02:13:32 +0000

Part 3: And the Award Goes To…

This is the third post in a series about participating in and organizing virtual events. The first post covered accessibility and inclusivity, and the second post covered the speaker experience. The last post includes the conclusion and a consolidated list of the tools and resources listed in this series.

There were SO many great, innovative, and thoughtful things I’ve seen at the 30+ virtual events I’ve participated in since pandemic started. Recognition and knowledge share for what goes into these sorts fo events often gets lost nowadays so I wanted to share some of the most impressive tactics, workarounds, and innovations I’ve seen since March 2020.