DEV Community: Ricardo Castro

How important is Observability for SRE?

Ricardo Castro — Fri, 11 Feb 2022 13:11:16 +0000

Observability is what defines a strong SRE team. In this blog, we have covered the importance of observability, and how SREs can leverage it to enhance their business.

Observability is the practice of assessing a system’s internal state by observing its external outputs. Through instrumentation, systems can provide telemetry such as metrics, traces, and logs that help organizations better understand, debug, maintain and evolve their platforms.

SREs use many tools and practices to manage services at scale and observability is a crucial part of it. Observability enhances SRE by allowing its practitioners to infer a system’s internal state. Actionable data is of the utmost importance for SRE in order to develop and maintain scalable, reliable, and secure systems. Observability provides the data that SREs need to better understand their systems, what is happening, and why.

What is Observability?

In traditional monitoring systems, you usually have a series of dashboards that help you understand when something wrong is happening. Usually in cloud native environments, using a microservices architecture, we assume services are meant to be run by software and not by humans. This increases the level of complexity and its dynamic nature makes it difficult to reason about problems. You need to make your systems observable so that you can dig into what’s going on.

Observability gives you the capacity to measure the internal state of your systems by checking their external outputs. It is built around three key pillars: metrics, traces, and logs.

Metrics are measurements of something about your system. They are numeric values, over an interval of time, usually with associated metadata (e.g., timestamp, name). They can be raw, calculated, or aggregated over a period of time. They can come from a variety of sources like servers or APIs. Metrics are structured by default and can be stored in open source systems like Prometheus and Riemann or in off-the-shelf solutions like Amazon CloudWatch and Azure Monitor. These optimized storage systems allow you to perform queries, create alerts, and store them for long periods of time.

Traces are the record of the execution path of a program or system. They represent the flow of a request through your services and allow you to see the end-to-end path of execution. Distributed tracing is particularly important in modern distributed architectures, like microservices. The primary building block of a trace is the span. In the OpenTracing specification, spans encapsulate the following information:

Operation name
Start and finish timestamp
key:value span Tags
key:value span Logs
SpanContext

A trace is a group of multiple spans that usually contain “References” to each other. They can be displayed using open source solutions like Jaeger or Zipkin as well as in SaaS offerings like Honeycomb or Datadog.

Logs are text records that describe discrete events, at a specific point in time (e.g. error, an important operation was executed). They’re typically the first place you’ll look to find what is going on with your systems. They include a timestamp and a payload to provide context. Logs can be in three major formats: plain text, structured and binary. Structured logs, which include additional metadata, can be stored in systems like Elasticsearch or Loki to be easily and efficiently queried.

SREs can leverage this information to better understand, maintain and design systems that work at scale.

How can SREs leverage Observability

According to the 2020 SRE Report, only 53% of respondents said they were using observability tools. This is a surprisingly low number considering that the pressure to iterate faster and meet customer expectations increased the demand for observability.

The increasing complexity of systems results in more unknowns and teams need to answer specific questions about their systems. Observability tools can help you take proactive actions to fix issues before they have a major user impact. In order to leverage observability, you’ll need to put in place the proper tooling and services to collect the necessary telemetry. Using open source software or commercial solutions you’ll need to:

Instrument your services to collect telemetry. This telemetry can come from servers, containers, or services and will provide information about your entire infrastructure
Correlate data between multiple sources, creating context, enhancing visualization, and enhancing automation

By using relevant metrics that track user satisfaction you’ll be able to understand when your services are not being reliable enough. By using traces, you’ll be able to understand the flow of requests through your systems and pinpoint where bottlenecks are forming. By using logs you’ll be able to track and understand meaningful events in your services. Armed with this information you’ll be able to detect issues faster before compromising SLOs. Mean time between failures (MTBF), mean time to failure (MTTF), and mean time to repair/recovery (MTTR) can be greatly reduced due to better insights and the alerts observability provides. Well-crafted alerts, based on SLOs and powered by observability, can help reduce alerts to a sustainable amount of actionable events. This helps reduce burnout and creates a culture that supports sustainable innovation.

Incident analysis and postmortems benefit greatly from observability. It enables you to know what’s happening under the hood, what needs to be improved or fixed. It allows end-to-end observability, enabling faster root cause analysis and fixing.

By gathering telemetry in a consistent and automated way, you’ll be able to implement MLOps and AIOps practices. These practices use Machine Learning and Artificial Intelligence techniques to simplify and enhance operations and accelerate problem resolution. They’ll allow you to replace repetitive manual tasks with intelligent and automated solutions that allow you to be proactive in the event of slowdowns or outrages. Observability generates huge amounts of information that humans can’t possibly analyze and correlate. By ingesting all that data, from the various observability solutions, these techniques can conclude what is relevant to focus and point SREs in the right direction.

How SRE and Observability can enhance business

SRE work and business goals are directly intertwined. Users determine the reliability of a system making it one of its most important features. Happy users generate value (e.g. revenue, product popularity), and as such, understanding and keeping users satisfied is of the utmost importance.

Observability provides the tooling necessary to understand user happiness by offering solutions to craft SLOs that measure user happiness. SLO, which stands for Service Level Objective, are measurements of user satisfaction. Instead of understanding how reliable your systems are by using indirect measurements (e.g., server metrics like CPU and memory usage), SLOs can be crafted to understand how satisfied users are (e.g., users can’t buy certain products). You can leverage projects like sloth to help craft SLOs, create dashboards and meaningful alerts. Businesses can use the metrics to make decisions about what features to develop and what type of work needs to be prioritized. SLO-based approaches allow organizations to have informed discussions, backed by data, about when reliability work should be a priority and when feature work should be prioritized.

Having better insights and understanding about systems, allows organizations to reduce the cognitive load on engineers to develop and maintain services. Smaller, multifunctional, autonomous teams will be able to operate their services with increased productivity. Toil reduction is made easier since you now have ways to quickly measure and assess the impact of any change introduced to the system.

Conclusion

The increasing complexity of systems drives the need for better ways to understand them. Observability bridges the gap between your mental models about a system and what they really are. Metrics, traces, and logs provide the necessary information for you to develop and maintain services at scale.

SREs can leverage observability in order to enhance their understanding of systems. Increased visibility allows engineers to more easily understand what is happening under the hood and what actions need to be performed. Well-crafted SLOs and alerts help SREs reduce burnout and be more effective.

Businesses benefit from observability by leveraging it to understand user satisfaction. By understanding how happy users are with your services, you can make informed decisions about the type of work that needs to be prioritized. This increased systems understanding will allow engineers to reduce the cognitive load necessary to develop and maintain them, opening the door to smaller, multifunctional teams to be more effective.

Keeping users happy and engineers more productive will help businesses thrive. Site Reliability Engineering will leverage observability tools to make that a reality.

Squadcast is an incident management tool that’s purpose-built for SRE. Your team can get rid of unwanted alerts, receive relevant notifications, work in collaboration using the virtual incident war rooms, and use automated tools like runbooks to eliminate toil.

Using a Kubernetes Service Mesh Worth It?

Ricardo Castro — Sat, 11 Dec 2021 08:37:25 +0000

"A service mesh is a dedicated infrastructure layer that decouples some of the critical operational tasks of a distributed application from its business logic. Large-scale, Kubernetes-hosted microservice applications are natural candidates for service meshes due to their complex requirements of inter-services communication (e.g., retries, timeouts, traffic splitting), observability (e.g., metrics, logs, traces), and security features (e.g., authentication, authorization, encryption). Service meshes can offload many operational concerns of the Kubernetes cluster, leaving the developers to focus on business logic."

Is using a service mesh in Kubernetes really worth the trouble? Not sure? Then check this out.

How to improve your influence as an SRE

Ricardo Castro — Mon, 06 Dec 2021 12:45:21 +0000

Improving your influence over the company will help you deliver high quality work as your goals will be closely aligned with those of the company. In this blog piece, Ricardo has explained how to improve your influence as an SRE.

Balancing fast-paced business requirements with the demands of keeping production services stable is not an easy task. SRE is an opinionated implementation of DevOps and is defined by Ben Sloss, VP of Engineering at Google as “what happens when you ask a software engineer to design an operations function”. And it even comes with a completely free manual and workbook.

Although SRE aims to be a “prescription” on how to run complex systems the right way, reliability can mean different things in different contexts. And, usually, unless things go wrong it’s hard to prioritize reliability work ahead of features and bug fixes.

How can SREs encourage teams to think about their operational excellence? How can SREs go about making reliability part of everyone’s daily practices? How can SREs effectively influence people to take reliability seriously and incorporate SRE concepts and practices into their routines? It turns out this is one of the most important question every SRE faces.

Influence vs Authority

When trying to disseminate a practice and cultivate change, you can usually go one of two routes: influence or authority. Influence is “the capacity to have an effect on the character, development, or behavior of someone or something, or the effect itself”. In your context, the goal is to provide best practices, resources, and tools in the hope teams adopt them.

In contrast, authority is “the power or right to give orders, make decisions, and enforce obedience”. Applied to your context, it would be to dictate that teams must follow and adopt several operational practices.

Both approaches can be valuable, depending on the context. For example, for critical systems (e.g. healthcare, aviation) authority might be required to ensure a certain level of safety. On the other hand, authority can be a detriment for teams, making them not feel part of the decision process, not taking into consideration their unique context, and alienating them. With that in mind, you want to be as influential as possible regarding reliability.

Improving SRE influence

When it comes to improving influence there are several ways you can go about it. Applied to an SRE context, there are several tactics you can employ that will help you make reliability part of the discussion when building complex systems.

Make sure people understand your purpose and motivations

When people don’t understand your motivations and goals it’s natural for them to become defensive; it’s human nature. It will be a lot easier to spread a reliability mindset if people understand what you’re trying to achieve.

If you’re a central SRE team make it crystal clear what your team does. Clarify if it’s an operational team that takes care of services in production or if it’s a team more focused on guidelines and tooling. Create artifacts that people can access asynchronously (e.g. internal documentation, blog posts) that makes it easy to understand what you’re working towards. Give internal presentations about your work and roadmap and make your team available (e.g. mailing list, slack channel) for consultation or even just for informal chats.

In general, people will be more receptive and address your concerns if they know what you’re all about and that they can reach you when necessary.

Drive an understanding that SRE work ties directly to business goals

A system’s reliability is determined, fundamentally, by its ability to do what its users need it to do. It will then be determined by how happy users are and you know those happy users are good for business. Accepting that reliability is one of the most important requirements of any service, users determine its reliability.

SRE work will be intimately tied to business goals. Satisfied users will generate value (e.g. revenue, product popularity) and reliability is a huge contributor to that perception. It will be important to drive this understanding that you’re not focusing on reliability just to be picky but because that concern will make your business prosper.

Create a common language to talk about reliability

You’ve probably been in a situation where you’re trying to communicate with someone that does not speak the same language as you. Maybe you’ve gone to a foreign country, you’re asking for directions and you don’t speak the local language. Eventually, you make whoever you’re trying to speak with understand, you just want the directions to that awesome attraction and they do their best to point you in the right direction. More often than not, it’s a difficult exchange.

Similarly, if you approach product development teams without a clear way to talk about and measure reliability, it will be hard to reason about it. Creating a shared language to talk about reliability, assess it and prioritize work will be detrimental to the success of your quest. The reliability stack will give you the basis for a framework that will make reliability conversations a lot easier. SLIs will provide you with the necessary reliability measurements while SLOs will allow you to assess, within a certain period of time, how reliable your system is. With those pieces in place, Error Budgets will make it easier to prioritize work that addresses reliability concerns.

Get buy-in

Because there will always be bugs to fix and features to deliver, reliability will often be an afterthought. Getting buy-in will ensure teams have reliability in mind and will advocate for it on your behalf.

Identify key stakeholders that will help you “spread the message” and treat reliability like an obvious requirement. This will be highly dependent on your organization, but before diving into processes and tools, you should first focus on people. Maybe you need to get product development teams onboard first so that management feels that, not only, reliability is important but that teams are taking it seriously and are receptive to prioritize reliability work. Or maybe you need to address it the other way around, getting buy-in from executives so that development teams understand the need for reliability and feel safe spending time working on it.

Whatever route makes more sense in your context, it will definitely help you improve reliability awareness when key stakeholders are “on your side” and they themselves drive those discussions with peers as well as management.

Make your work visible

SRE work is, effectively, work. There’s engineering work but there’s also a lot of work related to advocating, coaching, or consulting. And it should be clear to everyone what you’re working on, what your goals are, what you’re trying to achieve, what your roadmap is.

Whatever tools and processes product development teams are using, you should be using similar ones. This will help standardize how work is tracked and prioritized. It will allow teams to easily understand what you’re focusing on, if they require something from you that you’re not prioritizing or if your shared goals might be compromised.

Making sure your work is visible will assure teams you don’t have any hidden agenda and that you’re all working with the same goals in mind.

Communicate extensively; always be on sales mode; build bridges

It’s very important to build bridges. People are a lot more receptive to hearing your thoughts and ideas if they trust you. If you start, out of the gate, telling people what they need to fix and prioritize, you’ll be met with a lot of resistance.

Start by sitting down with teams and understanding what they do, what their products are, what pain points they have, and what they would like to improve. Actively listening can take you a long way. Ask questions, clarify issues, and understand what’s at stake. You’ll want teams to see you as a partner, that wants to make their lives easier and not as an adversary.

An “us vs them” mentality can be attained quickly if you start imposing reliability-concern gates. Either with lengthy manual processes or automated blocking checks, enforcing might come at the expense of the team’s goodwill. Instead, work with them to make sure those concerns are valid, share engineering work that will address those concerns, and communicate well in advance about when, and if any blocking gate will be put in place.

Be self-service, not a bottleneck

SRE work is never-ending. It’s a day-to-day practice that needs to scale with the organization. And one of the best ways to scale it is to be self-service.

You should be building tools that would make it easy for teams to incorporate reliability concerns into their work. For example, you could be building and maintaining libraries that export the necessary metrics to your monitoring system or that standardize logging. Or you could be building automation capabilities that would help diagnose problems in your systems. Or you could be building tools that automate manual tasks and address known issues within your systems. Most importantly, you don’t want to be a bottleneck. You want teams to be as independent as possible to do their work and deliver value to the business.

Conclusion

SRE involves a lot of engineering work encompassing, at the same time, a lot of communication. A lot of that communication is targeted at influencing teams to take reliability seriously and make it part of their work.

Making sure that teams understand what you're working towards is critical to get people on board. You should communicate extensively, create artifacts, give internal talks, partner with teams, and make your work visible. Make sure teams understand that reliability is measured by how happy users are with your services, that reliability work is focused on making sure they are satisfied, and that the business will thrive on that.

Get people on board, get their buy-in. Your message and goals will be easier to spread if you have advocates on your side. Identify key stakeholders, work with them, understand their concerns and build with them a common understanding of what reliability looks like, that they would happily share with others. Having a shared language or framework, like the reliability stack, will make it a lot easier to talk about, assess, and prioritize reliability work.

At the end of the day, you want to deliver value and enable teams to focus on delivering value to the business. Make yourself self-service. Build tools that help improve reliability, which is a no-brainer to use. Help teams solve pain points through automation. Build self-remediation tools to help address known issues in your system. And make sure you don’t become just another 'pain' or 'gate' that teams complain and dread about. You should be seen as a partner, as a team that works with the same goals in mind and that is there to help when necessary.

How to Measure System Reliability

Ricardo Castro — Sat, 13 Nov 2021 17:19:31 +0000

Originally published on Cprime.

As businesses grow, new requirements arise for teams. The technology ecosystem becomes ever more complex and it is really important to understand each change and how it affects the overall system, as well as the service provided to users, who have high expectations. They expect systems to be up, responsive, fast, consistent, and reliable.

System ReliabilityReliability for systems means that a system is doing what its users need it to do. The reliability of a system is essentially how happy the customer is and we know that a happy customer is better for business. If we accept that reliability is one of the most important requirements of any service, users determine this reliability, and it’s okay to not be perfect all the time. We need a way of thinking that can encompass these truths. We have limited resources to spend, be they financial, human, or political.

Service Level Indicators

A metric is a measurement of something about a system. Good examples are the amount of memory a server is using, the time it takes an HTTP request to be fulfilled, or if an HTTP response was an error. Metrics are extremely important to understand a system and develop higher-level constructs that help us assess a system’s reliability.

Since we can’t measure user happiness directly, Service Level Indicators (SLIs) are proxies to help us gauge user satisfaction. SLIs represent a quantifiable measure of service reliability and we can calculate them:

SLI = (Good events / Valid Events) * 100%

What is, then, a Good Event? This can be challenging to define. For example, measuring the latency for a service is not enough. What good latency is, needs to be defined so that the SLI is useful. It forces a binary state to be achieved, even if the underlying metrics don’t provide a binary state.

A good example of an SLI would be:

Requests to a service will be responded to within 200ms.

Good events would be all requests that are responded to within 200ms. This SLI provides information about whether or not the service is up if it is available, and also if it is responsive. With a simple SLI, we can get a lot of information from the service state. The trick is to keep it simple and iterate. A good way to go about this is to choose important features the service is providing and measure what it’s valued by its users.

SLIs are foundational to assess the reliability of a system and are therefore very important that they are chosen meaningfully. If an SLI isn’t good, concepts like SLOs and Error Budgets aren’t either. Why is this important? If reliability is always measured from the user’s perspective, a user can be a human or a machine, and we choose an indicator that doesn’t measure user happiness, all SLOs, and Error Budgets would be useless since users would be unhappy whether we are achieving the objective or not.

Service Level Objectives

With an SLI defined, a goal to be achieved is the next target. In other words, an SLIdefines what matters to users, while a Service Level Objective (SLO) defines how many times it has to be achieved for users to be happy with the service.

In a perfect world, 100% would be the target. Unfortunately, in the real world, reliability is expensive, and to achieve 100% no failures could occur. But the internet is complex. There are switches, cables, routers, ISPs, CDNs, etc, – and all of that sits between systems and the users using them. At any point in time, any of those can fail. With a slightly less reliable service, a user cannot distinguish between the failure of a system or the internet’s infrastructure. Systems should be built to be reliable enough to make the users happy, no more no less, and allow teams to invest time in providing business value.

SLOs are targets that are agreed upon, about what it means to be reliable enough, for a service that is being provided during a period of time. It means that the more reliable a system has to be, the more expensive it will become.

A good example of an SLO would be:

99% of requests to a service need to be responded to within 200ms within a 30 day period.

Error Budgets

What is left from an SLO is called an Error budget. For example, when defining an SLO of 99%, the Error Budget would be 1%. Error Budgets can be calculated:

Error Budget = 100% – SLO

For the previous example, when the measurement period starts, 1% is the Error Budget available. During the 30 days period, incidents will burn the Error Budget each time they happen. An Error Budget is effectively the percentage of reliability left, and it helps make educated decisions on whether, for example, to release a new feature or not based on the risk. They help make sure the operability process (e.g. incident response) is appropriate to the budget available for the service being provided.

SLIs, SLOs, and Error Budgets form a framework that makes it possible to define what reliability means for services. It allows reliability to be continuously measured and provides a platform for educated decisions to be made about how to prioritize features against reliability work. It allows development and operations teams to agree on what reliability means for a service, supported by data and avoiding unnecessary confrontation. It allows teams to collaborate, as well as focus on their specific work, by making sure user satisfaction is measured and assessed.

Troubleshooting Kubernetes FailedAttachVolume and FailedMount

Ricardo Castro — Thu, 11 Nov 2021 10:29:39 +0000

You've been working with Kubernetes and sometime your stateful workloads fail. You regularly get FailedAttachVolume and FailedMount errors and you have no idea what's going on.

If you want to find out what's going on and how you can go about fixing quickly them, check this out.

From Zero to SRE

Ricardo Castro — Tue, 05 Oct 2021 16:24:10 +0000

Traditionally, developing applications and running them in production was seen as completely separate worlds, usually being the focus and concern of different teams. This kind of separation gives birth to the proverbial wall that separates development and operations, where developers “throw” their code over the wall and expect operations teams to run and manage them in production. This results in teams having different and conflicting goals: development teams prioritize building and shipping new features while operations teams focus on system stability, where code changes are seen as potential threats.

With systems getting larger and larger, this type of situation does not scale, and a new way of doing things needed to emerge. And in 2009 Patrick Debois coined the term “DevOps”, a set of principles and practices with emphasis on people and culture and the goal of improving collaboration between development and operations teams.

A few years before, facing a similar situation, Google put Benjamin Treynor in charge of a team with the goal of ensuring that Google’s websites were available, reliable, and as serviceable as possible. Since Benjamin was a software engineer he approached an operations problem with a software engineering perspective, giving birth to what is today known as Site Reliability Engineering (SRE): “what happens when you ask a software engineer to design an operations team”. SRE rose to fame around 2016 after Google published the book Site Reliability Engineering: How Google Runs Production Systems, describing a lot of the practices they use to run their systems in production.

SRE is all about running reliable production systems. Big companies like Google, Facebook, or Amazon run large production systems and face many challenges most companies rarely do. Despite that, the way they run their systems can help us run ours. But even armed with all that information, how do we go about starting our SRE journey?

Firefighting

In the beginning, teams are reactive. There are a lot of moving parts, highly complex systems, failures appearing right, left, and center. The world is chaotic and it seems close to impossible to apply any sort of engineering practice to tackle the situation. At this stage, teams don’t have many options and need to keep struggling, responding to a myriad of crises, while trying to reserve some time, energy, and focus to try and improve the situation. Putting brakes on feature development can help stabilize a system, getting teams some relief. Adding people to teams that can focus on automation, can help reduce toil.

This is a difficult stage for teams. It requires a dual perspective since they will need to keep systems running while building a new approach to run operations. This is also a reflection on operations teams being brought late into the product cycle. They can be made aware of a new service or a new set of requirements just before a service needs to go live, or even worse, when something is already live and failing miserably.

Gatekeeping

Usually, one of the first measures that are put in place to deal with firefighting is gatekeeping. The goal is to make every change to production pass through and be approved by the SRE team. On a small scale, this approach can work for a while. But, as systems and teams grow, SRE teams start being a choke point, limiting change and slowing down other teams.

At this stage, SRE teams work to become more engaged with development teams by implementing processes. While this reflects positive engagement, it can derail into an us-versus-them struggle, leading teams to circumvent the processes and SRE teams all together.

Partnering

When there are frameworks in place (e.g. SLOs and Error Budgets) that can be used to measure reliability and make decisions on how to act, SRE teams can remove themselves from the critical paths to production. They can then partner with development teams and work together to meet the desired reliability criteria. At the same time, SRE teams will be continually involved in how to measure the impact on user experience. At this stage, SREs are involved in the process earlier in the cycle and work alongside development teams right from the start, on the reliability of the product.

Relationships between teams become a lot less antagonistic. SRE teams are regularly sought for cooperation since they enable much higher satisfaction and long-term value.

Enabling

When reliability is part of the process, SREs are involved in the full lifecycle of a service, from creation to decommissioning. SREs can act as consultants, bringing consciousness about business goals, reliability, and security. Processes are in place to allow teams to measure how reliable their services are and how to act when necessary. This allows scalability since teams can operate independently while being able to rely on SREs when necessary. Development teams have the ability to pull SREs and SREs don’t need to impose demands. A voluntary, data-supported engagement between teams becomes enjoyable and sustainable.

Enabling SRE teams to become highly functional is a long process. It takes time and some steps are “mandatory”. Starting in firefighting mode is almost inevitable for most organizations. It is then, when they realize that a new approach to manage operations needs to emerge. At this stage, some time has to be set aside so that teams can work on processes and tools to help ease the pain of constant firefighting. A lightweight gatekeeping stage can help bring teams together as well as identifying processes and automation that would pave the way to the partnering stage. Enabling stage is the end goal where teams work together, armed with tools that help measure reliability, and processes that help prioritize work and decide how to proceed when reliability is at risk.

While some short-circuiting can be done to accelerate the process, some stages build on others and use them as foundations. High-level management support can help accelerate the evolution by allowing teams to focus on prioritizing the work that will make them reach the enabling stage. Focus on success stories to help build momentum. A lot of SRE is cultural, a different way of doing things. Successes help build a narrative and motivate people to follow the same path.

Creating an SRE team can seem like a daunting task and there are several topologies that can be adopted. Each organization needs to find what better suits its needs. But there are some guiding principles to help navigate this journey:

Start small and iterate - in the beginning, keep it simple. Select a small group of people to implement SRE practices. Find what works and what doesn’t. Iterate and onboard new practices
Find the right people - SRE teams should have a mix of domain knowledge. Either through internal or external hiring, aim to build teams with a diverse set of skills
Scan for the right qualities - depending on the scope and topology of the team, the skill set might defer. Some good qualities to scan for are the ability to see the bi picture, the ability to troubleshoot, good communication skills, desire to remove toil, a knack to dig deeper, and do excellent teamwork
Training - a lot of SRE work is cultural. The ability to communicate well and provide training to teams is paramount
Guidelines and Governance - SRE teams help facilitate the adoption of a reliability mindset. They can help establish guidelines that make it easier for teams to onboard reliability practices

Either through a separate SRE team or any other topology, it’s important to start simple and iterate. Introduce SRE principles, assess, find out what works, and then incorporate. Navigating from Firefighting to Enabling is a journey and it will be slightly different for each organization.

From Monolith to Microservices and Beyond

Ricardo Castro — Tue, 05 Oct 2021 16:22:09 +0000

Originally published on Cprime.

Monolithic architectures were the de facto standard of how we built internet applications. Despite still being used nowadays, microservices have grown in popularity and are becoming the established architecture to build services. Service-oriented Architectures (SoA) are not new but the specialization into microservices, which are loosely coupled, and independently deployable smaller services, that focus on a single well-defined business case, became wildly popular since they enable:

Faster delivery
Isolation
Scaling
Culture
Flexibility

Distributed systems are complex. The above (simplified) diagram depicts how a monolith can be broken down into microservices but hides a lot of complexity, such as:

Service discovery
Load balancing
Fault tolerance
Distributed tracing
Metrics
Security

As systems grow bigger and bigger these challenges become exacerbated to the point where they become virtually impossible to tackle by teams focusing on specific business cases.

With the advent and popularization of containers, technologies emerged to tame the ever-growing operations demand, offering rich sets of features. Enter Kubernetes, the most popular container platform today, supported by every major cloud provider, offering:

Automated rollouts and rollbacks
Storage orchestration
Automatic bin packing
Self-healing
Service discovery and load balancing
Secret and configuration management
Batch execution
Horizontal scaling

Orchestration platforms like Kubernetes aim to ease the operational burden on teams while bringing some new development patterns to the mix. But while this is true, there’s some effort that still has to be put into operating services, from the development team’s point of view. Platforms like Kubernetes have their own “language” that needs to be understood so that applications can be deployed on it as well as a diverse set of configuration features and requirements. For example, for Kubernetes to optimally handle service lifecycle, services should provide health endpoints that Kubernetes will use to probe and decide when to restart such service.

Although Kubernetes was developed to orchestrate containers at large, it does not manage containers directly. Instead, it manages Pods which are groups of containers that share storage and network resources and have the same lifecycle. Kubernetes guarantees that all containers inside a Pod are co-located and co-scheduled and that they all run in a shared context.

These shared facilities between containers facilitate the adoption of patterns for composite containers:

Sidecar - extend and enhance the main container, making it better (e.g filesystem sync)
Ambassador - proxy connections to and from the outside world (e.g. HTTP requests)
Adapter - standardize and normalize output from sources (e.g. data from centralized logging)

By going deeper into microservices architectures, with the ultimate goal of each microservice focusing only on its own business logic, we can hypothesize that some functionalities could be abstracted and wrapped around the business logic using the above composite patterns. In general, services require:

Lifecycle management - deployments, rollbacks, configuration management, (auto)scaling
Networking - service discovery, retries, timeouts, circuit breaking, dynamic routing, observability
Resource binding - message transformation, protocol conversion, connectors
Stateful Abstractions - application state, workflow management, distributed caching

Service Mesh

A Service Mesh is a dedicated and configurable infrastructure layer with the intent of handling network-based communication between services. Istio and Linkerd are two examples of implementations. Most implementations usually have two main components: the Control plane and the Data plane. The Control plane manages and configures the proxies that compose the Data plane. Those Data plane proxies are deployed as sidecars and can provide functionalities like service discovery, retries, timeouts, circuit breaking, fault injection, and much more.

By using a Service Mesh, services can offload these concerns and focus on business rules. And since microservices can be developed using different languages and frameworks, by abstracting these functionalities, they do not have to be redeveloped and maintained for each scenario.

Serverless Computing

Serverless computing comes into play with the promise of freeing teams from having to deal with operational tasks. The general idea with Serverless computing is to be able to provide the service code, together with some minimal configuration, and the provider will take care of the operational aspects. Most cloud providers have serverless offerings and there are also serverless options on top of Kubernetes that use some of the patterns mentioned before. Some examples are Knative, Kubeless, or OpenFaaS.

Distributed Application Runtimes

Projects like Dapr aim to be the Holy Grail for application development. Their goal is to help developers build resilient services that run in the cloud. By codifying best practices for building microservices into independent and agnostic building blocks that can be used only if necessary, they allow services to be built using any language or framework and run anywhere.

They offer capabilities around networking (e.g. service discovery, retries), observability (e.g. tracing) as well as capabilities around resource binding like connectors to cloud APIs and publish/subscribe systems. Those functionalities can be provided to services via libraries or deployed using sidecars.

Microservices and Beyond

Microservices are entering an era of multi-runtime, where interactions between the business logic and the outside world are done through sidecars. Those sidecars offer a lot of abstractions around networking, lifecycle management, resource binding, and stateful abstractions. We get the benefits of microservices with bounded contexts handling their own piece of the puzzle.

Microservices will focus more and more on differentiating business logic, taking advantage of battle-tested, off-the-shelf sidecars that can be configured with a bit of YAML or JSON and updated easily since they’re not part of the service itself. Together they will compose the intricate web services that will power the future.

How awesome is cookiecutter?

Ricardo Castro — Wed, 07 Jul 2021 17:05:31 +0000

Originally published on mccricardo.com.

Have you ever been in the sittuation of trying to structure a project but having no idea how? Maybe you're new to a specific technology and have no clue how to organize things. Or maybe you start projects often and keep repeating the same set of tasks over and over again.

If you read Backstage Software Templates maybe you noticed something called cookiecutter. So, what is it? Apart from being awesome, cookiecutter is a command-line utility that is able to create projects from templates. This allows us and our teams to specify templates for projects, use those templates and share them among teams. We can even embed it into other tools, like Backstage.

Quickstart

There are several options to install cookiecutter but we will go with pip:

~ pip install --user cookiecutter

Now that we have cookiecutter installed, we can define our templates and start using them. Even better is the fact that we can use publicly available templates and start coding right away. For our purposes, we'll be creating a Go project using the cookiecutter-golang template:

~ cookiecutter https://github.com/lacion/cookiecutter-golang.git
full_name [Luis Morales]: Ricardo Castro
github_username [lacion]: mccricardo
app_name [mygolangproject]: demo
project_short_description [A Golang project.]: A demo
docker_hub_username [lacion]: mccricardo
docker_image [lacion/alpine-base-image:latest]: golang:1.16.5-apline3.14
docker_build_image [lacion/alpine-golang-buildimage]: golang:1.16.5-apline3.14
Select docker_build_image_version:
1 - 1.13
2 - 1.12.9
3 - 1.11.9
Choose from 1, 2, 3 [1]: 1
Select go_mod_or_dep:
1 - mod
2 - dep
Choose from 1, 2 [1]: 1
use_docker [y]: y
use_git [y]: y
use_logrus_logging [y]: y
use_viper_config [y]: y
use_cobra_cmd [y]: y
Select use_ci:
1 - travis
2 - circle
3 - none
Choose from 1, 2, 3 [1]: 3
Initialized empty Git repository in /home/mccricardo/test/demo/.git/
[master (root-commit) 2e82ac9] Initial Commit.
 13 files changed, 597 insertions(+)
 create mode 100644 .gitignore
 create mode 100644 AUTHORS.md
 create mode 100644 CONTRIBUTING.md
 create mode 100644 Dockerfile
 create mode 100644 Makefile
 create mode 100644 README.md
 create mode 100644 cmd/root.go
 create mode 100644 cmd/version.go
 create mode 100644 config/config.go
 create mode 100644 go.mod
 create mode 100644 log/log.go
 create mode 100644 main.go
 create mode 100644 version/version.go

After cookiecutter has finished doing its magic we can take a look at the result:

~ demo git:(master) ls
AUTHORS.md  cmd  config  CONTRIBUTING.md  Dockerfile  go.mod  log  main.go  Makefile  README.md  version
~ demo git:(master) cat main.go 
package main

import (
    "github.com/mccricardo/demo/cmd"
)

func main() {
    cmd.Execute()   
}

As we can confirm, based on the cookiecutter-golang template, cookiecutter went ahead and created a folder strucutre as well as several files with boilerplate code.

Go ahead and take a look at the documentation as well as the source code and add one more tool to your automation arsenal.

Giving back

Ricardo Castro — Sun, 04 Jul 2021 15:38:55 +0000

Originally published on mccricardo.com.

The last two and half months have been intense. Nine conference talks, starting on a new role and a lot of reading have consumed a lot of time. Talks in particular take a lot of time. From thinking about a topic, building the presentation, to practicing, a lot of effort goes into making something with enough quality to be accepted at a conference. Coupled with regular blogging, it's like having a second job. So, why do I do it? And more importantly, why should most of us give it a try?

Paying it forward

Most of my career has been built on the shoulders of giants. Everything from open-source software, splendid blog posts, to videos on complex topics have helped me progress and getter better at what I do. It's only fair to contribute back and help others. While once upon a time I contributed directly to a couple of OSS projects, nowadays I feel I add more value by blogging and giving talks. I still do the occasional contribution now and then.

Helping us learn

Blogging and giving talks also help us learn. Often when I want to learn about a topic I decide on giving a talk and/or blog about it. Curiously, the fact of forcing ourselves to explain something to people helps us reason more clearly about that topic as well as understanding it more deeply.

Building a personal brand

Putting ourselves out there makes people aware of we who are and what we do. Over time we start to build a personal brand and get known by the value we provide. It can help our professional careers as well as our self-confidence.

Impostor syndrome

"Impostor syndrome refers to an internal experience of believing that you are not as competent as others perceive you to be" and this one of the main reasons people don't blog or give talks. Fun fact: we don't have to be the supreme authority on a topic to deliver expertise on it; people will benefit a lot from our hard work and will thank us for it.

It's fun

If you're like me you'll always get a bit nervous once a new blog post gets published or your about to deliver a talk. That said, we learn to deal with that and it becomes fun. The benefits seriously outweigh the risks. It's hard work but it's fun!

Why not give it a try?

We don't have to give a hundred talks a year or write a blog post every day. We can start small, see how it feels, see what works and what doesn't, find our niches. Over time things start to gain momentum and the whole world benefits from it.

Backstage Software Templates

Ricardo Castro — Tue, 13 Apr 2021 21:37:28 +0000

Originally published on mccricardo.com.

If our goal is to make our lives and the lives of those that work with us easier and happier, Backstage Software Templates can give us a hand. This tool can help us create Components inside Backstage that will load code skeletons, fill in some variables and publish that template somewhere.

Let's get this party started

If you've been following along, in Backstage intro we set up our Backstage installation locally and, conveniently, bootstrapped a lot of components, including Software Templates. Let's right to it.

We'll start by selecting Create Component.

We'll then select Golang Microservice.

We'll be displayed a form requesting some necessary information. We'll fill it in and click Next Step.

A new form will be displayed that will allow us to set up VCS integration.

At last, we will be shown a review and we can proceed with Create.

And Backstage will do its magic.

A new component will be created.

As well as a new repo with a lot of code.

Let's take a moment to take a look around and let that sink in.

What just happened?

Based on a template, Backstage was able to create a new repo, full of code and integrations. The setup we did in Backstage intro brought all of that free of charge. Taking a look at app.config.yaml we can see:



catalog:
  rules:
    - allow: [Component, System, API, Group, User, Template, Location]
  locations:
    - type: url
      target: https://github.com/spotify/cookiecutter-golang/blob/master/template.yaml
      rules:
        - allow: [Template]

This points to the following template:



apiVersion: backstage.io/v1alpha1
kind: Template
metadata:
  name: golang-starter
  title: Golang Microservice
  description: Create a Golang repo with this template built by members of the Go community
  tags:
    - experimental
    - go
spec:
  owner: web@example.com
  templater: cookiecutter
  type: service
  path: "."
  schema:
    required:
      - component_id
      - project_short_description
      - docker_image
      - docker_build_image
      - docker_build_image_version
      - use_logrus_logging
      - use_viper_config
      - use_ci
      - use_cobra_cmd
    properties:
      component_id:
        title: Name
        type: string
        description: Unique name of the component
      project_short_description:
        title: Description
        type: string
        description: Description of the component
      docker_image:
        title: Docker Image
        type: string
        description: The docker base image to use when running the service
        default: alpine-base-image:latest
      docker_build_image:
        title: Docker Build Image
        type: string
        description: The docker base image to use when building the service
        default: golang
      docker_build_image_version:
        title: Docker Build Image Version
        description: The image version to use when building the service
        type: string
        enum:
          - alpine
        default: alpine
      use_logrus_logging:
        title: Enable Logrus Logging (https://github.com/sirupsen/logrus)
        type: string
        enum:
          - "y"
          - "n"
        default: "y"
      use_viper_config:
        title: Enable Viper Config (https://github.com/spf13/viper)
        type: string
        enum:
          - "y"
          - "n"
        default: "y"
      use_cobra_cmd:
        title: Enable Cobra CLI Tools (https://github.com/spf13/cobra)
        type: string
        enum:
          - "y"
          - "n"
        default: "y"
      use_ci:
        title: Add CI
        description: Add a CI config to the repo, Gitub Actions, Circle or Travis are the only supported right now
        type: string
        enum:
          - github
          - travis
          - circle
          - none
        default: github

Now something has to interpret this and do some magic. This is where packages/backend/src/plugins/scaffolder.ts comes into play:



import { SingleHostDiscovery } from '@backstage/backend-common';
import { CatalogClient } from '@backstage/catalog-client';
import {
  CookieCutter,
  CreateReactAppTemplater,
  createRouter,
  Preparers,
  Publishers,
  Templaters
} from '@backstage/plugin-scaffolder-backend';
import Docker from 'dockerode';
import { Router } from 'express';
import type { PluginEnvironment } from '../types';

export default async function createPlugin({
  logger,
  config,
  database,
  reader,
}: PluginEnvironment): Promise<Router> {
  const cookiecutterTemplater = new CookieCutter();
  const craTemplater = new CreateReactAppTemplater();
  const templaters = new Templaters();

  templaters.register('cookiecutter', cookiecutterTemplater);
  templaters.register('cra', craTemplater);

  const preparers = await Preparers.fromConfig(config, { logger });
  const publishers = await Publishers.fromConfig(config, { logger });

  const dockerClient = new Docker();

  const discovery = SingleHostDiscovery.fromConfig(config);
  const catalogClient = new CatalogClient({ discoveryApi: discovery });

  return await createRouter({
    preparers,
    templaters,
    publishers,
    logger,
    config,
    dockerClient,
    database,
    catalogClient,
    reader
  });
}

@backstage/plugin-scaffolder-backend through CookieCutter "knows" how to bootstrap Golang projects based on cookiecutter-golang. And that's how all the magic happens.

Eventually, we will need to add our own templates and we will need help writing them. And if the builtin actions are not enough this might include writing custom actions.

Backstage Software Templates takes a big leap into providing standardization and help teams become more productive faster.

Backstage TechDocs

Ricardo Castro — Mon, 05 Apr 2021 21:44:34 +0000

Originally published on mccricardo.com.

Documentation, that thing everyone complains about but few want to create or maintain. However, good documentation is priceless, and the less painful it is to create, maintain and access the more valuable it becomes.

TechDocs is Backstages’s docs-as-code built-in solution and this basically means we can write our documentation in files that live together with our code. It's based on MkDocs although there are plans to support other types of sources. TechDocs enables documentation to be found from a service page in Backstage's Catalog and be built with good old Markdown.

How TechDocs works

Backstage generates documentation in a three-step process:

Prepare - fetch the source files from the source code and pass them to the generator
Generate - run the build phase (through a container or mkdocs cli)
Publish - upload the generated files to storage

(image source: Backstage official documentation)

As shown above, for a production deployment, this process should be decoupled from the Backstage deployment. Backstage would only be responsible for fetching and presenting the generated documentation.

Documentation template

Documentation can be created as a standalone project and our local setup already has a template that we can play with.

Add TechDocs to our setup

For our demo, we'll go with a simpler setup (as shown below) and let Backstage do all the heavy lifting. To make things even easier will need Docker.

(image source: Backstage official documentation)

We well reuse our repository from the Flux CD post. We'll add a few files and edit another. We'll start by creating a mkdocs.yaml file.



site_name: 'gitops-flux-helm-docs'

nav:
  - Home: index.md

plugins:
  - techdocs-core

Next we create an index.md inside a docs folder.




# Example

This is a basic example of documentation.

We'll now edit catalog-info.yaml and add the following annotation under metadata.



metadata:
  annotations:
    backstage.io/techdocs-ref: url:https://gitlab.com/mccricardo-blog-demos/gitops-flux-helm

From the repository side, we're done. We've added the necessary configuration that will allow Backstage to find docs/index.md and generate the documentation. We now turn our attention to Backstage. On app-config.yaml we'll add the following code.



techdocs:
  builder: 'local'
  generators:
    techdocs: 'docker'
  publisher:
    type: 'local'

And that's it. We'll let Backstage do all the heavy lifting (with a bit of help of Docker).

Backstage Catalog

Ricardo Castro — Sat, 03 Apr 2021 16:31:51 +0000

Originally published on mccricardo.com.

Now that we have an idea of what Backstage is and can do and of what is under the hood we can start to explore some of its features. In this post, we'll explore the Backstage Catalog.

What is the Catalog?

The Backstage Catalog aims to be the centralized hub to track ownership and metadata of all the software in our ecosystem (services, APIs, libraries, data pipelines, etc). Metadata YAML files are at the heart of the Catalog, alongside the code, and will enable Backstage to collect it and make it visible. The Catalog allows teams to manage and maintain their software within a uniform view. Moreover, it will make all software in our organizations discoverable.

If you've gone through Backstage intro you can find the Catalog at http://localhost:3000/catalog.

How to add components to the catalog

As mentioned above, metadata YAML files are at the heart of the Catalog and should be stored in your VCS of choice. We can add components to the Catalog in several different ways:

Manually register components - via Catalog Import plugin
Static configuration
Creating new components through Backstage
Integrating with an external source - some organizations already have an existing system for keeping track of software and they can be integrated into Backstage

Static configuration

We'll get our hands "dirty" and will add a service to the Catalog through configuration. For that, we'll use a repository created for the Flux CD demo and integrate it in Backstage. We'll add a catalog-info.yaml file to the repository with a simple Component configuration:

apiVersion: backstage.io/v1alpha1
kind: Component
metadata:
  name: gitops-flux-helm
  description: Gitops Flux Helm demo
  tags:
    - gitops
spec:
  type: service
  lifecycle: production
  owner: user:guest

Moving our attention to Backstage, under app-config.yaml we'll update two sections:

integrations:
  gitlab:
    - host: gitlab.com
      token: <YOUR-API-TOKEN>

catalog:
  rules:
    - allow: [Component, System, API, Group, User, Template, Location]
  locations:
    # gitops-flux-helm project
    - type: url
      target: https://gitlab.com/mccricardo-blog-demos/gitops-flux-helm/-/blob/master/catalog-info.yaml

And voila, Backstage can now track our project.

This static configuration achieves a similar outcome to manually registering components. But when you couple this type of setup with discovery integrations like Github Discovery magic starts to happen.

Components created through Backstage are automatically registered in the catalog. In a future post, we'll explore Backstage Software Templates and see real magic happening.