DEV Community: ntcsteve

Automate your configuration with observability as code

ntcsteve — Sat, 11 Nov 2023 22:32:19 +0000

Follow along with dashboards as code examples using New Relic and Terraform. To read this full New Relic blog, click here.

Observability as code (also known as o11y as code) is the process of automating the configuration of observability tools. You manage your infrastructure with code, so why not manage your observability the same way—and your dashboards as well?

This three-part blog series is your guide to o11y as code, providing tips, examples, and guidance. In this series, we'll walk through examples of how you can automate the configuration of your observability tools, starting with dashboards here in part one. Part one covers the basics of Terraform, how to provision a sample app, and how to create dashboards as code.

By the end of the series, you'll have worked with a total of five examples of observability as code using New Relic and Hashicorp's Terraform: dashboards as code, alerts as code, synthetic monitoring as code, tags as code, and workloads as code. You'll be working with data from the sample FoodMe restaurant ordering app.

How did we get here? Infrastructure as code

Since infrastructure as code (also known as IaC) appeared on the scene more than a decade ago, it’s become a core requirement in the modern cloud era. The terminology “as code” means treating infrastructure configuration just like we treat code, pushing configuration into source control, then carefully pushing out changes again to the infrastructure layer.

With the rise of modern distributed systems, we also see more outages, and finding the root cause of the issue can be challenging when something goes wrong. Observability fits into the new paradigm because we need to determine the internal states of our systems from their outputs. Observability uses different system outputs such as tracing, logs, and metrics to understand the internal state of the distributed components, diagnose where the problems are, and get to the root cause.

Unfortunately, the operational practices we rely on didn’t change much, and developers and operations engineers might find they still look at hundreds of alerts or dashboards. This approach leads to non-repeatable, non-standardized dashboard configurations or adjusting alerts dynamically to avoid signals fatigue and drifting from organizational best practices.

But we can use what we know about infrastructure as code to automate observability. Meet the new approach: observability as code, which treats observability configurations as code. As explained in Observability as code simplifies your life, observability as code represents a shift of intention to an auditable code-managed solution that reduces the work needed to maintain and develop a configuration.

Understand the basics of Terraform

Terraform by Hashicorp is an infrastructure as code tool that you can use to define and manage infrastructure resources in configuration files that are easily readable by humans. You can declaratively manage services and automate your changes to those services.

In most examples, a Terraform module is a set of Terraform configuration files in one directory. When you run Terraform commands directly from that single directory, it is considered the root module. Here's what it looks like, as shown in the Terraform docs:

.
├── LICENSE
├── README.md
├── main.tf
├── variables.tf
├── outputs.tf

Terraform files used in this blog series

The examples in the tutorial exercises in this blog series focus on two important files:

The main.tf file contains the main set of configurations for your module. You can also create other configuration files and organize them in a way that makes sense for your project.
The variables.tf file contains the variable definitions for your module. If you want others to use your module, configure the variables as arguments in the module block.

Example of a New Relic Terraform provider

Here’s an example of a New Relic provider in Terraform from Configuring the New Relic Terraform Provider.

# get the New Relic terraform provider
terraform {
  required_version = "~> 1.0"
  required_providers {
    newrelic = {
      source  = "newrelic/newrelic"
    }
  }
}

# configure the New Relic provider
provider "newrelic" {
  account_id = <Your Account ID>
  api_key = <Your User API Key>    # usually prefixed with 'NRAK'
  region = "US"                    # Valid regions are US and EU
}

You can also use environment variables to configure the provider, which can simplify your provider block. Each provider has key schema attribute, such as account_id, api_key, and region.

Terraform commands to remember

To initialize and run Terraform effectively, remember these four commands:

The terraform init command performs initialization steps to prepare the current working directory for use with Terraform. This command is safe to run multiple times, to update the working directory with configuration changes.
The terraform plan command creates an execution plan, which lets you preview the changes that Terraform will make to your infrastructure. You can use this command to check whether the proposed changes match what you expect before you apply the changes.
The terraform apply command automatically creates an execution plan, prompting you to approve that plan, and then takes the indicated actions. Follow the prompts, and answer yes to apply the changes. Terraform will then provision the resources.
The terraform destroy command is a convenient way to remove all the remote objects managed by a particular Terraform configuration. Follow the prompts, and Terraform will delete all the resources.

For more information on Terraform commands, see Provisioning infrastructure with Terraform.

The examples in the next sections show key concepts in Terraform such as providers, data sources, and resources. You'll be automating configuration of New Relic dashboards to view data from the sample FoodMe restaurant app.

This blog post demo uses the newrelic_one_dashboard resource. As an alternative, if you want to use the newrelic_one_dashboard_json resource, see the Creating dashboards with Terraform and JSON templates tutorial.

Before you begin provisioning your first Terraform module

For this tutorial, we’re going to provision a sample app. But before you provision your first Terraform module, you’ll need to get an account ID, your user key, and point to the correct data center:

This video walkthrough covers prerequisite work.

Automate Configuration with Observability as Code | New Relic

Provision the sample app

Before we work on implementing observability as code, let’s start by provisioning our sample app!

Generate your unique URL for the FoodMe example app with this Glitch link: glitch.com/edit/#!/remix/nr-devrel-o11yascode
Set the environment variables. Go to .env and insert these values:

LICENSE_KEY: Insert your New Relic ingest API keys.
APP_NAME: Insert your name or initials to the name of the app FoodMe-XXX (for example, FoodMe-Jan).

Preview your URI.

Go to Tools (bottom of the panel), and select Preview in a new window.

Record your URL.

Note your newly generated URL. You’ll use this later on in part two of the series for synthetic monitoring as code.

Generate some workloads. Now that you’re in the sample app, enter an example name, delivery address, and select Find Restaurants! After you are on the main page, click around to generate some workloads for the sample app. We'll need some data to look at in the dashboards.

Create dashboards as code

Now we're ready for our first observability as code example: dashboards as code. With New Relic custom dashboards, you can collect and visualize the specific data that you want to see and display in New Relic. You'll learn how to automate configuring dashboards in New Relic using Terraform.

There are three main steps. To see everything we are covering in this section, watch this video. For more details, go to Getting started with New Relic and Terraform. You can also work along with these steps with code samples in GitHub and the hands-on workshop in Instruqt.

Automate Configuration with Observability as Code | New Relic

In Terraform, each resource block describes one or more observability objects, such as dashboards, alerts, notification workflows, or workloads. We'll use examples from Resource: newrelic_one_dashboard:

Create a resource block and declare a type (newrelic_one_dashboard) with a given name (exampledash). The type and the name of the resource are the identifier for the resource, so they must be unique within a module. Here's a simple example for deploying dashboards as code in New Relic, based on Resource: newrelic_one_dashboard.

# New Relic One Dashboard
resource "newrelic_one_dashboard" "exampledash" {
    # The title of the dashboard.
  name = "New Relic Terraform Example"

    # A nested block that describes a page
  page {
        # The name of the page.
    name = "New Relic Terraform Example"

        # A nested block that describes a Billboard widget
    widget_billboard {
      title = "Requests per minute"
      row = 1
      column = 1
      width = 6
      height = 3

            # A nested block that describes a NRQL Query
      nrql_query {
        query = "FROM Transaction SELECT rate(count(*), 1 minute)"
      }
    }
  }
}

For more details on attribute reference, see the attribute reference for the newrelic provider in Terraform.

For more details on New Relic Query Language (NRQL), see syntax, clauses, and functions.

Next, you'll include a variables.tf file in Terraform. You can customize Terraform modules with input variables instead of modifying the source code of the module. Then it's easy to share and reuse modules across other configurations in Terraform. At the end of this section, you'll see an example variables.tf file.
Finally, you'll combine what we covered about the New Relic provider, the resources, the main.tf file, and the corresponding variariables.tf file to deploy dashboards as code.

The next two example main.tf and variariables.tf files use concepts described in Google Site Reliability Engineering, The Four Golden Signals: latency, traffic, errors, and throughput. These examples are based on code samples in the Getting Started with the New Relic Provider documentation.

Example main.tf file complete code

# get the New Relic terraform provider
terraform {
  required_version = "~> 1.0"
  required_providers {
    newrelic = {
      source  = "newrelic/newrelic"
    }
  }
}

# configure the New Relic provider
provider "newrelic" {
  account_id = (var.nr_account_id)
  api_key = (var.nr_api_key)    # usually prefixed with 'NRAK'
  region = (var.nr_region)      # Valid regions are US and EU
}

# resource to create, update, and delete dashboards in New Relic
resource "newrelic_one_dashboard" "dashboard_name" {
  name = "O11y_asCode-FoodMe-Dashboards-TF"

  # determines who can see the dashboard in an account
  permissions = "public_read_only"

  page {
    name = "Dashboards as Code"

    widget_markdown {
      title = "Golden Signals - Latency"
      row = 1
      column = 1
      width = 4
      height = 3

      text = "## The Four Golden Signals - Latency\n---\n#### The time it takes to service a request. It’s important to distinguish between the latency of successful requests and the latency of failed requests. \n\n#### For example, an HTTP 500 error triggered due to loss of connection to a database or other critical backend might be served very quickly; however, as an HTTP 500 error indicates a failed request, factoring 500s into your overall latency might result in misleading calculations. \n\n#### On the other hand, a slow error is even worse than a fast error! Therefore, it’s important to track error latency, as opposed to just filtering out errors."
    }

    widget_line {
      title = "Golden Signals - Latency - FoodMe - Line"
      row = 1
      column = 5
      width = 4
      height = 3

      nrql_query {
        query = "SELECT average(apm.service.overview.web) * 1000 as 'Latency' FROM Metric WHERE appName like '%FoodMe%' since 30 minutes ago TIMESERIES AUTO"
      }
    }

    widget_stacked_bar {
      title = "Golden Signals - Latency - FoodMe - Stacked Bar"
      row = 1
      column = 9
      width = 4
      height = 3

      nrql_query {
        query = "SELECT average(apm.service.overview.web) * 1000 as 'Latency' FROM Metric WHERE appName like '%FoodMe%' since 30 minutes ago TIMESERIES AUTO"
      }
    }

    widget_markdown {
      title = "Golden Signals - Errors"
      row = 4
      column = 1
      width = 4
      height = 3

      text = "## The Four Golden Signals - Errors\n---\n\n#### The rate of requests that fail, either explicitly (e.g., HTTP 500s), implicitly (for example, an HTTP 200 success response, but coupled with the wrong content), or by policy (for example, \"If you committed to one-second response times, any request over one second is an error\").\n \n#### Where protocol response codes are insufficient to express all failure conditions, secondary (internal) protocols may be necessary to track partial failure modes. \n\n#### Monitoring these cases can be drastically different: catching HTTP 500s at your load balancer can do a decent job of catching all completely failed requests, while only end-to-end system tests can detect that you’re serving the wrong content."
    }

    widget_area {
      title = "Golden Signals - Errors - FoodMe - Area"
      row = 4
      column = 5
      width = 4
      height = 3

      nrql_query {
        query = "SELECT (count(apm.service.error.count) / count(apm.service.transaction.duration))*100 as 'Errors' FROM Metric WHERE (appName like '%FoodMe%') AND (transactionType = 'Web') SINCE 30 minutes ago TIMESERIES AUTO"
      }
    }

    widget_billboard {
      title = "Golden Signals - Errors - FoodMe - Billboard Compare With"
      row = 4
      column = 9
      width = 4
      height = 3

      nrql_query {
        query = "SELECT (count(apm.service.error.count) / count(apm.service.transaction.duration))*100 as 'Errors' FROM Metric WHERE (appName like '%FoodMe%') AND (transactionType = 'Web') SINCE 30 minutes ago COMPARE WITH 30 minutes ago"
      }
    }

    widget_markdown {
      title = "Golden Signals - Traffic"
      row = 7
      column = 1
      width = 4
      height = 3

      text = "## The Four Golden Signals - Traffic\n---\n\n#### A measure of how much demand is being placed on your system, measured in a high-level system-specific metric. \n\n#### For a web service, this measurement is usually HTTP requests per second, perhaps broken out by the nature of the requests (e.g., static versus dynamic content). \n\n#### For an audio streaming system, this measurement might focus on network I/O rate or concurrent sessions. \n\n#### For a key-value storage system, this measurement might be transactions and retrievals per second."
    }

    widget_table {
      title = "Golden Signals - Traffic - FoodMe - Table"
      row = 7
      column = 5
      width = 4
      height = 3

      nrql_query {
        query = "SELECT rate(count(apm.service.transaction.duration), 1 minute) as 'Traffic' FROM Metric WHERE (appName LIKE '%FoodMe%') AND (transactionType = 'Web') FACET path SINCE 30 minutes ago"
      }
    }

    widget_pie {
      title = "Golden Signals - Traffic - FoodMe - Pie"
      row = 7
      column = 9
      width = 4
      height = 3

      nrql_query {
        query = "SELECT rate(count(apm.service.transaction.duration), 1 minute) as 'Traffic' FROM Metric WHERE (appName LIKE '%FoodMe%') AND (transactionType = 'Web') FACET path SINCE 30 minutes ago"
      }
    }

    widget_markdown {
      title = "Golden Signals - Saturation"
      row = 10
      column = 1
      width = 4
      height = 3

      text = "## The Four Golden Signals - Saturation\n---\n\n#### How \"full\" your service is. A measure of your system fraction, emphasizing the resources that are most constrained (e.g., in a memory-constrained system, show memory; in an I/O-constrained system, show I/O). Note that many systems degrade in performance before they achieve 100% utilization, so having a utilization target is essential.\n\n#### In complex systems, saturation can be supplemented with higher-level load measurement: can your service properly handle double the traffic, handle only 10% more traffic, or handle even less traffic than it currently receives? For very simple services that have no parameters that alter the complexity of the request (e.g., \"Give me a nonce\" or \"I need a globally unique monotonic integer\") that rarely change configuration, a static value from a load test might be adequate. \n\n#### As discussed in the previous paragraph, however, most services need to use indirect signals like CPU utilization or network bandwidth that have a known upper bound. Latency increases are often a leading indicator of saturation. Measuring your 99th percentile response time over some small window (e.g., one minute) can give a very early signal of saturation.\n\n#### Finally, saturation is also concerned with predictions of impending saturation, such as \"It looks like your database will fill its hard drive in 4 hours.\""
    }

    widget_line {
      title = "Golden Signals - Saturation - CPU & Memory - Multi-Queries"
      row = 10
      column = 5
      width = 4
      height = 3

      nrql_query {
        query = "SELECT rate(sum(apm.service.cpu.usertime.utilization), 1 second) * 100 as 'cpuUsed' FROM Metric WHERE appName LIKE '%FoodMe%' SINCE 30 minutes ago TIMESERIES AUTO"
      }

      nrql_query {
        query = "SELECT average(apm.service.memory.physical) * rate(count(apm.service.instance.count), 1 minute) / 1000 as 'memoryUsed %' FROM Metric WHERE appName LIKE '%FoodMe%' SINCE 30 minutes ago TIMESERIES AUTO"
      }
    }

    widget_line {
      title = "Golden Signals - Saturation - Memory - Line Compare With"
      row = 10
      column = 9
      width = 4
      height = 3

      nrql_query {
        query = "SELECT average(apm.service.memory.physical) * rate(count(apm.service.instance.count), 1 minute) / 1000 as 'memoryUsed %' FROM Metric WHERE appName LIKE '%FoodMe%' SINCE 30 minutes ago COMPARE WITH 20 minutes ago TIMESERIES AUTO"
      }
    }
  }
}

Example variables.tf file complete code

# your unique New Relic account ID
variable "nr_account_id" {
  default = "XXXXX"
}
# your User API key
variable "nr_api_key" {
  default = "XXXXX"
}

# valid regions are US and EU
variable "nr_region" {
  default = "US"
}

What the final result looks like

Now that you have deployed dashboards as code, your final result should look like this in New Relic:

Follow along with dashboards as code examples using New Relic and Terraform. To read this full New Relic blog, click here.

Top 10 tips for instrumenting Golang with New Relic, part 5

ntcsteve — Sat, 11 Nov 2023 21:17:37 +0000

To read this full New Relic blog, click here.

Are you building modern apps in Golang? This multi-part blog series will help you instrument Golang faster by providing tips, guidance, and best practices with New Relic.

Typically, the content in this blog would be presented in a workshop, but we've made it a blog series so anyone can follow along "hands on," using sample code, Instruqt, this blog, and videos. For the best experience:

See and run the full code that I'll be walking through in this multi-part series.

Review previous guides - Part 1, Part 2, Part 3 and Part 4.

Go Instrumentation Checklist

In Part 1, Part 2, Part 3 and Part 4 we provided extensive coverage and tips for better instrumentation of Golang in New Relic. The Golang agent differs from others in that it does not offer out-of-the-box auto instrumentation. Because Golang is a compiled language, it does not allow for dynamic runtime or method replacement like dynamic languages such as Java, .NET, Node.js, and PHP.

While Golang has unique language advantages, code modification is required to monitor Go applications. On the positive side, you have complete control over what to instrument and can highly customize it to your needs. In the last part of this series, we will summarize everything we have covered so far with a production checklist to use before onboarding your Go apps into production.

1. Using appropriate API key

License keys are used to report almost all data, with the exception of browser and mobile monitoring data, which have their own keys. Each key is unique to an account, allowing for tracking and monitoring of multiple sources of data. This flexibility ensures comprehensive and accurate reporting. Please ensure you have the right license type before starting your Golang instrumentation.

2. Leverage best practices

As we've previously discussed, when working with compiled languages such as Go, manual instrumentation is essential. It's not only important to have a solid grasp of how to use the platform itself, but also to understand how to effectively utilize New Relic within your specific environment. By doing so, you can maximize the benefits of New Relic and ensure optimal performance of your Go applications. Here is the best practice guide from New Relic, including the basics such as tags.

3. Time to run the app

If you are instrumenting a small sample or a short-lived application in your development environment, it is important to include the following code segment. This is because the harvest cycle in the New Relic Go agent is set to 5 seconds, which means that data is collected and sent to New Relic every 5 seconds. By including this code segment, you ensure that the data from your app is accurately captured and monitored during development.

However, for production deployment, you can safely omit this code segment if your application runs for longer than 5 seconds. In production, the New Relic Go agent automatically adjusts the harvest cycle based on the needs and duration of your application. This ensures that your app's performance is effectively monitored without the need for manual intervention.

4. Distributed Tracing

Starting from version 3.16.0, the Go agent has introduced a significant enhancement - the automatic enabling of Distributed Tracing. This means that the agent now proactively adds distributed tracing headers to outbound requests and intelligently scans incoming requests for distributed tracing headers. By default, this feature is enabled to provide seamless tracing capabilities.

Distributed Tracing is a powerful technique that allows you to trace the path of a single request as it traverses through various services and components of your application. It provides invaluable insights into the performance and behavior of your distributed systems, helping you identify bottlenecks, latency issues, and dependencies.

With the Go agent's automatic distributed tracing, you no longer need to manually configure tracing headers or worry about missing out on important tracing data. The agent takes care of all the necessary tracing operations behind the scenes, ensuring that your distributed systems are effectively monitored and traced.

However, if for any reason you want to disable distributed tracing, the Go agent provides a simple configuration option. By setting the value to false, you can turn off distributed tracing and revert to the default behavior. This gives you the flexibility to control the tracing capabilities of your application according to your specific requirements.

5. Capture contextual data

When utilizing New Relic, it can be highly beneficial to leverage custom attributes in order to enhance your events with additional metadata. By using the AddAttribute() code, you can easily incorporate your own contextual data into your transactions, allowing for a more comprehensive and meaningful understanding of your application's performance.

Custom attributes provide an opportunity to decorate your transactions with relevant information that goes beyond the default metrics and data captured by New Relic. This can include details specific to your application, such as user-specific information, product details, pricing data, or any other relevant metadata. By customizing your events with these attributes, you gain deeper insights into the factors that impact your application's performance and user experience.

With the ability to capture and analyze this additional contextual data, you can better understand the root causes of any performance issues or anomalies. This, in turn, enables you to make data-driven decisions and optimizations to improve the overall performance and user satisfaction of your application.

txn.AddAttribute("product", "widget")
txn.AddAttribute("price", 19.99)
txn.AddAttribute("importantCustomer", true)

6. Instrumentation at scale

To improve the instrumentation at scale throughout your environment, it is recommended to examine the Gin Framework as a starting example. By following this approach, you can effectively instrument your middleware layer and gain valuable insights into its performance. Here is an example of how you can implement it:

router := gin.Default()
// Add the nrgin middleware before other middlewares or routes:
router.Use(nrgin.Middleware(app))

This package provides a Gin middleware that can be used to instrument all requests. To use it, call nrgin.Middleware to get a gin.HandlerFunc. Then, add the handler to your application as a middleware to instrument inbound requests handled by a gin.Engine. Implementing this approach allows you to gain deep insights into the behavior and performance of your middleware layer, facilitating effective troubleshooting and optimization. It enhances your ability to identify bottlenecks, track custom metrics, and make data-driven decisions to improve the overall performance of your application.

7. Production rollout

Once you have determined how to incorporate these instrumentations at scale across your environment, it is essential to have a discussion with your team or engineering lead regarding the optimal approach for deploying them through your deployment pipeline. This step is crucial as it allows you to gather input and insights from your colleagues, ensuring that the deployment process is efficient, effective, and aligned with your team's goals and strategies.

Engaging in this discussion allows you to leverage the collective expertise and experience of your team members, enabling the identification of any potential challenges or considerations that need to be addressed. This collaborative approach fosters a sense of ownership and accountability among team members, as everyone has the opportunity to contribute their ideas and suggestions. Additionally, involving your engineering lead in this discussion has the added benefit of gaining their guidance and leadership. Their experience and knowledge can offer valuable insights into best practices and industry standards, enabling you to make informed decisions and avoid potential pitfalls. Here are the common operations you need to perform on the Go agent.

To install the Go agent
To configure the Go agent
To update the Go agent
To uninstall the Go agent

8. Apdex for Production

In this informative video, you will learn how to accurately calculate Apdex and identify the ideal threshold for your applications. The video provides a comprehensive guide that takes you through the process step by step, ensuring that you have a clear understanding of how to determine the correct threshold for your specific use case. Additionally, the video offers a simple and effective NRQL query that allows you to easily find the best thresholds for your app. By following the instructions provided in this video, you will be able to optimize the performance of your applications and ensure an exceptional user experience.

9. Alerts and Dashboards

To improve visibility and control over your application's performance, you can utilize the powerful features of New Relic through configuring alerts and dashboards. By setting up alerts, you will receive timely notifications whenever there are fluctuations in key performance metrics across your entire stack. These alerts can be tailored to meet your specific needs, allowing you to monitor and track important aspects of your application.

In addition, New Relic's dashboards provide a comprehensive overview of your application's performance data, enabling you to visualize and analyze the most relevant metrics. With the ability to gather and chart specific data from different sources, the dashboards offer a flexible and customizable experience, allowing you to monitor your application's performance from anywhere within the New Relic platform.

By utilizing both alerts and dashboards, you can proactively monitor your application's performance, identify potential issues or bottlenecks, and take appropriate actions to ensure optimal performance and user experience. Whether it involves detecting sudden increases in response time, monitoring server health and resource utilization, or tracking the success rate of specific transactions, New Relic's alerts and dashboards provide you with the necessary tools and insights to effectively manage your application's performance.

Introduction to Alerts
Introduction to Dashboards

10. Analyze Deployments

New Relic Change Tracking allows you to capture and record deployments and other changes in your system, providing valuable context to performance data. This feature enables you to easily view and analyze the impact of changes on system performance and quality. The change analysis UIs provide detailed insights, including faceted errors, log attribute trends, related issues and anomalies, and calculated impacts on key Golang signals. By utilizing change tracking metadata, such as commit SHAs, changelogs, and deep links, you can quickly identify and resolve issues. This powerful capability enhances your understanding of how changes affect your system and helps optimize development velocity.

There are several options available for you to begin recording changes (including deployments) to different parts of your system. These options provide both automated and manual ways to track changes effectively. Here are the available methods:

NerdGraph (GraphQL) API: By utilizing the NerdGraph API, you can take advantage of GraphQL mutations to designate the specific changes you want to track. This powerful API allows you to run queries and retrieve detailed information about the changes you are monitoring.
New Relic CLI: Our command line interface, known as the New Relic CLI, offers a convenient way to designate and track changes. With just a few commands, you can easily specify the changes you want to monitor within your system.
Jenkins: If you are using Jenkins as your CI/CD tool, we have developed a dedicated plugin that simplifies the process of tracking changes. By integrating this plugin into your Jenkins pipeline, you can effortlessly monitor and capture changes made to your system.
GitHub Actions: For those utilizing GitHub Actions in their CI/CD workflow, we provide comprehensive guidance on how to configure and leverage GitHub Actions for change tracking. This integration allows you to seamlessly track and manage changes directly within your CI/CD pipeline.

These flexible and versatile options empower you to choose the most suitable method for your specific requirements. Whether you prefer using APIs, command line interfaces, or integrating with popular CI/CD tools like Jenkins or GitHub Actions, New Relic has you covered when it comes to effectively tracking changes in your system.

To read this full New Relic blog, click here.

Top 10 tips for instrumenting Golang with New Relic, part 4

ntcsteve — Sat, 11 Nov 2023 20:38:47 +0000

To read this full New Relic blog, click here.

Are you building modern apps in Golang? This multi-part blog series will help you instrument Golang faster by providing tips, guidance, and best practices with New Relic.

See and run the full code that I'll be walking through in this multi-part series.
Follow the hands-on lab in Instruqt.
Read along with this blog and follow the videos below.

The following two videos will give an overview of Instruqt and some instructions about how to use it if you haven't done that before.

Overview of Instruqt

Top 10 tips for instrumenting Golang with New Relic, part 4 | New Relic

Instruqt UI navigation

Top 10 tips for instrumenting Golang with New Relic, part 4 | New Relic

Now that you're set up, let's learn a little more about the instrumentation of Golang.

Tip 9: Tracing is your Friend

Top 10 tips for instrumenting Golang with New Relic, part 4 | New Relic

The Go programming language is increasingly popular for building distributed systems and microservices. However, this approach presents some challenges, as a single problematic service along the path can negatively impact the overall response time for that request, potentially driving customers away or creating issues.

Hence, tracing has emerged as a valuable technique to address this issue. It collects data as requests travel from one service to another by recording each segment of the journey as a span. These spans contain essential details about each request segment, such as the time it took to process and the HTTP status code returned. Eventually, these spans are combined into one trace, which gives you a complete picture of the entire request. Distributed Tracing is a specialized form designed to work in complex architectures using monoliths and microservices. It benefits software teams working in modern environments, where complex architectures are becoming increasingly common.

Distributed Tracing in New Relic is designed to give these software teams an easy way to capture, visualize, and analyze traces, enabling them to understand better how their applications are performing and quickly identify any issues that might arise. Distributed Tracing is enabled by default in Go agent versions 3.16.0 and higher. This means that the agent will automatically add distributed tracing headers to outbound requests and scan incoming requests for distributed tracing headers.

In some cases, auto-instrumentation may not be available for distributed tracing, but manual instrumentation is still possible. One simple way to create an external segment for an outbound HTTP request is to use the newrelic.NewRoundTripper method.

Here is an example of making a request to http://api.example.com that includes the distributed tracing headers as described in the documentation.

func useNewRoundTripper(txn *newrelic.Transaction) (*http.Response, error) {
    client := &http.Client{}
    client.Transport = newrelic.NewRoundTripper(client.Transport)
    request, _ := http.NewRequest("GET", "http://example.com", nil)
    request = newrelic.RequestWithTransactionContext(request, txn)
    return client.Do(request)
}

This is an example where we can see Distributed Tracing with gRPC in Golang.

Infinite Tracing is a great option to consider if you have a large amount of data to analyze and want to find the most relevant traces. With this feature, you can send your trace data to our cloud-based service and rely on Infinite Tracing to make sampling decisions. This ensures you're keeping the trace data you need to see.

Unlike our standard distributed tracing options, Infinite Tracing can process more trace data. It uses tail-based sampling, a superior method of sampling data after it's collected. Unlike head-based sampling, which our standard tracing feature uses, tail-based sampling reduces the chance of missing important data.

Infinite Tracing is highly configurable, and you can set it up in various ways to ensure that it keeps the trace data you need to see. This way, you can be sure you're getting the most relevant traces for analysis. So, if you're dealing with data and want to ensure you get all vital information, Infinite Tracing is an excellent solution. You explore further with the introduction and setting up Infinite Tracing for more details.

Tip 10: Tracking Performance Changes

Deploying an application can be risky, especially when the app breaks or doesn't perform as expected. Many factors can contribute to a bad deployment, including code changes, environment configuration, and infrastructure issues. This is why it's crucial to have visibility into your app's performance before, during, and after deployment. New Relic Change Tracking allow you to track deployments and correlate them with changes in your app's performance. Using these markers, you can monitor how your app responds to changes made during deployment and detect any issues that may arise in Golang.

Markers are created automatically by New Relic whenever you deploy a new app version. These markers appear in APM charts and dashboards, allowing you to quickly see the impact of deployments on your app's performance. You can also view change tracking in context with your app's logs, metrics, and traces, making it easier to understand the root cause of any issues.

By using markers in New Relic, you can reduce the risk of bad deployments and ensure that your app always performs at its best. With better visibility into your app's performance, you can make more informed decisions about when and how to deploy new changes. To document a change, such as a deployment, it is necessary to include the mandatory fields, and then decide whether to include any optional fields related to Golang. The following GraphQL mutations provide examples of different ways to create deployment markers.

mutation {
  changeTrackingCreateDeployment(
    deployment: { version: "0.0.1", entityGuid: "INSERT_YOUR_GUID" }
  ) {
    deploymentId
    entityGuid
  }
}

Once you've completed the steps for recording a deployment, you are ready to see its performance in the UI. For more information, you can explore track changes using NerdGraph (GraphQL).

Bonus Tip 11: Logs in Context

Top 10 tips for instrumenting Golang with New Relic, part 4 | New Relic

Software developers rely on log data to troubleshoot issues, making it an invaluable source of information. However, in today's world, developers face a new challenge of dealing with an overwhelming amount of log data from various sources. As the volume of log data increases, it becomes more challenging to collect, manage, and analyze this data.

New Relic logs in context is designed to address this challenge. It correlates your log data with data collected from our APM and infrastructure agents and services instrumented with OpenTelemetry. This means you can see log data or links to log data in other UI experiences, such as APM, Distributed Tracing, and Errors Inbox.

With the Go language APM agent, you can also get logs in context and see your app logs in the context of your other New Relic data. There are two ways to get logs into New Relic.

Option 1: using native APM agent to collect logs

For the majority of users, the best solution for log forwarding is the automatic feature built into the Go agent. It is simple to set up and works well for most use cases. This feature is now turned on by default in versions v3.20.0+ of the Go agent, but it is still possible to modify its configuration to enable application log forwarding.

app, err := newrelic.NewApplication(
  newrelic.ConfigAppLogForwardingEnabled(true),
)

After configuring your agent to send logs to New Relic, you can instrument your logging library using the Logs in Context plugin. This can be done by using supported plugins such as Zerolog, Logrus, or the Standard Library Log from the Golang ecosystem.

    // Create a logWriter, then pass it to the log.Logger
    writer := logWriter.New(os.Stdout, app)
    logger := log.New(&writer, "Background:  ", log.Default().Flags())

    logger.Print("Hello world!")

    txnName := "logsInContext Sample Transaction"
    txn := app.StartTransaction("logsInContext")

    // Always create a new log object in order to avoid changing the context of the logger for
    // other threads that may be logging outside of this transaction
    txnLogger := log.New(writer.WithTransaction(txn), "Transaction: ", log.Default().Flags())
    txnLogger.Printf("In transaction %s.", txnName)

    // Random sleep to simulate delays
    randomDelay := rand.Intn(300)
    time.Sleep(time.Duration(randomDelay) * time.Millisecond)

    txnLogger.Printf("Ending transaction %s.", txnName)
    txn.End()

    logger.Print("Goodbye!")

Option 2: using a log forwarder to collect logs

Our infrastructure monitoring agent enables you to forward your logs to New Relic, which consolidates all of your logging data in one location. This provides you with deeper visibility into both your application and platform performance.

By forwarding your logs to New Relic, you can enhance your log management capabilities by collecting, processing, exploring, querying, and setting alerts on your log data from the client side.

To forward your logs through the New Relic infrastructure monitoring agent, follow these steps:

Create a New Relic account if you haven't already. It's free, forever.
Check the system requirements to ensure your configuration is correct.
Install the infrastructure agent (version 1.11.4 or higher).
Create a logging.yml configuration file in the infrastructure agent's logging.d directory.
Configure your log sources and other parameters.
Generate some traffic and wait a few minutes, then check your account for data.
Explore your log data in the Logs UI and benefit from the log attributes automatically inserted by the infrastructure agent.

For more information, don't forget to visit the official page for the New Relic Go agent.

To read this full New Relic blog, click here.

Top 10 tips for instrumenting Golang with New Relic, part 3

ntcsteve — Wed, 01 Nov 2023 17:48:30 +0000

To read this full New Relic blog, click here.

Are you building modern apps in Golang? This multi-part blog series will help you instrument Golang faster by providing tips, guidance, and best practices with New Relic.

See and run the full code that I'll be walking through in this multi-part series.
Follow the hands-on lab in Instruqt.
Read along with this blog and follow the videos below.

The following two videos will give an overview of Instruqt and some instructions about how to use it if you haven't done that before.

Overview of Instruqt

Top 10 tips for instrumenting Golang with New Relic, part 3 | New Relic

Instruqt UI navigation

Top 10 tips for instrumenting Golang with New Relic, part 3 | New Relic

Now that you're set up, let's learn a little more about the instrumentation of Golang.

Tip 7: It MUST be the database layer!

Top 10 tips for instrumenting Golang with New Relic, part 3 | New Relic

At some point during software development, it's not uncommon to experience issues that may seem to stem from the database layer. However, blaming the database layer is not always the best approach. It's possible to instrument and provide more context for database requests in Go, thanks to the use of segments.

By using segments, you can instrument your Go application's datastore calls. These datastore segments will appear in the APM Transactions breakdown table and Databases tab of the Transactions page in New Relic, providing a comprehensive overview of your database performance.

If you're using a MySQL, PostgreSQL, or SQLite database driver, using a pre-built integration package is the easiest way to add datastore segments. This also applies to MongoDB, so you can easily instrument your database calls without writing any custom code. For MySQL Database, here is the package driver, as seen in our docs. You can go to this link for a full code sample with MySQL.

import (
    _ "github.com/newrelic/go-agent/v3/integrations/nrmysql"
)

func main() {
    db, err := sql.Open("nrmysql", "user@unix(/path/to/socket)/dbname")
}

To provide a context containing a newrelic.Transaction to all Exec and Query methods on sql.DB, sql.Conn, sql.Tx, and sql.Stmt, you need to use the context methods ExecContext, QueryContext, and QueryRowContext in place of Exec, Query, and QueryRow, respectively. This will ensure proper database transaction monitoring with New Relic.

txn := app.StartTransaction("mysqlQuery")
ctx := newrelic.NewContext(context.Background(), txn)
row := db.QueryRowContext(ctx, "SELECT count(*) from tables")

If you're using a MongoDB database, you can simplify the process of creating Datastore Segments by using this package to instrument your MongoDB calls. To do this, simply set the monitor in the connect options using SetMonitor, as outlined in our documentation.

import "github.com/newrelic/go-agent/v3/integrations/nrmongo"

nrMon := nrmongo.NewCommandMonitor(nil)
client, err := mongo.Connect(ctx, options.Client().SetMonitor(nrMon))

Note that the nrmongo monitor must be set last to prevent it from being overwritten. If you need to use more than one event.CommandMonitor, you can pass the original monitor to the nrmongo.NewCommandMonitor function, as demonstrated in the documentation.

origMon := &event.CommandMonitor{
    Started:   origStarted,
    Succeeded: origSucceeded,
    Failed:    origFailed,
}
nrMon := nrmongo.NewCommandMonitor(origMon)
client, err := mongo.Connect(ctx, options.Client().SetMonitor(nrMon))

Then add the current transaction to the context used in any MongoDB call.

ctx = newrelic.NewContext(context.Background(), txn)
resp, err := collection.InsertOne(ctx, bson.M{"name": "pi", "value": 3.14159})

Alternatively, you can manually create Datastore segments for each database call. Like basic segments, datastore segments start when the StartTime field is populated and end when the End method is called.

To monitor a datastore segment, you should add the following code snippet to the beginning of the function you want to monitor, as shown in the linked example.

s := newrelic.DatastoreSegment{
    StartTime: txn.StartSegmentNow(),
    Product: newrelic.DatastoreMySQL,
    Collection: "users",
    Operation: "INSERT",
    ParameterizedQuery: "INSERT INTO users (name, age) VALUES ($1, $2)",
    QueryParameters: map[string]interface{}{
        "name": "Dracula",
        "age": 439,
    },
    Host: "mysql-server-1",
    PortPathOrID: "3306",
    DatabaseName: "my_database",
}
defer s.End()

Here is the full sample code of Golang instrumenting a Database function,

func addAlbum(alb Album) (int64, error) {
    result, err := db.Exec("INSERT INTO album (title, artist, price) VALUES (?, ?, ?)", alb.Title, alb.Artist, alb.Price)

    // Provide a context containing a newrelic. Transaction to all exec
    // and query methods on sql.DB, sql.Conn, sql.Tx, and sql.Stmt.
    txn := nrApp.StartTransaction("mysqlQuery")
    ctx := newrelic.NewContext(context.Background(), txn)
    row := db.QueryRowContext(ctx, "INSERT INTO album (title, artist, price) VALUES (?, ?, ?)")
    row.Scan()

    // Random sleep to simulate delays
    randomDelay := rand.Intn(200)
    time.Sleep(time.Duration(randomDelay) * time.Millisecond)

    // End the DB transaction.
    txn.End()

    if err != nil {
        return 0, fmt.Errorf("addAlbum: %v", err)
    }
    id, err := result.LastInsertId()
    if err != nil {
        return 0, fmt.Errorf("addAlbum: %v", err)
    }
    return id, nil
}

Tip 8: What about Goroutines?

Top 10 tips for instrumenting Golang with New Relic, part 3 | New Relic

Goroutines are a key feature in Go programming language that allows functions to run concurrently with other functions or methods. Goroutines are a lightweight alternative to traditional threads, making them an attractive option for concurrent programming in Go.

However, to effectively monitor and instrument a goroutine, you need to use the Transaction.NewGoroutine() transaction method. This method allows you to create a new reference to the transaction, which must be called any time you pass the transaction to another goroutine that makes segments.

To gain valuable insights into how your application is performing, it's important to track the performance of your concurrent code by instrumenting your goroutines. This is especially important for larger applications where concurrency can cause performance bottlenecks and other issues. You can see more details here on instrumenting Goroutines.

go func(txn *newrelic.Transaction) {
    defer txn.StartSegment("goroutines").End()
    time.Sleep(100 * time.Millisecond)
}(txn.NewGoroutine())

One more important suggestion for instrumenting your Golang code is to use a WaitGroup, which is a valuable tool for managing concurrency. The WaitGroup waits for a collection of goroutines to finish before proceeding. The main goroutine calls Add to set the number of goroutines to wait for. Then, each of the goroutines runs and calls Done when finished. Finally, Wait can be used to block until all goroutines have finished. By using a WaitGroup, you can ensure that all of your goroutines have completed their tasks before moving on.

Here is an example where WaitGroup is used to wait for all the goroutines launched here to finish. Note that if a WaitGroup is explicitly passed into functions, it should be done by a pointer.

package main

import (
    "fmt"
    "sync"
    "time"
)

func worker(id int) {
    fmt.Printf("Worker %d starting\n", id)
    time.Sleep(time.Second)
    fmt.Printf("Worker %d done\n", id)
}

func main() {
    var wg sync.WaitGroup
    for i := 1; i <= 5; i++ {
        wg.Add(1)
        i := i

                // Goroutines
        go func() {
            defer wg.Done()
            worker(i)
        }()
    }
    wg.Wait()
}

Here is an example on how to instrument Goroutines with New Relic,

func async(w http.ResponseWriter, r *http.Request) {
    // To access the transaction in your handler, use the newrelic.FromContext API.
    txn := newrelic.FromContext(r.Context())

    // This WaitGroup is used to wait for all the goroutines to finish.
    wg := &sync.WaitGroup{}
    println("goRoutines created!")

    for i := 1; i < 9; i++ {
        wg.Add(1)
        i := i

        // The Transaction.NewGoroutine() allows transactions to create segments 
        // in multiple goroutines.
            defer wg.Done()
            defer txn.StartSegment("goroutine" + strconv.Itoa(i)).End()
            println("goRoutine " + strconv.Itoa(i))

            randomDelay := rand.Intn(500)
            time.Sleep(time.Duration(randomDelay) * time.Millisecond)
        }(txn.NewGoroutine())
    }

    // Ensure the WaitGroup is done
    segment := txn.StartSegment("WaitGroup")
    wg.Wait()
    segment.End()
    w.Write([]byte("goRoutines success!"))
}

For more information, don't forget to visit the official page for the New Relic Go agent.

To read this full New Relic blog, click here.

Top 10 tips for instrumenting Golang with New Relic, part 2

ntcsteve — Tue, 31 Oct 2023 18:49:19 +0000

To read this full New Relic blog, click here.

Are you building modern apps in Golang? This multi-part blog series will help you instrument Golang faster by providing tips, guidance, and best practices with New Relic.

See and run the full code that I'll be walking through in this multi-part series.
Follow the hands-on lab in Instruqt.
Read along with this blog and follow the videos below.

The following two videos will give an overview of Instruqt and some instructions about how to use it if you haven't done that before.

Overview of Instruqt

Top 10 tips for instrumenting Golang with New Relic, part 2 | New Relic

Instruqt UI navigation

Top 10 tips for instrumenting Golang with New Relic, part 2 | New Relic

Now that you're set up, let's learn a little more about the instrumentation of Golang.

Tip 4: Transactions and segments

Top 10 tips for instrumenting Golang with New Relic, part 2 | New Relic

The New Relic Go instrumentation revolves around two fundamental concepts: Transactions and segments. These concepts are essential to understanding Go integrations and examples.

Transactions are a crucial aspect of Go instrumentation. It refers to a logical unit of work in a software application comprising function calls and method calls executed during a specific time frame. A transaction usually represents a key transaction encompassing the activity from when the application receives a request to when it sends the response.

Transactions are crucial because they help developers identify the slowest parts of the code. By tracking these transactions, developers can identify bottlenecks and optimize the performance of the application. This information can then be used to optimize the user experience and improve customer satisfaction.

Here's an example of monitoring a web transaction as seen in our docs:

// Begin a new transaction with a specified name and defer its ending.
txn := app.StartTransaction("transaction_name")
defer txn.End()

// req variable of type *http.Request to mark the transaction as a web transaction.
txn.SetWebRequestHTTP(req)

// writer is a http.ResponseWriter, use the returned writer in place of the original.
writer = txn.SetWebResponse(writer)
writer.WriteHeader(500)

In software applications, a transaction may consist of multiple segments representing a specific part of the process. For instance, a transaction in an ecommerce platform may involve adding items to a shopping cart, processing payment, and updating inventory levels.

To gain insights into these transactions' performance, measuring the time taken by each segment is essential. By instrumenting segments, you can track the time taken by functions and code blocks that make up the segments, such as external calls, datastore calls, adding messages to queues, and background tasks. This information is powerful for gaining insights into the time taken by specific functions and code blocks.

Here's an example of a segment as seen in our docs:

segment := newrelic.Segment{}
segment.Name = "yourSegmentName"
segment.StartTime = txn.StartSegmentNow()
// insert your code logic here
segment.End()

To effectively instrument Go, it is important to understand the three types of segments. Choosing the appropriate segment is essential to ensure that they appear correctly in the UI. Properly ending each transaction or segment is mandatory, as failure to do so will result in the instrumented function not appearing in New Relic.

Function segments: Instrumenting a function as a segment is equivalent to instrumenting any other block of code as a segment. This means that the process of measuring the execution time of a function or any arbitrary code block is essentially the same.

Here's an example of a function segment as seen in our docs:

segment := txn.StartSegment("yourSegmentName")
// insert your code logic here
segment.End()

Datastore Segments: If you want to improve the performance of your datastore calls, you can use a datastore segment to instrument them. By doing so, you'll be able to gather more detailed insights into how your datastore is performing and identify any areas that may be causing issues. A datastore segment allows you to see the resulting datastore segments in the transactions breakdown table and Databases tab of the transactions page within New Relic. This gives you access to more granular data about the performance of your datastore, which can help you make more informed decisions about optimizing it for your specific needs.

Here's an example of a datastore segment as seen in our docs:

s := newrelic.DatastoreSegment{
    Product: newrelic.DatastoreMySQL,
    Collection: "users",
    Operation: "INSERT",
    ParameterizedQuery: "INSERT INTO users (name, age) VALUES ($1, $2)",
    QueryParameters: map[string]interface{}{
        "name": "Dracula",
        "age": 439,
    },
    Host: "mysql-server-1",
    PortPathOrID: "3306",
    DatabaseName: "my_database",
}
s.StartTime = txn.StartSegmentNow()
// insert your code logic here
s.End()

External segments: For your Go application's external service calls, such as web services, cloud resources, and other network requests, you can instrument them with external segments. This allows the resulting external segments to be displayed in the transactions breakdown table and external services page within New Relic.

Here's an example of a external segment as seen in our docs:

func external(txn *newrelic.Transaction, req *http.Request) (*http.Response, error) {
    s := newrelic.StartExternalSegment(txn, req)
    response, err := http.DefaultClient.Do(req)
    s.Response = response
    s.End()
    return response, err
}

In summary, here's a sample code of how everything fits together:

func randomFormat() string {

    //Monitor a transaction
    nrTxnTracer := nrApp.StartTransaction("randomFormat")
    defer nrTxnTracer.End()

    // Random sleep to simulate delays
    randomDelayOuter := rand.Intn(40)
    time.Sleep(time.Duration(randomDelayOuter) * time.Microsecond)

    // Create a segment
    nrSegment := nrTxnTracer.StartSegment("Formats")

    // Random sleep to simulate delays
    randomDelayInner := rand.Intn(80)
    time.Sleep(time.Duration(randomDelayInner) * time.Microsecond)

    // A slice of message formats.
    formats := []string{
        "Hi, %v. Welcome!",
        "Great to see you, %v!",
        "Good day, %v! Well met!",
        "%v! Hi there!",
        "Greetings %v!",
        "Hello there, %v!",
    }

    // End a segment
    nrSegment.End()

    // Return a randomly selected message format by specifying
    // a random index for the slice of formats.
    return formats[rand.Intn(len(formats))]
}

See full sample code on transactions and segments.

Tip 5: Custom attributes

Utilizing custom attributes can be highly beneficial when using a New Relic such as APM, browser, mobile, infrastructure, and synthetics. This enhancement allows for the addition of personalized metadata to existing events. Custom attributes provide essential business and operational context to existing events, consisting of key-value pairs that offer metadata related to the associated events. This provides the ability to improve custom charts and queries to analyze the data, as discussed in Tip 1: Learn why you should instrument.

Various types of business context can be added as custom attributes, including customer ID, customer market segment, customer value classification, or transaction identifiers. Operational context can also be added as custom attributes, including feature flags that were utilized, datastore used, infrastructure accessed, or errors detected and ignored.

Here's an example of AddAttribute() to add metadata to your transactions:

txn.AddAttribute("product", "shoes")
txn.AddAttribute("price", 69.90)
txn.AddAttribute("onSale", true)

See additional details using the Go agent API, including AddAttribute().

Here's an example of capturing a custom attribute:

func Hello(name string) (string, error) {

    // Monitor a transaction
    nrTxnTracer := nrApp.StartTransaction("Hello")
    defer nrTxnTracer.End()

    // If no name was given, return an error with a message.
    if name == "" {
        return name, errors.New("empty name")
    }

    // Create a message using a random format.
    message := fmt.Sprintf(randomFormat(), name)

    // Custom attributes by using this method in a transaction
    nrTxnTracer.AddAttribute("message", message)
    return message, nil
}

See full sample code on transactions and segments, including custom attributes.

Tip 6: Modularize your instrumentation

Top 10 tips for instrumenting Golang with New Relic, part 2 | New Relic

At some point, you will need to attach instrumentation at scale throughout your environment. This may seem daunting, but it' a critical step in ensuring your application performs optimally. Consider discussing with your team or engineering lead to determine the best strategic location for instrumenting all of your request calls.

This could include examining the architecture of your application to identify potential chokepoints and considering the types of metrics you want to collect. You may also want to consider any potential tradeoffs between the depth and breadth of instrumentation. Ultimately, the key is to approach this process thoughtfully and systematically, ensuring that your instrumentation strategy aligns with your broader goals and objectives.

We recommend instrumenting the middleware layer with a wrapper, as demonstrated in this example using the Gin framework. This package, nrgin, instruments https://github.com/gin-gonic/gin applications.

router := gin.Default()
// add the nrgin middleware as the first middleware or route in your application.
router.Use(nrgin.Middleware(app))

The middleware creates a Gin middleware that instrument all requests. Use this package to instrument inbound requests handled by a gin.Engine. Call nrgin.Middleware to get a gin.HandlerFunc, which can be added to your application as a middleware.

router := gin.Default()

// Package nrgin instruments https://github.com/gin-gonic/gin applications.
router.Use(nrgin.Middleware(nrApp))

router.GET("/games", getgames)
router.GET("/games/:id", getgameByID)
router.POST("/games", postgames)

Note these two pieces of helpful documentation:

Instrumenting Gin

See full code sample with Gin

More integration examples to explore with New Relic

For more information, don't forget to visit the official page for the New Relic Go agent.

To read this full New Relic blog, click here.

Top 10 tips for instrumenting Golang with New Relic, part 1

ntcsteve — Tue, 31 Oct 2023 17:47:40 +0000

To read this full New Relic blog, click here.

Are you building modern apps in Golang? This multi-part blog series will help you instrument Golang faster by providing tips, guidance, and best practices with New Relic.

Typically, the content in this blog would be presented in a workshop, but we've made it a blog series so anyone can follow along "hands-on," using sample code, Instruqt, this blog, and videos. For the best experience:

See and run the full code that I'll be walking through in this multi-part series

Follow the hands-on lab in Instruqt

Read along with this blog and follow the videos below

The following two videos will give an overview of Instruqt and some instructions about how to use it if you haven't done that before.

Overview of Instruqt

Top Tips for Instrumenting Golang | New Relic

Instruqt UI navigation

Top Tips for Instrumenting Golang | New Relic

Now that you're set up, let's learn a little more about instrumentation of Golang.

An introduction to Golang instrumentation

Golang is increasingly becoming popular and loved by many developers when building modern cloud-native services. Golang, also known as Go, is a compiled, high-performance language meant to be simple, easy to learn, and effective to use. Since its inception back in 2009, Go has continued to grow significantly, as seen by the latest Stack Overflow 2022 survey.

While developers continue to build more services in Go, finding effective ways to instrument better for Go can be confusing and frustrating. Interpreter languages like PHP, Ruby, Python, and JavaScript, don’t require a pre-runtime translation of code, meaning you can automatically instrument your applications. But, Go, like C and Rust, is a compiled language, meaning it relies on translators that generate machine code from source code. So you can't use a dynamic and automatic instrumentation technique like bytecode injection.

Manual instrumentation is required to get better visibility to find the root cause of an issue.

Tip 1: Learn why you should instrument

The answer is simple: you’ll want to quickly identify issues when there’s a production incident or an outage. It would be great if finding a root cause of an outage was easy and could be followed quickly with a fix and a plan for an effective remediation path. But, the reality is engineers often firefight among an abundance of noise and staggering amounts of information. They traverse multiple tools just to find an answer.

Often, you might rely on infrastructure metrics or troll through endless messy logs to understand and triage a problem. But it can be hard to understand what’s happening inside the application. Without the additional help from software engineers to instrument their Go apps, Ops engineers often resort to guesswork or intuition when an incident or outage occurs.

So, before you start your instrumentation effort, start with a simple rule: Ask yourself, "What is the minimal viable instrumentation (or MVI) that I can instrument to help my fellow Ops engineers quickly identify issues?"

Admittedly, most of the time engineers are only interested in identifiers. For example, if there is a failed transaction, an engineer wants to know which userID, transactionID, or some form of contextual information can pinpoint exactly the affected request for a remediation path. But relying on status codes such as 404 or 500 isn’t going to be sufficient anymore. You'll need additional contextual metadata to work effectively as your environment starts to get complex.

Here are the four key identifiers you should consider as part of your instrumentation.

Request Identifier: correlation ID between requests (such as accountID)

State Identifier: current operations or status (such as completed)

Environment Identifier: operating domain metadata (such as software version)

Parameters Identifier: variables used to pass information (such as total or account)

These identifiers are the building blocks to improve observability.

After you've decided on your identifiers, plan ahead with your top five most important software functions that you should expose with telemetry, using the New Relic Go agent, and add more as you need over time.

Tip 2: Start the flow of data into New Relic using StartTransaction

The following video will step through tips two and three, followed by further explanation about this process in this blog.

Top Tips for Instrumenting Golang | New Relic

When you start using the New Relic Go agent for the first time, there’s a high possibility you won’t see any data, which is different than interpreted languages such as PHP, Ruby, Python, and JavaScript.

For interpreted languages, the New Relic SDK begins instrumenting supported framework functions, methods, or classes. But for Go, you need to start instrumentation by writing a StartTransaction.

txn := app.StartTransaction("transaction_name")
defer txn.End()

Note these two pieces of helpful documentation:

Instrument and monitor Go transactions
Install New Relic for Go

The app statement refers to the variable assigned during the New Relic configuration process, while the defer statement defers the segment ending until the function closes. This means that setting up New Relic for your Golang app requires you to add New Relic methods to your source code manually. Here is a simple instrumentation of a popular function: print().

func main() {
   app, err := newrelic.NewApplication(
       newrelic.ConfigAppName("Short Lived App"),
       newrelic.ConfigLicense(os.Getenv("NEW_RELIC_LICENSE_KEY")),
       newrelic.ConfigDebugLogger(os.Stdout),
   )
   if nil != err {
       fmt.Println(err)
       os.Exit(1)
   }

   // Wait for the application to connect.
   if err := app.WaitForConnection(5 * time.Second); nil != err {
       fmt.Println(err)
   }

   // Do the tasks at hand.  Perhaps record them using transactions and/or
   // custom events.
   tasks := []string{"white", "black", "red", "blue", "green", "yellow"}
   for _, task := range tasks {
       txn := app.StartTransaction("task")
       time.Sleep(10 * time.Millisecond)
       txn.End()
       app.RecordCustomEvent("task", map[string]interface{}{
           "color": task,
       })
   }

   // Shut down the application to flush data to New Relic.
   app.Shutdown(5 * time.Second)
}

Note the additional code segment - app.WaitForConnection and app.Shutdown, where we wait for the application to connect and shut down for five seconds. You need to do this initially because the harvest cycle in the New Relic Go agent is five seconds. This is only required if you are instrumenting a small sample app or a short-lived app in your development environment. For production deployment, you won’t need to use this code segment if your app runs longer than five seconds.

For a full code sample in one file, see this repository on GitHub.

Tip 3: Embed observability practices

As you start instrumenting more with your Go apps, take some time to discuss and review incidents with your team or engineering leads. Conduct a post-mortem or retrospective to understand what additional metadata engineers need from the app to triage issues more effectively.

Observability is about better understanding the inner workings of a system through exposed external outputs—getting one step closer to the why. Observability is also a scientific engineering approach to refine, experiment, learn, and add observations over time, so you can gain more confidence in your system.

As mentioned earlier, this is a great opportunity to embed observability practices as part of your instrumentation and development cycle. Observability is not about monitoring metrics or using an existing operations approach to look at hundreds or thousands of dashboards or alerts.

Often engineers will debate whether metrics, tracing, or logs are the best, but each telemetry type has its own pros and cons. Ultimately observability is all about encouraging deep collaboration between Dev and Ops, separating noise and signals. You gain more confidence over time in your systems by exposing contextual information either through better usage of metrics, tracing, or logs.

Unlike automatic instrumentation, the good news about manual instrumentation for Go is that there are no hidden, weird, or sudden surprises. Starting fresh allows you to embed observability concepts, knowing exactly what the instrumentation is doing, and you have absolute control over what you instrument in your app. The package from New Relic is stable, open, and available so you can understand what the agent is doing underneath the hood.

For more information, don't forget to visit the official page for the New Relic Go agent.

To read this full New Relic blog, click here.

How to obfuscate logs using Fluent Bit in New Relic

ntcsteve — Tue, 23 May 2023 23:06:02 +0000

To read this full New Relic blog, click here.

If you need to protect sensitive information and follow data privacy regulations, it's critical to obfuscate your log data, which means obscuring personally identifiable information (PII). But effectively concealing PII in logs might take time to implement, can increase compute resources, and might not work well with all types of logs.

It's vital to obfuscate PII and other sensitive data in your logs because you can ensure confidentiality and integrity of the data, while continuing to use logs to analyze system behavior and troubleshoot issues. You can mitigate the risk of data breaches and unauthorized access and still get your ongoing debugging work done.

This blog post shares helpful tips and code samples for obfuscating logs through client-side Fluent Bit with New Relic infrastructure monitoring. Whether you’re evaluating New Relic with other platforms, or you’re already using New Relic and seeking to safeguard sensitive information, read on to explore advanced configuration options in New Relic. You’ll gain advice on logs obfuscation, including:

The importance of log obfuscation and a few popular approaches
Using Fluent Bit in New Relic infrastructure monitoring
How to obfuscate your logs
How to remove sensitive data from your logs

Log obfuscation is both important and difficult

Log obfuscation is a critical security measure that helps protect computer systems against data breaches and unauthorized access to sensitive information. You might need to remove or obfuscate attributes from your logs to protect sensitive data like email addresses or credit card numbers from customer accounts, so they’re not visible in the logs. Removing or obfuscating these attributes can prevent unauthorized access to sensitive data in case of log leakages or malicious users.

An added bonus of eliminating unnecessary attributes is reducing log sizes, making them easier to manage and analyze. This can boost overall system performance and reduce costs for storing and processing large log files.

Implementing obfuscation for logs is especially challenging. It’s tricky to ensure that the obfuscated logs provide enough information for effective troubleshooting and analysis, the reason you need logs in the first place. This challenge is even more significant when dealing with large, complex systems that generate huge amounts of log data.

A few log obfuscation approaches
To overcome this challenge, engineering teams use various techniques such as redaction, hashing, and encryption to obfuscate sensitive data while maintaining the overall integrity of the log files.

One popular technique involves creating custom rules using Grok patterns or Lua scripts to mask or remove sensitive information from log data.

The process begins by identifying the types of sensitive data that need to be obfuscated. For example, you know that you have IP addresses, credit card numbers, and usernames for your customers.
Then you define custom rules in Grok patterns or Lua scripts to match the sensitive data patterns and specify how they should be masked or removed.
The next step is to test the custom rules against sample log data to ensure they correctly match the intended data patterns and apply the appropriate obfuscation.
After the rules have been tested, you can integrate them into the log processing pipeline with a log management tool like Logstash, Fluent Bit, or FluentD. You can use all of these log management tools to forward the processed logs to New Relic for further analysis.

Finding the right balance between security and usability in your logs

To make sure the logs remain a valuable tool for monitoring system performance and troubleshooting, it’s crucial to balance the level of obfuscation with how useful they are for troubleshooting and analysis. Effective log obfuscation requires careful consideration of the trade-offs between security and usability.

Grok patterns are a helpful pattern-matching tool for parsing data from unstructured log files, and Lua scripts offer more flexibility and control over the log processing pipeline but may require more development effort. The choice between Grok patterns and Lua scripts will depend on the needs and maturity of the log processing pipeline.

How Fluent Bit works with New Relic infrastructure monitoring

Other techniques such as Grok patterns or Lua scripts are available to modify logs, but using Fluent Bit in New Relic infrastructure monitoring is a more straightforward approach to protect sensitive information in logs.

Fluent Bit is an open-source data collector for a unified logging layer, designed to collect, filter, and forward logs and metrics data to various destinations. Embedded as part of New Relic infrastructure monitoring, Fluent Bit is deployed as a client-side agent by default. This means you can accomplish advanced configurations to collect logs and metrics data from multiple sources and send them to New Relic for analysis. The next diagram shows how log forwarding works in New Relic:

When the New Relic infrastructure agent runs, it processes configuration files in the logging.d directory. As a result, it generates a run-time Fluent Bit configuration file with the necessary [INPUT], [FILTER], and [OUTPUT] sections. If you provided an external Fluent Bit configuration file via the fluentbit option, the agent also declares an @include in the generated configuration file.

By default, the runtime file uses the default Fluent Bit configuration values because it doesn’t define a [SERVICE] section. But you can customize the configuration settings by creating your own [SERVICE] section in an external Fluent Bit configuration file and including it with the fluentbit option. To use the advanced configuration in Fluent Bit, such as logs obfuscation, you must generate separate configuration and parsers files externally and use them with the fluentbit, config_file, and parsers_file options in the agent.

Here’s an example of using Fluent Bit's additional configuration to forward obfuscated logs:

logs:
  - name: external-fluentbit-config-and-parsers-file
    fluentbit:
      config_file: /etc/newrelic-infra/logging.d/fluent-bit.conf
      parsers_file: /etc/newrelic-infra/logging.d/parsers.conf

The config_file option refers to the path of an existing Fluent Bit configuration file that you can use to customize your Fluent Bit configuration settings. This file includes the necessary [INPUT], [FILTER], and [OUTPUT] sections, among others.

The parsers_file option refers to the path of an existing Fluent Bit parsers file. This file defines custom parsing rules that Fluent Bit can use to extract information from incoming logs. Using parsers, you can transform raw log data into structured data, making it easier to analyze and understand.

Important: To forward logs for common use cases, the infrastructure agent provides simple log forwarding configurations in the logging.d/ directory's YAML files. These files are automatically translated into Fluent Bit configuration files with the correct format and configuration defaults.

Obfuscate your logs using Fluent Bit in New Relic

To start using Fluent Bit in New Relic, you need two files in place: the parsers_file for parsing files and the config_file for modifying records. Follow these steps, using advanced settings with Fluent Bit in New Relic.

Step 1: Disable existing New Relic Logs Configuration

Before using Fluent Bit, you must disable any existing New Relic logs configurations. You'll either need to remove or comment out the following configuration in your YAML file. In this example, the configuration is commented out.

# logs:
#   - name: nginx
#     file: /var/log/nginx.log
#     attributes:
#       logtype: nginx
#       environment: workshop
#       engineer: yourname
#       country: yourcountry
#     pattern: 500|404|400

Step 2: Create the fluentbit.yml file

Create or rename your file to fluentbit.yml, and include this path:

logs:
  - name: external-fluentbit-config-and-parsers-file
    fluentbit:
      config_file: /etc/newrelic-infra/logging.d/fluent-bit.conf
      parsers_file: /etc/newrelic-infra/logging.d/parsers.conf

Here’s an example of the file structure to use with Fluent Bit in New Relic infrastructure monitoring.

Step 3: Create a parsers_file

Create a file called parsers.conf, and include your Fluent Bit parser configuration. You can use the power of the parser filter plugin in Fluent Bit to parse fields in event records. You can extract specific information from your logs and transform it into structured data that’s easier to analyze and understand. You can test these custom rules against sample log data to ensure they match the intended data patterns and apply the appropriate obfuscation.

Here’s the default sample parser provided by Fluent Bit:

[PARSER]
    Name dummy_test
    Format regex
    Regex ^(?<INT>[^ ]+) (?<FLOAT>[^ ]+) (?<BOOL>[^ ]+) (?<STRING>.+)$

Here's an NGINX parser in a regular expression (regex) example:

[PARSER]
    Name   nginx
    Format regex
    Regex ^(?<source>[^ ]*) (?<host>[^ ]*) (?<user>[^ ]*) \\[(?<time>[^\\]]*)\\] "(?<method>\\S+)(?: +(?<path>[^\\"]*?)(?: +\\S*)?)?" (?<code>[^ ]*) (?<size>[^ ]*)(?: "(?<referer>[^\\"]*)" "(?<agent>[^\\"]*)")
    Time_Key time
    Time_Format %d/%b/%Y:%H:%M:%S %z

Using a regular expression pattern, this code defines a custom parser for NGINX access logs. The regular expression captures fields such as the client's IP address, time of the request, HTTP method, URL path, status code, and user agent. The Time_Key directive specifies the time field, and Time_Format defines the time format in the log line. This configuration enables log processing tools to parse NGINX access logs and extract relevant fields for analysis and visualization. For more examples, you can explore parsers provided by Fluent Bit, New Relic, or open source communities.

In the next example, you’ll use for obfuscating and removing logs attributes.

Step 4: Create a config_file

Create a file called fluent-bit.conf file and include a modify configuration where you’ll use one of the most valuable features of Fluent Bit, the Modify Filter plugin. Use this plugin to modify records using rules and conditions. You can change the content of log records based on specific criteria or requirements. This makes it easier to analyze and troubleshoot issues in your system.

Here’s the default sample from Fluent Bit:

[INPUT]
    Name mem
    Tag  mem.local

[OUTPUT]
    Name  stdout
    Match *

[FILTER]
    Name modify
    Match *
    Add Service1 SOMEVALUE
    Add Service3 SOMEVALUE3
    Add Mem.total2 TOTALMEM2
    Rename Mem.free MEMFREE
    Rename Mem.used MEMUSED
    Rename Swap.total SWAPTOTAL
    Add Mem.total TOTALMEM

Here’s an example configuration file to use to obfuscate attributes in your logs:

[INPUT]
    Name              tail
    Tag               nginx
    Path              /var/log/nginx.log
    DB                /var/log/nginx_error.db
    Path_Key          filename
    Parser            nginx
    Mem_Buf_Limit     8MB
    Skip_Long_Lines   On
    Refresh_Interval  30

[FILTER]
    Name record_modifier
    Match *
    Record logtype nginx
    Record hostname ${HOSTNAME}
    Record service_name Coming_From_FluentBit

[FILTER]
    Name modify
    Match *
    Set source XXXXX

[OUTPUT]
    Name newrelic
    Match *
    apiKey INGEST KEY

After applying the configuration file to obfuscate logs through Fluent Bit, your end result will look something like this example.

Step 5: Restart the agent

To use configurations from Fluent Bit, restart the New Relic infrastructure agent. Before restarting, ensure that both your files (fluent-bit.conf and parsers.conf) are correctly referenced in the fluentbit.yml file, as shown back in step 2.

logs:
  - name: external-fluentbit-config-and-parsers-file
    fluentbit:
      config_file: /etc/newrelic-infra/logging.d/fluent-bit.conf
      parsers_file: /etc/newrelic-infra/logging.d/parsers.conf

Here’s the command to restart the New Relic infrastructure agent:

sudo systemctl restart newrelic-infra

Remove sensitive data from logs

In the previous example in Step 4, you learned how to obfuscate a log using Set. Another alternative is to use the Remove operation, a powerful tool for modifying logs. Use it to easily remove unnecessary or sensitive data from a record. Delete a key-value pair from a record, if it exists, using the Record Modifier plugin.

Here’s an example configuration file to remove attributes in your logs:

[INPUT]
    Name              tail
    Tag               nginx
    Path              /var/log/nginx.log
    DB                /var/log/nginx_error.db
    Path_Key          filename
    Parser            nginx
    Mem_Buf_Limit     8MB
    Skip_Long_Lines   On
    Refresh_Interval  30

[FILTER]
    Name record_modifier
    Match *
    Record logtype nginx
    Record hostname ${HOSTNAME}
    Record service_name Coming_From_FluentBit
    Remove_key source

[OUTPUT]
    Name newrelic
    Match *
    apiKey INGEST KEY

After applying the configuration file to remove log data using Fluent Bit, your final result should look something like this example:

Important: Restarting the New Relic infrastructure agent after applying a new configuration is a crucial step to ensure the changes take effect. This step is particularly important because it guarantees that the new configuration settings are loaded and used by the agent. Make sure to restart the agent after applying new configurations, the same as shown in Step 5.

Conclusion

You've reviewed examples of straightforward ways to ensure confidentiality and integrity of log data using Fluent Bit in New Relic infrastructure monitoring.

Specifically, you've learned how to protect sensitive information in logs by obfuscating data with Fluent Bit log configurations. For example, after a key value is identified, you can easily use the Modify filter plugin to obfuscate it or the Record Modifier filter plugin to remove log attributes.

To read this full New Relic blog, click here.

Not an existing New Relic user? Sign up for a free account to get started!