New Relic, Programmability and Open Source

Natalie A. Wiser — Tue, 21 Jul 2020 18:16:29 +0000

Charles Humble for New Relic

While New Relic has been involved in various open-source projects for some time, these efforts have received increased focus in recent months, as the company has made a concerted push into open source and made the New Relic One platform programmable. The recently launched is intended to act as a hub for developers to discover, use, and contribute to the open-source projects maintained and supported by the vendor. Areas the projects cover include instrumentation, third-party integrations, custom apps, tooling templates and example code.

Among the approximately 180 projects, you’ll find the source code for a large number of the Nerdpacks available through the application catalogue on New Relic One. These provide working, usable examples of the programmatic capabilities of the platform. Nerdpacks are Webpack applications that use GraphQL to interact with the data available in New Relic, and React components for visualization and other UI. These are the same tools that New Relic’s developers use to build the platform itself.

Examples of Nerdpacks include Cloud Optimize, which evaluates the size and utilization of your AWS, GCP and Azure cloud instances and shows the amount of money you could save by right-sizing; Container Explorer, which gives you a global view of all containers associated with an account; and (Top)[https://opensource.newrelic.com/projects/newrelic/nr1-top], a simple process monitor inspired by the Unix top command:

The source for the New Relic open-source website itself, built using Gatsby, is also available.

In addition, you can find a number of libraries, including GraphQL Java Core, which provides helpers for integrating (graphql-java)[https://github.com/graphql-java/graphql-java] with a JVM application, simplifying input mapping and schema configuration. Ben Evans, Principal Engineer & JVM Technologies Architect at New Relic, explained that

We use GraphQL as a key "glue" technology - where the schema acts as an interface definition and a contract between the back end and front end (and because we have programmability capability customers can use the same contract to build their own rich front ends if they want).

This means we quite naturally fall into the "schema first" view of the world - where we want to enable teams to write the schema and then auto-generate based upon that schema, rather than adding Java annotations to pre-existing services... Of course, the schema-first approach may not work for everyone - but it works for us so we thought we'd share those pieces that enable it.

There are three major community-led projects that New Relic doesn’t host, but to which it contributes. The first is the OpenTelemetry standard, currently incubating with the CNCF. This project, formed through a merger of OpenTracing and OpenCensus, is a multi-vendor effort that provides a single set of APIs, libraries, agents, and collector services to capture distributed traces and metrics from your application. New Relic contributes by defining specifications, implementing parts of the specification in language SDKs, reviewing pull requests from other contributors, and serving as a maintainer in the various OpenTelemetry sub projects. The company has two full-time engineers working on the project - John Watson is a Java maintainer, and Tyler Yahn is a Go maintainer. John is also highly active on the project’s Gitter.

Secondly, New Relic is a participant in the W3C Distributed Tracing Working Group. The W3C project is closely linked to OpenTelemetry, which supports the emerging W3C standard as one of the available options for context propagation. As Alois Reitbauer has written in an explainer about the W3C Trace Context sub-project

The W3C Trace Context specification is a set of new standards being developed by open source and commercial tool providers that defines a unified approach to context and event correlation within distributed systems, such as microservices environments. Having such a standard will enable end-to-end transaction tracing within distributed applications across a range of monitoring tools.

The third community effort is AdoptOpenJDK (soon to be moving to the Eclipse Foundation and renamed Eclipse Adoptium). AdoptOpenJDK started in 2017 as an advocacy and on-boarding program for OpenJDK (the OSS project behind Java itself), but has since shifted its efforts to providing a long-term, stable build platform for OpenJDK. New Relic is a Silver Sponsor, with a pool of around five engineers involved in the project. New Relic's involvement came about, Evans explained, after Oracle changed the terms under which they distribute their Java binaries.

We decided that we would adopt the same upgrade cadence as we see our customers using and that meant sticking with LTS Java releases. In turn, that meant choosing a vendor to be our supplier for open-source Java binaries.

AdoptOpenJDK offers free, high quality binaries - and they are focused on the binaries, not on making source code contributions. After Oracle, AdoptOpenJDK is the stand-out distribution that we see our customers using, so it was an easy choice to make them our supplier as well.

The decision to get involved on a technical level came about due to a confluence of: a) New Relic engineers who wanted to spend more time working on open-source software and b) Technical efforts within AdoptOpenJDK that are relevant to New Relic. Our services and backend is pretty much all Java and Kotlin so we have a key strategic interest in the success of the project. So at one level it's no more complicated than the familiar: "Contribute to what matters to you" mantra of open source.

In terms of what New Relic contributes to the project Evans told me that

As New Relic is an Observability company, it's probably no surprise that we're interested in anything that intersects our business. So, we're interested in things like Java Flight Recorder (JFR) and innovations in serviceability. We are also very interested in performance improvements and avoiding regressions - both for customers and for our own high-performance services.

For example, when we transitioned to using AdoptOpenJDK last year, we had a programme of chasing down and fully investigating every performance regression that was reported when moving from Oracle to OpenJDK. From our knowledge of how the OpenJDK project works we were basically sure that any regressions should be false positives, but our approach was very much "Trust But Verify". As you might hope, we were able to prove that all the reports were false positives and there were no real regressions caused by the upgrade to OpenJDK.

It's not a very glamorous example, but sometimes making more perfect software requires detailed, meticulous work. Developers and teams need the confidence when upgrading that their systems won't break or regress in subtle ways. We're happy to do our bit to help provide that confidence for our customers, ourselves, and the whole community.

Connecting to New Relic with Open-Source Software

Natalie A. Wiser — Mon, 13 Jul 2020 18:33:47 +0000

Written by Charles Humble for New Relic

New Relic offers a variety of mechanisms by which you can gather telemetry data from open-source software and send it back to New Relic One for analysis.

New Relic’s various agents include built-in instrumentation for many popular parts of their respective ecosystems. The Java Agent, for instance, includes built-in instrumentation for many open-source libraries and frameworks such as Grails, Hibernate, Hystrix, Play, the Spring Framework, Spring Boot, Thrift, and Vert.x. Likewise, the node.js agent includes telemetry for Express, Restify, Connect, Hapi and Kora, while the .NET agent supports both .NET Framework and .NET Core, and so on.

New Relic also offers a number of additional integrations for many popular open-source applications and frameworks including Elasticsearch, Consul, MongoDB, PostgreSQL, RabbitMQ, Memcached, Apache, Kafka and more. The source for these integrations is available on the New Relic GitHub and they are licensed under the MIT license. The information captured by them is often extensive: for Kafka, for example, data is collected on brokers (both ZooKeeper and Bootstrap), producers, consumers, and topics.

In many cases, therefore, you will probably find the built-in instrumentation gives you all the information you need. However, if you find that it doesn't, then you also have the option of building custom integrations using the New Relic telemetry SDKs. These are open-source, language-specific libraries for reporting metrics, trace data, and other telemetry data to New Relic, and are available for Java, Node/TypeScript, Python, Go and .NET via New Relic's GitHub. The .NET SDK is also available as a NuGet package. They are licensed under the Apache v.2 license.

A final, new option to consider is the OpenTelemetry project that started from a merger of OpenTracing and OpenCensus and is currently incubating at the Cloud Native Computing Foundation (CNCF). This project is in beta, and therefore not yet recommended for production use, but it is particularly interesting since it represents a vendor-neutral standard for gathering telemetry data. As John Watson and Lavanya Chockalingam have commented, OpenTelemetry thereby offers a means of future-proofing your instrumentation. It also, they write, provides a number of benefits:

Simplified choice. You don’t need to decide between one standard or the other. Are you currently using OpenTracing or OpenCensus? Don’t worry, OpenTelemetry offers backward compatibility for both projects.
Cross-platform. OpenTelemetry supports a variety of languages and back ends. It represents a vendor-neutral path to capturing and transmitting telemetry to back ends without altering existing instrumentation. This is an important project that satisfies developers looking for this kind of freedom.
Streamlined observability. As OpenTelemetry puts it, "Effective observability requires high-quality telemetry." Expect to see more vendors move toward OpenTelemetry, as it’s far easier to accommodate and test against a single standard.

Equally important, OpenTelemetry means that you are not locked in to a single vendor’s agent for collecting telemetry data.

New Relic is contributing to the project and is currently shipping open-source exporters for Java and Go that can send OpenTelemetry data back to New Relic One.

DEV Community: Natalie A. Wiser

New Relic, Programmability and Open Source

Charles Humble for New Relic

Connecting to New Relic with Open-Source Software

Written by Charles Humble for New Relic