DEV Community: Vladimir Mihailenco

Uptrace v1.7 is available

Vladimir Mihailenco — Fri, 29 Mar 2024 11:51:57 +0000

The Uptrace team is pleased to present the latest v1.7 release, which introduces dark mode, revamped navigation, dashboards list, system tracing metrics, and much more.

New nav and dark mode

Uptrace v1.7 improves navigation and adds support for the dark mode.

Dark mode should be enabled automatically if your operating system uses a dark theme. It can also be enabled manually using the switch in the upper right corner.

Dashboards improvements

Uptrace now provides a separate view that allow to list and manage your dashboards.

Uptrace also provides quick access to dashboards that have the same prefix, for example, here is how dashboards with the "Hostmetrics: " prefix are grouped together.

You may need to reset your existing dashboards via UI to benefit from this.

Querying spans from metric dashboards

Added ability to query spans using system metrics provided by Uptrace: uptrace_tracing_spans, uptrace_tracing_logs, and uptrace_tracing_events.

Such system metrics allow to query spans table directly without actually creating any metrics. You have access to all span attributes and can use them for filtering and grouping.

JSON API and auth token

Added support for JSON API using the user.auth_token for authentication.

For now JSON API allows to query spans, create metric monitors, and annotations.

For example, you can get a list of host names using the group by host_name query like this:

curl https://api.uptrace.dev/api/v1/tracing/<project_id>/groups?time_gte=2023-07-10T00:00:00Z&time_lt=2023-07-11T00:00:00Z&query=group%20by%20host_name \
  --header "Authorization: Bearer <token>"

.NET metrics

Added pre-built dashboards for .NET system metrics reported by OpenTelemetry instrumentation.

Other changes

Updated Tempo API to the latest version. You should be able to use Tempo integration with the recent Grafana version.
Added new project setting force_span_name by @HeCorr. See the config for details.
Added ability to disable TLS when configuring SMTP mailer by @rgl.

The full changelog is available on GitHub.

Uptrace is an open source log management tool for OpenTelemetry that supports distributed tracing, metrics, and logs. You can use it to monitor applications and troubleshoot issues.

In just a few minutes, you can try Uptrace by visiting the cloud demo (no login required) or running it locally with Docker. The source code is available on GitHub.

Uptrace v1.6 is available

Vladimir Mihailenco — Mon, 25 Mar 2024 14:37:06 +0000

The Uptrace team is pleased to present the latest v1.6 release, which adds support for service graphs, Prometheus remote write, Grafana data source for Prometheus, annotations, and much more.

Service Graphs

Service Graphs provide a visual representation of service interactions, dependencies, and performance metrics. They are built by analyzing span relationships and require certain span attributes.

The same data is also available as a metrics dashboard which comes by default with Uptrace:

Prometheus remote write

Starting with the v1.6 release, Uptrace natively supports Prometheus Remote Write protocol, which allows Prometheus to send its collected metrics data to a long-term storage solution such as Uptrace.

You can configure Prometheus Remote Write using the following endpoint:

remote_write:
  - url: 'http://localhost:14318/api/v1/prometheus/write'
    headers: { 'uptrace-dsn': 'http://project2_secret_token@localhost:14318?grpc=14317' }

The configuration above allows to send Prometheus data directly to Uptrace without using OpenTelemetry Collector.

Grafana Prometheus data source

You can also use Uptrace as a Prometheus data source in Grafana. Uptrace uses the original Prometheus engine so all Prometheus queries should be supported and you should be able to use existing Grafana dashboards with the Uptrace data source.

In Grafana, go to "Configururation" -> "Data source".
Click on "Add data source" and then select "Prometheus".
As an URL, use the Uptrace HTTP address, for example, http://localhost:14318/api/prometheus or https://api.uptrace.dev/api/prometheus.
In "Custom HTTP Headers" section, click "Add header" to specify Uptrace DSN.

Header: uptrace-dsn
Value: http://project2_secret_token@localhost:14318?grpc=14317

Chart annotations

Chart annotations are labels or notes added to a chart to provide additional information or context. Annotations help clarify the data presented in the chart and help the viewer understand key points or trends.

Uptrace displays annotations as square dots on the x-axis. Clicking on an annotation displays the annotation description and tags. The description field can contain markdown links to other systems with more details.

You can create annotations by sending an HTTP POST request to the Uptrace API, for example, when deploying a new version of your application:

curl -X POST http://localhost:14318/api/v1/annotations \
   -H 'uptrace-dsn: https://<token>@api.uptrace.dev/<project_id>' \
   -d '{"name":"Deployed to production", "attrs": {"service.version": "540d2ee"}}'

See documentation for more details.

HTTPcheck Receiver

HTTP Check Receiver is a component of the OpenTelemetry Collector that can be used to perform synthetic checks against HTTP endpoints, for example:

receivers:
  httpcheck:
    targets:
      - endpoint: 'https://api.uptrace.dev/health/status'
        method: GET
    collection_interval: 15s

Starting with the v1.6 release, Uptrace comes with a pre-built dashboard to visualize the collected data:

Just like with other metrics, you can create monitors to receive notifications when certain conditions are met:

See OpenTelemetry HTTP check for details.

Uptrace DSN

Uptrace DSN (Data Source Name) is a connection string that is used to connect and send data to an Uptrace backend. It contains a backend address (host:port) and a secret token that grants access to a project.

If the recent version Uptrace DSN was extended to also contain the gRPC port in case you're using the OTLP/gRPC exporter:

# before
UPTRACE_DSN="http://project1_secret_token@localhost:14318/1"

# after
UPTRACE_DSN="http://project1_secret_token@localhost:14318?grpc=14317"

Previously issued DSNs are still fully supported.

Conclusion

The full list of changes is available on GitHub, where you can also download the latest version or run Uptrace locally using Docker.

You can also sign up for a free Cloud account and get 1TB of storage and 50,000 timeseries. No credit card required.

Open Source Splunk Alternatives

Monitoring FastAPI with OpenTelemetry

Vladimir Mihailenco — Thu, 14 Sep 2023 14:33:20 +0000

By integrating OpenTelemetry with FastAPI, you can gain valuable insight into the performance, behavior and dependencies of your API. You can monitor and troubleshoot issues, optimize performance, and ensure the reliability of your FastAPI applications.

What is FastAPI?

FastAPI is a modern, high-performance web framework for building APIs with Python. It is designed to be easy to use, highly efficient, and able to handle high loads.

FastAPI's combination of performance, productivity, and modern features has made it popular among developers building APIs with Python.

What is OpenTelemetry?

OpenTelemetry is an open source observability framework hosted by Cloud Native Computing Foundation. It is a merger of OpenCensus and OpenTracing projects.

OpenTelemetry aims to provide a single standard across all types of observability signals such as OpenTemetry logs, distributed tracing, and metrics.

OpenTelemetry specifies how to collect and send telemetry data to backend platforms. With OpenTelemetry, you can instrument your application once and then add or change vendors without changing the instrumentation.

OpenTelemetry provides detailed instrumentation, enabling you to monitor and measure the performance of your FastAPI applications in real-time. You can identify bottlenecks, optimize code, and ensure your application runs smoothly even under heavy traffic.

FastAPI instrumentation

To install OpenTelemetry instrumentation for FastAPI:



pip install opentelemetry-instrumentation-fastapi

Usage

To instrument FastAPI application:



from fastapi import FastAPI
from opentelemetry.instrumentation.fastapi import FastAPIInstrumentor

app = FastAPI()
FastAPIInstrumentor.instrument_app(app)

Also see OpenTelemetry FastAPI example at GitHub.

OpenTelemetry allows you to trace requests as they flow through your FastAPI application, providing a clear picture of how different components and services interact. This end-to-end tracing helps diagnose issues quickly and streamline troubleshooting.

Once your FastAPI application is instrumented with OpenTelemetry, you can use the observability data in a variety of ways. You can visualize distributed traces, analyze performance metrics, and gain insight into the behavior of your API using OpenTelemetry backends such as Uptrace, Jaeger, Prometheus, or Grafana.

What is Uptrace?

Uptrace is an open source APM for OpenTelemetry that supports distributed tracing, metrics, and logs. You can use it to monitor applications and troubleshoot issues.

Uptrace comes with an intuitive query builder, rich dashboards, alerting rules, notifications, and integrations for most languages and frameworks.

Uptrace can process billions of spans and metrics on a single server and allows you to monitor your applications at 10x lower cost.

In just a few minutes, you can try Uptrace by visiting the cloud demo (no login required) or running it locally with Docker. The source code is available on GitHub.

uvicorn

If you are using uvicorn with Gunicorn, you need to initialize OpenTelemetry in the post-fork hook:



import uptrace

def post_fork(server, worker):
    uptrace.configure_opentelemetry(...)

workers = 4
worker_class = "uvicorn.workers.UvicornWorker"

See Application servers for details.

Conclusion

With insights from OpenTelemetry, you can make informed decisions about scaling your FastAPI application. Load balancing strategies can be adjusted based on real-time data to handle traffic spikes effectively.

OpenTelemetry also allows you to instrument specific parts of your code for custom telemetry collection. You can use OpenTelemetry Python APIs to manually create spans and add custom attributes, events, or metrics to capture additional information.

Structured logging best practices

Vladimir Mihailenco — Tue, 12 Sep 2023 10:51:17 +0000

In structured logging, log messages are broken down into key-value pairs, making it easier to search, filter, and analyze logs. This is in contrast to traditional logging, which usually consists of unstructured text that is difficult to parse and analyze.

What is structured logging?

Structured logging is the practice of capturing and storing log messages in a structured and organized format.

Traditional logging often involves printing raw text messages to log files, which can be difficult to parse and analyze programmatically.

In contrast, structured logging formats log messages as key-value pairs or in structured data formats such as JSON or XML.

Why to use structured logging?

Structured logging provides the following benefits:

Improved readability. The structured format makes log messages more human-readable, allowing developers and operators to easily understand the content without relying solely on raw text parsing.
Better searching and filtering. Structured data makes it easier to search for specific log entries or filter logs based on specific criteria. This is especially useful for large-scale applications with large amounts of log data.
Easy integration with tools. Structured logs can be ingested and processed by various log management and analysis tools, enabling powerful analysis, visualization, and monitoring of application behavior.
Improved debugging and troubleshooting. When log messages are structured, it is easier to include relevant contextual information, such as timestamps, error codes, and specific attributes related to the logged events, which facilitates effective debugging and troubleshooting.
Consistency and scalability. Structured logging promotes a consistent and uniform log format throughout the application, making it easier to scale logging capabilities and maintain logs in a standardized manner.

Structured log formats

Structured logging can be implemented using various data formats, with JSON being one of the most commonly used due to its simplicity and human-readability.

However, other formats can also be used depending on the requirements of the application and the logging framework in use.

JSON format

JSON is a lightweight data exchange format that is easy for both humans and machines to read and write. It represents data as key-value pairs and arrays, making it an excellent choice for structured logging due to its simplicity and widespread support.

Example:

{
  "timestamp": "2023-08-04T12:34:56Z",
  "level": "INFO",
  "module": "payment-service",
  "message": "Payment successful",
  "amount": 50.25,
  "transaction_id": "abc123"
}

You can also use JSON to include structured data in your log messages:

request failed {"http.method": "GET", "http.route": "/users/:id", "enduser.id": 123, "foo": "hello world"}

logfmt

This format represents log entries as a series of key-value pairs separated by delimiters such as spaces or tabs. It is simple and easy to implement.

If a value contains a space, you must enclose it in quotation marks. For example:

request failed http.method=GET http.route=/users/:id enduser.id=123 foo="hello world"

Free format

If your library does not support structured logging, you can still improve grouping by quoting params:

# good
can't parse string: "the original string"
"foo" param can't bempty

# bad
can't parse string: the original string
foo param can't be empty

Logging backend

Uptrace is an open source APM for OpenTelemetry that supports logs, traces, and metrics. You can use it to monitor applications and troubleshoot issues.

Uptrace natively supports structured logging and automatically parses log messages to extract the structured data and store it as attributes.

Uptrace comes with an intuitive query builder, rich dashboards, alerting rules, notifications, and integrations for most languages and frameworks.

Uptrace can process billions of logs on a single server and allows you to monitor your applications at 10x lower cost.

In just a few minutes, you can try Uptrace by visiting the cloud demo (no login required) or running it locally with Docker. The source code is available on GitHub.

Conclusion

Structured logging enables better log management, improved troubleshooting, and better application monitoring, resulting in more efficient and reliable software development and maintenance processes.

Introduction to Go app monitoring

Vladimir Mihailenco — Mon, 11 Sep 2023 12:04:57 +0000

When it comes to monitoring Golang applications, there are various tools and practices you can use to gain insights into your application's performance, resource usage, and potential issues.

By using OpenTelemetry for monitoring in your Go applications, you can gain valuable insights into the behavior, performance, and resource utilization of your distributed systems, allowing you to troubleshoot issues, optimize performance, and improve the overall reliability of your software.

What is OpenTelemetry?

OpenTelemetry is an open source project that aims to provide a set of vendor-neutral, standardized APIs and instrumentation libraries for collecting distributed traces, metrics, and logs.

OpenTelemetry is designed to facilitate observability in cloud-native and microservices architectures, where applications are often composed of multiple services that communicate with each other.

OpenTelemetry provides a standardized way to capture observability signals:

Metrics indicate that there is a problem.
Traces tell you where the problem is.
Logs help you find the root cause.

What is distributed tracing?

OpenTelemetry provides distributed tracing, allowing you to follow the flow of a single request as it propagates through different services in a distributed system. Traces provide a comprehensive view of the interactions between different components, helping you understand performance bottlenecks and troubleshoot problems.

OpenTelemetry tracing works by instrumenting your application code to create and manage distributed traces. A trace represents the flow of a single request as it propagates through different services and components in a distributed system.

Traces are composed of spans, which represent individual operations or stages within the trace. Each span captures timing and contextual information to help you understand the behavior and performance of your application.

Traces

OpenTelemetry Go provides a robust API for instrumenting your Go applications to create and manage distributed traces. It allows you to create spans that represent individual operations or stages within a trace, and these spans capture timing and contextual information.

In the following example, we use the Start method from the tracer to create spans. The Start method returns a new context with the created span, which is then passed to other functions to propagate the context throughout your application.

package main

import (
    "context"
    "time"

    "go.opentelemetry.io/otel"
)

func main() {
    // Start a new root span representing the entire request.
    tracer := otel.Tracer("my-tracer")
    ctx, span := tracer.Start(context.Background(), "my-root-span")
    defer span.End()

    // Simulate some work.
    doWork(ctx)
}

func doWork(ctx context.Context) {
    // Create a child span to represent a processing stage.
    tracer := otel.Tracer("my-tracer")
    ctx, span := tracer.Start(ctx, "my-child-span")
    defer span.End()

    // Simulate some work.
    time.Sleep(100 * time.Millisecond)
}

Tracing data is typically exported to an OpenTelemetry backend or storage system for analysis and visualization.

Uptrace is a distributed tracing tool that supports distributed tracing, metrics, and logs. You can use it to monitor applications and troubleshoot issues.

Uptrace comes with an intuitive query builder, rich dashboards, alerting rules with notifications, and integrations for most languages and frameworks.

Uptrace can process billions of spans on a single server, allowing you to monitor your software at 10x less cost.

Uptrace provides an OpenTelemetry distributive for Go that simplifies OpenTelemetry SDK configuration. Here is how you can use it to configure OpenTelemetry Go to export spans and metrics to Uptrace:

import "github.com/uptrace/uptrace-go/uptrace"

uptrace.ConfigureOpentelemetry(
    // copy your project DSN here or use UPTRACE_DSN env var
    //uptrace.WithDSN("https://<token>@api.uptrace.dev/<project_id>"),

    uptrace.WithServiceName("myservice"),
    uptrace.WithServiceVersion("v1.0.0"),
    uptrace.WithDeploymentEnvironment("production"),
)

Metrics

OpenTelemetry Metrics provides a standardized way to collect and record metrics from your application. Metrics are essential for monitoring the health and performance of your services, helping you gain insight into your application's resource utilization and behavior.

Here's an example of how you can use OpenTelemetry metrics to create and record a counter in your Go application:

package main

import (
    "context"
    "fmt"
    "time"

    "go.opentelemetry.io/otel"
    "go.opentelemetry.io/otel/exporters/stdout"
    "go.opentelemetry.io/otel/metric"
)

var requestsCounter = metric.Must(meter).NewInt64Counter("requests")

func main() {
    // Create a new stdout exporter to print metrics to the console.
    exporter, err := stdout.NewExporter(stdout.WithPrettyPrint())
    if err != nil {
        panic(err)
    }

    // Create a new meter provider with the stdout exporter.
    meter := otel.GetMeterProvider().Meter("example")

    // Increment the counter to simulate an event.
    requestsCounter.Add(context.Background(), 1)

    // Wait for a few seconds to see the metric output.
    time.Sleep(5 * time.Second)
}

Logs

Monitoring logs in Go applications is crucial for identifying issues, understanding application behavior, and troubleshooting problems.

While traces and metrics provide valuable insights into system performance, logs offer detailed context and information about specific events, errors, and application behavior.

It is important to use a logging library that supports structured logging. Structured logging allows you to log key-value pairs or JSON objects instead of plain text messages. This makes it easier to parse and analyze logs programmatically.

Here's an example of how structured logging looks like in JSON format:

{
  "timestamp": "2023-08-02T12:34:56.789Z",
  "severity": "INFO",
  "service": "example",
  "component": "web",
  "message": "User logged in",
  "user_id": "12345",
  "ip_address": "192.168.0.1"
}

Popular Go logging libraries that support structured logging are Logrus and Zap.

Logrus

Logrus is a popular logging library for Go that provides a simple and flexible API for logging in your applications. It is known for its ease of use and extensibility.

You can use OpenTelemetry Logrus instrumentation to use Logrus in your applications:

import (
    "github.com/uptrace/opentelemetry-go-extra/otellogrus"
    "github.com/sirupsen/logrus"
)

// Instrument logrus.
logrus.AddHook(otellogrus.NewHook(otellogrus.WithLevels(
    logrus.PanicLevel,
    logrus.FatalLevel,
    logrus.ErrorLevel,
    logrus.WarnLevel,
)))

// Use ctx to pass the active span.
logrus.WithContext(ctx).
    WithError(errors.New("hello world")).
    WithField("foo", "bar").
    Error("something failed")

Zap

Zap is another popular logging library for Go, known for its high performance and low allocation overhead. Developed by Uber, Zap aims to provide fast and efficient logging for production use, making it a popular choice for high performance applications.

You can use OpenTelemetry Zap instrumentation to use Zap in your applications:

import (
    "go.uber.org/zap"
    "github.com/uptrace/opentelemetry-go-extra/otelzap"
)

// Wrap zap logger to extend Zap with API that accepts a context.Context.
log := otelzap.New(zap.NewExample())

// And then pass ctx to propagate the span.
log.Ctx(ctx).Error("hello from zap",
    zap.Error(errors.New("hello world")),
    zap.String("foo", "bar"))

// Alternatively.
log.ErrorContext(ctx, "hello from zap",
    zap.Error(errors.New("hello world")),
    zap.String("foo", "bar"))

Profiling

Profiling allows you to measure the performance and resource usage of your application. It helps you identify performance bottlenecks and areas that may need optimization.

Go provides built-in profiling tools that allow you to collect and analyze profiling data.

CPU profiling

CPU profiling is used to find out which functions use the most CPU time during the execution of your program. To enable CPU profiling in your Go application, you can use the runtime/pprof package:

package main

import (
    "fmt"
    "os"
    "runtime/pprof"
)

func main() {
    f, err := os.Create("cpu.prof")
    if err != nil {
        fmt.Println("Error creating file:", err)
        return
    }
    defer f.Close()

    pprof.StartCPUProfile(f)
    defer pprof.StopCPUProfile()

    // Your application code here...
}

Once the profiling is enabled, run your program, and it will generate a cpu.prof file. You can then analyze this file using the go tool pprof command-line tool.

Memory profiling

Memory profiling helps you understand memory usage patterns in your application and identify potential memory leaks. To enable memory profiling, you can again use the runtime/pprof package:

package main

import (
    "fmt"
    "os"
    "runtime/pprof"
)

func main() {
    f, err := os.Create("memory.prof")
    if err != nil {
        fmt.Println("Error creating file:", err)
        return
    }
    defer f.Close()

    pprof.WriteHeapProfile(f)
    // Your application code here...
}

After running your program with memory profiling enabled, it will generate a memory.prof file. Analyze this file using the go tool pprof command-line tool.

Block profiling

Block profiling is used to find out how goroutines are blocked or waiting. This helps identify potential concurrency issues. To enable block profiling, you can use the runtime/pprof package:

package main

import (
    "fmt"
    "os"
    "runtime/pprof"
)

func main() {
    f, err := os.Create("block.prof")
    if err != nil {
        fmt.Println("Error creating file:", err)
        return
    }
    defer f.Close()

    pprof.Lookup("block").WriteTo(f, 0)
    // Your application code here...
}

After running your program with block profiling enabled, it will generate a block.prof file. Analyze this file using the go tool pprof command-line tool.

Conclusion

OpenTelemetry Go is a valuable tool for Go developers who want to implement observability best practices in their applications.

With OpenTelemetry Go, developers can collect telemetry data, understand the behavior of their application, and ensure optimal performance and reliability in modern cloud-native and microservices environments.

Zero-downtime PostgreSQL migrations

Vladimir Mihailenco — Sun, 10 Sep 2023 13:07:17 +0000

Zero-downtime PostgreSQL migrations are essential for maintaining the availability and reliability of your application while making changes to the database schema or performing other maintenance tasks.

Follow these simple rules to avoid common pitfalls and apply changes to your database without unplanned downtime.

Use transactions

Enclose your migration scripts in a transaction block. If any part of the migration fails, the entire transaction will be rolled back, ensuring that your database remains in its previous state.

Avoid long-running transactions

Running a migration in a transaction means that changes made within a transaction are not visible until the end of a transaction. That is exactly what we need when we apply a migration, but in practise it does not work well.

Using a transaction causes PostgreSQL to maintain two versions of a database for the duration of a transaction. One version with your changes and one without. PostgreSQL transactions are well-suited for such task but they have their limits too.

PostgreSQL also has to hold all the locks ackquired during a migration. For example, updating a row locks the row so your changes are not overwritten by another transaction. PostgreSQL releases the lock only in the end of a transaction. And if a concurrent transaction actually changes the same row migrations fails.

Using transactions only works well if your migration is small and fast (less than 5-10 seconds). Even for a medium database, using long transactions either makes a migration slower (in some cases 10x slower) or causes a migration to fail.

Split long-running queries into smaller batches

PostgreSQL runs every query in a transaction. So not using BEGIN and COMMIT does not imply that you are not using transactions. Meaning that you need to split long running queries into smaller ones and avoid large transactions for the reasons we discussed above.

For example, you need to update 1 million rows. Don't do it with a single UPDATE query. Instead split the job into 10 batches each containing 100k rows. And execute the same UPDATE query separately on each batch. Now you have 10 queries instead of 1, but you can be sure that migration will succeed.

Update rows in a consistent order

When possible, update rows in a consistent order. This helps avoiding deadlocks when 2 conurrent transactions try to update the same rows but in a different order.

For example, deadlock happens when transaction 1 locks row #1 and transaction 2 locks row #2. Now transaction 1 waits for a lock on row #2 and transaction 2 waits for a lock on row #1. They lock each other and PostgreSQL has to kill one of them.

Bad:

-- transaction 1
UPDATE test WHERE id IN (1, 2);

-- transaction 2
UPDATE test WHERE id IN (2, 1);

Good:

-- transaction 1
UPDATE test WHERE id IN (1, 2);

-- transaction 2
UPDATE test WHERE id IN (1, 2);

The same rule applies when you are using INSERT ON CONFLICT DO UPDATE. In such case, you may need to sort rows before inserting them.

Don't add columns with NOT NULL

Queries like ADD column NOT NULL fail on tables that already have some rows. Because existing rows do not have values for the newly added column, PostgreSQL refuses to add the column.

> ALTER TABLE test ADD COLUMN foo text NOT NULL;

ERROR:  column "foo" of relation "test" contains null values

Your alternatives are:

Add a default value, for example, foo text NOT NULL DEFAULT ''.
Drop NOT NULL althogether and add some validation against NULL elsewhere.
Split the query into multiple migrations:

-- migration 1
ALTER TABLE test ADD COLUMN foo text;

-- migration 2
UPDATE test SET foo = '';

-- migration 3
ALTER TABLE test ALTER COLUMN foo SET NOT NULL;

Create temporary tables

In some cases, you can create temporary tables with the new schema, copy data from the old table, and then swap the tables atomically. This minimizes the downtime during migration.

Rolling deployments

Implement rolling deployments for your application. This involves deploying new code and migrating the database incrementally, one instance or node at a time, while keeping the others running.

Monitoring PostgreSQL

To monitor PostgreSQL, you can use OpenTelemetry PostgreSQL receiver that comes with OpenTelemetry Collector.

OpenTelemetry Collector is an agent that pulls telemetry data from systems you want to monitor and sends it to an OpenTelemetry backend using the OpenTelemetry protocol (OTLP).

Uptrace is a DataDog competitor that supports distributed tracing, metrics, and logs. You can use it to monitor applications and troubleshoot issues.

Uptrace comes with an intuitive query builder, rich dashboards, alerting rules with notifications, and integrations for most languages and frameworks.

Uptrace can process billions of spans and metrics on a single server and allows you to monitor your applications at 10x lower cost.

In just a few minutes, you can try Uptrace by visiting the cloud demo (no login required) or running it locally with Docker. The source code is available on GitHub.

Conclusion

Zero-downtime PostgreSQL migrations require careful planning, testing, and coordination. It's crucial to balance the need for making schema changes with the goal of maintaining a highly available and performant application.

PostgreSQL Table Partitioning

Vladimir Mihailenco — Sat, 09 Sep 2023 12:37:04 +0000

This tutorial explains how to use PostgreSQL Table Partitioning.

Why partition a table?

Table partitioning allows to split one large table into smaller ones bringing the following benefits:

Smaller tables are faster both for reading and writing.
You can very efficiently drop the whole partition instead of deleting data row by row.
Because PostgreSQL knows how to prune unused partitions, you can use partitions as a crude index. For example, by paritioning a table by date, you may not need an index on the date field any more and use a sequential scan instead.
Rarely used partitions can be moved to a cheaper storage.

Partitioning methods

Let's suppose we have a table:

CREATE TABLE measurements (
  id int8 NOT NULL,
  value float8 NOT NULL,
  date timestamptz NOT NULL
);

You can partition that table by providing columns to use as the partition key:

CREATE TABLE measurements (
  id int8 NOT NULL,
  value float8 NOT NULL,
  date timestamptz NOT NULL
) PARTITION BY RANGE (date);

PostgreSQL supports several partitioning methods which only differ in the way they specify row values for the partition key.

Partition by range

Partitioning by range allows to specify a range of values for the partition, for example, we can store data for each month in a separate partition:

CREATE TABLE measurements_y2021m01 PARTITION OF measurements
FOR VALUES FROM ('2021-01-01') TO ('2021-02-01');

Partition by list

List partitioning allows to specify a list of values for the partition, for example, we can store small fraction of the frequently accessed data in the hot partition and move the rest to the cold partition:

CREATE TABLE measurements (
  id int8 PRIMARY KEY,
  value float8 NOT NULL,
  date timestamptz NOT NULL,
  hot boolean
) PARTITION BY LIST (hot);

CREATE TABLE measurements_hot PARTITION OF measurements
FOR VALUES IN (TRUE);

CREATE TABLE measurements_cold PARTITION OF measurements
FOR VALUES IN (NULL);

You can then move rows between partitions by updating the hot column:

-- Move rows to measurements_hot
UPDATE measurements SET hot = TRUE;

-- Move rows to measurements_cold
UPDATE measurements SET hot = NULL;

Partition by hash

Partitioning by hash allows to uniformly distribute rows into a set of tables, for example, we can create 3 partitions for our table and pick a partition for the row using a hash and a remainder of division:

CREATE TABLE measurements (
  id int8 PRIMARY KEY,
  value float8 NOT NULL,
  date timestamptz NOT NULL
) PARTITION BY HASH (id);

CREATE TABLE measurements_1 PARTITION OF measurements
FOR VALUES WITH (MODULUS 3, REMAINDER 0);

CREATE TABLE measurements_2 PARTITION OF measurements
FOR VALUES WITH (MODULUS 3, REMAINDER 1);

CREATE TABLE measurements_3 PARTITION OF measurements
FOR VALUES WITH (MODULUS 3, REMAINDER 2);

Thanks to using hashes, the partitions will receive approximately the same amount of rows.

Managing partitions

PostgreSQL allows to detach and attach partitions:

ALTER TABLE measurements DETACH PARTITION measurements_y2021m01;

ALTER TABLE measurements ATTACH PARTITION measurements_y2021m01
FOR VALUES FROM ('2021-01-01') TO ('2021-02-01');

You can use those commands to partition an existing table without moving any data:

-- Use the existing table as a partition for the existing data.
ALTER TABLE measurements RENAME TO measurements_y2021m01;

-- Create the partitioned table.
CREATE TABLE measurements (LIKE measurements_y2021m01 INCLUDING DEFAULTS INCLUDING CONSTRAINTS)
PARTITION BY RANGE (date);

-- Attach the existing partition with open left constraint.
ALTER TABLE measurements ATTACH PARTITION measurements_y2021m01
FOR VALUES FROM ('0001-01-01') TO ('2021-02-01');

-- Use proper constraints for new partitions.
CREATE TABLE measurements_y2021m02 PARTITION OF measurements
FOR VALUES FROM ('2021-02-01') TO ('2021-03-01');

Using partitioned tables with Bun

Bun is a SQL-first Golang ORM (Object-Relational Mapping) that supports PostgreSQL, MySQL, MSSQL, and SQLite. It aims to provide a simple and efficient way to work with databases while utilizing Go's type safety and reducing boilerplate code.

Bun allows to create partitioned tables:

type Measure struct {
    ID    int64
    Value float64
    Date  time.Time
    Hot   bool `bun:",nullzero"`
}

_, err := db.NewCreateTable().
    Model((*Measure)(nil)).
    PartitionBy("LIST (hot)").
    Exec(ctx)

And query partitions directly using ModelTableExpr:

var measures []*Measure
num, err := db.NewSelect(&measures).
    ModelTableExpr("measurements_hot").
    Count(ctx)

You can even create separate models for partitions:

type MeasureHot struct {
    bun.BaseModel `bun:"measures_hot"`
    Measure
}

type MeasureCold struct {
    bun.BaseModel `bun:"measures_cold"`
    Measure
}

PostgreSQL monitoring

To monitor PostgreSQL, you can use OpenTelemetry PostgreSQL receiver that comes with OpenTelemetry Collector.

OpenTelemetry Collector is commonly used for monitoring and observability purposes in modern software applications and distributed systems. It plays a crucial role in gathering telemetry data from various sources, processing that data, and exporting it to monitoring and observability backends for analysis and visualization.

Uptrace is a Grafana alternative that supports distributed tracing, metrics, and logs. You can use it to monitor applications and troubleshoot issues.

Uptrace comes with an intuitive query builder, rich dashboards, alerting rules with notifications, and integrations for most languages and frameworks.

Uptrace can process billions of spans and metrics on a single server and allows you to monitor your applications at 10x lower cost.

In just a few minutes, you can try Uptrace by visiting the cloud demo (no login required) or running it locally with Docker. The source code is available on GitHub.

pgBackRest: PostgreSQL S3 backups

Vladimir Mihailenco — Thu, 10 Aug 2023 11:48:37 +0000

This tutorial explains how to backup PostgreSQL database using pgBackRest and S3.

Introduction

pgBackRest is a modern PostgreSQL Backup & Restore solution that has all the features you may ever need:

Parallel backup and restore.
Full, differential, and incremental backups.
Delta restore.
ZSTD compression.
Encryption.
And many more.

Installation

Ubuntu provides pre-compiled packages for pgbackrest:

sudo apt install pgbackrest

Terms

Stanza is a pgBackRest configuration for a PostgreSQL database cluster. Most db servers only have one db cluster and therefore one stanza.

Repository is where pgBackRest stores backups and archives WAL segments.

Configuration

Let's create a basic directory structure for configs and logs:

mkdir -m 770 /var/log/pgbackrest
chown postgres:postgres /var/log/pgbackrest
mkdir /etc/pgbackrest

And save the following config in /etc/pgbackrest/pgbackrest.conf:

[demo]
pg1-path=/var/lib/postgresql/14/main

[global]
repo1-retention-full=3 # keep last 3 backups
repo1-type=s3
repo1-path=/s3-path
repo1-s3-region=us-east-1
repo1-s3-endpoint=s3.amazonaws.com
repo1-s3-bucket=s3_bucket_name
repo1-s3-key=$AWS_ACCESS_KEY
repo1-s3-key-secret=$AWS_SECRET_KEY

# Force a checkpoint to start backup immediately.
start-fast=y
# Use delta restore.
delta=y

# Enable ZSTD compression.
compress-type=zst
compress-level=6

log-level-console=info
log-level-file=debug

For point-in-time recovery, you also need to configure PostgreSQL to upload WAL files to S3:

archive_mode = on
archive_command = 'pgbackrest --stanza=demo archive-push %p'
archive_timeout = 300

Full backup

Full backup copies all files in a database cluster.

sudo -u postgres pgbackrest --type=full --stanza=demo backup

Differential backup

Differential backup only copies files that have changed since the last full backup. It is smaller than a full backup, but to restore it you will need the base full backup.

sudo -u postgres pgbackrest --type=diff --stanza=demo backup

Incremental backup

Incremental backup only copies files that have changed since the last backup (full, differential, or incremental). It is smaller than a full or differential backup, but to restore it you will need all dependant backups.

sudo -u postgres pgbackrest --type=incr --stanza=demo backup

Backup restore

To restore the cluster from the last backup:

sudo -u postgres pgbackrest --stanza=demo --delta restore

To view all available backups:

sudo -u postgres pgbackrest --stanza=demo info

PostgreSQL monitoring

To monitor PostgreSQL, you can use OpenTelemetry PostgreSQL receiver that comes with OpenTelemetry Collector.

OpenTelemetry Collector is designed to collect, process, and export telemetry data from multiple sources. It acts as a centralized and flexible data pipeline that simplifies the management of telemetry data in distributed systems.

Uptrace is a OpenTelemetry backend that supports distributed tracing, metrics, and logs. You can use it to monitor applications and troubleshoot issues.

Uptrace comes with an intuitive query builder, rich dashboards, alerting rules with notifications, and integrations for most languages and frameworks.

Uptrace can process billions of spans and metrics on a single server and allows you to monitor your applications at 10x lower cost.

In just a few minutes, you can try Uptrace by visiting the cloud demo (no login required) or running it locally with Docker. The source code is available on GitHub.

Conclusion

pgBackRest is a reliable backup tool that requires miminum configuration. To achieve a good balance between backup size and restoration time, you can create a full backup weekly and a differential/incremental backup daily.

Cursor pagination for PostgreSQL/MySQL

Vladimir Mihailenco — Mon, 07 Aug 2023 13:46:32 +0000

Cursor pagination is a useful technique for improving performance and usability of web applications that display large sets of data.

With cursor pagination, the server sends a page of data to the client along with a cursor, which identifies the position of the last item in the page. The client can use this cursor to request the next page of data, passing the cursor as a parameter to the server.

Introduction

Usually, you can paginate through results using LIMIT X OFFSET Y:

SELECT * FROM entries ORDER BY id ASC LIMIT 10 OFFSET 0; -- first page
SELECT * FROM entries ORDER BY id ASC LIMIT 10 OFFSET 10; -- second page
SELECT * FROM entries ORDER BY id ASC LIMIT 10 OFFSET 20; -- third page

Such pagination method works well, but you may notice that the query becomes slower and slower as the offset grows. That happens because OFFSET 100000 tells database to read and discard 100,000 rows which makes performance with large offsets unacceptable. The usual response here is to limit the allowed offset range, for example, you could limit the number of allowed pages to 1000.

But what if you can't limit the number of pages? For example, GitHub must allow users to view all commits in a repo no matter how big a repo can be. The answer is cursor pagination.

Cursor pagination

Cursor-based pagination works by returning to the client a pointer (cursor) to the last item on the page. To get the next page, the client passes the cursor to the server and the server returns results after the given cursor. The main limitation of this approach is that the client can't jump to a specific page and does not know the total number of pages.

::: tip
Cursor-based pagination provides much worse user experience than the classic pagination.
Use it only when you must.
:::

Because the cursor must unambiguously identify the row, you can only use cursor-based pagination on primary keys or columns with an unique constraint. That also ensures that the query uses an index and can quickly skip already paginated rows.

Cursor pagination vs Offset pagination

Compared to traditional page-based pagination, cursor pagination has several advantages:

Performance. Cursor pagination reduces the amount of data that needs to be retrieved from the database, resulting in faster page load times and reduced server load.
Stability. Cursor pagination provides more stable and predictable pagination compared to page-based pagination, which can result in inconsistent pagination if data is added or removed while navigating pages.

All that comes at a cost of reduced flexibility. Cursor pagination does NOT allow users to jump to any point in the data set without having to traverse all previous pages.

Example

Let's paginate the following model using primary key as a pointer:

type Entry struct {
    ID   int64
    Text string
}

Our helper Cursor struct may look like this:

type Cursor struct {
    Start int64 // pointer to the first item for the previous page
    End   int64 // pointer to the last item for the next page
}

To retrieve the next page, we need to continue from the cursor pointing to the last item:

func selectNextPage(ctx context.Context, db *bun.DB, cursor int64) ([]Entry, Cursor, error) {
    var entries []Entry
    if err := db.NewSelect().
        Model(&entries).
        Where("id > ?", cursor).
        OrderExpr("id ASC").
        Limit(10).
        Scan(ctx); err != nil {
        return nil, Cursor{}, err
    }
    return entries, NewCursor(entries), nil
}

To retrieve the previous page, we need to iterate backwards starting from the cursor pointing to the first item:

func selectPrevPage(ctx context.Context, db *bun.DB, cursor int64) ([]Entry, Cursor, error) {
    var entries []Entry
    if err := db.NewSelect().
        Model(&entries).
        Where("id < ?", cursor).
        OrderExpr("id DESC").
        Limit(10).
        Scan(ctx); err != nil {
        return nil, Cursor{}, err
    }
    return entries, NewCursor(entries), nil
}

We can use those methods like this:

page1, cursor, err := selectNextPage(ctx, db, 0)
if err != nil {
    panic(err)
}

page2, cursor, err := selectNextPage(ctx, db, cursor.End)
if err != nil {
    panic(err)
}

prevPage, _, err := selectPrevPage(ctx, db, cursor.Start)
if err != nil {
    panic(err)
}

See example for details.

Monitoring performance

To monitor Bun performance, you can use OpenTelemetry instrumentation that comes with Bun.

By using OpenTelemetry, developers can gain valuable insight into the performance of their applications and the interactions between different components, making it easier to troubleshoot problems, optimize performance, and improve the overall reliability of distributed systems.

Uptrace is a OpenTelemetry backend that supports distributed tracing, metrics, and logs. You can use it to monitor applications and troubleshoot issues.

Uptrace comes with an intuitive query builder, rich dashboards, alerting rules with notifications, and integrations for most languages and frameworks.

Uptrace can process billions of spans and metrics on a single server and allows you to monitor your applications at 10x lower cost.

In just a few minutes, you can try Uptrace by visiting the cloud demo (no login required) or running it locally with Docker. The source code is available on GitHub.

Monitoring Spring Boot with OpenTelemetry

Vladimir Mihailenco — Wed, 02 Aug 2023 12:45:46 +0000

Integrating OpenTelemetry with Spring Boot allows you to capture distributed traces and other telemetry data from your application, providing valuable insights into its performance and behavior in a distributed environment.

What is OpenTelemetry?

OpenTelemetry defines APIs and protocols for collecting telemetry data such as metrics, traces, and logs, and provides a variety of libraries, agents, and integrations for popular programming languages and technologies.

OpenTelemetry is an open and vendor-neutral solution that provides a unified approach to observability, making it easier for organizations to manage the complexity of their cloud-native infrastructure. It enables organizations to collect telemetry from their applications and send it to a variety of of distributed tracing tools.

Conduit app

Spring Boot is a popular Java framework that simplifies the development of Java applications. It provides a convention-over-configuration approach and comes with built-in support for dependency injection, configuration management, and several other features.

RealWorld example app is a full-stack application called "Conduit" that consists of a backend that serves JSON API and a frontend UI. There are numerous implementations for different languages and frameworks, but in this tutorial you will be using the Spring backend and the React frontend.

RealWorld backend

Let's start by downloading the backend source code:

git clone https://github.com/gothinkster/spring-boot-realworld-example-app.git

Then you need to build a JAR from the downloaded source code:

cd spring-boot-realworld-example-app
./gradlew bootJar

If you are getting Could not find snakeyaml-1.27-android.jar (org.yaml:snakeyaml:1.27), apply the following diff to build.gradle and try again:

diff --git a/build.gradle b/build.gradle
index 12781f0..52a8f71 100644
--- a/build.gradle
+++ b/build.gradle
@@ -33,6 +33,7 @@ dependencies {
     implementation 'io.jsonwebtoken:jjwt:0.9.1'
     implementation 'joda-time:joda-time:2.10.6'
     implementation 'org.xerial:sqlite-jdbc:3.34.0'
+    implementation 'org.yaml:snakeyaml:1.28'

Now you can start the app using the compiled JAR:

java -jar build/libs/spring-boot-realworld-example-app-0.0.1-SNAPSHOT.jar

You can check that the backend is working by visiting http://localhost:8080/tags:

curl http://localhost:8080/tags
{"tags":[]}

Let's stop the app for now by pressing CTRL+C.

OpenTelemetry Java Agent

To integrate OpenTelemetry with a Spring Boot application, you can use the OpenTelemetry Java Agent, which provides instrumentation for various Java frameworks, including Spring Boot, to automatically collect telemetry data.

OpenTelemetry Java Agent is a standalone process that provides automatic instrumentation and tracing capabilities for Java applications without requiring any code changes. It works by attaching to a Java application at runtime and intercepting method calls to collect telemetry data such as traces and metrics.

The agent sits between the instrumented application and the backend systems or observability platforms, allowing for centralized and streamlined telemetry data handling.

To download the latest OpenTelemetry Agent:

wget https://github.com/open-telemetry/opentelemetry-java-instrumentation/releases/latest/download/opentelemetry-javaagent.jar

Otel Agent accepts various configuration options either via system properties or environment variables.

Uptrace

Uptrace is an open source APM for OpenTelemetry that supports distributed tracing, metrics, and logs. You can use it to monitor applications and troubleshoot issues.

Uptrace comes with an intuitive query builder, rich dashboards, alerting rules with notifications, and integrations for most languages and frameworks.

Uptrace can process billions of spans and metrics on a single server and allows you to monitor your applications at 10x lower cost.

Uptrace DSN (Data Source Name) is a connection string that is used to connect and send data to an Uptrace backend. You can obtain a DSN after installing Uptrace and creating a project.

Use the following environment variables to configure OpenTelemetry Java to send data to Uptrace.

export OTEL_RESOURCE_ATTRIBUTES=service.name=myservice,service.version=1.0.0
export OTEL_TRACES_EXPORTER=otlp
export OTEL_METRICS_EXPORTER=otlp
export OTEL_LOGS_EXPORTER=otlp
export OTEL_EXPORTER_OTLP_COMPRESSION=gzip
export OTEL_EXPORTER_OTLP_ENDPOINT=https://otlp.uptrace.dev:4317
export OTEL_EXPORTER_OTLP_HEADERS=uptrace-dsn=https://<token>@uptrace.dev/<project_id>
export OTEL_EXPORTER_OTLP_METRICS_TEMPORALITY_PREFERENCE=DELTA
export OTEL_EXPORTER_OTLP_METRICS_DEFAULT_HISTOGRAM_AGGREGATION=BASE2_EXPONENTIAL_BUCKET_HISTOGRAM

Let's start the backend application again, but this time you're going to use the Java Agent to automatically instrument the JAR:

java -javaagent:opentelemetry-javaagent-all.jar -jar build/libs/spring-boot-realworld-example-app-0.0.1-SNAPSHOT.jar

As usually you can open http://localhost:8080/tags to check that API is working.

RealWorld frontend

You have a working backend, but it is not very interesting without a frontend. Let's go ahead and install a React + Redux frontend that will serve as a UI:

git clone https://github.com/gothinkster/react-redux-realworld-example-app.git

The app comes with various JS dependencies:

cd react-redux-realworld-example-app
npm install

Now you need to configure the frontend app to use our backend that is running at http://localhost:8080/. You can do that by editing src/agent.js file:

const API_ROOT = 'http://localhost:8080'

After which you can start the React app and enjoy the UI at http://localhost:4100/register:

npm start

After clicking a few links you should see traces like this one arriving in your Uptrace project:

Conclusion

By using OpenTelemetry Java Agent, you can instrument a Java application without changing a single line of code. Curious to try instrumenting your app? Sign up for a free Uptrace account and follow this tutorial. Good luck.

Monitoring Express.js with OpenTelemetry and Uptrace

Vladimir Mihailenco — Tue, 01 Aug 2023 11:20:04 +0000

What is tracing?

In modern applications, especially those based on microservices or serverless architectures, different services often interact with each other to fulfill a single user request. This makes it challenging to identify performance bottlenecks, diagnose issues, and analyze the overall system behavior.

Distributed tracing aims to address these challenges by creating a trace, which is a representation of a single user request's journey through the various services and components. Each trace consists of a series of interconnected spans, where each span represents an individual operation or activity within a specific service or component.

What is OpenTelemetry?

OpenTelemetry is an open source observability framework hosted by Cloud Native Computing Foundation. It is a merger of OpenCensus and OpenTracing projects.

OpenTelemetry aims to provide a single standard across all types of observability signals such as OpenTemetry logs, distributed tracing, and metrics.

Creating spans

To start creating spans, you need a tracer. You can create a tracer by providing the name and version of the library/application doing the instrumentation:

const otel = require('@opentelemetry/api')

const tracer = otel.trace.getTracer('app_or_package_name', '1.0.0')

Once you have a tracer, creating spans is easy:

// Create a span with name "operation-name" and kind="server".
tracer.startActiveSpan('operation-name', { kind: otel.SpanKind.SERVER }, (span) => {
  doSomeWork()
  span.end()
})

OpenTelemetry Node.js

Using the OpenTelemetry Node.js library, you can easily add distributed tracing capabilities to your Express.js applications.

To instrument an Express.js app, you need to install OpenTelemetry Node.js SDK and available instrumentations:

# Using npm
npm install @opentelemetry/sdk-node @opentelemetry/api @opentelemetry/auto-instrumentations-node

# Using yarn
yarn add @opentelemetry/sdk-node @opentelemetry/api @opentelemetry/auto-instrumentations-node

You can let OpenTelemetry to instrument your application automatically or do it explicitly by installing required instrumentations, for example, OpenTelemetry Express instrumentation:

# Using npm
npm install @opentelemetry/instrumentation-http @opentelemetry/instrumentation-express

# Using yarn
yarn add @opentelemetry/instrumentation-http @opentelemetry/instrumentation-express

Instrumenting Express.js with OpenTelemetry

After installing OpenTelemetry, you need to configure OpenTelemetry SDK to export data to an OpenTelemetry backend for strorage and visualization:

'use strict'

const otel = require('@opentelemetry/api')
const { BatchSpanProcessor } = require('@opentelemetry/sdk-trace-base')
const { Resource } = require('@opentelemetry/resources')
const { NodeSDK } = require('@opentelemetry/sdk-node')
const { OTLPTraceExporter } = require('@opentelemetry/exporter-trace-otlp-http')
const { AWSXRayIdGenerator } = require('@opentelemetry/id-generator-aws-xray')
const { HttpInstrumentation } = require('@opentelemetry/instrumentation-http')
const { ExpressInstrumentation } = require('@opentelemetry/instrumentation-express')

const dsn = process.env.UPTRACE_DSN
console.log('using dsn:', dsn)

const exporter = new OTLPTraceExporter({
  url: 'http://localhost:14318/v1/traces',
  headers: { 'uptrace-dsn': dsn },
  compression: 'gzip',
})
const bsp = new BatchSpanProcessor(exporter, {
  maxExportBatchSize: 1000,
  maxQueueSize: 1000,
})

const sdk = new NodeSDK({
  spanProcessor: bsp,
  resource: new Resource({
    'service.name': 'myservice',
    'service.version': '1.0.0',
  }),
  idGenerator: new AWSXRayIdGenerator(),
  instrumentations: [new HttpInstrumentation(), new ExpressInstrumentation()],
})
sdk.start()

To avoid potential issues, it is strongly recommended to put the OpenTelemetry initialization code into a separate file called otel.js and use the --require flag to load the tracing code before the application code:

# JavaScript
node --require ./otel.js app.js

# TypeScript
ts-node --require ./otel.ts app.ts

See OpenTelemetry Express.js example for details.

What is Uptrace?

Uptrace is a popular DataDog alternative designed to monitor and analyze the performance of microservices-based architectures.

Uptrace provides end-to-end visibility into the flow of requests and transactions across a distributed system, enabling developers and operators to understand and optimize the behavior of their applications.

You can get started with Uptrace by downloading a DEB/RPM package or a pre-compiled Go binary.

What's next?

By integrating OpenTelemetry with Express.js, you can gain insight into your application's distributed traces, OpenTelemetry metrics, and logs. This helps you understand the behavior of your Express.js application across multiple services or microservices, and facilitates troubleshooting and performance optimization.

Next, instrument more operations to get a more detailed picture. Try to prioritize network calls, disk operations, database queries, error and logs.

You can also create your own instrumentations using OpenTelemetry JS Tracing API.

Monitoring Gin and GORM with OpenTelemetry

Vladimir Mihailenco — Tue, 01 Aug 2023 08:06:35 +0000

Still using Jaeger/Sentry? Uptrace is an OpenTelemetry tracing tool that stores data in extremely efficient ClickHouse database and provides powerful filtering and grouping.

What is tracing?
OpenTelemetry and Uptrace
Creating spans
Example application
Configuring OpenTelemetry
Instrumenting Gin
Instrumenting GORM
Recording logs
Running the example
What's next

OpenTelemetry and Uptrace

OpenTelemetry is a vendor-neutral API for distributed traces and metrics. You can use OpenTelemetry to collect traces, errors, and logs. Uptrace stores that data, aggregates and processes it to help you pinpoint failures and find performance bottlenecks.

Uptrace is an open source APM and blazingly fast distributed tracing tool powered by OpenTelemetry and ClickHouse. It is a popular alternative to Jaeger and can be installed by downloading a DEB/RPM package or a pre-compiled binary.

What is tracing?

OpenTelemetry Tracing allows you to see how a request progresses through different services and components, timings of every operation, any logs and errors as they occur. In a distributed environment, tracing also helps you understand relationships and interactions between distributed micro-services and systems.

Using tracing, you can break down requests into spans. Span is an operation (unit of work) your app performs handling a request, for example, a database query or a
network call.

Trace is a tree of spans that shows the path that a request makes through an app. Root span is the first span in a trace.

To learn more about tracing, see Distributed tracing using OpenTelemetry.

Creating spans

You can create spans using OpenTelemetry Go Tracing API like this:



import "go.opentelemetry.io/otel"

var tracer = otel.Tracer("app_or_package_name")

func someFunc(ctx context.Context) error {
    ctx, span := tracer.Start(ctx, "some-func")
    defer span.End()

    // the code you are measuring

    return nil
}

Example application

In this tutorial, you will be instrumenting a toy app that uses Gin router and GORM database client. You can retrieve the source code with the following command:



git clone git@github.com:uptrace/uptrace.git
cd example/gin-gorm

Configuring OpenTelemetry

Uptrace provides OpenTelemetry Go distro that configures
OpenTelemetry SDK for you. To install the distro:



go get github.com/uptrace/uptrace-go

Then you need to initialize the distro whenever you app is started:



import "github.com/uptrace/uptrace-go/uptrace"

uptrace.ConfigureOpentelemetry(
    // copy your project DSN here or use UPTRACE_DSN env var
    //uptrace.WithDSN("https://<key>@uptrace.dev/<project_id>"),

    uptrace.WithServiceName("myservice"),
    uptrace.WithServiceVersion("v1.0.0"),
)

See documentation for details.

Instrumenting Gin

You can instrument Gin router using OpenTelemetry Gin instrumentation provided by OpenTelemetry:



import (
    "github.com/gin-gonic/gin"
    "go.opentelemetry.io/contrib/instrumentation/github.com/gin-gonic/gin/otelgin"
)

router := gin.Default()
router.Use(otelgin.Middleware("service-name"))

otelgin instrumentation will save the active span in the Go context.Context. You can retrieve the context from the http.Request, extract the span from it,
and use the span to record attributes:



func (h *Handler) Index(c *gin.Context) {
    ctx := c.Request.Context()

    // Extract span from the request context.
    span := trace.SpanFromContext(ctx)

    // Check if the span was sampled and is recording the data.
    if span.IsRecording() {
        span.SetAttributes(
            attribute.String("string_key", "string_value"),
            attribute.Int("int_key", 42),
            attribute.StringSlice("string_slice_key", []string{"foo", "bar"}),
        )
    }

    otelgin.HTML(c, http.StatusOK, indexTmpl, gin.H{
        "traceURL": otelplay.TraceURL(trace.SpanFromContext(ctx)),
    })
}

Instrumenting GORM

You can instrument GORM database client using OpenTelemetry GORM instrumentation:



import (
    "github.com/uptrace/opentelemetry-go-extra/otelgorm"
    "gorm.io/gorm"
)

if err := db.Use(otelgorm.NewPlugin()); err != nil {
    panic(err)
}

After the database is instrumented, you should use WithContext method to propagate the active trace context:



user := new(User)
if err := h.db.WithContext(ctx).Where("username = ?", username).First(user).Error; err != nil {
    _ = c.Error(err)
    return
}

Recording logs

You can also record log messages using OpenTelemetry Zap instrumentation for Zap logging library:



// Create Zap logger.
log := otelzap.New(zap.NewExample())

// Extract the active context from the request.
ctx := c.Request.Context()

// Use the logger and the context to record log messages on the active span.
log.Ctx(ctx).Error("hello from zap",
    zap.Error(errors.New("hello world")),
    zap.String("foo", "bar"))

// otelzap also supports an alternative syntax.
log.ErrorContext(ctx, "hello from zap",
    zap.Error(errors.New("hello world")),
    zap.String("foo", "bar"))
}

Running the example

You can start Uptrace backend with a single command using Docker example:



docker-compose up -d

And then start the app passing Uptrace DSN as an env variable:



UPTRACE_DSN=http://project2_secret_token@localhost:14317/2 go run .

The app should be serving requests on http://localhost:9999 and should render a link to Uptrace UI. After opening the link, you should see this:

What's next?

Next, you can learn about OpenTelemetry Go Tracing to create your own instrumentations or browse existing OpenTelemetry instrumentations provided by the community.