DEV Community: li-jin-ou

Hertz how to use Opentelemetry for beginners

li-jin-ou — Mon, 28 Nov 2022 13:12:33 +0000

Introduction

In the previous article we focused on the use of hlog in conjunction with requestid to associate logs.
Hertz now also provides extended support for Opentelemetry obs-opentelemetry to facilitate the use of Opentelemetry in the Hertz service.

How to use

The Opentelemetry extension to Hertz provides three main functions: Metric, Tracing and Logging.

Metric

Use Opentelemetry's Metric at Hertz.

It supports HTTP Metrics and Runtime Metrics, more details can be found here.

import (
    "github.com/hertz-contrib/obs-opentelemetry/provider"
    hertztracing "github.com/hertz-contrib/obs-opentelemetry/tracing"
)

func main() {
    // ...
  p := provider.NewOpenTelemetryProvider(
            provider.WithServiceName(serviceName), 
            // Support setting ExportEndpoint via environment variables: OTEL_EXPORTER_OTLP_ENDPOINT 
            provider.WithExportEndpoint("localhost:4317"),
            provider.WithInsecure(), 
    )
    defer p.Shutdown(context.Background())
    tracer, cfg := hertztracing.NewServerTracer()
    h := server.Default(tracer)
    h.Use(hertztracing.ServerMiddleware(cfg))
    // ...
}

The results are as follows:

Tracing

Hertz integrates with Opentelemetry's tracing.

import (
    ...
    "github.com/hertz-contrib/obs-opentelemetry/provider"
    "github.com/hertz-contrib/obs-opentelemetry/tracing"
)


func main()  {
    serviceName := "echo"

    p := provider.NewOpenTelemetryProvider(
        provider.WithServiceName(serviceName),
        provider.WithExportEndpoint("localhost:4317"),
        provider.WithInsecure(),
    )
    defer p.Shutdown(context.Background())

    tracer, cfg := hertztracing.NewServerTracer()
    h := server.Default(tracer)
    h.Use(hertztracing.ServerMiddleware(cfg))

    ...

    h.Spin()
}

The results are as follows:

Logger

Hertz provides logger for association with tracing.

import (
    hertzlogrus "github.com/hertz-contrib/obs-opentelemetry/logging/logrus"
)

func init()  {
    hlog.SetLogger(hertzlogrus.NewLogger())
    hlog.SetLevel(hlog.LevelDebug)

}

func main() {
    //...
    h.GET("/ping", func(c context.Context, ctx *app.RequestContext) {
        req := &api.Request{Message: "my request"}
        resp, err := client.Echo(c, req)
        if err != nil {
            hlog.Errorf(err.Error())
            return
        }
        hlog.CtxDebugf(c, "message received successfully: %s", req.Message)
        ctx.JSON(consts.StatusOK, resp)
    })
    //...
}

The results are as follows:

{"level":"debug","msg":"message received successfully: my request","span_id":"445ef16484a171b8","time":"2022-07-04T06:27:35+08:00","trace_flags":"01","trace_id":"e9e579b32c9d6b0598f8f33d65689e06"}

Example Code

Example: https://github.com/cloudwego/hertz-examples/tree/main/opentelemetry
Documentation: https://www.cloudwego.io/docs/hertz/tutorials/service-governance/Opentelemetry/

Summary

The Hertz provides the Opentelemetry extension obs-opentelemetry to facilitate the integration of Metric, Tracing and Logger, in the next article we will use Gorm's Open-Telemetry extension together with Hertz's Open-Telemetry extension to develop A project.

Reference

Hertz client's rich retry functionality

li-jin-ou — Wed, 09 Nov 2022 04:08:32 +0000

Hertz client's rich retry functionality

Hertz is an ultra-large-scale enterprise-level microservice HTTP framework, featuring high ease of use, easy expansion, and low latency etc.

Hertz uses the self-developed high-performance network library Netpoll by default. In some special scenarios, Hertz has certain advantages in QPS and latency compared to go net.

Hertz client provides users with customized retry logic,Let's take a look at how to use Client Retry(This section mainly configures the times of retry and the delay policy )

Retry times and delay policy configuration

First create a Client, and use the configuration item WithRetryConfig() to configure the Retry related logic

package main

import (
    "github.com/cloudwego/hertz/pkg/app/client"
    "github.com/cloudwego/hertz/pkg/app/client/retry"
)

func main() {
    cli, err := client.NewClient(
        client.WithRetryConfig(
            retry.WithXxx(), // the method of setting retry config
        ),
    )
}

Configuration Name	Type	Description
WithMaxAttemptTimes	uint	Set the maximum attempts times. Default：1 times (That is, only request once without retry)
WithInitDelay	time.Duration	Set initial delay time. Default: 1ms
WithMaxDelay	time.Duration	Set the maximum delay time. Default: 100ms
WithMaxJitter	time.Duration	Set the maximum jitter time, which needs to be used in conjunction with `RandomDelayPolicy`, and will generate a random time not exceeding the maximum jitter time. Default: 20ms
WithDelayPolicy	type DelayPolicyFunc func(attempts uint, err error, retryConfig *Config) time.Duration	Set the delay policy, you can use any combination of the following four policies, `FixedDelayPolicy`, `BackOffDelayPolicy`, `RandomDelayPolicy`, `DefaultDelayPolicy` ( See the next section Delay Policy for details ) . Default: DefaultDelayPolicy (That is, the retry delay is 0)

Delay Policy

retry.WithDelayPolicy() usage

cli, err := client.NewClient(
        client.WithRetryConfig(
            ...
            retry.WithDelayPolicy(retry.CombineDelay(retry.FixedDelayPolicy, retry.BackOffDelayPolicy, retry.RandomDelayPolicy)),
            ...
        ),
    )

Function Name	Description
CombineDelay	It is used to combine any of the following four policies and sum the values calculated by the selected policy. When you only need one of the following four policies, you can choose to use `CombineDelay` or directly pass any policy into `WithDelayPolicy` as a parameter
FixedDelayPolicy	Set the fixed delay time and use the value set by `WithInitDelay` to generate an equivalent delay time
BackOffDelayPolicy	Set the exponential delay time. Use the value set by `WithInitDelay`. The exponential delay time is generated according to the number of retries currently
RandomDelayPolicy	Set the random delay time. Use the value set by `WithMaxJitte`r to generate a random delay time that does not exceed this value
DefaultDelayPolicy	Set the default delay time (That is, 0) . Generally, it is used alone and has no effect when combined with other policies

Complete example

package main

import (
    "github.com/cloudwego/hertz/pkg/app/client"
    "github.com/cloudwego/hertz/pkg/app/client/retry"
)
func main() {

    cli, err := client.NewClient(
        client.WithRetryConfig(
            retry.WithMaxAttemptTimes(3), // Maximum number of attempts, including initial calls
            retry.WithInitDelay(1*time.Millisecond), // Initial delay
            retry.WithMaxDelay(6*time.Millisecond), // Maximum delay.No matter how many retries and what the policy is, the delay will not exceed this delay
            retry.WithMaxJitter(2*time.Millisecond), // Maximum jitter delay, which will have effect only when combined with RandomDelayPolicy
            /*
               To configure the delay policy, you can select any combination of the following four, and the final result is the sum of each delay policy.
               FixedDelayPolicy uses the value set by retry.WithInitDelay,
               BackOffDelayPolicy increases exponentially with the number of retries based on the value set by retry.WithInitDelay,
               RandomDelayPolicy generates a random value of [0, 2*time.Millisecond). 2*time.Millisecond is the value set by retry.WithMaxJitter,
               DefaultDelayPolicy generates a value of 0. If it is used alone, retry again immediately,
               retry.CombineDelay() sums the values generated by the set delay policy, and the final result is the delay time of the current retry,
               The first call failed -> Retry delay：1 + 1<<1 + rand[0,2)ms -> The second call failed -> Retry delay：min(1 + 1<<2 + rand[0,2) , 6)ms -> The third call succeeded/failed
            */
            retry.WithDelayPolicy(retry.CombineDelay(retry.FixedDelayPolicy, retry.BackOffDelayPolicy, retry.RandomDelayPolicy)),
        ),
    )
}

Retry condition configuration

If you want to customize the conditions under which retries occur, you can use client SetRetryIfFunc() configuration. The parameter of this function is a function, and the signature is:

func(req *protocol.Request, resp *protocol.Response, err error) bool

Relevant parameters include the req, resp and err fields in the Hertz request. You can use these parameters to determine whether the request should be retried. In the following example, when the status code returned by the request is not 200 or err!=nil during the call, we return true, that is, we retry.

cli.SetRetryIfFunc(func(req *protocol.Request, resp *protocol.Response, err error) bool {
   return resp.StatusCode() != 200 || err != nil
})

Note that if you do not set client SetRetryIfFunc() . We will judge according to Hertz's default retry conditions, that is, whether the request meets the following DefaultRetryIf() function and whether the call is idempotent. ( Idempotent call: when canIdempotentRetry is true in pkg/protocol/http1/client.go::Do() and pkg/protocol/http1/client.go::doNonNilReqResp() )

// DefaultRetryIf Default retry condition, mainly used for idempotent requests.
// If this cannot be satisfied, you can implement your own retry condition.
func DefaultRetryIf(req *protocol.Request, resp *protocol.Response, err error) bool {
   // cannot retry if the request body is not rewindable
   if req.IsBodyStream() {
      return false
   }

   if isIdempotent(req, resp, err) {
      return true
   }
   // Retry non-idempotent requests if the server closes
   // the connection before sending the response.
   //
   // This case is possible if the server closes the idle
   // keep-alive connection on timeout.
   //
   // Apache and nginx usually do this.
   if err == io.EOF {
      return true
   }

   return false
}
func isIdempotent(req *protocol.Request, resp *protocol.Response, err error) bool {
   return req.Header.IsGet() ||
      req.Header.IsHead() ||
      req.Header.IsPut() ||
      req.Header.IsDelete() ||
      req.Header.IsOptions() ||
      req.Header.IsTrace()
}

When canIdempotentRetry in Hertz source code doNonNilReqResp() is true.

doNonNilReqResp() return true
err = conn.SetWriteDeadline(currentTime.Add(c.WriteTimeout))
err = reqI.Write(req, zw)
err = reqI.ProxyWrite(req, zw)
err = zw.Flush()
err = conn.SetReadTimeout(c.ReadTimeout)
( err = respI.ReadHeaderAndLimitBody() \

References

HTTP request ID association with logs

li-jin-ou — Fri, 04 Nov 2022 16:06:41 +0000

Introduction

Hertz is an ultra-large-scale enterprise-level microservice HTTP framework, featuring high ease of use, easy expansion, and low latency etc.

Hertz uses the self-developed high-performance network library Netpoll by default. In some special scenarios, Hertz has certain advantages in QPS and latency compared to go net. For performance data, please refer to the echo data below.

In internal practice, for some typical services, such as services with a high proportion of frameworks, gateways and other services, after migrating Hertz, compared to the Gin framework, the resource usage is significantly reduced. For more information, see hertz-benchmark.

Hertz provides requestid middleware、built-in hlog log library and some hlog log component extensions, this article focuses on how to associate request IDs with logs to make it easier for users to find logs.

Hands-on

Introduction to the Request ID middleware

The requestid middleware for Hertz is based on the Gin framework's requestid and adapted to Hertz. Its main purpose is to add rquestid to the HTTP response and context of a request to uniquely identify a HTTP request.

It is used in the following way:

package main

import (
    "context"

    "github.com/cloudwego/hertz/pkg/app"
    "github.com/cloudwego/hertz/pkg/app/server"
    "github.com/cloudwego/hertz/pkg/common/utils"
    "github.com/cloudwego/hertz/pkg/protocol/consts"
    "github.com/hertz-contrib/requestid"
)

func main() {
    h := server.Default()

    h.Use(requestid.New())

    // Example ping request.
    h.GET("/ping", func(ctx context.Context, c *app.RequestContext) {
        c.JSON(consts.StatusOK, utils.H{"ping": "pong"})
    })

    h.Spin()
}

Accessing 127.0.0.1:8888/ping, we will see an extra X-request-ID field in the HTTP response header.

Hlog extensions

Hertz also provides hlog for printing the framework's internal logs. Users can use this in simple log printing scenarios.

The default hlog is based on the log package implementation and has normal performance, while Hertz provides the logger extension, which provides zap and logrus implementations.

The logrus extension is used in the following way:

package main

import (
    "context"

    "github.com/cloudwego/hertz/pkg/common/hlog"
    hertzlogrus "github.com/hertz-contrib/logger/logrus"
)

func main() {
    logger := hertzlogrus.NewLogger()
    hlog.SetLogger(logger)
    hlog.CtxInfof(context.Background(), "hello %s", "hertz")
}

Practical code

Associate the log of a request by using the requestid middleware with the logger extension.

Custom Hooks

Logrus supports a user-defined Hook that can print requestid in the log by implementing a custom Hook.

// Custom Hook
type RequestIdHook struct{}

func (h *RequestIdHook) Levels() []logrus.Level {
    return logrus.AllLevels
}

func (h *RequestIdHook) Fire(e *logrus.Entry) error {
    ctx := e.Context
    if ctx == nil {
        return nil
    }
    value := ctx.Value("X-Request-ID")
    if value != nil {
        e.Data["log_id"] = value
    }
    return nil
}

Full code

package main

import (
    "context"

    "github.com/cloudwego/hertz/pkg/app"
    "github.com/cloudwego/hertz/pkg/app/server"
    "github.com/cloudwego/hertz/pkg/common/hlog"
    "github.com/cloudwego/hertz/pkg/common/utils"
    "github.com/cloudwego/hertz/pkg/protocol/consts"
    hertzlogrus "github.com/hertz-contrib/logger/logrus"
    "github.com/hertz-contrib/requestid"
    "github.com/sirupsen/logrus"
)

type RequestIdHook struct{}

func (h *RequestIdHook) Levels() []logrus.Level {
    return logrus.AllLevels
}

func (h *RequestIdHook) Fire(e *logrus.Entry) error {
    ctx := e.Context
    if ctx == nil {
        return nil
    }
    value := ctx.Value("X-Request-ID")
    if value != nil {
        e.Data["log_id"] = value
    }
    return nil
}

func main() {
    h := server.Default()
    logger := hertzlogrus.NewLogger(hertzlogrus.WithHook(&RequestIdHook{}))
    hlog.SetLogger(logger)

    h.Use(requestid.New())

    // Example ping request.
    h.GET("/ping", func(ctx context.Context, c *app.RequestContext) {
        hlog.CtxInfof(ctx, "test log")
        c.JSON(consts.StatusOK, utils.H{"ping": "pong"})
    })
    h.Spin()
}

Effect

{"level":"info","msg":"HERTZ: Using network library=netpoll","time":"2022-11-04T13:58:51+08:00"}
{"level":"info","msg":"HERTZ: HTTP server listening on address=[::]:8888","time":"2022-11-04T13:58:51+08:00"}
{"level":"info","log_id":"8f0012a3-f97b-49ca-b13b-1f009585b5d9","msg":"test log","time":"2022-11-04T13:59:11+08:00"}

In this way we associate the log of an HTTP request with requstid. In fact Hertz provides more powerful capabilities, which we will cover in the next article. You can check out obs-opentelemetry in advance if you are interested.