DEV Community: dikac

Profiling Memory In Go

dikac — Tue, 17 Dec 2024 02:00:47 +0000

Efficient memory management is critical in Golang applications, particularly in high-concurrency environments, long-running services, or data-intensive tasks. Profiling memory usage helps diagnose issues, optimize performance, and prevent out-of-memory (OOM) errors. This guide provides a comprehensive approach to profiling memory usage from a Go endpoint.

Why Memory Profiling Matters

Memory profiling identifies inefficient memory usage, memory leaks, and excessive allocations in your application. Without proper profiling, memory issues can lead to performance degradation, higher costs, and service downtime.

Common Causes of High Memory Usage

Memory leaks: Unintended memory retention due to data structures not being cleared up.
Excessive allocations: Large slices, maps, or other data structures consuming significant memory.

Setting Up Memory Profiling in Go

To profile memory usage in a Go application, you can use tools like pprof for runtime profiling and parca for continuous profiling. Here’s how to set up and use these tools effectively.

Profiling Tools

pprof

A built-in Go tool that provides profiling for memory, CPU, goroutines, and more.
- Documentation
Parca

A continuous profiling tool that provides real-time insights by collecting data from pprof.
- Documentation
Stress Testing

Generate load to simulate real-world usage and observe memory behavior under stress. For our case we use SoapUI.

Using `pprof`

Since pprof is built-in tool, installation is not required, include the following snippet to enable pprof in your application:

import (
    _ "net/http/pprof"
)

func main() {
    go func () {
    log.Print(http.ListenAndServe(":1234", nil))
    }()
}

This exposes pprof on port 1234. Access profiling data by visiting http://localhost:1234/debug/pprof/ or using tools like go tool pprof.

Using `parca` for Continuous Profiling

To install parca see https://github.com/parca-dev/parca, after successfully installing parca, configure parca.yaml job_name.static_configs.targets set the same port number as pprof (in this example 1234)

then you can run command

parca --config-path="parca.yaml"

if successful you will see message similar to

level=info name=parca ts=2024-10-30T06:19:44.5149184Z caller=factory.go:53 msg="loading bucket configuration"
level=info name=parca ts=2024-10-30T06:19:44.5159183Z caller=badger.go:54 msg="Set nextTxnTs to 0"
level=info name=parca ts=2024-10-30T06:19:44.517917Z caller=server.go:90 msg="starting server" addr=:7070

addr=:7070 is where you can access parca web interface, port number might be different depend on configuration

if all setup successful, you can access parca on web browser

There is a multiple profiling type, for Memory usage you can use

if you encounter any issue, you should consult documentation since different environment might require different solution

pprof https://pkg.go.dev/net/http/pprof
parca https://github.com/parca-dev/parca

Identifying Memory Usage

Stress Testing

Before profiling, simulate high traffic using stress-testing tools in our case we use SoapUI. Stress tests help replicate conditions leading to memory issues.

Analyzing Memory Usage

After completing a stress test, monitor memory usage over time using the parca dashboard.

Click on the graphs to access detailed profiles.

Using the icicle graph, examine the stack and corresponding memory usage. Wider lines indicate higher memory consumption. This visualization helps pinpoint processes consuming significant memory.

In our application, a process with substantial memory usage was identified:

Memory Optimization

Memory optimization is a complex topic that varies depending on the application and its environment. Here are some practical techniques:

Selective Data Loading: Load only the necessary data to significantly reduce memory allocation.
Avoiding Pointers: Use value types instead of pointers to minimize heap allocations.
Predefining Data Lengths: Specify lengths for known-size data structures to improve memory efficiency.

Upon further investigation, we discovered that the data retrieved from the cache was excessively large. We needed to validate whether such a large dataset was truly necessary for our logic flow.

In our case, it turned out that this large dataset was not required. Therefore, we optimized the process by selectively removing unnecessary data. After re-running the tests, memory usage was reduced by approximately 50%.

Previous Implementation

After selectively removing unneeded data

With help of this method we can easily narrow down and correct memory usage, in our case Selective Data Loading is the correct method for reducing memory usage.

Conclusion

Memory profiling is a critical practice for maintaining the performance and stability of Go applications. By leveraging tools like pprof and parca, you can identify memory issues, optimize resource usage, and ensure your application performs reliably under various loads. Regular profiling and proactive optimization help address memory-related challenges effectively.

Handling Currency In Golang And Other Programming Language

dikac — Fri, 28 Jun 2024 20:37:56 +0000

Handling currency in programming languages is a critical aspect of software development, especially for applications dealing with financial transactions, e-commerce, banking, and accounting systems. Accurate representation and manipulation of currency values are essential to avoid errors that can lead to significant financial discrepancies.

This article will explore best practices in handling currency with example in Go languages.

Precision and Accuracy

One of the primary concerns when dealing with currency is precision. Unlike floating-point numbers, which can introduce rounding errors, currency values require precise representation.

consider following code

package main

import "fmt"

func main() {

    var a float64 = 1.1
    var b float64 = 1.2
    var c float64 = 1.3
    fmt.Println(a + b + c)

}

Above code would print

3.5999999999999996

Since memory of computers is limited, most programming language including Go store floating point based on the IEEE-754 standard using 32 or 64 bits, even using 64-bit precision it’s impossible to store an infinite number of digits, which means these numbers are rounded at some point, making them inherently imprecise, and the more calculation is performed the more they become imprecise.

There is many way to handle this problem like using 3rd party library or the programming language you use has a native support. In Go, there are several libraries, including :

credit to kennfatt for curating this library list

Choosing Library

Choosing the right library for handling currency programming involves several considerations. Here's a guide to help you choose an appropriate decimal library for your needs

1. Accuracy and Precision

Ensure the library supports precision and accuracy suit your requirement. Look for arbitrary precision capabilities if dealing with very large numbers or very precise calculations.

2. Ease of Use

The library should have clear and comprehensive documentation. It should be easy to integrate with your existing codebase and workflows.

3. Performance

Consider the performance impact, especially if you are performing a large number of calculations or operating in a high-frequency trading environment.

4. Feature

Ensure the features suit with your needs, from basic arithmetic like rounding, addition, subtraction multiplication, division to more complex math operation.

5. Community and Support

A well-supported library with an active community can be crucial for getting help and finding solutions to problems. Look for libraries that are actively maintained and updated.

Using 3rd Party

This article will use 3rd party library govalues/decimal for code example, since doc simple and easy to read and suite the need for code demonstration

package main

import (
    "fmt"
    "github.com/govalues/decimal"
)

func main() {

    a, _ := decimal.Parse("1.1")
    b, _ := decimal.Parse("1.2")
    c, _ := decimal.Parse("1.3")

    a, _ = a.Add(b)
    a, _ = a.Add(c)
    fmt.Println(a.String())

}

Above code would print

3.6

In above example has no precision loss, still since memory has a cost and not infinite, digit after the decimal point are still limited, controlling decimal digit is important, in this example you can set the size using decimal.ParseExact()

Storing And Retrieving

Storing in databases also requires careful consideration to maintain precision and consistency.

Use Appropriate Column Types

Most relational databases have specific types for currency values, such as DECIMAL or NUMERIC.

Avoid Floating-Point Storage

Just like in programming, avoid storing currency values as floating-point numbers in databases.

For example in MySQL

CREATE TABLE `users` (
    `id` int,
    `balance` decimal(6, 2)
);

Full MySQL Demonstration in go

package main

import (
    "database/sql"
    _ "github.com/go-sql-driver/mysql"
)

func main() {

    db, err := sql.Open("mysql", "root:root@tcp(127.0.0.1:3306)/test")
    if err != nil {
        panic(err)
    }

    _, err = db.Exec(`
        CREATE TABLE test (
            id int,
            balance_decimal decimal(16, 8),
            balance_float float(16, 8),
            PRIMARY KEY (id)
        );
    `)

    _, err = db.Exec("INSERT INTO test (id, balance_decimal, balance_float) VALUES (1, 1.1, 1.1)")
    _, err = db.Exec(`
        UPDATE test 
            SET 
                balance_decimal = balance_decimal + 1.2 , 
                balance_float   = balance_float + 1.2 
            WHERE id = 1;
    `)
}

Above code would produce

id	balance_decimal	balance_float
1	2.30000000	2.29999995

Data Transfer

Data transfer also requires careful consideration of precision. a correct format is required

For example in JSON format, using type string guaranteed precision across any programing language

package main

import (
    "encoding/json"
    "log"
)

type Data struct {
    Decimal string  `json:"decimal"`
    Float   float64 `json:"float"`
}

func main() {

    payload := []byte(`{"decimal":"999.99999999999999","float":999.99999999999999}`)

    result := Data{}

    _ = json.Unmarshal(payload, &result)

    log.Print("Decimal: ", result.Decimal)
    log.Print("Float: ", result.Float)

}

Above code would print

Decimal: 999.99999999999999
Float: 1000

Conclusion

Handling currency in programming languages requires careful attention to detail to ensure precision, accuracy, and consistency. By using appropriate data types, libraries, and best practices, developers can avoid common pitfalls and ensure their applications handle currency correctly. Proper handling of currency not only prevents financial errors but also enhances user trust and confidence in the application.

There's no single best library for everyone. Each project has different needs. You should think about what's good and bad about each library based on what your project requires.

External Link

Handling RabbitMQ Reconnections With Go and amqp091-go

dikac — Tue, 11 Jun 2024 09:15:42 +0000

In a distributed system, maintaining a stable connection between components is crucial for ensuring reliable communication. This article explores strategies for reconnecting to RabbitMQ, ensuring that your applications can recover from network disruptions or broker restarts without losing valuable data.

Setting Up

Importing Dependencies

import (
    "fmt"
    amqp "github.com/rabbitmq/amqp091-go"
    "log"
    "os"
    "os/signal"
    "syscall"
    "time"
)

We use amqp091-go as a connection for RabbitMQ

Preparing option

type Config struct {
    username string
    password string
    host     string
    port     int
}

type Container struct {
    connection *amqp.Connection
}

type Channel struct {
    name       string
    kind       string
    durable    bool
    autoDelete bool
    internal   bool
    noWait     bool
    args       amqp.Table
}

Config holds connection configuration.
Container holds the connection to RabbitMQ. In each application, they should have a wrapper for holding the connection.
Channel holds channel configuration.

Connection Creation

// dial 
// create connection and register channel
func dial(config Config, channels []Channel) (conn *amqp.Connection, err error) {

    // create connection
    conn, err = amqp.Dial(fmt.Sprintf("amqp://%s:%s@%s:%d", config.username, config.password, config.host, config.port))
    if err != nil {
        return
    }
    log.Print("rabbitmq connect success")

    // register channel
    channel, err := conn.Channel()
    for _, c := range channels {
        err = channel.ExchangeDeclare(c.name, c.kind, c.durable, c.autoDelete, c.internal, c.noWait, c.args)
        if err != nil {
            return
        }
    }
    log.Print("rabbitmq channels register success")

    return
}

We create a Go function dial to assist in establishing a connection to RabbitMQ. It takes in a Config struct containing connection details like username, password, host, and port, and a slice of Channel structs representing different channels to be registered with the server.

Connection Status

func check(container *Container) {
    for {
        time.Sleep(time.Duration(1) * time.Second)
        log.Printf("rabbitmq connected: %v", !container.connection.IsClosed())
    }
}

We create a Go function check to check and print connection status periodically, to have visibility of the connection status.

Reconnection Code

Below is the full code containing reconnection code. We will walk through step by step to understand how reconnection works in Go with RabbitMQ.

func reconnect(container *Container, config Config, channels []Channel) {

    go func() {
        for {
            reason, ok := <-container.connection.NotifyClose(make(chan *amqp.Error))
            if !ok {
                log.Print("rabbitmq connection closed")
                break
            }
            log.Printf("rabbitmq connection closed unexpectedly, reason: %v", reason)

            for {

                time.Sleep(time.Duration(1) * time.Second)

                connection, err := dial(config, channels)

                if err == nil {
                    container.connection = connection
                    log.Print("rabbitmq reconnect success")
                    break
                }

                log.Printf("rabbitmq reconnect failed, err: %v", err)
            }

        }
    }()
}

This function is a reconnect loop for a RabbitMQ client, and since reconnection creates a brand-new connection, the reconnect function is required to have all the values like the initial connection.

It starts a goroutine to prevent blocking that listens for the closure of the connection.
When the connection is closed, it enters a loop to attempt reconnection.
It uses a delay of 1 second (time.Sleep(time.Duration(1) * time.Second)) between reconnection attempts.
It attempts to reconnect by calling the dial function with the provided configuration and channels.
If the reconnection is successful, it updates the container.connection with the new connection.
If the reconnection fails, it logs the error and continues the loop.

Stitching Together

func main() {

    config := Config{
        username: "guest",
        password: "guest",
        host:     "localhost",
        port:     5672,
    }

    channels := []Channel{
        {
            name:       "reconnection-exchange",
            kind:       "direct",
            durable:    true,
            autoDelete: false,
            internal:   false,
            noWait:     false,
            args:       nil,
        },
    }

    container := &Container{}

    connection, err := dial(config, channels)
    if err != nil {
        log.Fatal(err)
        return
    }
    container.connection = connection
    go check(container)

    defer func() {
        if err := container.connection.Close(); err != nil {
            log.Print(err)
        }
        log.Print("rabbitmq connection closed")

    }()

    reconnect(container, config, channels)

    {
        quit := make(chan os.Signal, 1)
        signal.Notify(quit, os.Interrupt, syscall.SIGINT, syscall.SIGTERM)
        <-quit

    }
}

Now we combine all code together in main function.

Testing

Running

Before running the code, make sure RabbitMQ is running and accessible. Upon running the code, you should see a message like the one below.

rabbitmq connect success
rabbitmq channels register success
rabbitmq connected: true
rabbitmq connected: true
rabbitmq connected: true
rabbitmq connected: true
rabbitmq connected: true
...

This message indicates that the RabbitMQ connection is connected normally.

Disconnection & Down

After running and connected normally, you can emulate disconnection & server down by manually shutting down the RabbitMQ server. After the server stops, you should see a message like the one below.

rabbitmq connection closed unexpectedly, reason: Exception (320) Reason: "CONNECTION_FORCED - broker forced connection closure with reason 'shutdown'"
rabbitmq connected: false
rabbitmq connected: false
rabbitmq connected: false
rabbitmq connected: false
rabbitmq reconnect failed, err: dial tcp [::1]:5672: connectex: No connection could be made because the target machine actively refused it.
rabbitmq connected: false
rabbitmq connected: false
rabbitmq connected: false
rabbitmq reconnect failed, err: dial tcp [::1]:5672: connectex: No connection could be made because the target machine actively refused it.
rabbitmq connected: false
...

This message indicates disconnection from the RabbitMQ connection, and you should set up an alert for this event.

Reconnection

After disconnection, you can emulate reconnection by running the RabbitMQ server again. After the server is up and ready, and the reconnection mechanism is successful, you should see a message like the one below.

rabbitmq connect success
rabbitmq channels register success
rabbitmq reconnect success
rabbitmq connected: true
rabbitmq connected: true
rabbitmq connected: true
rabbitmq connected: true
rabbitmq connected: true
rabbitmq connected: true

Conclusion

In conclusion, the article demonstrates a robust strategy for handling RabbitMQ reconnections in Go, ensuring reliable communication in distributed systems. By implementing a reconnection loop and monitoring the connection status, the application can gracefully recover from network disruptions or broker restarts. This approach enhances the resilience of the system, ensuring that important data is not lost due to temporary connection issues. Using the amqp091-go library, developers can easily implement these reconnection mechanisms, improving the overall reliability of their RabbitMQ-based applications.

External Link

How To Solve Merge Conflict With Intermediate Branch

dikac — Thu, 18 Apr 2024 07:22:36 +0000

Introduction

Usually, in a company, there are several developers, repositories, and branches. The more features that are developed, the more complex Git branching becomes, so there needs to be a correct Git branching mechanism to make the process of merging code easier.

This will explain git method to prevent tainting branch from other branch changes, making branch clean and can be directly merged to master branch, avoiding cherry-pick and reduce Merge Complexity.

Knowledge Requirement

This document assume that you have basic knowledge of git commit, merge, branch, and merge conflict.

Here some basic tutorial link:

Problem

The issue came up when two or more people were working on the same shared branches like master (for the production server) and staging/dev (for the testing server), and pushing changes directly to these branches was not allowed.

This create a situation where staging/dev contain changes from other people and might not yet stable, thus cannot be merged to master, and to your branch because your branch eventually will be merged to branch.

For example:
you checkout from master to develop new functionality to calendar feature, after finishing the development you PR to staging, but at the same time Steve (other developer) working on calendar feature to improve performance, and both Steve and you happen to change the same file. Now Steve or You cannot directly merge/rebase into staging, now imagine if there is more than 2 developer.

Example No Conflict

If the branch does not conflict with staging/dev on PR You don't need to use special strategy, You can directly PR to main & staging/dev without any conflict.

This happens if changes happen in different file, and if up-to branch date is not enforced.

Branch created from master, after branch updated, PR has no merge conflict with master/dev and staging/dev, therefore intermediate branch can be skipped.

Example Conflict

If the Branch PR has merge conflict staging/dev, you cannot proceed the merge, merging/rebase from staging/dev to Branch will make Branch contain changes from staging/dev that may not stable/final yet

Branch created from master, after branch updated, PR has merge conflict with staging/dev, therefore intermediate branch is needed

If you merging/rebase from staging/dev to Branch, then you cannot directly merge Branch to Master because Branch contain changes from staging/dev that may not stable/final.

Branch created from master, after branch updated, PR has merge conflict with staging/dev, merging staging/dev to Branch cause Branch contain changes staging/dev

Solution

One solution is using new Intermediate branch to contain merge conflict resolution between branch and staging/dev

Intermediate branch accept changes from Branch or staging/dev, but not pushing changes to Branch, and PR to Staging/Dev using Intermediate Branch

Branch created from master, after branch updated, PR has merge conflict with staging/dev, create intermediate branch from Branch, merging staging/dev to intermediate branch will not change Branch, Branch are clean and can directly PR to Master

Now with intermediate branch, Branch does not contain changes from staging/dev, and branch is clean and can be directly PR to master.

Now each developers Branch does not contain changes from other developer Branch

Each developers Branch are clean and can directly PR to Master

Git workflow visualized

Pros & Cons

Pros

Branch clean can be directly PR to master
No cherry picking

Cons

Increase branch number
In rare case you need to resolve the same conflict multiple times

Conclusion

By using intermediate branch able to make master/branch clean in our use case. There is many branching model for git, this model works particularly well in our use case. You should also explore another branching model/strategy option, to make inform decision to decide which is best for your use case.

Legend

--> = merge/rebase
Branch = Working branch outside master, staging/dev for feature, hotfix or anything.
⚪ = Commit

DEV Community: dikac

Profiling Memory In Go

Why Memory Profiling Matters

Common Causes of High Memory Usage

Setting Up Memory Profiling in Go

Profiling Tools

Using pprof

Using parca for Continuous Profiling

Identifying Memory Usage

Stress Testing

Analyzing Memory Usage

Memory Optimization

Conclusion

Handling Currency In Golang And Other Programming Language

Precision and Accuracy

Choosing Library

1. Accuracy and Precision

2. Ease of Use

3. Performance

4. Feature

5. Community and Support

Using 3rd Party

Storing And Retrieving

Use Appropriate Column Types

Avoid Floating-Point Storage

Full MySQL Demonstration in go

Data Transfer

Conclusion

External Link

Handling RabbitMQ Reconnections With Go and amqp091-go

Setting Up

Importing Dependencies

Preparing option

Connection Creation

Connection Status

Reconnection Code

Stitching Together

Testing

Running

Disconnection & Down

Reconnection

Conclusion

External Link

How To Solve Merge Conflict With Intermediate Branch

Introduction

Knowledge Requirement

Problem

Example No Conflict

Example Conflict

Solution

Pros & Cons

Pros

Cons

Conclusion

Legend

Using `pprof`

Using `parca` for Continuous Profiling