DEV Community: immortalt

Use pprof for golang program memory analysis

immortalt — Fri, 03 Dec 2021 03:23:42 +0000

When using golang to write complex projects, it is often useful to use multi-coroutine concurrency scenarios. At this
time, it is easy to cause the problem of coroutine leaks due to negligence, and then produce similar memory leaks. This
article focuses on the investigation of coroutine leaks, and provides ideas and practices for visual analysis of golang
program memory.

Introduction to pprof

pprof is a tool for visualization and analysis of profiling data.

pprof reads a collection of profiling samples in profile.proto format and generates reports to visualize and help
analyze the data. It can generate both text and graphical reports (through the use of the dot visualization package).

How to use pprof

Add monitoring code

First, we need to add monitoring code in the golang program, and expose it through the http interface.

package main

import _ "net/http/pprof"
import "net/http"

func main() {
    go func() {
        _ = http.ListenAndServe("0.0.0.0:8081", nil)
    }()
    // your code
}

Then we start the program that needs to be analyzed, and we are ready to analyze it.

How to check the memory size of each module

By analyzing the size of the memory occupied by each module and function, memory leaks can be found very effectively.

Command line method to generate visual analysis images

go tool pprof -alloc_space -cum http://localhost:8081/debug/pprof/heap

After the command is run, enter web in the console and press Enter, and a svg picture will be opened with the
default viewing software of .svg, showing the memory usage diagram of each module. If you enter web and report an
error of Failed to execute dot. Is Graphviz installed? Error: exec: "dot": executable file not found in %PATH%, it is
because Graphviz is not installed on the computer, which is a component that image generation depends on. The solution
is: Open https://graphviz.gitlab.io/download/ and follow the prompts to download and install. After the installation is
complete, for Windows, add the bin folder of the Graphviz installation path after setting the environment variable
path.

View specific data list in web browser

http://localhost:8081/debug/pprof/heap?debug=1

How to view the number of coroutines created by each module

By analyzing the number of coroutines created by each module and function, coroutine leaks can be checked very
effectively. If there is coroutine leaks, the number of coroutines in the corresponding modules is astonishing.

Command line method to generate visual analysis images

go tool pprof http://localhost:8081/debug/pprof/goroutine

After the command runs, enter web in the console and press Enter.

View specific data list in web browser

http://localhost:8081/debug/pprof/goroutine?debug=1

Summary

The above is an introduction to the simple use of pprof, I believe it will be helpful to troubleshoot memory leaks and
coroutine leaks in golang. If you need more detailed usage, please refer to the official pprof documentation.

Common operations of Git

immortalt — Fri, 03 Dec 2021 03:22:57 +0000

Commit code

git add .
git commit -m "feat: xxxxxx"

How to revoke after commit

git reset --soft HEAD^

This successfully revoked the commit. Note that only the commit operation is
withdrawn, and the code modification remains.

git reset parameters

--mixed
do not delete the workspace to change the code, cancel the commit, and cancel the git add . operation. This is the
default parameter.

--soft
do not delete the workspace to change the code, cancel the commit, and do not cancel the git add ..

--hard
delete workspace change code, revoke commit, revoke git add .. After completing this operation, it is restored to the
last commit state.

How to migrate git repository to another git

git remote set-url origin remote_git_address
git push

This method will only migrate the current branch to the new git.

How to save username and password of git in Ubuntu

nano ~/.gitconfig

Edit the config file and set this config:

[credential]  
    helper = store

Install Node.js on Linux

immortalt — Fri, 03 Dec 2021 03:22:31 +0000

Using NodeSource

Refer to the NodeSource documentation for more
information on the available versions.

Node.js v14.x:

# Using Ubuntu
curl -fsSL https://deb.nodesource.com/setup_14.x | sudo -E bash -
sudo apt-get install -y nodejs

# Using Debian, as root
curl -fsSL https://deb.nodesource.com/setup_14.x | bash -
apt-get install -y nodejs

Upgrade npm

npm install -g npm

Monitor status of nvme ssd in Ubuntu

immortalt — Fri, 03 Dec 2021 03:21:57 +0000

Install nvme-cli

sudo apt-get install nvme-cli

List disks

sudo nvme list

View the details of the hard drive

sudo nvme smart-log /dev/nvme0

Auto refresh status

sudo watch -n 1 nvme smart-log /dev/nvme0

Only view the temperature info

sudo nvme smart-log /dev/nvme0 | grep "^temperature"

Online expansion of Ubuntu LVM disk

immortalt — Fri, 03 Dec 2021 03:19:56 +0000

LVM overview

Physical Volume (PV)

Refers to a disk partition, or a device (such as RAID) that has the same function as a disk partition. It is the basic
storage logical block of LVM, but compared with basic physical storage media (such as partitions, disks, etc.), it
contains LVM-related Management parameters.

Volume Group (VG)

Similar to a physical disk in a non-LVM system, it is composed of one or more physical volumes PV. One or more LVs (
logical volumes) can be created on the volume group.

Logical Volume (LV)

Similar to disk partitions in non-LVM systems, logical volumes are built on the volume group VG. A file system (such as
/home or /usr, etc.) can be established on the logical volume LV.

LVM expansion operation

Create a new primary partition

If you expanded the hard disk size of the virtual machine, and you are still using one hard disk such as /dev/sda, you
can do this:

sudo fdisk /dev/sda

Enter m to view the help, enter n to create a new partition, select the primary partition, step by step, press w
to write and save, and get /dev/sda2.

You can also add a new hard drive to the machine, and the new hard drive may be shown as /dev/sdb. The operation of
creating a new partition is similar.

Format the partition as ext4 format

sudo mkfs -t ext4 /dev/sda2

/dev/sda2 represents the new partition, you can replace it with other paths according to your situation, for
example /dev/sdb1.

Find the VG Name

~# vgdisplay
  --- Volume group ---
  VG Name               ubuntu--vg-root
  System ID
  Format                lvm2
  ......

Add the new partition to the original VG of lvm

vgextend ubuntu--vg-root /dev/sda2

Check the VG

You should find that there are some free space in the volume group.

~# vgdisplay
  --- Volume group ---
  VG Name               ubuntu--vg-root
  System ID
  Format                lvm2
  Metadata Areas        2
  Metadata Sequence No  5
  VG Access             read/write
  VG Status             resizable
  MAX LV                0
  Cur LV                3
  Open LV               3
  Max PV                0
  Cur PV                2
  Act PV                2
  VG Size               758.99 GiB
  PE Size               4.00 MiB
  Total PE              194302
  Alloc PE / Size       15103 / <59.00 GiB
  Free  PE / Size       179199 / <700.00 GiB

Check the LV

~# lvdisplay
  --- Logical volume ---
  LV Path                /dev/ubuntu--vg-root/lv-root
  LV Name                lv-root
  VG Name                ubuntu--vg-root
  LV UUID                KiPSR5-XXvZ-JaXz-fyTw-EE0U-NV7E-6ohpCN
  LV Write Access        read/write
  LV Creation host, time ubuntu-server, 2018-11-15 06:06:41 +0000
  LV Status              available
  # open                 1
  LV Size                45.00 GiB
  Current LE             11520
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           253:0

Extend the LV

~# lvextend -L 700G /dev/ubuntu--vg-root/lv-root
  Size of logical volume ubuntu--vg-root/lv-root changed from 45.00 GiB (11520 extents) to 700.00 GiB (179200 extents).
  Logical volume ubuntu--vg-root/lv-root successfully resized.

Find the root file system name

~# df -h
Filesystem                       Size  Used Avail Use% Mounted on
udev                              16G     0   16G   0% /dev
tmpfs                            3.2G   19M  3.2G   1% /run
/dev/mapper/ubuntu--vg-root       45G  5.3G   37G  13% /

Resize file system

~# resize2fs -p /dev/mapper/ubuntu--vg-root
resize2fs 1.44.1 (24-Mar-2018)
Filesystem at /dev/mapper/ubuntu--vg-root is mounted on /; on-line resizing required
old_desc_blocks = 6, new_desc_blocks = 88
The filesystem on /dev/mapper/ubuntu--vg-root is now 183500800 (4k) blocks long.

View the latest hard drive size

~# df -h
Filesystem                       Size  Used Avail Use% Mounted on
udev                              16G     0   16G   0% /dev
tmpfs                            3.2G   59M  3.1G   2% /run
/dev/mapper/ubuntu--vg-root      689G  5.4G  655G   1% /

How to reduce the LV size

You can use the lvreduce command.

Monitor the temperature and health status of nvme ssd in Ubuntu

immortalt — Fri, 03 Dec 2021 03:14:54 +0000

Install nvme-cli

sudo apt-get install nvme-cli

List disks

sudo nvme list

View the details of the hard drive

sudo nvme smart-log /dev/nvme0

Auto refresh status

sudo watch -n 1 nvme smart-log /dev/nvme0

Only view the temperature info

sudo nvme smart-log /dev/nvme0 | grep "^temperature"

Imporve your Ubuntu Desktop's performance

immortalt — Fri, 03 Dec 2021 03:14:24 +0000

When you install the Ubuntu system on a physical machine, don't forget to do these things to improve your computer's
performance.

My experiments found that the improvement was very noticeable, based on Ubuntu 20.10.

use s-tui to monitor and test your max CPU frequency

how to install

sudo apt install python3-pip stress
sudo pip3 install s-tui

how to use

sudo s-tui

You can use the arrow keys and space bar to switch between Monitor mode and Stress mode. The Stress mode performs a
stress test on the CPU.

use indicator-cpufreq to set your CPU frequency

how to install

sudo apt install indicator-cpufreq
sudo reboot

how to use

Look at the status bar icon in the upper right corner and click on the emerging icon to set the CPU's performance mode.

Create socks server on Ubuntu 20.04

immortalt — Fri, 03 Dec 2021 03:11:33 +0000

In daily life, we may need to create a socks server to provide network for other hosts.

I summarized the method of creating socks service through dante-server under Ubuntu 20.04.

install dante-server

sudo apt install dante-server

add user for dante-server

Use this method to create a user to prevent it from logging in.

sudo useradd -s /sbin/nologin dante-socks

Edit the configuration file

sudo nano /etc/danted.conf

minimal configuration for dante-server

logoutput: syslog stdout
internal: ztuku335yr port = 1080
external: ens3
user.notprivileged: dante-socks
clientmethod: none
socksmethod: none
#allow connections from local network
client pass {
        from: 10.0.0.229/24 to: 0.0.0.0/0
        log: error # connect disconnect
}
socks pass {  
        from: 0.0.0.0/0 to: 0.0.0.0/0
        command: bind connect udpassociate
        log: error # connect disconnect iooperation
}
socks pass {
        from: 0.0.0.0/0 to: 0.0.0.0/0
        command: bindreply udpreply
        log: error # connect disconnect iooperation
}

Create a private Ubuntu mirror

immortalt — Fri, 03 Dec 2021 03:10:57 +0000

To create a private Ubuntu image, the commonly used tools are debmirror and apt-mirror.

As of today, these two tools have their own advantages and disadvantages.

debmirror

debmirror document

debmirror has many downloading methods, and it will compare the local file with the file on the server before
downloading so that only the missing files are downloaded. However, it does not seem to support multi-threaded
downloading. Debmirror is downloaded one by one, so the download speed is relatively slow.

apt-mirror

apt-mirror document

apt-mirror supports 20 threads to download at the same time by default. If you choose a suitable mirror station, the
download speed can reach the maximum of my network. However, it does not seem to make a comparison. Although it can
speed up the download start speed, sometimes some files may be lost, resulting in an incomplete mirror, so the client
cannot be updated.

how to write the config file

In order to download the complete package, we need to modify the configuration file in /etc/apt/mirror.list. Use this
format to ensure that the i386 package will be downloaded.

deb-amd64 http://mirrors.aliyun.com/ubuntu focal main restricted universe multiverse
deb-i386 http://mirrors.aliyun.com/ubuntu focal main restricted universe multiverse

Combine them together

Therefore, in order to achieve a balance between efficiency and completeness, we can combine apt-mirror and debmirror.
We can first use apt-mirror to download quickly, then we can use debmirror to check and download the missing packages.
Maybe this is the best way to download the Ubuntu mirror.

How to migrate mgo to qmgo

immortalt — Wed, 23 Dec 2020 17:08:41 +0000

For more information, visit my new blog Immortal Blog

Why migrate to qmgo

Mgo is a convenient library for golang to operate mongodb, but it does not support the latest features of mongodb, such as Transaction.
My old project is written in mgo. If I want to migrate to the official mongo-driver, its syntax is more primitive, the usage difference is large, and the migration cost will be higher.
I found a library called qmgo, which is based on the official mongo-driver package, but it is closer to mgo in terms of syntax and suitable for simple and rude migrations.

Below I will record the methods used in the migration process, which I found out by myself. If you have any questions,
please correct me.

Batch replacement

Batch replace github.com/globalsign/mgo/bson with go.mongodb.org/mongo-driver/bson
Batch replace bson.NewObjectId() with primitive.NewObjectID()
Batch replace bson.ObjectId with primitive.ObjectID
Batch replace .Find(bson.M with .Find(c, bson.M
Batch replace .Find(search to .Find(c, search
Batch replace .Insert( to .InsertOne(c,
Batch replace .Update( to .UpdateOne(c,
Batch replace .RemoveAll( to .RemoveAll(c,
Batch replace .Remove( to .Remove(c,
Batch replace errors.Is(err, mgo.ErrNotFound) with qmgo.IsErrNoDocuments(err)
Batch replace .EnsureIndex(mgo.Index{ for .CreateOneIndex(c, options.IndexModel{

Copy and paste

Where there is no context, copy and paste everywhere c := context.Background()
If it is in the interface method of gin, you can directly use gin's c *gin.Context (but if multi-coroutine operation needs to Copy gin's Context)

Function substitution

Replace the bson.IsObjectIdHex() method:

package db

import (
    "go.mongodb.org/mongo-driver/bson/primitive"
)

func IsObjectIDHex(hex string) bool {
    _, err := primitive.ObjectIDFromHex(hex)
    if err != nil {
        return false
    }
    return true
}

Replace the bson.ObjectIdHex() method:

package db

import (
    "fmt"
    "go.mongodb.org/mongo-driver/bson/primitive"
)

func ObjectIDHex(hex string) primitive.ObjectID {
    id, err := primitive.ObjectIDFromHex(hex)
    if err != nil {
        panic(fmt.Sprintf("Invalid input to ObjectIDHex: %+v", hex))
    }
    return id
}

These functions can be placed in the package of the auxiliary method or the package of the database operation, and then
replace the references in batches.

Summary

At this point, the project completed the migration within two hours and started running smoothly.

Thanks to the author of qmgo for making the wheels for me.

If I have the opportunity, I will also open up some wheels.