Building a Bitcoin P2P Network Analyzer with Golang and Fiber Framework

In the world of Bitcoin, understanding the dynamics of the peer-to-peer (P2P) network is crucial. Monitoring node information, connection metrics, and P2P traffic can provide valuable insights into the network's health, performance, and behavior. In this article, we will explore how to build a Bitcoin P2P network analyzer using Golang and the Fiber framework. We'll cover fetching node information, tracking connection metrics, and exposing relevant information through APIs.

Prerequisites

To follow along with this tutorial,the necessary dependencies installed:

• Golang 1.17 or later
• SQLite
• Btcd ( a bitcoin node implementation written in golang )
• A running testnet bitcoin node daemon

Additionally, a basic understanding of Bitcoin, P2P networks, and the Golang Fiber framework will be helpful.

Setting Up the Project:

1. Open a new folder and create a new Go module for our project:

go mod init bitcoin-p2p-analyzer

1. Install the required dependencies:

go get -u github.com/gofiber/fiber/v2
go get -u github.com/lncm/lnd-rpc/v0.10.0/lnrpc
go get -u github.com/btcsuite/btcd
go get -u gorm.io/gorm
go get -u github.com/joho/godotenv

I would be using this folder format during the course of this tutorial:

📁 app
📁 bitcoin
📁 controllers
📁 db
📁 lightning
📁 services
📁 utils
go.mod
go.sum
main.go 
.env

The app directory will house the configuration and initialization of the fiber application which will run in the main.go. The bitcoin and lightning folder will contain configuration of our bitcoin and lightning clients respectively. The controllers will contain function handlers for our api endpoints, while the db will hold our connection to our sqlite database using gorm. The services and utils fodler will contain helper methods for our application.
It is important to note that all our implementation will focus on the bitcoin test network(testnet).

Node Information Monitoring

The first step in our network analyzer is to fetch and monitor node information such as user agent, version, and services. We can use the Lightning Network Daemon (LND) client and Bitcoin Client to retrieve this information. Let us setup our Lightning Client:

// lightning/client.go


package lightning

import (
    "context"
    "encoding/hex"
    "fmt"
    "io/ioutil"
    "log"
    "os/user"
    "time"

    "google.golang.org/grpc"
    "google.golang.org/grpc/credentials"
    "gopkg.in/macaroon.v2"

    "github.com/lncm/lnd-rpc/v0.10.0/lnrpc"
)

type rpcCreds map[string]string

func (m rpcCreds) RequireTransportSecurity() bool { return true }
func (m rpcCreds) GetRequestMetadata(_ context.Context, _ ...string) (map[string]string, error) {
    return m, nil
}
func newCreds(bytes []byte) rpcCreds {
    creds := make(map[string]string)
    creds["macaroon"] = hex.EncodeToString(bytes)
    return creds
}

func getClient(hostname string, port int, tlsFile, macaroonFile string) lnrpc.LightningClient {
    macaroonBytes, err := ioutil.ReadFile(macaroonFile)
    if err != nil {
        panic(fmt.Sprintln("Cannot read macaroon file", err))
    }

    mac := &macaroon.Macaroon{}
    if err = mac.UnmarshalBinary(macaroonBytes); err != nil {
        panic(fmt.Sprintln("Cannot unmarshal macaroon", err))
    }

    ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
    defer cancel()

    transportCredentials, err := credentials.NewClientTLSFromFile(tlsFile, hostname)
    if err != nil {
        panic(err)
    }

    fullHostname := fmt.Sprintf("%s:%d", hostname, port)

    connection, err := grpc.DialContext(ctx, fullHostname, []grpc.DialOption{
        grpc.WithBlock(),
        grpc.WithTransportCredentials(transportCredentials),
        grpc.WithPerRPCCredentials(newCreds(macaroonBytes)),
    }...)
    if err != nil {
        panic(fmt.Errorf("unable to connect to %s: %w", fullHostname, err))
    }

    return lnrpc.NewLightningClient(connection)
}

func Client() lnrpc.LightningClient {
    usr, err := user.Current()
    if err != nil {
        log.Fatal(err)
    }
    homeDir := usr.HomeDir
    lndDir := fmt.Sprintf("%s/app_container/lightning", homeDir)
    var (
        hostname     = "localhost"
        port         = 10009
        tlsFile      = fmt.Sprintf("%s/tls.cert", lndDir)
        macaroonFile = fmt.Sprintf("%s/data/chain/bitcoin/testnet/admin.macaroon", lndDir)
    )

    client := getClient(hostname, port, tlsFile, macaroonFile)

    return client
}

The Bitcoin Client:

package bitcoin

import (
    "log"

    "bitcoin-p2p-analyzer/utils"
    "github.com/btcsuite/btcd/rpcclient"
)

func Client() *rpcclient.Client {

    // Connect to a running Bitcoin Core node via RPC
    connCfg := &rpcclient.ConnConfig{
        Host:         utils.GetEnv("BTC_HOST"),
        User:         utils.GetEnv("BTC_USER"),
        Pass:         utils.GetEnv("BTC_PASS"),
        HTTPPostMode: true,
        DisableTLS:   true,
    }

    client, err := rpcclient.New(connCfg, nil)
    if err != nil {
        log.Fatal("Error connecting to bitcoind:", err)
    }
    // Get the current block count
    blockCount, err := client.GetBlockCount()
    if err != nil {
        log.Println("Error getting block count:", err)

    }

    log.Println("Current block count:", blockCount)
    return client
}

Then we go ahead and create our function in the services directory that displays our node information:

// services/lightning.go

package services

import (
    "context"
    "log"

    "bitcoin-p2p-analyzer/lightning"
    "github.com/lncm/lnd-rpc/v0.10.0/lnrpc"
)

type LNodeMetrics struct {
    PubKey      string `json:"pub_key"`
    UserAgent   string `json:"user_agent"`
    Alias       string `json:"alias"`
    NetCapacity int    `json:"network_capacity"`
}

func GetNodeInfo() *LNodeMetrics {

    client := lightning.Client()

    infoReq := &lnrpc.GetInfoRequest{}

    info, err := client.GetInfo(context.Background(), infoReq)

    if err != nil {
        log.Fatalf("Error getting node info: %v", err)
    }

    moreInfo, _ := client.GetNetworkInfo(context.Background(), &lnrpc.NetworkInfoRequest{})

    result := &LNodeMetrics{
        UserAgent:   info.Version,
        Alias:       info.Alias,
        NetCapacity: int(moreInfo.TotalNetworkCapacity),
        PubKey:      info.IdentityPubkey,
    }

    return result
}

Bitcoin Node Info:

// services/bitcoin.go

package services

import (
    "log"
    "math"

    "bitcoin-p2p-analyzer/bitcoin"
    "github.com/btcsuite/btcd/chaincfg/chainhash"
)

type NodeMetrics struct {
    Difficulty    float64     `json:"difficulty"`
    Version       interface{} `json:"version"`
    Chain         string      `json:"chain"`
    Blocks        int32       `json:"no_of_blocks"`
    BestBlockHash string      `json:"bestblockhash"`
    UserAgent     interface{} `json:"user_agent"`
    HashRate      float64     `json:"hash_rate"`
}

func GetInfo() *NodeMetrics {

    client := bitcoin.Client()

    defer client.Shutdown()

    info, err := client.GetBlockChainInfo()

    if err != nil {
        log.Println(err)
    }

    networkInfo, _ := client.GetNetworkInfo()

    lastBlockHash, err := chainhash.NewHashFromStr(info.BestBlockHash)
    if err != nil {
        log.Println(err)
    }

    lastBlock, err := client.GetBlock(lastBlockHash)
    if err != nil {
        log.Println(err)
    }

    timeToFindBlock := lastBlock.Header.Timestamp.Unix() - int64(lastBlock.Header.PrevBlock[len(lastBlock.Header.PrevBlock)-1])
    hashrate := float64(info.Difficulty) / (float64(timeToFindBlock) * math.Pow(2, 32))

    metrics := &NodeMetrics{
        Difficulty:    info.Difficulty,
        Version:       networkInfo.Version,
        Chain:         info.Chain,
        Blocks:        info.Blocks,
        BestBlockHash: info.BestBlockHash,
        UserAgent:     networkInfo.SubVersion,
        HashRate:      hashrate,
    }

    return metrics

}

Then we create an api function handler in our controllers folder:

// controllers/controller.go
package controllers

import (
    "bitcoin-p2p-analyzer/services"
    "github.com/gofiber/fiber/v2"
)

type Response struct {
    Success bool        `json:"success"`
    Data    interface{} `json:"data"`
}

func GetMetrics(c *fiber.Ctx) error {

    type NodeResponse struct {
        Lightning interface{} `json:"lightning"`
        Bitcoin   interface{} `json:"bitcoin"`
    }

    bitcoin := services.GetInfo()
    lightning := services.GetNodeInfo()

    response := &NodeResponse{
        Bitcoin:   bitcoin,
        Lightning: lightning,
    }

    return c.Status(fiber.StatusOK).JSON(response)
}

Then we update the app directory:

// app/app.go

package app

import (
    "bitcoin-p2p-analyzer/controllers"
    "github.com/gofiber/fiber/v2"
    "github.com/gofiber/fiber/v2/middleware/cors"
)

func App() *fiber.App {
    app := fiber.New()

    app.Use(cors.New())

    app.Use(cors.New(cors.Config{
        AllowOrigins: "*",
        AllowHeaders: "Origin, Content-Type, Accept",
    }))

    app.Get("/", func(c *fiber.Ctx) error {
        return c.SendString("Hello, World!")
    })

    app.Get("/node-info", controllers.GetMetrics)
    return app
}

update the main.go:

// main.go

package main

import (
    "log"

    "bitcoin-p2p-analyzer/app"
)

func main() {

    err := app.App().Listen("0.0.0.0:1700")

    if err != nil {
        log.Fatal(err)
    }

}

Save the file and run the app in your terminal:

go run main.go

Open your postman on the following address http://127.0.0.1:1700/node-info

Connection Metrics Monitoring

Next, we'll focus on tracking connection metrics, such as uptime and churn, for both on-premises servers and EC2 instances running Bitcoin Core (btcd). We would be implementing in a time-series manner where by we can compare the metrics overtime and to get insights on the connection performance. We would use goroutines to scrape the metrics every 3 minutes and save in a sqlite database. We'll use the btcd and LND clients to fetch the necessary information. Here's an example code snippet:

// db/database.go

package db

import (
    "log"

    "gorm.io/driver/sqlite"
    "gorm.io/gorm"
)

func DB() *gorm.DB {
    db, err := gorm.Open(sqlite.Open("metrics.db"), &gorm.Config{})
    if err != nil {
        log.Fatalf("Failed to open database: %v", err)
    }

    return db

}

// services/metrics.go
package services

import (
    "context"
    "log"
    "time"

    "bitcoin-p2p-analyzer/bitcoin"
    "bitcoin-p2p-analyzer/db"
    "bitcoin-p2p-analyzer/lightning"
    "bitcoin-p2p-analyzer/utils"
    "github.com/lncm/lnd-rpc/v0.10.0/lnrpc"
)

func ConnectionMetrics() {

    db := db.DB()

    // Create the metrics table if it doesn't already exist
    db.AutoMigrate(&utils.ConnectionMetrics{})

    // Get Bitcoin Client
    bitcoin := bitcoin.Client()

    defer bitcoin.Shutdown()

    // Get Lightning Client
    lnd := lightning.Client()

    for {
        //Get Bitcoin Peer Info

        peerInfo, err := bitcoin.GetPeerInfo()

        if err != nil {
            log.Printf("Failed to fetch btcd peer info: %v", err)
            continue
        }

        infoReq := &lnrpc.GetInfoRequest{}

        lndInfo, err := lnd.GetInfo(context.Background(), infoReq)

        if err != nil {
            log.Printf("Failed to fetch lnd info: %v", err)
            continue
        }

        // Calculate the incoming and outgoing bandwidth for the btcd node
        var btcdBandwidthIn, btcdBandwidthOut uint64
        for _, peer := range peerInfo {
            btcdBandwidthIn += peer.BytesRecv
            btcdBandwidthOut += peer.BytesSent
        }

        metrics := &utils.ConnectionMetrics{
            Timestamp:           time.Now(),
            NumBTCPeers:         int32(len(peerInfo)),
            NumLNDPeers:         int32(lndInfo.NumPeers),
            NumActiveChannels:   int32(lndInfo.NumActiveChannels),
            NumPendingChannels:  int32(lndInfo.NumPendingChannels),
            NumInactiveChannels: int32(lndInfo.NumInactiveChannels),
            BtcdBandwidthIn:     btcdBandwidthIn,
            BtcdBandwidthOut:    btcdBandwidthOut,
            BlockHeight:         int64(lndInfo.BlockHeight),
            BlockHash:           lndInfo.BlockHash,
            BestHeaderAge:       lndInfo.BestHeaderTimestamp,
            SyncedToChain:       lndInfo.SyncedToChain,
        }

        db.Create(&metrics)

        // Wait for 3 minute before fetching the next set of connection metrics
        time.Sleep(time.Minute * 3)
    }

}

Fetching Metrics:
To retrieve the stored connection metrics, we can create an API endpoint using the Fiber framework:

...
//services/metrics.go

func FetchMetrics() []utils.ConnectionMetrics {

    var allMetrics []utils.ConnectionMetrics

    db := db.DB()

    //fetch all metrics
    if err := db.Find(&allMetrics).Error; err != nil {
        log.Fatal(err)
    }

    return allMetrics

}

// controllers/controller.go
...

func GetConnMetrics(c *fiber.Ctx) error {

    metrics := services.FetchMetrics()

    response := &Response{
        Success: true,
        Data:    metrics,
    }

    return c.Status(fiber.StatusOK).JSON(response)
}

Then you update the app.go with the required endpoints:

// app/app.go
...
app.Get("/conn-metrics", controllers.GetConnMetrics)

go services.ConnectionMetrics() // run the function as a goroutine

Open your postman on http://127.0.0.1:1700/conn-metrics and test after 3 minutes.

Conclusion

In this tutorial, we explored how to build a Bitcoin P2P network analyzer using Golang and the Fiber framework. We learned how to fetch node information, track connection metrics, and expose the relevant information through an API endpoint. By monitoring node information, connection metrics, and P2P traffic, we can gain valuable insights into the behaviour, performance, and bandwidth usage of the Bitcoin network. For example, analysing node information allows us to identify the distribution of different node implementations, such as Bitcoin Core, btcd, or other client software versions. This insight can help us understand the level of network diversity and the adoption rate of software updates within the Bitcoin ecosystem.

Connection metrics provide visibility into the connectivity and network topology of the Bitcoin network. By monitoring connection counts and peer relationships, we can identify well-connected nodes, influential network participants, and potential bottlenecks. This information helps us evaluate the resilience and decentralisation of the network, and it can guide decisions related to optimising node connections for better performance and redundancy.

P2P traffic analysis allows us to examine the flow of data within the Bitcoin network. By studying message types, transaction propagation, and block dissemination, we can gain insights into the efficiency and effectiveness of the network's information dissemination protocols. This analysis can reveal potential delays or inefficiencies in block propagation, highlighting areas for optimisation to enhance the overall network throughput and reduce confirmation times.

Furthermore, by extending the functionality of a Bitcoin P2P network analyser, we can unlock even more valuable insights. For instance, incorporating transaction analysis capabilities can enable the identification of transaction patterns, fee dynamics, and network congestion levels. By visualising these metrics, we can better understand the factors influencing transaction confirmation times and fee market dynamics.

Additionally, integrating data from Lightning Network nodes and channels can provide insights into the growth, liquidity distribution, and routing efficiency of the Layer 2 network. This information can help evaluate the scalability and usability of the Lightning Network, as well as identify areas for improvement.

By leveraging the data collected and analyzed through a Bitcoin network monitoring tool, researchers, developers, and network participants can make informed decisions to enhance the security, efficiency, and scalability of the Bitcoin network.

Note: This article provides a high-level overview of the implementation process. Please refer to the official documentation and relevant libraries' documentation for detailed instructions and best practices.

Happy coding and exploring the fascinating world of Bitcoin P2P networks!