DEV Community: Kamalesh-Seervi

Real-Time Trading App: Golang, Kafka, Websockets — Setting up Consumer & Websockets(PART-3)

Kamalesh-Seervi — Thu, 04 Jan 2024 13:40:48 +0000

Real-Time Trading App: Golang, Kafka, Websockets — Setting up Consumer & Websockets(PART-3)

Step-by-step guide on configuring a Kafka consumer in Golang for real-time data processing.

PART-3

Creating Consumer Service & Websockets

In this ongoing series, we’ve delved into the introductory aspects of our tech stack and explored the high-level architecture. Our journey began with the implementation of a producer service using Kafka and Golang. Having accomplished this initial phase, the focus now shifts to the creation of a consumer service. We will delve into the process of actively listening for Kafka events and seamlessly transmitting real-time ticker data to the frontend using websockets.

Why did we choose to employ websockets instead of directly integrating Kafka on the frontend?

In the initial post, it was highlighted that Kafka lacks substantial support for web browsers due to various reasons. Notably, Kafka primarily utilizes TCP, and browsers tend to keep TCP connections open for very brief durations. Additionally, Kafka’s distribution policy, where messages are spread across consumers, poses a challenge when each browser tab is treated as a separate consumer. This distribution not only affects tabs but also extends to other devices consuming the data.
Moreover, Kafka consumers demand a significant amount of resources to manage offsets, message reads, and overall states. In contrast, websockets offer a lightweight alternative. This brief explanation serves as a justification for opting to use websockets to efficiently deliver real-time data to the frontend. Now, let’s delve into the implementation details of the consumer side.

Folder Structure:

core/settings.go

package core

import (
 "fmt"
 "log"
 "os"
 "strings"

 "github.com/joho/godotenv"
)

var TICKERS []string
var KAFKA_HOST string
var KAFKA_PORT string

func Load() {

 err := godotenv.Load("../.env")
 if err != nil {
  log.Fatal("Failed to load environment file")
 }
 t := os.Getenv("TICKERS")
 TICKERS := strings.Split(t, ",")
 LoadTikers(TICKERS)

 KAFKA_HOST = "127.0.0.1"
 KAFKA_PORT = "9092"
 fmt.Println(TICKERS)
}

core/ticker.go

package core

import "strings"

var tickerSet map[string]struct{}

func GetAllTickers() []string {
 tickerList := []string{}
 for key := range tickerSet {
  tickerList = append(tickerList, key)
 }
 return tickerList
}

func IsTickerAllowed(ticker string) bool {
 _, ok := tickerSet[strings.ToLower(ticker)]
 return ok
}

func LoadTikers(tickers []string) {
 if tickerSet == nil {
  tickerSet = make(map[string]struct{})
 }
 for _, t := range tickers {
  tickerSet[strings.ToLower(strings.Trim(strings.Trim(t, "\\"), "\""))] = struct{}{}
 }
}

In our main.go file, the settings.go module serves the purpose of loading all environment variables, ensuring a streamlined configuration process. On the other hand, ticker.go houses a comprehensive set of tickers and functions that will prove instrumental in upcoming stages of our implementation.

Now, let’s integrate and configure these components within our main.go.

Initial code for main.go

package main

import (
 "github.com/gin-contrib/cors"
 "github.com/gin-gonic/gin"
 "github.com/kamalesh-seervi/consumer/api"
 "github.com/kamalesh-seervi/consumer/core"
)

func main() {
 core.Load()
}

Keeping it simple, we incorporate core.Load() in our main.go to ensure the environment file is loaded seamlessly. As a quick check, let's print the tickers using

fmt.Println(core.GetAllTickers())

Now lets build is the API and Websocket Connection:

api/ticker.go

package api

import (
 "context"
 "fmt"
 "log"
 "strings"

 "github.com/gin-gonic/gin"
 "github.com/gorilla/websocket"
 "github.com/segmentio/kafka-go"

 "github.com/kamalesh-seervi/consumer/core"
)

func GetAllTickers(c *gin.Context) {
 c.JSON(200, core.GetAllTickers())
}

func ListenTicker(c *gin.Context) {
 conn, err := websocket.Upgrade(c.Writer, c.Request, nil, 1024, 1024)
 if err != nil {
  log.Println("WebSocket Upgrade Error: ", err)
  return
 }
 defer conn.Close()

 currTicker := c.Param("ticker")
 log.Println("Current ticker: ", currTicker)

 if !core.IsTickerAllowed(currTicker) {
  conn.WriteMessage(websocket.CloseUnsupportedData, []byte("Ticker is not allowed"))
  log.Println("Ticker not allowed ticker: ", currTicker)
  return
 }

 topic := "trades-" + strings.ToLower(currTicker)
 reader := kafka.NewReader(kafka.ReaderConfig{
  Brokers: []string{core.KAFKA_HOST + ":" + core.KAFKA_PORT},
  Topic: topic,
 })
 reader.SetOffset(-1)
 defer reader.Close()

 conn.SetCloseHandler(func(code int, text string) error {
  reader.Close()
  log.Printf("Received connection close request. Closing connection .....")
  return nil
 })

 go func() {
  code, wsMessage, err := conn.NextReader()
  if err != nil {
   log.Println("Error reading last message from WS connection. Exiting ...")
   return
  }
  fmt.Printf("CODE : %d MESSAGE : %s\n", code, wsMessage)
 }()

 for {
  message, err := reader.ReadMessage(context.Background())
  if err != nil {
   log.Println("Error: ", err)
   return
  }
  fmt.Println("Reading..... ", string(message.Value))

  err = conn.WriteMessage(websocket.TextMessage, message.Value)
  if err != nil {
   log.Println("Error writing message to WS connection: ", err)
   return
  }
 }
}

Let’s break down the functionality of the ListenTicker function:

WebSocket Upgrade:

The function begins by attempting to upgrade the HTTP connection to a WebSocket connection using websocket.Upgrade.
If the upgrade fails, an error is logged, and the function returns.

Parameters and Ticker Validation:

The current ticker is extracted from the request parameters.
The function checks if the current ticker is allowed using the core.IsTickerAllowed function.
If the ticker is not allowed, a message is sent to the WebSocket client indicating that the ticker is not allowed, and the function returns.

3. Kafka Topic Configuration:

A Kafka topic is constructed based on the lowercase version of the current ticker.
A Kafka reader is created using the kafka.NewReader function, configured with the Kafka broker information and topic.
The reader’s offset is set to -1 to read messages from the latest available.

4. WebSocket Connection Configuration:

A close handler is set on the WebSocket connection to handle closure events. It ensures that the Kafka reader is closed when the WebSocket connection is closed.

5. Goroutine for Handling WebSocket Messages:

A goroutine is launched to handle WebSocket messages. It reads the last message from the WebSocket connection, logs the details, and exits.

6. Reading Kafka Messages and Broadcasting to WebSocket:

The function enters a loop where it continuously reads messages from the Kafka topic using reader.ReadMessage.
Each Kafka message is then written to the WebSocket connection using conn.WriteMessage.

7. Error Handling:

Errors during the WebSocket message reading or writing process are logged.

Now let’s link all this and expose ws and api connections.

api/routing.go

package api

import (
 "github.com/gin-contrib/cors"
 "github.com/gin-gonic/gin"
)

func AddRoutes(router *gin.Engine) {
 router.Use(cors.Default())

 apiV1 := router.Group("/api/v1")
 {
  apiV1.GET("/tickers", GetAllTickers)
 }

 // WebSocket route
 router.GET("/ws/trades/:ticker", func(c *gin.Context) {
  if c.Request.Header.Get("Upgrade") != "websocket" {
   c.JSON(400, gin.H{"error": "WebSocket upgrade required"})
   return
  }

  // Specific WebSocket logic here
  ListenTicker(c)
 })
}

Final main.go

package main

import (
 "github.com/gin-contrib/cors"
 "github.com/gin-gonic/gin"
 "github.com/kamalesh-seervi/consumer/api"
 "github.com/kamalesh-seervi/consumer/core"
)

func main() {
 core.Load()
 router := gin.Default()

 // CORS middleware
 config := cors.DefaultConfig()
 config.AllowOrigins = []string{"*"}
 config.AllowMethods = []string{"GET", "POST", "HEAD", "PUT", "DELETE", "PATCH", "OPTIONS"}
 config.AllowHeaders = []string{"Origin", "Content-Type", "Accept", "Content-Length", "Accept-Language", "Accept-Encoding", "Connection", "Access-Control-Allow-Origin"}
 config.AllowCredentials = true
 router.Use(cors.New(config))

 // Add routes

 api.AddRoutes(router)

 // Start server
 router.Run(":8000")
}

Open your web browser and navigate to the URL 127.0.0.1:8000/api/v1/tickers. You will receive a response similar to the following.

Tickers

Before proceeding, ensure that you have built the producer service and confirmed that it is actively running to facilitate data pushing to Kafka. If you are running the service with Docker, remember to update the .env file, replacing 127.0.0.1 with kafka. If you are running it locally, you can disregard this step.

To swiftly execute this, use the provided command.

sudo docker-compose up -d 
&
Run Both producer and consumer build files.

Now let’s test the web socket stream.

Websocket stream

Consumer Live read data

If you successfully receive the data, your backend is now seamlessly linked to Kafka.

With this, we are approaching the final stages. The data flows from the backend to the frontend, and in the upcoming article, we will delve into visualizing and interpreting the data using charts to better understand its dynamics.

Conclusion

In conclusion, this series of articles has guided us through the establishment of a robust tech stack and a well-designed high-level architecture. We initiated the implementation with the creation of a producer service, integrating Kafka and Golang for efficient data transmission. The decision to employ websockets on the frontend was motivated by the limitations of directly using Kafka in browsers, considering factors such as TCP connection handling and resource utilization.

With the backend successfully linked to Kafka, we now have a functional system where data flows seamlessly from the producer service to the frontend through websockets. The backend is not only capable of fetching real-time data but also validating and broadcasting it to connected clients.

GitHub - Kamalesh-Seervi/Real-time-trade-app: Kafka, WebSockets

Radare2 — Cross-References, Static Analysis, and Binary Information Retrieval (Part 2–3)

Kamalesh-Seervi — Sat, 23 Dec 2023 16:05:20 +0000

Radare2 — Cross-References, Static Analysis, and Binary Information Retrieval (Part 2–3)

Navigating the Depths of Binary Analysis: Advanced Techniques and Insightful Information Extraction

Static analysis & Binary Information

Cross Reference Insights

Discover the power of axt and axf commands for comprehensive cross-reference analysis. Uncover the relationships within the binary and understand its structure with these advanced tools.

Static Analysis Unveiled

Import and Export Libraries

Use ii to reveal import libraries and iE for exports. Unravel the binary's dependencies and interactions by deciphering its import and export components.

Strings Analysis

Unearth hidden insights with the is command, revealing strings embedded within the binary. This crucial step unveils textual elements that provide valuable context and clues about the binary's functionality.

Getting In-Depth Binary Information

Rabin2: Your Binary Information Swiss Army Knife

Leverage the power of rabin2 to obtain detailed information about the binary. From basic details to hexadecimal representations, rabin2 provides a wealth of insights.

To get basic binary information:

rabin2 -I ./letter_frequencies

Hexadecimal view of the binary:

rabin2 -H ./letter_frequencies

Extracting strings with the zz tag:

rabin2 -zz ./letter_frequencies

Rafind2: Advanced String Search

Move beyond simple string searches with rafind2. This advanced tool allows for intricate string analysis within binary files, providing a more nuanced approach to information retrieval.

rafind2 -s frequencies ./letter_frequencies

Loading Headers

Learn to navigate binary headers with ease using commands like r2 -nn ./letter_frequencies, pf., and pf.elf_header @ elf_header. Understand the binary's structure and gain a deeper appreciation for its complexity.

Conclusion:

In conclusion, our exploration of Radare2’s capabilities in binary analysis has unveiled a powerful set of tools for cross-referencing, static analysis, and binary information retrieval. Through commands like axt , ii , and iE , we’ve navigated the intricacies of cross-references, dissected import and export libraries, and revealed critical strings within the binary.

Leveraging rabin2 and rafind2 has provided us with comprehensive insights into the binary’s structure, offering detailed information and advanced string search capabilities. As we conclude this segment, the journey continues with an anticipation of further revelations in dynamic analysis and more advanced techniques in the upcoming parts of this series.

Stay tuned for a deeper dive into the fascinating world of binary analysis with Radare2!

GitHub - Kamalesh-Seervi/radare2

Static Navigation Disassembly with Radare2 — PART-1

Kamalesh-Seervi — Thu, 21 Dec 2023 15:59:11 +0000

Static Navigation Disassembly with Radare2 — PART-1

To delve into the static analysis of binaries using Radare2, follow these fundamental commands:

Radare2 Beginner Guide

I’ve outlined a 10-part plan to facilitate learning. Each part is accompanied by a dedicated folder containing a comprehensive README file for better understanding.

GitHub - Kamalesh-Seervi/radare2

Radare2 consists of an hexadecimal editor (radare) with a wrapped IO layer supporting multiple backends for local/remote files, debugger (OS X, BSD, Linux, W32), stream analyzer, assembler/disassembler (rasm) for various architectures, code analysis modules, and scripting facilities. Additional tools include radiff (bindiffer), rax (base converter), rasc (shellcode development helper), rabin (binary information extractor), and rahash (block-based hash utility).

Installation:

sudo apt install radare2

Running a Binary

Check my github link above to get the binary files.
To execute a binary file in Radare2, use the following command:

r2 ./letter_frequencies

After executing the command, the Seek commander will be activated.

Views and Disassembly

Press V to access various views of the binary file.
Type p to switch between different views like hex, disassembly, debugger, ASCII hex, diffuse, and color visual.

Hex View Example

Enter to select the first line in the disassembler (similar to double-clicking a line).

Steps for Analysis

Enter Seeker mode by typing Shift+:.
In the Seeker mode, use Shift+: under the disassembly mode.
Analyze the binary by typing aaa and pressing Enter.
Display the functions in the binary with afl.

To focus on the main code, type s main and press Enter twice.

Explore the main code of the disassembly.
When entering a printf or similar function during reverse engineering, press Enter to view the stub code.

Note

To navigate back, press U.

Conclusion

By using Radare2 for static navigation and disassembly, you’ve gained a foundational understanding of binary analysis. This tool provides a robust set of commands for inspecting, analyzing, and navigating through binaries. As you continue your journey with Radare2, you’ll unlock its full potential in reverse engineering and binary analysis.

Experiment with different commands, explore various views, and deepen your comprehension of binary structures. The insights gained from static analysis will prove invaluable as you progress in your understanding of Radare2.

Stay tuned for the next part, where we’ll dive into advanced features and real-world examples. In Part 2, we’ll explore dynamic analysis, debugging, and more cool aspects of Radare2. Get ready for the next chapter in your Radare2 learning adventure!

GitHub - Kamalesh-Seervi/radare2

Real-Time Trading App: Golang, Kafka, Websockets — Setting up Kafka in Golang (PART-2)

Kamalesh-Seervi — Tue, 19 Dec 2023 15:36:39 +0000

Real-Time Trading App: Golang, Kafka, Websockets — Setting up Kafka in Golang (PART-2)

Configuring Kafka in a Golang environment and creating a build file for the producer.

This marks Part 2 of our real-time trade app series in Golang. In this installment, we’ll dive into coding for Kafka, focusing on establishing a connection and creating a producer build file. This step is crucial for testing the API connection to Binance’s web sockets. Stay tuned for hands-on coding and insights into optimizing the integration!

Establishing a connection with the data source.

As previously noted, we’ll utilize Binance’s WebSocket API as our data source. Within your app folder, create a trades subfolder and include the following files: listener.go, publish.go, and ticker.go.

Ticker.go

Within ticker.go, you’ll find the model for the ticker data sourced from Binance. This model serves a dual purpose, as it will be employed for both receiving and publishing the data.

package trades

type Ticker struct {
 Symbol string `json:"s"`
 Price string `json:"p"`
 Quantity string `json:"q"` 
 Time int64 `json:"T"`
}

Listener.go

As the name implies, listener.go houses the code where we actively listen for specific tickers through a WebSocket connection. Let’s delve into the details, starting with the establishment of the WebSocket connection.

package trades

import (
 "encoding/json"
 "log"
 "net/url"

 "github.com/gorilla/websocket"
)

type RequestParams struct {
 Id int `json:"id"`
 Method string `json:"method"`
 Params []string `json:"params"`
}

var conn *websocket.Conn

const (
 subscribeId = 1
 unSubscribeId = 2
)

func getConnection() (*websocket.Conn, error) {
 if conn != nil {
  return conn, nil
 }

 u := url.URL{Scheme: "wss", Host: "stream.binance.com:443", Path: "/ws"}
 log.Printf("connecting to %s", u.String())
 c, resp, err := websocket.DefaultDialer.Dial(u.String(), nil)
 if err != nil {
  log.Printf("handshake failed with status %d", resp.StatusCode)
  log.Fatal("dial:", err)
 }
 conn = c

 return conn, nil
}

func CloseConnections() {
 conn.Close()
}

func EstablishConnection() (*websocket.Conn, error) {
 newConnection, err := getConnection()
 if err != nil {
  log.Fatal("Failed to get connection %s", err.Error())
  return nil, err
 }
 return newConnection, nil
}

func AddOnConnectionClose(h func(code int, text string) error) {
 conn.SetCloseHandler(h)
}

func unsubscirbeOnClose(conn *websocket.Conn, tradeTopics []string) error {
 message := struct {
  Id int `json:"id"`
  Method string `json:"method"`
  Params []string `json:"params"`
 }{
  Id: unSubscribeId,
  Method: "UNSUBSCRIBE",
  Params: tradeTopics,
 }

 b, err := json.Marshal(message)
 if err != nil {
  log.Fatal("Failed to JSON Encode trade topics")
  return err
 }

 err = conn.WriteMessage(websocket.TextMessage, b)

 return nil
}

func SubScribeAndListen(topics []string) error {
 conn, err := getConnection()
 if err != nil {
  log.Fatal("Failed to get connection %s", err.Error())
  return err
 }

 conn.SetPongHandler(func(appData string) error {
  log.Println("Received pong:", appData)
  pingFrame := []byte{1, 2, 3, 4, 5}
  err := conn.WriteMessage(websocket.PingMessage, pingFrame)
  if err != nil {
   log.Println(err)
   // no need to fail
  }
  return nil
 })

 tradeTopics := make([]string, 0, len(topics))
 for _, topic := range topics {
  tradeTopics = append(tradeTopics, topic+"@"+"aggTrade")
 }
 log.Println("Listening to trades for ", tradeTopics)
 message := RequestParams{
  Id: subscribeId,
  Method: "SUBSCRIBE",
  Params: tradeTopics,
 }
 log.Println(message)
 b, err := json.Marshal(message)
 if err != nil {
  log.Fatal("Failed to JSON Encode trade topics")
  return err
 }

 err = conn.WriteMessage(websocket.TextMessage, b)
 if err != nil {
  log.Fatal("Failed to subscribe to topics " + err.Error())
  return err
 }

 defer unsubscirbeOnClose(conn, tradeTopics)
 defer conn.Close()

 for {
  _, payload, err := conn.ReadMessage()
  if err != nil {
   log.Println(err)
   return err
  }

  trade := Ticker{}

  err = json.Unmarshal(payload, &trade)
  if err != nil {
   log.Println(err)
   return err
  }

  log.Println(trade.Symbol, trade.Price, trade.Quantity)
 }
}

This code defines the listener functionality for the real-time trading app, focusing on establishing a WebSocket connection and handling incoming ticker data from Binance. Let’s break it down:

Imports:

Necessary packages are imported, including “github.com/gorilla/websocket” for WebSocket communication and “github.com/segmentio/kafka-go” for Kafka integration.

2. Data Structures:

RequestParams: Struct to represent the parameters for the WebSocket request.
conn: Variable to store the WebSocket connection.

3. Constants:

subscribeId and unSubscribeId : Constants representing subscription and unsubscription identifiers.

4. Functions:

getConnection(): Establishes a WebSocket connection to Binance’s streaming service.
CloseConnections(): Closes the WebSocket connection.
EstablishConnection(): Ensures a connection is established; if not, initiates a new one.
AddOnConnectionClose(h func(code int, text string) error): Adds a handler for WebSocket connection closure. unsubscirbeOnClose(conn *websocket.Conn, tradeTopics []string) error: Unsubscribes from specified trade topics upon connection closure.
SubScribeAndListen(topics []string) error: Subscribes to specified trade topics and listens for incoming data, then converts and publishes it to Kafka.

5. WebSocket Operations:

— Connection handling, ping-pong setup, subscription to trade topics, and handling incoming messages are managed.

6. Data Processing:

— Upon receiving trade data, it’s deserialized into a Ticker struct. The data is then logged, and a goroutine is spawned to convert and publish the data to Kafka.

This comprehensive listener code sets the foundation for actively retrieving and processing real-time trading data from Binance through WebSockets, preparing it for further integration and analysis in the application.

Testing

Execute this code by invoking the SubScribeAndListen function in your main.go file.

package main

import (
 "fmt"
 "os"
 "strings"
 "github.com/joho/godotenv"
  "github.com/kamalesh-seervi/trade-app/producer/trades" // <==== add this or it will autoadd
)

func main() {
 err := godotenv.Load("../.env")
 if err != nil {
  fmt.Print("Failed to load environment")
 }
 t := os.Getenv("TICKERS")
 topics := strings.Split(t, ",")
 for i,topic := range topics {
  topics[i] = strings.Trim(strings.Trim(topic,"\\"),"\"") 
 }

trades.SubScribeAndListen( // We need to add this to make it work and listen the websocket.
  topics,
 )

}

Build and run the app.

go build . && ./trade-app // change it accordingly

Realtime Data from the Binance API

Publish.go

package trades

import (
 "context"
 "log"
 "github.com/segmentio/kafka-go"
)

var (
 HOST string
 PORT string
);

func LoadHostAndPort(host string, port string){
 HOST = host
 PORT = port
} 

func Publish(t string, message kafka.Message, topic string) error {

 messages := []kafka.Message{
  message,
 }

 w := kafka.Writer{
  Addr: kafka.TCP(HOST + ":" + PORT), //127.0.0.1:9092 or kafka:9092 in docker
  Topic: topic,
  AllowAutoTopicCreation: true,
 }
 defer w.Close()

 err := w.WriteMessages(context.Background(), messages...)
 if err != nil {
  log.Println("Error writing msg to Kafka: ", err.Error())
  return err
 }

 log.Println("Publish msg to Kafka on topic: ", topic)

 return nil
}

Integrate this function into the listener.go file to facilitate Kafka publishing.

func SubScribeAndListen(topics []string) error {

  ...
  ...
  ...

 for {
  _, payload, err := conn.ReadMessage()
  if err != nil {
   log.Println(err)
   return err
  }

  trade := Ticker{}

  err = json.Unmarshal(payload, &trade)
  if err != nil {
   log.Println(err)
   return err
  }

  log.Println(trade.Symbol, trade.Price, trade.Quantity)
  go func() { // <=== here
   convertAndPublishToKafka(trade)
  }() 
 }
}

// add this function
func convertAndPublishToKafka(t Ticker) { 
 bytes, err := json.Marshal(t)
 if err != nil {
  log.Println("Error marshalling Ticker data", err.Error())
 }

 Publish(t.Symbol, kafka.Message{
  Key: []byte(t.Symbol + "-" + strconv.Itoa(int(t.Time))),
  Value: bytes,
 }, "trades-"+strings.ToLower(t.Symbol))
}

In the existing code snippet, a goroutine is initiated using the “go” keyword to concurrently execute the convertAndPublishToKafka function. Here's an explanation for your blog:

Logging Trade Data:

log.Println(trade.Symbol, trade.Price, trade.Quantity)

This line logs essential details of the incoming trade data, providing visibility into the traded symbol, price, and quantity.

2. Goroutine for Kafka Publishing:

go func() { convertAndPublishToKafka(trade) }(

By using a goroutine, this section ensures non-blocking execution of the convertAndPublishToKafka function. This approach enhances the overall efficiency of the application, allowing it to continue processing incoming trade data without waiting for the Kafka publishing process to complete.

3. convertAndPublishToKafka Function:

func convertAndPublishToKafka(t Ticker) { // ... (existing code) }

This function takes the received trade data (Ticker object), marshals it into JSON format, and then publishes it to Kafka. Key details, such as the trading symbol and timestamp, are utilized for Kafka message formatting. Any errors during the marshaling process are appropriately logged.

We’ve wrapped up the implementation of our producer service. If you have any questions, feel free to leave a comment below. I’ll be sharing the complete code shortly as we progress through the series, so stay tuned for that.

What’s coming up next?

Our next focus will be on the consumer service. In this part, we’ll be exposing a WebSocket API for the frontend. Data will be pushed to the frontend through a Kafka listener. Stay tuned for the upcoming content!

GitHub - Kamalesh-Seervi/Real-time-trade-app: Kafka, WebSockets

BabyEncryption Hack The Box

Kamalesh-Seervi — Fri, 15 Dec 2023 14:33:13 +0000

HTB | Crypto Challenge

You are tasked with discovering the flag by deciphering the provided ciphertext. The challenge includes a Python script for decryption, but you need to modify the script before executing it. As a newcomer to Hack The Box (HTB), it’s uncertain whether this approach is standard or specific to this challenge. Let’s proceed without delay.

Unzipping the file

Password _ : hackthebox_

Upon extracting the archive, two additional files emerge: chall.py housing the Python script for deciphering the ciphertext, and msg.enc which stores the encrypted text. Now, let’s examine the contents of these files.

chall.py & msg.enc

Executing the script will result in the following error messages.

error script

After spending a few moments attempting to resolve the errors, I opted to abandon that approach and chose to modify the script instead.

Decode Script

def decrypt(msg):
    pt = []
    for char in msg:
        char= char-18
        char=179*char%256
        pt.append(char)
    return bytes(pt)

with open("msg.enc") as f:
    ct=bytes.fromhex(f.read())
print(decrypt(ct))

The value 179 is used in the decryption script as part of the reverse operation to undo the multiplication and modulo operation performed during encryption. In the encryption script, there’s a line:

ct.append((123 * char + 18) % 256)

This means during encryption, each character (char) is multiplied by 123, then 18 is added, and finally, the result is taken modulo 256.

In the decryption script, the goal is to reverse this process. The line:

char = 179 * char % 256

performs the reverse calculation. It uses 179 as the modular multiplicative inverse of 123 modulo 256. In modular arithmetic, the modular multiplicative inverse of a number a modulo m is a number b such that (a * b) % m == 1. In this case, 179 is chosen because (123 * 179) % 256 == 1, allowing the decryption process to reverse the effect of the original multiplication and modulo operation.

Let’s execute it …

Flag

The task was not too challenging, and we successfully obtained the flag:

HTB{l00k_47_y0u_r3v3rs1ng_3qu4710n5_c0ngr475}

Real-Time Trading App: Golang, Kafka, Websockets — Intro & Setup (PART-1)

Kamalesh-Seervi — Wed, 13 Dec 2023 14:15:09 +0000

Real-Time Trading App: Golang, Kafka, Websockets — Intro & Setup (PART-1)

High-performance real-time trading engine with Golang, Kafka, and Websockets…

This blog will be structured into four parts:

Introduction & Setup
Golang Integration with Kafka
Consumer Service & Websockets Implementation
Frontend Development

System Design

This article serves as a comprehensive guide to building a real-time trading platform using Golang, Kafka, and Websockets. Let’s delve into the rationale behind our choice of components:

1. Golang: Golang’s direct compilation to machine code and its simplicity make it an ideal choice for low-latency systems. While debates exist about language preferences, Golang stands out for its efficiency and effectiveness in getting the job done.

2. Kafka: Despite the bidirectional capabilities of Websockets, we opt for Kafka for two key reasons. Firstly, Websockets solely facilitate data exchange without data storage capabilities. For timely financial data analysis, multiple services may be necessary. Secondly, Kafka’s distributed architecture aligns seamlessly with stock market platforms operating across various countries.

3. Websockets: The need for Websockets arises from Kafka’s default design for multi-host environments. In Kafka, a single partition cannot be subscribed to by more than one consumer. This poses challenges when multiple tabs of the same app are open, as each tab receives messages sequentially. Additionally, the scarcity of client-side libraries for Kafka, especially for the web, makes Websockets a more practical choice across various platforms.

4. Web App: For the web application, we’ve chosen ReactJS to craft an intuitive and responsive user interface. Leveraging the power of React components and its virtual DOM, we aim to create a seamless and interactive trading experience. The real-time data received from Kafka via Websockets will be efficiently rendered using React components.

Setup

Let’s kick off the implementation. We’ll utilize Binance’s open-source WebSocket connection to establish our data source. We’ll subscribe to the following tickers to receive real-time updates:

btcusdt,ethusdt,busdusdt,bnbusdt,ltcusdt,xrpusdt,maticusdt

Now that we have decided on our data source, it’s time to set up Kafka.

I will be using docker images from bitnami.

version: '3.8'
services:
  zookeeper:
    env_file:
    - ./.env
    image: bitnami/zookeeper
    expose:
    - "2181"
    ports:
    - "2181:2181"

  kafka:
    image: bitnami/kafka
    env_file:
    - ./.env
    depends_on:
    - zookeeper
    ports:
    - '9092:9092'
    environment:
      KAFKA_ADVERTISED_LISTENERS: INSIDE://kafka:9093,OUTSIDE://localhost:9092
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: INSIDE:PLAINTEXT,OUTSIDE:PLAINTEXT
      KAFKA_LISTENERS: INSIDE://0.0.0.0:9093,OUTSIDE://0.0.0.0:9092
      KAFKA_INTER_BROKER_LISTENER_NAME: INSIDE
      KAFKA_ZOOKEEPER_CONNECT: "zookeeper:2181"
      KAFKA_BROKER_ID: 1

To commence, we’ll set up two essential services: Zookeeper and Kafka. For those unfamiliar with Zookeeper, it serves as a centralized cluster management system developed by Apache. Typically employed in distributed systems, Zookeeper plays a crucial role in addressing questions related to Kafka’s operation, including:

Determining the broker responsible for handling the publish/subscribe functionality for a given topic and partition.
Managing the count of nodes or server instances available in the cluster.
Providing insights into available topics, data retention settings, and more.

While this provides a brief overview, you can delve deeper into Zookeeper’s functionalities here.

The question may arise: why deploy Zookeeper when, for testing and development purposes, we don’t necessarily require multiple nodes?

While Kafka can technically operate without Zookeeper, it’s essential to note that Apache does not recommend doing so in production environments. Hence, for consistency and best practices, we opt to incorporate Zookeeper from the outset, aligning with Apache’s guidelines for a robust and reliable setup.

Here is the ** .env file**

KAFKA_HOST=kafka
KAFKA_PORT=9092

#Zookeeper
ALLOW_ANONYMOUS_LOGIN=yes
ZOO_PORT_NUMBER=2181

#Kafka
KAFKA_CFG_ZOOKEEPER_CONNECT=zookeeper:2181
ALLOW_PLAINTEXT_LISTENER=yes
KAFKA_CFG_ADVERTISED_LISTENERS=PLAINTEXT://kafka:9092

TICKERS="btcusdt,ethusdt,busdusdt,bnbusdt,ltcusdt,xrpusdt,maticusdt"

Proceed with the following command to initiate the setup:

sudo docker-compose up -d

These steps should suffice for configuring Kafka and Zookeeper in your local environment.

Conclusion

We’ve successfully laid the foundation for our real-time trading application by setting up Kafka and Zookeeper in our local environment. In the upcoming week, be on the lookout for Part 2 of this series, where we’ll delve into the integration of Golang with Kafka. Stay tuned for a deeper exploration of how these technologies synergize to create a robust and efficient real-time trading platform. Happy coding!

GitHub - Kamalesh-Seervi/Real-time-trade-app: Kafka, WebSockets

WPA-WPA2 Wi-Fi Hacking: A Step-by-Step Guide.

Kamalesh-Seervi — Mon, 20 Nov 2023 17:58:11 +0000

Disclaimer: The content provided in this blog is intended for educational purposes only. The information shared here is meant to contribute to the understanding of cybersecurity and ethical hacking in a responsible and legal manner.

Aircrack-ng

Introduction

Wi-Fi security is a crucial aspect of digital privacy, and understanding how vulnerabilities can be exploited is essential for network administrators and cybersecurity enthusiasts. In this tutorial, we’ll explore the process of hacking WPA-WPA2-protected Wi-Fi networks for educational purposes using Aircrack-ng on Kali Linux.

Tools Required

Before diving into the process, make sure you have the following tools:

Network Adapter (e.g., TL-WN722N V2) with monitoring mode support.
Kali Linux installed on your machine.
Aircrack-ng
Airodump-ng
Airmon-ng
Crunch

Monitoring Mode Setup

Step 1: Kill Interrupting Services

Before enabling monitoring mode, identify and kill services that might interrupt the process:

sudo airmon-ng check wlan0
sudo airmon-ng check kill

Step 2: Enable Monitoring Mode

Stop the WLAN interface and enable monitor mode:

ifconfig wlan0 down
iwconfig wlan0 mode monitor
ifconfig wlan0 up

Verify the mode using:

iwconfig

Packet Capture and 4-way Handshake

Step 3: Capture BSSID and Monitor Network

Use airodump-ng to capture BSSID information:

airodump-ng wlan0

Select a specific BSSID for monitoring and run:

airodump-ng -c 1 -w Scan_network --bssid EW:WV:4H:J7:A5:28 wlan0

Run the above command in the background.

Step 4: Deauthentication Process

Deauthenticate the target Wi-Fi to capture the 4-way handshake:

sudo aireplay-ng -0 0 -a EW:WV:4H:J7:A5:28 wlan0

Run the command until you see the 4-way handshake in the background code.

Final Stage: Password Cracking with Crunch and Aircrack-ng

After capturing the 4-way handshake, the final step involves cracking the Wi-Fi password using Crunch and aircrack-ng. It's important to note that the success of this process heavily depends on various factors, including the complexity and length of the password.

Using Crunch for Password Generation

In this example, I used Crunch to generate possible passwords. Since I knew the Wi-Fi password consisted of only numeric characters, the command was tailored accordingly:

sudo crunch 8 8 123456780 | aircrack-ng -w - Scan_Kamalesh-01.cap -e KamaleshD

Note:

Modify the crunch command based on your knowledge of the password, such as adjusting the length or character set.

Password Length and OSINT

If you don’t know the password, attempting to crack it by testing all possible combinations is an extremely time-consuming process. The number of possibilities can be in the billions, making it practically impossible to crack on a standard PC within a reasonable timeframe.

In such cases, consider leveraging Open Source Intelligence (OSINT) techniques. Analyze social footprints, gather information about the target’s preferences, and try to determine the likely length and complexity of the password. This approach can significantly reduce the search space and increase the chances of successful password guessing.

Remember, ethical hacking involves responsible and legal use of knowledge. Always respect privacy and adhere to ethical standards when engaging in cybersecurity activities.

Conclusion

Ethical hacking involves understanding vulnerabilities to enhance security measures. This guide aims to provide insights into Wi-Fi security, emphasizing responsible and legal use of knowledge. Always respect privacy and adhere to ethical standards when exploring cybersecurity.

GitHub - Kamalesh-Seervi/WPA-WPA2-wifi-hacking: Aircrack-ng

CamPhish: Understanding and Safeguarding Your Privacy

Kamalesh-Seervi — Wed, 15 Nov 2023 15:46:13 +0000

Disclaimer: Do not use this technique to hack anyone unless you have permission.

To begin, acquire the CamPhish tool by visiting its GitHub page through the providedlink . Alternatively, you can obtain it by entering the following command into your terminal.

git clone https://github.com/techchipnet/CamPhish

After Installing the tool just run it and you should see the following options

[01] Ngrok

You can proceed with any option, but I suggest opting for option 1, which involves using Ngrok. Ngrok is the tool that facilitates exposure in this process.

Disregard any misconceptions; it’s not what you might be assuming. The purpose is to expose your home network to the Internet. Visit the official Ngrok website, create an account, and obtain your authentication token. Save this token for future use. Next, when prompted to choose a template, I suggest selecting option 2 for the YouTube link, as these options might raise fewer suspicions. Simply choose 2 and press ENTER to proceed.

[02] Live Youtube TV

Once you opt for the Live YouTube TV option, the system will prompt you to provide a watch ID, which you can obtain directly from YouTube. Here’s a demonstration: pick any video on youtube.com; for instance, I’ll choose the song “What is Love.” Check the URL, and you’ll find the watch ID. Refer to the screenshot below for clarification:

Original URL: https://www.youtube.com/watch?v=UyQm4O9G7OM&list= Extracted ID: RDUyQm4O9G7OM

watch ID [YouTube]

The watch ID is the text following watch?v=in the URL. Copy this ID and paste it into the terminal when prompted by the program for the watch ID.

Visit the ngork website, create an account, and navigate to the dashboard. Once there, go to “Authtoken,” copy the token, and paste it into the terminal as shown in the image below.

ngrok AuthToken

Token ID

Following this, it will initiate a PHP server and an Ngrok server. You will receive a link that you can then share with your target. To succeed in this, you’ll require social engineering skills to entice clicks on your provided link.

In the provided Direct link, once the victim clicks on it, they will be directed to a YouTube page featuring the selected song. Simultaneously, a unique process begins, and as depicted in the screenshot below, I successfully obtained images of the victim, securely stored in my CamPhish folder.

Images

Note

Install Ngrok directly from the official site instead of using the initially downloaded inbuilt Ngrok. Upon running CamPhish for the first time, delete the existing Ngrok files. Download the executable file from the official Ngrok site and move it to the CamPhish folder. This step is necessary due to issues with the inbuilt Ngrok, ensuring the proper functioning of the tool.

Recognize how simple it is to compromise someone’s camera? Stay vigilant against cybercrime and scams, and make sure to educate your parents or friends on these potential threats. This knowledge will help safeguard them from falling victim to cyber-attacks, especially since older and non-technical individuals are often targeted.

Safety and tips:

Be Skeptical of Unsolicited Communications:

Be cautious of unexpected emails, messages, or social media requests, especially if they contain urgent or alarming messages.

2. Sender Information:

Double-check the sender’s email address or contact details. Legitimate organizations usually have official and recognizable communication channels.

3. Don’t Click on Suspicious Links:

Avoid clicking on links in emails or messages from unknown sources. Hover over links to preview the URL before clicking.

4. Use Two-Factor Authentication (2FA):

Enable 2FA whenever possible to add an extra layer of security to your accounts.

5. Keep Software and Systems Updated:

Regularly update your operating system, antivirus software, and other applications to patch vulnerabilities.

6. Educate Yourself and Others:

Stay informed about common phishing tactics and educate friends and family members to recognize and avoid potential threats.

7. Use Reliable Security Software:

Install and regularly update reputable antivirus and anti-malware software to detect and block phishing attempts.

8. Be Wary of Requests for Personal Information:

Avoid sharing sensitive information such as passwords, credit card details, or social security numbers through email or unfamiliar websites.

9. Check Website Security:

Before entering personal information on a website, ensure the site is secure. Look for “https://” in the URL and check for a padlock symbol.

10. Trust Your Instincts:

If something feels off or too good to be true, it probably is. Trust your instincts and proceed with caution.

By following these tips, you can significantly reduce the risk of falling victim to phishing attempts and enhance your overall online security.

Mastering Nmap: Essential Commands and Examples for Network Security

Kamalesh-Seervi — Sun, 05 Nov 2023 06:51:33 +0000

Discover the Power of Nmap Scanning and Enumeration Techniques to Strengthen Your Network Defense

What is Nmap?

Nmap stands for Network Mapped (Nmap) and is a network scanning and host detection tool that is very useful during several steps of penetration testing.

Nmap is open source and can be used to:

Detect the live host on the network (host discovery)

Detect the open ports on the host (port discovery or enumeration)

Detect the software and the version to the respective port (service discovery)

Detect the operating system, hardware address, and the software version

Detect the vulnerability and security holes (Nmap scripts)

Nmap Syntax:

nmap [scan type] [options] [target specification]

Nmap Scan types:

TCP SCAN
UDP SCAN
SYN SCAN
ACK SCAN
FIN SCAN
NULL SCAN
XMAS SCAN
RPC SCAN
IDLE SCAN

Before delving into advanced Nmap concepts, let’s first explore the fundamentals of NSlookup. NSlookup is a command-line tool used to query DNS servers and retrieve information about domain names. To illustrate, consider the following example:

NSlookup is a command-line tool for querying DNS servers to retrieve information about domain names, such as their associated IP addresses.

Example: To find the IP address of a domain, you can use NSlookup like this:

nslookup google.com

O/P

Following our exploration of NSlookup, we’ll now transition to Nmap, an advanced network scanning tool. We’ll begin by discussing Nmap’s core concepts and provide an example to illustrate its functionality.

Nmap — Advanced Scanning (Best practices)

1. Scanning and Logging Network Data with Nmap

In the provided Nmap command, several options and parameters are used to perform a network scan and save the results:

nmap: This is the command itself, invoking the Nmap tool.
-oG -: This option instructs Nmap to generate output in the "grepable" format and sends it to the standard output (stdout).
192.168.29.238: This is the target IP address (or hostname) you want to scan. Nmap will perform its scanning and testing on this target.
-vv: The -v option stands for "verbose." Using it twice (-vv) increases the verbosity level, providing more detailed information during the scan.
> Desktop/results: The > symbol is used to redirect the output of the Nmap command to a file named "results" on the desktop. This will create a file containing the scan results in the current user's Desktop directory.

nmap -oG - <ip> -vv > Desktop/results

O/P

2. Scanning Particular with Nmap

This Nmap command is designed to perform an extensive network scan targeting a range of IP addresses and specifically focusing on particular port services:

nmap: This is the Nmap command to initiate network scanning.
-oG -: The -oG option instructs Nmap to generate output in the "grepable" format, while the hyphen - indicates that the output should be sent to the standard output (stdout).
192.168.29.0-255: This specifies a range of IP addresses from 192.168.29.0 to 192.168.29.255. The scan will be conducted on all IP addresses within this range.
-p 22: The -p option is used to specify the port number to scan, and in this case, it's set to 22. Port 22 is the default port for SSH (Secure Shell), a protocol used for secure remote access to system.
-vv: The -v option is for "verbose" mode, and using it twice (-vv) increases the verbosity, providing detailed information during the scan.
> Desktop/results: The > symbol is used to redirect the output of the Nmap command to a file named "results" on the desktop.

 nmap -oG - <ip>-<range> -p 22 -vv > Desktop/results

O/P

Nmap — Aggressive Scanning

This Nmap command performs an “Aggressive” scan on the target “scanme.nmap.org.” The -A option in the command instructs Nmap to enable version detection, script scanning, and traceroute to provide a more detailed and comprehensive assessment of the target system. "scanme.nmap.org" is a service provided by Nmap that allows users to test their Nmap scanning skills on a safe and controlled target. The result of this scan will include information about open ports, services, operating system details, and potential vulnerabilities, making it an extensive reconnaissance effort.

nmap -A scanme.nmap.org

-A (aggressive scan)

2. The Nmap below command is used to perform a version detection scan on the target "scanme.nmap.org." Here's an explanation of the command:

-sV: The -sV option is used to enable version detection during the scan. When this option is included, Nmap attempts to determine the version of the services running on the target by analyzing their responses to various probes. This can help identify not only the service but also the specific version of the service (e.g., Apache 2.4.7).

nmap -sV scanme.nmap.org

-sV (Services)

3. The Nmap below command is used to perform a fast scan on the target "scanme.nmap.org." Here's an explanation of the command:

-F: The -F option is a shorthand for the "fast" scan mode. When you use this option, Nmap performs a quick scan focused on identifying the most common open ports and their associated services. This is also known as a "fast scan" or a "top 1000 ports" scan, and it's designed to be less time-consuming than a comprehensive scan.

nmap -F scanme.nmap.org

-F (fast mode)

The Nmap below command is used to scan the target "www.google.com" while displaying only the open ports. Here's an explanation of the command:

--open: The --open option instructs Nmap to display only the open ports and services discovered during the scan. This means that it will filter out closed or filtered ports from the scan results, providing a concise list of the open ports that are actively accepting connections.
www.google.com: This is the target hostname or domain name, in this case, "www.google.com." The scan is performed on Google's web servers.

nmap --open www.google.com

— open (shows open ports)

Resources:

Hacktoberfest 2023 Pledge

Kamalesh-Seervi — Wed, 25 Oct 2023 14:57:22 +0000

Officially registered for #Hacktoberfest2023!

Looking forward to discovering awesome repositories and contributing new features and fixes.

Tags:

webdev
beginners
opensource
hacktoberfest23
Hacktoberfest2023

Hacktoberfest2023

Kamalesh-Seervi — Wed, 25 Oct 2023 14:54:29 +0000

Intro

Hello, I'm Kamalesh Seervi, and I'm excited to share my experience with Hacktoberfest 2023. This is my first Hacktoberfest; I've have participated in this event for first time. You can find my contributions on my GitHub profile.

Highs and Lows

This Hacktoberfest had its fair share of highs and lows. One of the biggest accomplishments for me was successfully contributing to an open-source project that I've been following for a long time. It was a light-bulb moment because I had always thought contributing to this project was beyond my capabilities, but with determination and guidance from the project maintainers, I was able to make meaningful contributions.

However, there were also some challenges during the month. I encountered a complex issue in one of the projects I was contributing to. It seemed impossible to fix at first, and I was frustrated. But instead of giving up, I decided to reach out to the project's community for help. This experience taught me the importance of seeking assistance and collaborating with others in the open-source community. Eventually, we solved the issue together, which was a valuable learning experience.

Growth

Before Hacktoberfest 2023, my skillset was primarily centered around web development and a bit of scripting. However, during this month, I expanded my knowledge and skills. I delved into the world of open-source contributions, which included working on projects in various programming languages, like Python, JavaScript, and even C++. I improved my understanding of version control systems, code review processes, and collaborating with a diverse group of developers.

My learning and career goals have evolved as well. I've realized the importance of giving back to the open-source community, and I plan to continue contributing to projects that align with my interests. This experience has opened up new opportunities for me, and I'm considering a shift towards a more open-source-centric career path. Hacktoberfest 2023 has been a transformative journey, and I'm looking forward to the continued growth and learning that open source has to offer.

Setting up Prometheus and Grafana Integration on Kubernetes with Helm

Kamalesh-Seervi — Sun, 22 Oct 2023 15:45:45 +0000

In this comprehensive guide, you will gain insight into the process of seamlessly integrating Prometheus and Grafana within your Kubernetes environment using Helm. Furthermore, you’ll discover how to construct a straight forward dashboard in Grafana. Prometheus and Grafana stand as two highly favored open-source monitoring solutions for Kubernetes.

Acquiring the skill to deploy them via Helm empowers you to efficiently oversee your Kubernetes cluster and swiftly resolve issues. This proficiency will also deepen your comprehension of your cluster’s overall health and performance, allowing you to diligently monitor resource allocation and performance metrics in your Kubernetes environment.

Prerequisites

To get started with this guide, make sure you have the following prerequisites:

Docker Installation: Follow the official Docker documentation to install Docker on your machine.
Kubectl Installation: Install Kubectl on your local machine for communication with your Kubernetes cluster. Refer to the official Kubectl documentation for guidance.
Basic Kubernetes Knowledge: It’s helpful to have some fundamental knowledge of Kubernetes. You can either consult the Kubernetes official documentation or access Semaphore’s free ebook titled “CI/CD with Docker and Kubernetes,” which doesn’t require prior Docker or Kubernetes expertise.
Kubernetes Cluster Setup: You’ll be deploying Prometheus and Grafana on your Kubernetes cluster. In this guide, we’ll use Minikube, a free local Kubernetes cluster. Alternatively, you can opt for managed cloud-based Kubernetes services such as Google Kubernetes Engine (GKE), Azure Kubernetes Service (AKS), Amazon Elastic Kubernetes Service (EKS), or DigitalOcean Kubernetes Service (DOKS). Keep in mind that some of these cloud-based services may involve a cost, while others offer free plans.

By meeting these prerequisites, you’ll be ready to integrate Prometheus and Grafana seamlessly into your Kubernetes environment.

What is Prometheus?

Prometheus stands as an open-source DevOps utility, offering robust monitoring and real-time alerting features tailor-made for container orchestration platforms like Kubernetes. This tool excels in collecting and storing metrics as time series data, making it an ideal choice for tracking and analyzing platform performance. One of its standout attributes is its innate ability to monitor the container orchestration platform, making it an invaluable data source for various data visualization libraries, including Grafana.

The metrics Prometheus captures from the Kubernetes cluster encompass:

Health status of the Kubernetes cluster.
CPU utilization statistics.
Memory consumption metrics.
Node status within the Kubernetes infrastructure.
Insights into potential performance bottlenecks.
Performance metrics.
Resource allocation and utilization across server components.

In essence, Prometheus plays a pivotal role in ensuring the health and performance of a Kubernetes cluster, making it an essential tool for DevOps and system administrators.

What is Grafana?

Grafana is a versatile, open-source tool for visualizing data. When it’s connected to data sources like Prometheus, it offers features like interactive dashboards, charts, graphs, and web alerts. You can use Grafana to view and understand your data from various sources, not just Prometheus, including InfluxDB, Azure Monitor, and others.

You can build your own dashboards or use pre-made ones and customize them as needed. Many DevOps professionals use Grafana and Prometheus to create powerful databases and visual displays for tracking data over time. In this guide, we’ll show you how to make a dashboard for visualizing metrics from Prometheus.

Choosing the Right Deployment Method for Prometheus and Grafana Integration on Kubernetes

Manual Kubernetes Deployment: In this method, you’re required to create Kubernetes Deployment and Services YAML files for both Prometheus and Grafana. These YAML files must include all the necessary configurations to enable integration with Kubernetes. Subsequently, you deploy these files to your Kubernetes cluster to make Prometheus and Grafana operational. This process may result in multiple YAML files, which can be somewhat burdensome for many DevOps professionals. Additionally, a single mistake in any YAML file can impede the integration of Prometheus and Grafana on Kubernetes.
Using Helm: This stands out as the simplest and most convenient method for deploying applications in containers to Kubernetes. Helm serves as the official package manager for Kubernetes and streamlines the installation, deployment, and management of Kubernetes applications. Helm packages and encapsulates the Kubernetes application within a Helm Chart. A Helm Chart encompasses all the essential YAML files, including Deployments, Services, Secrets, and ConfigMaps manifests. These files are instrumental in deploying the application container in Kubernetes. Instead of crafting individual YAML files for each application container, Helm offers the convenience of downloading pre-existing Helm charts that come equipped with the necessary manifest YAML files.

Installing Helm

Before you install Helm, you must start your Minikube Kubernetes using the following command:

Installing Helm on macOS

brew install helm

Installing Helm on Linux

sudo apt-get install helm

Installing Helm on Windows

choco install Kubernetes-helm

Helm Commands

To get all the Helm commands, run this command:

helm

The command output:

Helm commands

Here are the fundamental Helm commands:

helm search: Search for Helm Charts in the ArtifactHub repository.
helm pull: Retrieve and download a Helm Chart from the ArtifactHub repository.
helm install: Upload and deploy a Helm Chart to your Kubernetes cluster.
helm list: Display a list of all deployed Helm charts within your Kubernetes cluster.

Prometheus Helm Charts

Let’s begin by searching for the Prometheus Helm Charts. To find the Prometheus Helm, use the following command:

helm search hub prometheus

The command lists the following Prometheus Helm Charts:

Prometheus Artifacts

You can also go to the ArtifactHub repository and search for the official Prometheus Helm Chart as shown in the image below:

Artifacts site

The first one on the list is the official Prometheus Helm Chart. To get this Helm chart, run this command:

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update

Install Prometheus Helm Chart on Kubernetes Cluster

To install Prometheus Helm Chart on Kubernetes Cluster, run this helm install command:

helm install prometheus prometheus-community/prometheus

After successfully installing Prometheus on your Kubernetes Cluster, you can access the Prometheus server via port 80. The subsequent task is to inspect the Kubernetes resources that have been deployed. These resources consist of the pods and services generated by the Helm Chart within your Kubernetes cluster.

To examine the deployed Kubernetes resources, execute the following kubectl command:

kubectl get all

The installation of the Helm Chart results in the creation of several essential Kubernetes resources, including:

Pods: These pods host the Prometheus Kubernetes application within the cluster.
Replica Sets: A collection of instances of the same application inside the Kubernetes cluster, enhancing application reliability.
Deployments: These deployments serve as the blueprint for creating application pods.
Services : Services are responsible for exposing the pods running within the Kubernetes cluster, allowing us to access the deployed Kubernetes application.

The subsequent action involves accessing and launching the Prometheus Kubernetes application. You can access the application through the Kubernetes services designated for Prometheus. To obtain a list of all the Kubernetes Services associated with Prometheus, execute the following command:

kubectl get service

Exposing the prometheus-server Kubernetes Service

To expose the prometheus-server Kubernetes service, run this command:

kubectl expose service prometheus-server --type=NodePort --target-port=9090 --name=prometheus-server-external

This command will convert the ClusterIP type to the NodePort type. It makes the prometheus-server accessible outside the Kubernetes Cluster on port 9090.

Now we have exposed the prometheus-server Kubernetes service. Let’s access the Prometheus application using the following command:

minikube service prometheus-server-external

Please be aware that it may require some time for the URL to become accessible. You might need to make multiple attempts in your web browser until you successfully access the Prometheus Kubernetes application through the provided URL. It’s important to keep the terminal open and the tunnel command running to ensure continuous access to the service.

With Prometheus successfully installed on Kubernetes via Helm, Prometheus is up and running within the cluster, and it’s accessible through a browser via a URL. Moving on to the next steps in the tutorial:

We will proceed to install Grafana.
Subsequently, we’ll establish the integration between Prometheus and Grafana. Grafana will utilize Prometheus as its primary data source.
Finally, we will employ Grafana to craft the dashboards essential for monitoring and observing the Kubernetes cluster.

Install Grafana

To install, we follow the same steps as those for installing Prometheus:

Search for Grafana Helm Charts

To search for the Prometheus Helm charts, run this command:

helm search hub grafana

You can also go to the ArtifactHub repository and search for the official Grafana Helm Chart as shown in the image below:

To get this Grafana Helm chart, run this command:

helm repo add grafana https://grafana.github.io/helm-charts 
helm repo update

Install Grafana Helm Chart on Kubernetes Cluster

You’ll run this helm install command:

helm install grafana grafana/grafana

We have installed Grafana on the Kubernetes Cluster. We can access the Grafana server via port 80. The next step is to access and launch the Grafan application. You will access the application using the Kubernetes services for Grafana. To get all the Kubernetes Services for Grafana, run this command:kubectl get service

Exposing the grafana Kubernetes Service

To expose the grafana Kubernetes service, run this command:

kubectl expose service grafana --type=NodePort --target-port=3000 --name=grafana-ext

This command will convert the ClusterIP type to the NodePort type. It makes the grafana accessible outside the Kubernetes Cluster on port 3000. Now we have exposed the grafana Kubernetes service. Let’s access the Grafana application using the following command:

minikube service grafana-ext

The image above shows the Grafana Login page. To get the password for admin, run this command on a new terminal.

kubectl get secret --namespace default grafana -o jsonpath="{.data.admin-password}" | base64 --decode ; echo

Login into Grafana

To login into Grafana, input admin as the user name and your generated password. It will launch a Welcome to Grafana home page as shown below:

To add Prometheus as the data source. To add Prometheus as the data source, follow these steps:

On the Welcome to Grafana Home page, click Add your first data source:
Select Prometheus as the data source:
You will then add the URL where your Prometheus application is running. This is the first URL (internal to the cluster) shown when we ran minikube service prometheus-server-external earlier.

Click the “Save & test” button to preserve your modifications.

With that, you have successfully completed the integration of Prometheus and Grafana on Kubernetes using Helm. The final phase involves crafting a Grafana Dashboard, a pivotal step in visualizing the metrics for your Kubernetes cluster.

Grafana Dashboard

As previously mentioned, you have the flexibility to either create your own dashboards from the ground up or import existing ones provided by Grafana. In this section, we’ll walk you through the process of importing a Grafana Dashboard. To import a Grafana Dashboard, follow these steps:

Retrieve the Grafana Dashboard ID from the public Grafana Dashboard library.

Kubernetes Monitoring with Grafana

On this web page, search for Kubernetes:
Scroll until you find the Kubernetes cluster monitoring (via Prometheus) dashboard

Select Dashboard and copy the Dashboard ID:
Go Back to Grafana and click Home on the top left corner:
It will display a menu.

On the menu, click Dashboards

Click New
It will display three options: New Dashboard, New Folder and Import.
Click Import
dd the Grafana ID: You will add the Grafana ID that you have copied and click Load. The Grafana ID is 315.

It will the launch the Dashboard shown below:

You use this dashboard to monitor and observe the Kubernetes cluster metrics. It displays the following Kubernetes cluster metrics:

Network I/O pressure.
Cluster CPU usage.
Cluster Memory usage.
Cluster filesystem usage.
Pods CPU usage.

Advantages of Configuring Prometheus and Grafana for Container Orchestration Platform Monitoring

The deployment of Prometheus and Grafana for monitoring purposes offers several notable benefits:

It delivers a comprehensive, all-encompassing solution for monitoring and overseeing a Kubernetes cluster.
You gain the capability to perform metric queries using Prometheus’s PromQL query language, facilitating in-depth analysis.
In the context of a microservices architecture, Prometheus efficiently tracks all your microservices concurrently, ensuring no aspect goes unmonitored.
Immediate alerts are triggered when a service encounters a failure, allowing for swift corrective actions.
The Grafana dashboard provides comprehensive performance and health reports for your clusters, offering valuable insights and visual representations of your system’s state.

Conclusion

In this comprehensive guide, you’ve acquired the knowledge needed to seamlessly integrate Prometheus and Grafana into your Kubernetes environment using Helm. Furthermore, you’ve gained insights into creating a straightforward dashboard in Grafana, enabling you to monitor resource utilization and performance metrics across your Kubernetes cluster.

Monitoring plays a pivotal role in DevOps, ensuring that you maintain visibility into your Kubernetes cluster and the performance of microservices. Implementing these practices is essential for real-time updates on your cluster’s health, allowing you to stay informed about its current status.

That brings us to the conclusion of this Prometheus and Grafana guide. Thank you for reading, and here’s to your continued learning and success!