Building a Microservices Architecture with gRPC and Kafka in Golang

Today, I want to take you through the journey of building a robust microservices architecture using gRPC for efficient internal communication and Kafka for reliable event streaming. One of my biggest challenges, which often made me feel like an imposter, was understanding microservices and gRPC and how to set them up correctly. This project is my first approach into constructing a microservices architecture, and I’m excited to share the learnings and outcomes with you.

The concepts and setup explained here were heavily influenced by https://www.youtube.com/@anthonygg_, who provided an excellent foundation and clear guidance that I could build upon.

What’s Under the Hood?

The architecture is composed of several key services:

• gateway: The entry point for all client requests, translating JSON HTTP requests to gRPC.
• producer-micro: Handles incoming driver data and produces messages to Kafka.
• consumer-micro: Consumes messages from Kafka and persists them into the database.
• write-db-micro: Microservice responsible for database write operations.
• read-db-micro: Microservice responsible for database read operations.
• distance-micro: Calculates the distance traveled by drivers.
• spam: This is a simple spam services that spams the gateway with locations.

Build and Run

Getting everything up and running is a breeze with the provided Makefile. Each microservice can be built and started with simple commands.

• Build and run gateway: make gate
• Build and run producer-micro: make prod
• Build and run consumer-micro: make con
• Build and run write-db-micro: make write
• Build and run read-db-micro: make read
• Build and run distance-micro: make dist
• Build and run spam: make spam

To generate gRPC code from .proto files, run: make proto

Docker Setup

This project includes a docker-compose file for easy setup of the microservices and MongoDB as containers. The docker-compose file orchestrates the creation and interconnection of all the necessary containers.

• Build the container : docker-compose up -d

API Endpoint

The following endpoint is available on the gateway for drivers to send their data:

- http://localhost:3000/api/v1/drivers

- http://localhost:3000/api/v1/drivers/1

Feel free to dive into the README on GitHub for more detailed instructions on building and running the services.

Dive Into the Code

The complete code for this project is available on GitHub. Check it out, give it a star if you like it, and contribute if you have ideas on how to improve it!

(https://github.com/pyrolass/grpc-microservice-go)