Logging is a critical component of any application, allowing developers to easily identify and troubleshoot issues that arise during runtime. However, as applications become more complex, it can be challenging to manage and analyze logs in a way that is both efficient and effective. This is where Logstash comes in. Logstash is a powerful tool that simplifies the process of collecting, processing, and storing logs in a centralized location. In this blog post, we will explore how to push logs from a Go app to Logstash using the go-logstash package.
What is Logstash?
Logstash is an open-source tool that allows developers to easily ingest, process, and store logs. It is part of the Elastic Stack (also known as ELK), which includes Elasticsearch and Kibana. Logstash provides a variety of input and output plugins that allow it to collect logs from a wide range of sources, including files, TCP/UDP sockets, and messaging systems like Kafka and RabbitMQ. Once collected, Logstash can process and enrich the logs before forwarding them to Elasticsearch for storage and analysis.
Introducing go-logstash
go-logstash is a Golang package that provides a simple interface for pushing logs to Logstash. It supports both TCP and UDP protocols and can output logs in either JSON or string format. go-logstash is easy to use and provides customizable options for configuring the Logstash connection.
Setting up Logstash
Before we can start pushing logs to Logstash, we need to set up a Logstash instance. This can be done by downloading and installing Logstash from the Elastic website or setting it up using Docker Compose.
Create a new directory, let's call it go-logstash-demo
and let's also create the necessary files that we need:
mkdir go-logstash-demo && cd go-logstash-demo
touch docker-compose.yml
touch docker-setup/logstash/Dockerfile
touch docker-setup/logstash/logstash.conf
touch docker-setup/go-logger/Dockerfile
touch main.go
Here is the docker-compose.yml
file that creates the ELK stack, Elastic Search, Logstash, and Kibana each into its own container, along with a container for the Go application.
# docker-compose.yml
version: "3.9"
services:
elasticsearch:
image: elasticsearch:7.1.0
volumes:
- ./esdata:/usr/share/elasticsearch/data
ports:
- "9200:9200"
- "9300:9300"
environment:
- "discovery.type=single-node"
networks:
- elk
logstash:
build:
context: .
dockerfile: docker-setup/logstash/Dockerfile
ports:
- 9600:9600
- 5228:5228
environment:
LOGSTASH_PORT: 5228
LOGSTASH_INDEX: "test-index"
ELASTIC_HOST: "elasticsearch:9200"
ELASTIC_USERNAME: "elastic"
ELASTIC_PASSWORD: "elastic"
networks:
- elk
depends_on:
- elasticsearch
links:
- elasticsearch
kibana:
image: kibana:7.1.0
hostname: kibana
ports:
- "5601:5601"
networks:
- elk
depends_on:
- elasticsearch
links:
- elasticsearch
environment:
ELASTIC_HOST: "http://elasticsearch:9200"
ELASTIC_USERNAME: "elastic"
ELASTIC_PASSWORD: "elastic"
go-app:
container_name: go-app
build:
context: .
dockerfile: docker-setup/go-logger/Dockerfile
networks:
- elk
networks:
elk:
driver: bridge
volumes:
esdata:
driver: local
Here is a custom Dockerfile for the Logstash container because we want to provide a custom logstash.conf
file to it:
# docker-setup/logstash/Dockerfile
FROM docker.elastic.co/logstash/logstash-oss:7.1.0
COPY ./docker-setup/logstash/logstash.conf /etc/logstash/conf.d/
CMD logstash -f /etc/logstash/conf.d/logstash.conf
Here's an example of a basic logstash.conf
file that creates a Logstash pipeline to collect logs from a TCP or UDP socket and output them to Elasticsearch:
# docker-setup/logstash/logstash.conf
input {
tcp {
host => "0.0.0.0"
port => "${LOGSTASH_PORT}"
codec => json_lines
}
udp {
host => "0.0.0.0"
port => "${LOGSTASH_PORT}"
codec => json_lines
}
}
output {
stdout { codec => json_lines }
elasticsearch {
hosts => [ "${ELASTIC_HOST}" ]
user => "${ELASTIC_USERNAME}"
password => "${ELASTIC_PASSWORD}"
codec => json_lines
index => "${LOGSTASH_INDEX}"
}
}
The logstash.conf
config file is capable of supporting environment variables as well, which we are providing through our docker-compose.yml
file. This pipeline listens for logs on TCP port 5228 and expects them to be in JSON format and outputs the logs to Elasticsearch in JSON.
We also need to create a Dockerfile for the Go application, as it would be using the internal Docker network to communicate with Logstash in order to push logs.
# docker-setup/go-logger/Dockerfile
FROM golang:1.19-alpine
WORKDIR /go-logstash-json
COPY . .
RUN go build -o out/logger *.go
CMD [ "./out/logger" ]
Pushing logs from a Go app to Logstash
Now that we have our ELK set up ready on our local, let's take a look at how we can push logs to it from a Go application using go-logstash.
Step 1: Create a Go application
In the root directory of go-logstash-demo
, run:
go mod init example.com/go-logstash-demo
Step 2: Install go-logstash
The first step is to install go-logstash using the following command:
go get github.com/KaranJagtiani/go-logstash
Step 3: Test the Library
Next, we need to import the go-logstash package in our Go application and test the library:
# main.go
package main
import (
logstash_logger "github.com/KaranJagtiani/go-logstash"
)
func main() {
logger := logstash_logger.Init("logstash", 5228, "tcp", 5)
payload := map[string]interface{}{
"message": "TEST_MSG",
"error": false,
}
logger.Log(payload) // Generic log
logger.Info(payload) // Adds "severity": "INFO"
logger.Debug(payload) // Adds "severity": "DEBUG"
logger.Warn(payload) // Adds "severity": "WARN"
logger.Error(payload) // Adds "severity": "ERROR"
}
logger := logstash_logger.Init("logstash", 5228, "tcp", 5)
This line creates a new logger that will send logs to a Logstash instance running on logstash
host in Docker through TCP port 5228 with a connection timeout of 5 seconds.
Conclusion
In conclusion, sending logs from a Go application to Logstash in the ELK stack is a powerful way to centralize and manage your application logs. With the help of the go-logstash library, it's easy to push logs to Logstash using either TCP or UDP. Logstash can then process and enrich the logs, making it easier to analyze and troubleshoot issues in your application.
Thank you for reading this beginner's guide on sending logs from a Go application to Logstash in the ELK stack with Docker setup. I hope that this guide has provided you with a useful overview of the process and has helped you get started with integrating Go and Logstash.
If you have any questions or feedback, please don't hesitate to leave a comment or reach out to me directly. I'm always happy to hear from readers and help in any way I can.
Thanks again for reading, and happy logging!
Top comments (0)