Inside Container Debugging with GO

As developers, a good amount of our time is spent debugging. But sometimes, due to setup constraints and a lack of debugging tools for container development, it just isn’t possible.

When working locally, all you have to do is click Debug and everything works like magic.

When dealing with a remote environment or running inside containers, however, we no longer have the Debug option and we can’t even compile the binary in debug mode.

Some IDEs try to solve this problem, but most of the time their solution isn’t good enough so we are stuck relying on ad-hoc logging.

The real issue is that containers aren’t built for active development. They raise barriers around restarting processes, re-compiling binaries on the fly, and otherwise dynamically modifying the runtime. As a result, working with containerized environments is cumbersome, and we spend a bunch of time on workarounds:

1. We can add logs and rebuild the image

Because debugging takes so long to setup, we might give up. Instead, we iterate by adding additional ad-hoc logging around the area we are developing. Not only do we get limited information, but our cycle time is super long, since building the image and re-deploying the container (whether locally or remote) can take minutes instead of seconds.

Sometimes this is the best option... if we end up finding the bug in few iterations - otherwise, we’ll end up making progress, but very slowly. The problem is that you can never know how many iterations it will take, and usually we underestimate ;).

2. We can de-containerize the service we are currently developing

Leaving all the other containers in the environment up, we can remove stop the container with the service we are currently developing, and run it locally. This allows us to debug and run with great built-in-IDE tooling!

Unfortunately, this means we need to have a working local setup for all of the containers we develop, and if we have more than a few - keeping these working can be a hassle. We also may need to handle proxying the network traffic using tools like telepresence. We also need to remember to re-containerize it when we are done, or we end up with a very inconsistent setup.

3. We can debug inside the container by installing the required dependencies in the image, and re-building the binary

This is the most scalable approach to debugging inside containers as, following a one-time setup cost, allows fast iterations and full debugging capabilities.

We’ll dive deep into option #3 and explore how to debug GO code inside the container. To make this work, we’ll need to install the debugging infrastructure (delve) and the toolchain so we can compile within the container.

Debugging with GO inside a running container

Let’s start with a simple example of a containerized Go service, and make the necessary changes to debug the service using Delve:

FROM golang:1.18-alpine3.15 as builder

WORKDIR /code/
COPY ./go.mod ./go.sum /code/
RUN --mount=type=cache,target=/go/pkg/mod go mod download

COPY . .
RUN go build -o /code/build/server main.go

FROM alpine:3.15
COPY --from=builder /code/build/server /server

ENTRYPOINT ["/server"]

Step 1: Change how you build the binary

As a first step, let’s modify the go build command so it keeps the debug symbols in the resulting binary:

FROM golang:1.18-alpine3.15 as builder

WORKDIR /code/
COPY ./go.mod ./go.sum /code/
RUN --mount=type=cache,target=/go/pkg/mod go mod download

COPY . .
RUN go build -gcflags "all=-N -l" -o /code/build/server main.go

FROM alpine:3.15
COPY --from=builder /code/build/server /server

ENTRYPOINT ["/server"]

Step 2: Install the required dependencies(such as Delve)

Next, let’s install Delve -

FROM golang:1.18-alpine3.15 as builder

RUN apk add build-base
RUN go install github.com/go-delve/delve/cmd/dlv@latest

WORKDIR /code/
COPY ./go.mod ./go.sum /code/
RUN --mount=type=cache,target=/go/pkg/mod go mod download

COPY . .
RUN go build -gcflags "all=-N -l" -o /code/build/server main.go

FROM alpine:3.15
COPY --from=builder /go/bin/dlv /
COPY --from=builder /code/build/server /

ENTRYPOINT ["/server"]

Step 3: Run the process with dlv - so we can actually debug!

Finally, change the entrypoint to the Delve executable we copied over, and configure it to act as a debug server for our binary.

FROM golang:1.18-alpine3.15 as builder

RUN apk add build-base
RUN go install github.com/go-delve/delve/cmd/dlv@latest

WORKDIR /code/
COPY ./go.mod ./go.sum /code/
RUN --mount=type=cache,target=/go/pkg/mod go mod download

COPY . .
RUN go build -gcflags "all=-N -l" -o /code/build/server main.go

FROM alpine:3.15
COPY --from=builder /go/bin/dlv /
COPY --from=builder /code/build/server /

EXPOSE 2345
ENTRYPOINT ["/dlv", "--listen=:2345", "--headless=true", "--api-version=2", \
            "--accept-multiclient", "exec", "/server"]

Container debugging, like it’s local

In the world of R&D, it’s so crucial to have efficient and easy-to-use dev environments and workflows so we can work frustration-free. The length of the development cycle (from code change to feedback) is a critical component, and debugging gives us very rich feedback for a given code change.
But the lack of support in debugging tools within containerized environments themselves brings constant distractions and halts the workflow and thinking process of your people.

That’s why having the ability - and the simplicity - of debugging in containerized services as if they were local can provide immense benefits for your teams.

Even if containers were never intended to be used by dev in such a fashion, with Raftt, debugging containerized services becomes as easy as running it natively on your machine.

About Raftt

Raftt creates modern flexible environments synced with your tools, features and workflows so you can explore and develop freely. Our cloud platform liberates you from the limitations of your hardware, allowing you to spawn an unlimited number of environments, collaborate and share, and enjoy stability, consistency, and performance.