In previous articles we created a HTTP REST API server, and another kind of applications and we deployed them manually locally.
Deploying applications manually is cool but today, we will try and use Pulumi to deploy, programmatically, in Go of course ^^, our awesome apps.
Ready?
Pulumi
Pulumi is an Infrastructure as code (IasC) tool that allows you to build your infrastructures with a programming language. It supports a variety of programming languages: Python, Node.js, Go, Java, .Net...
Like Terraform, Pulumi have an architecture based on providers/plugins. There are official providers (AWS, GCP, Kubernetes, Docker...) but it is possible to create our own providers too.
How Pulumi is working?
Concretely, users defined the desired state in Pulumi programs and Pulumi create the desired resources.
To provision, update or delete infrastructures, Pulumi have an intuitive Command Line Interface (CLI). If you are familiar with Docker Compose CLI and Terraform CLI, you will adopt Pulumi CLI too.
Let's install the Pulumi CLI.
In this guide we will install it with brew but you can install in many ways, follow the installation guide.
$ brew install pulumi/tap/pulumi
Let's check the CLI is correctly installed locally:
$ pulumi version
v3.77.1
Pre-requisites
If you want to reproduce this article, you have to create an account on Pulumi and have an access token.
On my side I am using a free Pulumi account.
What do we want?
An Infrastructure as Code (IaC) tool is originally used to deploy infrastructures in Cloud providers but we can also handle (deploy, change and destroy) our Go apps and that's what we will do in this article.
We will deploy two applications:
- our cute Gophers API that list existing Gophers, display information, create, update and delete a Gopher
- a Node.js HMI Gophers API Watcher that displays our cute Gophers (who don't love UI? ^^)
As you maybe know, I like to run apps in containers so we will run our apps in containers.
Pre-requisites - Create a Docker image from a Go app
We will use the Docker provider in Pulumi to deploy and run our applications so before we need to have Docker images with our apps.
Let's do it!
Uh... come on Aurélie ... You want really to explain deployment with Pulumi and Docker images creation in only one article?
Yes, with docker init command we can generate necessary Docker files and create our images easily without headaches or "marabout tips".
I have already explained it in video:
So thanks to the docker init command I generated a Dockerfile, built (for different platform and architecture), tagged and pushed the image.
Our Gophers API is available on Docker Hub (with several tags, depending on your host platform): https://hub.docker.com/r/scraly/gophers-api
And the Gophers API watcher too: https://hub.docker.com/r/scraly/gophers-api-watcher
Initialization
First of all, we can create our repository in GitHub (in order to share and open-source it).
For that, I logged in GitHub website, clicked on the repositories link, click on "New" green button and then I created a new repository called “pulumi-gophers”.
Now, in your local computer, git clone this new repository where you want:
$ git clone https://github.com/scraly/pulumi-gophers.git
$ cd pulumi-gophers
Initialize our project (A prompt should ask the Pulumi access token):
$ pulumi new go --force
Manage your Pulumi stacks by logging in.
Run `pulumi login --help` for alternative login options.
Enter your access token from https://app.pulumi.com/account/tokens
or hit <ENTER> to log in using your browser :
Welcome to Pulumi!
Pulumi helps you create, deploy, and manage infrastructure on any cloud using
your favorite language. You can get started today with Pulumi at:
https://www.pulumi.com/docs/get-started/
Tip: Resources you create with Pulumi are given unique names (a randomly
generated suffix) by default. To learn more about auto-naming or customizing resource
names see https://www.pulumi.com/docs/intro/concepts/resources/#autonaming.
This command will walk you through creating a new Pulumi project.
Enter a value or leave blank to accept the (default), and press <ENTER>.
Press ^C at any time to quit.
project name: (pulumi-gophers)
project description: (A minimal Go Pulumi program)
Created project 'pulumi-gophers'
Please enter your desired stack name.
To create a stack in an organization, use the format <org-name>/<stack-name> (e.g. `acmecorp/dev`).
stack name: (gophers)
Created stack 'gophers'
Installing dependencies...
go: downloading github.com/pulumi/pulumi/sdk/v3 v3.60.1
go: downloading golang.org/x/net v0.7.0
...
go: downloading github.com/kr/text v0.2.0
Finished installing dependencies
Your new project is ready to go!
To perform an initial deployment, run `pulumi up`
The command create a gophers stack and the code organization of your project:
$ tree
.
├── go.mod
├── go.sum
├── main.go
├── Pulumi.yaml
└── README.md
Create our application (Pulumi Go program)
Our application will:
- retrieve
Gophers APIDocker image - retrieve
Gophers API WatcherDocker image - create a Docker network (thanks to that our containers should communicate with each other)
- create a
gophers-apicontainer and run it - create a
gophers-api-watchercontainr and run it
For that, we will use the Pulumi official Docker provider.
Let's install Pulumi SDK and Pulumi Docker provider in order to use it in our code:
$ go get github.com/pulumi/pulumi-docker/sdk/v3@v3.6.1
$ go get github.com/pulumi/pulumi/sdk/v3@v3.44.2
Good, now we can create a main.go file and copy/paste the following code into it.
Go code is organized into packages. So, first, we initialize the package, called main, and all dependencies/libraries we need to import and use in our main file:
package main
import (
"fmt"
"github.com/pulumi/pulumi-docker/sdk/v3/go/docker"
"github.com/pulumi/pulumi/sdk/v3/go/pulumi"
"github.com/pulumi/pulumi/sdk/v3/go/pulumi/config"
)
With the imports added, you can start creating the main() function that contains the intelligence of our app:
func main() {
pulumi.Run(func(ctx *pulumi.Context) error {
When you will execute pulumi up command to deploy our applications, Pulumi will run all the code we will wrote in the pulumi.Run(func(ctx *pulumi.Context) error { block code.
Let's write our program.
First, we define the configuration:
//Configuration
protocol := "http://"
//tag := "latest" //for linux/arm64
tag := "linux-amd64" //if you run this program in a linux/amd64 arch, on GitPod for example ;-)
cfg := config.New(ctx, "")
gophersAPIPort := cfg.RequireFloat64("gophersAPIPort")
gophersAPIWatcherPort := cfg.RequireFloat64("gophersAPIWatcherPort")
Then, we pull the Gophers API Docker image from the Docker Hub with the tag you defined in the configuration:
// Pull the Gophers API image
gophersAPIImageName := "gophers-api"
gophersAPIImage, err := docker.NewRemoteImage(ctx, fmt.Sprintf("%v-image", gophersAPIImageName), &docker.RemoteImageArgs{
Name: pulumi.String("scraly/" + gophersAPIImageName + ":" + tag),
})
if err != nil {
return err
}
ctx.Export("gophersAPIDockerImage", gophersAPIImage.Name)
Then, we pull the Gophers API Watcher Docker image from the Docker Hub with the tag you defined in the configuration:
// Pull the Gophers API Watcher (frontend/UI) image
gophersAPIWatcherImageName := "gophers-api-watcher"
gophersAPIWatcherImage, err := docker.NewRemoteImage(ctx, fmt.Sprintf("%v-image", gophersAPIWatcherImageName), &docker.RemoteImageArgs{
Name: pulumi.String("scraly/" + gophersAPIWatcherImageName + ":" + tag),
})
if err != nil {
return err
}
ctx.Export("gophersAPIWatcherDockerImage", gophersAPIWatcherImage.Name)
Our containers will need to connect to each other, so we will need to create a Docker Network:
// Create a Docker network
network, err := docker.NewNetwork(ctx, "network", &docker.NetworkArgs{
Name: pulumi.String(fmt.Sprintf("services-%v", ctx.Stack())),
})
if err != nil {
return err
}
ctx.Export("containerNetwork", network.Name)
Create the Gophers API container:
// Create the Gophers API container
_, err = docker.NewContainer(ctx, "gophers-api", &docker.ContainerArgs{
Name: pulumi.String(fmt.Sprintf("gophers-api-%v", ctx.Stack())),
Image: gophersAPIImage.RepoDigest,
Ports: &docker.ContainerPortArray{
&docker.ContainerPortArgs{
Internal: pulumi.Int(gophersAPIPort),
External: pulumi.Int(gophersAPIPort),
},
},
NetworksAdvanced: &docker.ContainerNetworksAdvancedArray{
&docker.ContainerNetworksAdvancedArgs{
Name: network.Name,
Aliases: pulumi.StringArray{
pulumi.String(fmt.Sprintf("gophers-api-%v", ctx.Stack())),
},
},
},
})
if err != nil {
return err
}
Create the Gophers API Watcher container:
// Create the Gophers API Watcher container
_, err = docker.NewContainer(ctx, "gophers-api-watcher", &docker.ContainerArgs{
Name: pulumi.String(fmt.Sprintf("gophers-api-watcher-%v", ctx.Stack())),
Image: gophersAPIWatcherImage.RepoDigest,
Ports: &docker.ContainerPortArray{
&docker.ContainerPortArgs{
Internal: pulumi.Int(gophersAPIWatcherPort),
External: pulumi.Int(gophersAPIWatcherPort),
},
},
Envs: pulumi.StringArray{
pulumi.String(fmt.Sprintf("PORT=%v", gophersAPIWatcherPort)),
pulumi.String(fmt.Sprintf("HTTP_PROXY=backend-%v:%v", ctx.Stack(), gophersAPIPort)),
pulumi.String(fmt.Sprintf("PROXY_PROTOCOL=%v", protocol)),
},
NetworksAdvanced: &docker.ContainerNetworksAdvancedArray{
&docker.ContainerNetworksAdvancedArgs{
Name: network.Name,
Aliases: pulumi.StringArray{
pulumi.String(fmt.Sprintf("gophers-api-watcher-%v", ctx.Stack())),
},
},
},
})
if err != nil {
return err
}
return nil
})
}
And that's it! We defined everything we want in our infrastructures: 2 applications running in containers thanks to Pulumi.
The configuration
As you saw, we didn't hardcode the apps port number:
cfg := config.New(ctx, "")
gophersAPIPort := cfg.RequireFloat64("gophersAPIPort")
gophersAPIWatcherPort := cfg.RequireFloat64("gophersAPIWatcherPort")
Instead we define them as config parameters. The program will get them in a Pulumi.<your-stack-name>.yaml file.
To define them and generate the config file, execute the following commands:
$ pulumi config set gophersAPIPort 8080
$ pulumi config set gophersAPIWatcherPort 8000
After editing our main.go file and defining our configuration fields and values, it's time to ask to Go to download and install all the Go providers and dependencies:
$ go mod tidy
Let's deploy our apps
Now we can deploy our apps, to do that just execute the pulumi up comand.
This will display the plan/the preview of the desireed state. A prompt will ask you to choose the stack (dev by default) and to confirm of you want to perform/apply the changes.
$ pulumi up
Please choose a stack, or create a new one: gophers
Previewing update (gophers)
View in Browser (Ctrl+O): https://app.pulumi.com/scraly/pulumi-gophers/gophers/previews/cb2a49a5-e17e-4e58-9525-a1931b214b23
Type Name Plan
+ pulumi:pulumi:Stack pulumi-gophers-gophers create
+ ├─ docker:index:RemoteImage gophers-api-watcher-image create
+ ├─ docker:index:RemoteImage gophers-api-image create
+ ├─ docker:index:Network network create
+ ├─ docker:index:Container gophers-api-watcher create
+ └─ docker:index:Container gophers-api create
Outputs:
containerNetwork : "services-gophers"
gophersAPIDockerImage : "scraly/gophers-api:linux-amd64"
gophersAPIWatcherDockerImage: "scraly/gophers-api-watcher:linux-amd64"
Resources:
+ 6 to create
Do you want to perform this update? yes
Updating (gophers)
View in Browser (Ctrl+O): https://app.pulumi.com/scraly/pulumi-gophers/gophers/updates/3
Type Name Status
+ pulumi:pulumi:Stack pulumi-gophers-gophers created (9s)
+ ├─ docker:index:Network network created (2s)
+ ├─ docker:index:RemoteImage gophers-api-watcher-image created (8s)
+ ├─ docker:index:RemoteImage gophers-api-image created (5s)
+ ├─ docker:index:Container gophers-api created (0.97s)
+ └─ docker:index:Container gophers-api-watcher created (1s)
Outputs:
containerNetwork : "services-gophers"
gophersAPIDockerImage : "scraly/gophers-api:linux-amd64"
gophersAPIWatcherDockerImage: "scraly/gophers-api-watcher:linux-amd64"
Resources:
+ 6 created
Duration: 13s
We can check if images have been successfully pulled from the registry:
$ docker image ls
REPOSITORY TAG IMAGE ID CREATED SIZE
scraly/gophers-api-watcher linux-amd64 ee8c626fdeab 3 hours ago 288MB
scraly/gophers-api linux-amd64 83e5cf52694c 3 hours ago 22.6MB
scraly/gophers-api-watcher latest 4de2009ea463 23 hours ago 286MB
scraly/gophers-api latest 0e32fa8f8e18 4 months ago 22.3MB
And check if containers are running as well:
$ docker container ls
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
ba0b23d0af11 scraly/gophers-api-watcher "docker-entrypoint.s…" 9 minutes ago Up 9 minutes 0.0.0.0:8000->8000/tcp gophers-api-watcher-gophers
7b6d448f3d01 scraly/gophers-api "/bin/server" 9 minutes ago Up 9 minutes 0.0.0.0:8080->8080/tcp gophers-api-gophers
Let's test it locally
Ahh I like this moment where we will be able to test what we have created!
First, let's test our API.
$ curl localhost:8080/gophers
[{"displayname":"5th Element","name":"5th-element","url":"https://raw.githubusercontent.com/scraly/gophers/main/5th-element.png"}]
Cool, and now let's display our cute HMI, for that go to localhost:8000/demo with your favorite browser:
Awesome, it's working (and our Gopher is so cute ^^)!
Now, if you want, you can now play with the Gophers API (add and edit existing gophers) and watch them appears in the cute HMI done by Horacio Gonzalez 🙂.
Cleanup
To easily destroy created resources, you can use pulumi destroy command.
$ pulumi destroy
Please choose a stack: gophers
Previewing destroy (gophers)
View in Browser (Ctrl+O): https://app.pulumi.com/scraly/pulumi-gophers/dev/previews/2344bad2-xxxx-xxxx-xxxx-846c828f7102
Type Name Plan
- pulumi:pulumi:Stack pulumi-gophers-gophers delete
- ├─ docker:index:Container gophers-api delete
- ├─ docker:index:Container gophers-api-watcher delete
- ├─ docker:index:Network network delete
- ├─ docker:index:RemoteImage gophers-api-image delete
- └─ docker:index:RemoteImage gophers-api-watcher-image delete
Outputs:
- containerNetwork : "services-gophers"
- gophersAPIDockerImage : "scraly/gophers-api:linux-amd64"
- gophersAPIWatcherDockerImage: "scraly/gophers-api-watcher:linux-amd64"
Resources:
- 6 to delete
Do you want to perform this destroy? yes
Destroying (gophers)
View in Browser (Ctrl+O): https://app.pulumi.com/scraly/pulumi-gophers/gophers/updates/2
Type Name Status
- pulumi:pulumi:Stack pulumi-gophers-gophers deleted
- ├─ docker:index:Container gophers-api-watcher deleted (0.26s)
- ├─ docker:index:Container gophers-api deleted (0.51s)
- ├─ docker:index:RemoteImage gophers-api-watcher-image deleted (0.64s)
- ├─ docker:index:Network network deleted (2s)
- └─ docker:index:RemoteImage gophers-api-image deleted (0.86s)
Outputs:
- containerNetwork : "services-gophers"
- gophersAPIDockerImage : "scraly/gophers-api:linux-amd64"
- gophersAPIWatcherDockerImage: "scraly/gophers-api-watcher:linux-amd64"
Resources:
- 6 deleted
Duration: 6s
The resources in the stack have been deleted, but the history and configuration associated with the stack are still maintained.
If you want to remove the stack completely, run `pulumi stack rm gophers`.
Known issues
creating failed
The first time you will "play" with Pulumi, specially the Docker provider, you can face to the issue I had:
Do you want to perform this update? yes
Updating (gophers)
View in Browser (Ctrl+O): https://app.pulumi.com/scraly/pulumi-gophers/gophers/updates/1
Type Name Status
+ pulumi:pulumi:Stack pulumi-gophers-gophers created (9s)
+ ├─ docker:index:Network network created (2s)
+ ├─ docker:index:RemoteImage gophers-api-watcher-image created (8s)
+ ├─ docker:index:RemoteImage gophers-api-image created (5s)
+ ├─ docker:index:Container gophers-api ** creating failed** 1 error
creating failed... OK...
I admit that when you have this problem, it can be disappointing.
A way, to find what is happening is to add the --debug flag to the pulumi up command.
Another way is to try, simply, to run the container locally:
$ docker run scraly/gophers-api:latest
WARNING: The requested image's platform (linux/arm64/v8) does not match the detected host platform (linux/amd64/v3) and no specific platform was requested
exec /usr/local/bin/docker-entrypoint.sh: exec format error
As you can see, I built and pushed my image into another platform/architecture (mac m1) than the server/machine I'm running it on (ubuntu).
So the solution was to use, on my mac, docker buildx build command with the platform flag, to build and push the image with the good platform, like this:
$ docker buildx build --platform linux/amd64 -t scraly/gophers-api:linux-amd64 . --push
What am I thinking about Pulumi
Before to conclude this blog post I need to tell you my thought about Pulumi.
I am doing a lot of Terraform since 2017, I trained my ex colleagues, used it in many projects, for several Cloud providers (AWS, OVHcloud...), and even maintaning a Terraform provider daily. So I admit I thought I don't need Pulumi because I know and uses already an IaC tool.
As you know, I am curious, so I wanted, since several years to test it but without time to do it. I wanted to try it for a concrete need and I found one: deploying a Kubernetes cluster & a node pool (and other resources) on OVHcloud.
The journey was not easy, I had several troubles (mainly the tf2pulumi converter and then the existing Pulumi OVH community provider). I lost several hours and days, but didn't give up and thanks to Engin we finally found solutions.
I think the experience with Pulumi you can have, will depends on the provider(s) you will use.
If you already know Terraform, you will easily understand the Pulumi concepts. If you are a developer, I think it can be easier for you to write your infrastructure in your favorite language, instead of write it in HCL (Hashicorp Configuration Language).
But, not all the company who have a Terraform provider have a Pulumi provider yet, it's a fact.
On my side the journey was not easy, but I am stubborn and hopefully Engin Diri helped me. Without his help, I would probably have given up or postponed my umpteenth attempt for several months.
Conclusion
As you have seen in this article and previous articles, it's possible to create applications in Go and even now deploying them in Go too with Pulumi.
All the code of our app is available in: https://github.com/scraly/pulumi-gophers
So if you want, now, you can use Pulumi, or another tool, to deploy your apps and automatize this task.
For me, there is no magic wand, no magic tool and technology better then other, depending on your team, the context, the need, use the technology, the tool or the language you want :-).
In the following articles we will create others kind/types of applications in Go.
Hope you'll like it.
Top comments (1)
Thank you ma'am! Your Golang series was amazing! I learned so many things!