Recently, I have been tasked with checking out other IAC tools where I work. One of those given to me was Pulumi. At first, I was not sure. I come from a Terraform background and have been using Terraform for around six years now, so I was hesitant to pick this up at first.
At first, Pulumi did not make much sense to me. My initial thought was they were trying to copy Hashicorp. Yet the more I used it, the more it has grown on me.
I like it because if you are a Dev and not a DevOps person, I believe you will like Pulumi perhaps more than Terraform. The main reason is that you can bring your language of choice to the platform and start making cloud Infrastructure without going off and learning a whole new language.
After two weeks of messing with Pulumi and Go, let me show you what I did.
The most impressive part I have seen from Pulumi and Go is the Auto Package. Auto Pkg Docs The Auto package brings a particular element of A. I to your program. It essentially lets you automate the whole process of running a Pulumi program.
If you want to run a Pulumi program, in the cmd line, you run Pulumi Up. In the picture below, you will see when Pulumi runs, it will ask if you want to create a Stack. A Stack is like a state file if you come from a Terraform background. You say yes and give your Stack a name.
Now, that's all good if you're running this locally and don't mind interacting with Pulumi.
But what about if you want to run this in a Pipeline? You can't have the pipeline prompt you to create a new stack. This is where the Auto Pkg starts to solve this for you. There may be other options for this. It would be worth checking out pulumi --config argument.Pulumi Up Doc
First, you must set your context, as many Pulumi Functions use the Context package. (I am not sure why this is; their documentation says it's necessary for Go to work, yet I've seen Go work fine without context. Yet that's what is required.)
So, you start with context.Background() then, you need to declare the following variables. Now, with the pipeline, You could put these in as ENV variables, but for the sake of this demo, we will hard-code these. (Later on, I will write more blog posts on this, and it will probably show a pipeline run in GitLab, so keep tuned in if you want to see that.) The Variables are: stackName , and the projectName. For obvious reasons, this stackName is what we will call the Stack. The projectName It is my understanding, and this could be wrong, but I believe the project is the top-level folder of what you will create; for example, this project will build an EC2 instance in AWS, so I called my Project ec2_deploy. If you run Pulumi manually, it will just go and grab the top-level folder name and call that the project.
Now, we are going to pass. context.Background(), stackName, and the ProjectName with a separate deploy func variable function called deployFunc (More on this later). To the function UpsertStackInlineSource(), this function takes all our variables and returns a created stack and an error to us. If the Stack is already created, the function won't error. (This may enable us to handle errors better than just running the pulumi up command and capturing a CLI error.) It will select the Stack and return the created Stack to us. This is handy if you run your pipeline over an already-created environment. So, the code will look a bit like this. (Side note: ensure you have "github.com/pulumi/pulumi/sdk/v3/go/auto" on your imports; otherwise, this won't work.)
`package main
import (
"context"
"fmt"
"os"
"github.com/pulumi/pulumi/sdk/v3/go/auto"
)
func main() {
ctx := context.Background()
projectName := "ec2_deploy"
// we use a simple stack name here, but recommend using auto.FullyQualifiedStackName for maximum specificity.
stackName := "dev"
// stackName := auto.FullyQualifiedStackName("myOrgOrUser", projectName, stackName)
// create or select a stack matching the specified name and project.
// this will set up a workspace with everything necessary to run our inline program (deployFunc)
createdStack, err := auto.UpsertStackInlineSource(ctx, stackName, projectName, deployFunc)
if err != nil {
fmt.Printf("Failed to set up a workspace: %v\n", err)
os.Exit(1)
}
}`
As we are working with AWS, we need to create a workspace; my only explanation for this workspace idea is the Python equivalent of a Virtual Environment. I still don't fully understand this concept, but according to various examples, we need it. We also need to install the AWS tools to deploy our EC2 instance. The code will look like this.
`workspace := createdStack.Workspace()
fmt.Println("Installing the AWS plugin")
// for inline source programs, we must manage plugins ourselves
err = workspace.InstallPlugin(ctx, "aws", "v6.22.0")
if err != nil {
fmt.Printf("Failed to install program plugins: %v\n", err)
os.Exit(1)
}
fmt.Println("Successfully installed AWS plugin")`
Depending on how you are signing into AWS from Pulumi, where I am, we are using SSO profiles. Again, this can be a manual process when running through the Pulumi command line, but when operating the Auto Pkg, you can set the config to use the AWS profile to sign in and deploy the EC2 instance.
How we are going to do that is by using a function called, SetAllConfig() There is another function called. SetConfig() But that function will only let you set one item to the configuration. In our use case, we need to send over the AWS Region; for the project I'm working on, I need to send over a couple of custom values. It was also cool to send many values to see how this would work.
These values, though, are weirdly not sent to the workspace but to the Stack we made at the beginning. When I was initially coding this, I presumed, with the linear process the program is running in, that we had moved from the Stack to the workspace and that all settings I was configuring were now in the workspace, not the Stack. Yet that's incorrect, all settings and configs are set to the workspace.
To send our config to the Stack, we are going to use the map called ConfigMap in the Auto PKG, which contains a field called ConfigValue you need to set the name fields of this map in a YAML format; for example, aws:region don't do aws:region: as this won't work. (Hours of troubleshooting and pain brought me to this conclusion). An example of what I am talking about is below. Side Note: You must also pass the context variable we set earlier to this function.
`cfg := auto.ConfigMap{
"aws:region": auto.ConfigValue{Value: "us-east-1"},
"company:ticket": auto.ConfigValue{Value: "CLOUD"},
"company:hubname": auto.ConfigValue{Value: "MYHUB"},
}
createdStack.SetAllConfig(ctx, cfg)`
Then, as we are sending it to the Stack, we also need to refresh our Stack to receive the new config. For that, we are going to use the Refresh() function again, you need to send your context variable to the function for this to work.
I can hear the cogs turning. Why don't you just set the configuration to the Stack at the beginning with the initial function? That is a great question; simply, it doesn't work; the only way you can set the config to the Stack that I could see with the documentation was this way. Again, I could be wrong, as I have just started this process with Pulumi.
`_, err = createdStack.Refresh(ctx)
if err != nil {
fmt.Printf("Failed to refresh stack: %v\n", err)
os.Exit(1)
}`
The last thing we need to code for working with Stacks is the Pulumi-up part to make it all work. For that, we are going to use the Up() function. This works the same way as typing in pulumi up -y into your terminal.
For this function to work, it uses the package optup function ProgressStreams() , which uses io.writer to write the output of what Pulumi is sending to a console. This should still work in a pipeline scenario, which I will see at a later date when I build one. Pkgs Documentation for more details: Optup Pkg
Your code for this part should look like this
`// wire up our update to stream progress to stdout
stdoutStreamer := optup.ProgressStreams(os.Stdout)
// run the update to deploy our EC2 Cluster
_, err = createdStack.Up(ctx, stdoutStreamer)
if err != nil {
fmt.Printf("Failed to update stack: %v\n\n", err)
os.Exit(1)
}`
Your program code at this point should look like this:
`package main
import (
"context"
"fmt"
"os"
"github.com/pulumi/pulumi/sdk/v3/go/auto"
"github.com/pulumi/pulumi/sdk/v3/go/auto/optup"
)
func main() {
ctx := context.Background()
projectName := "ec2_deploy"
// we use a simple stack name here, but recommend using auto.FullyQualifiedStackName for maximum specificity.
stackName := "dev"
// stackName := auto.FullyQualifiedStackName("myOrgOrUser", projectName, stackName)
// create or select a stack matching the specified name and project.
// this will set up a workspace with everything necessary to run our inline program (deployFunc)
createdStack, err := auto.UpsertStackInlineSource(ctx, stackName, projectName, deployFunc)
if err != nil {
fmt.Printf("Failed to set up a workspace: %v\n", err)
os.Exit(1)
}
workspace := createdStack.Workspace()
fmt.Println("Installing the AWS plugin")
// for inline source programs, we must manage plugins ourselves
err = workspace.InstallPlugin(ctx, "aws", "v6.22.0")
if err != nil {
fmt.Printf("Failed to install program plugins: %v\n", err)
os.Exit(1)
}
fmt.Println("Successfully installed AWS plugin")
cfg := auto.ConfigMap{
"aws:region": auto.ConfigValue{Value: "us-east-1"},
"company:ticket": auto.ConfigValue{Value: "CLOUD"},
"company:hubname": auto.ConfigValue{Value: "MYHUB"},
}
createdStack.SetAllConfig(ctx, cfg)
_, err = createdStack.Refresh(ctx)
if err != nil {
fmt.Printf("Failed to refresh stack: %v\n", err)
os.Exit(1)
}
// wire up our update to stream progress to stdout
stdoutStreamer := optup.ProgressStreams(os.Stdout)
// run the update to deploy our EC2 Cluster
_, err = createdStack.Up(ctx, stdoutStreamer)
if err != nil {
fmt.Printf("Failed to update stack: %v\n\n", err)
os.Exit(1)
}
}`
Yet, if you were to run this program like in my picture below, it would fail...
WHY?
It's failing because our first function UpsertStackInlineSource() to piece all this together is missing the deployFunc variable we have not yet set to deploy our EC2 instance. Now, I must say I don't like this, the reason being that the way you code a function essentially is with a variable, not the usual GO way with func(){} . The UpsertStackInlineSource() it's not very GO Idiomatic. I couldn't figure out how to pass a function to this function. I will at a later point in time, but for the purpose of this blog post, we will keep it a variable with the note to change it later so that we can keep to Go's Idiomaticness. (If that's a word.) Again, as always, I could be wrong and missing something here, but I could not get this function to work with another function during my tired hours.
// Deploy the EC2 Instance
deployFunc := func(ctx *pulumi.Context) error {
ubuntu, err := ec2.LookupAmi(ctx, &ec2.LookupAmiArgs{
MostRecent: pulumi.BoolRef(true),
Filters: []ec2.GetAmiFilter{
{
Name: "name",
Values: []string{
"ubuntu/images/hvm-ssd/ubuntu-jammy-22.04-amd64-server-*",
},
},
{
Name: "virtualization-type",
Values: []string{
"hvm",
},
},
},
Owners: []string{
"099720109477",
},
}, nil)
if err != nil {
return err
}
amiinstanceid := pulumi.Sprintf(ubuntu.Id)
_, err = ec2.NewInstance(ctx, "web", &ec2.InstanceArgs{
Ami: amiinstanceid,
InstanceType: pulumi.String("t3.micro"),
Tags: pulumi.StringMap{
"Name": pulumi.String("HelloWorld"),
},
})
if err != nil {
return err
}
return nil
}
The above code originated from the Pulumi website on how to deploy an EC2 instance. Yet there is a big error: It's on the line before _, err = ec2.NewInstance(ctx, "web", &ec2.InstanceArgs{ where you have to get the Instance ID of the AMI. There seems to be an issue with Pulumi and passing IDs around. The way I got around this was to use pulumi.Sprintf() , which converted the ID into a string that was acceptable to the AMI field.
Also, please make sure you have the following library in your import statement:
"github.com/pulumi/pulumi/sdk/v3/go/pulumi" and "github.com/pulumi/pulumi-aws/sdk/v6/go/aws/ec2" otherwise, this won't work.
I won't go into the EC2 code too much now, as this Blog post is long enough. I will be, however, writing another post about deploying the EC2 instance with Go and Pulumi at a later date :).
The code for this variable has to go up the top now in main(){}, before our UpsertStackInlineSource() function call because Go runs in a sequential order, so if you put the deployFunc after the UperStackInlineSource() call, it will error and say deployFunc does not exist. Your whole program code should now look like this:
`package main
import (
"context"
"fmt"
"os"
"github.com/pulumi/pulumi-aws/sdk/v6/go/aws/ec2"
"github.com/pulumi/pulumi/sdk/v3/go/auto"
"github.com/pulumi/pulumi/sdk/v3/go/auto/optup"
"github.com/pulumi/pulumi/sdk/v3/go/pulumi"
)
func main() {
// Deploy the EC2 Instance
deployFunc := func(ctx *pulumi.Context) error {
ubuntu, err := ec2.LookupAmi(ctx, &ec2.LookupAmiArgs{
MostRecent: pulumi.BoolRef(true),
Filters: []ec2.GetAmiFilter{
{
Name: "name",
Values: []string{
"ubuntu/images/hvm-ssd/ubuntu-jammy-22.04-amd64-server-*",
},
},
{
Name: "virtualization-type",
Values: []string{
"hvm",
},
},
},
Owners: []string{
"099720109477",
},
}, nil)
if err != nil {
return err
}
amiinstanceid := pulumi.Sprintf(ubuntu.Id)
_, err = ec2.NewInstance(ctx, "web", &ec2.InstanceArgs{
Ami: amiinstanceid,
InstanceType: pulumi.String("t3.micro"),
Tags: pulumi.StringMap{
"Name": pulumi.String("HelloWorld"),
},
})
if err != nil {
return err
}
return nil
}
ctx := context.Background()
projectName := "ec2_deploy"
// we use a simple stack name here, but recommend using auto.FullyQualifiedStackName for maximum specificity.
stackName := "dev"
// create or select a stack matching the specified name and project.
// this will set up a workspace with everything necessary to run our inline program (deployFunc)
createdStack, err := auto.UpsertStackInlineSource(ctx, stackName, projectName, deployFunc)
if err != nil {
fmt.Printf("Failed to set up a workspace: %v\n", err)
os.Exit(1)
}
workspace := createdStack.Workspace()
fmt.Println("Installing the AWS plugin")
// for inline source programs, we must manage plugins ourselves
err = workspace.InstallPlugin(ctx, "aws", "v6.22.0")
if err != nil {
fmt.Printf("Failed to install program plugins: %v\n", err)
os.Exit(1)
}
fmt.Println("Successfully installed AWS plugin")
cfg := auto.ConfigMap{
"aws:region": auto.ConfigValue{Value: "us-east-1"},
"company:ticket": auto.ConfigValue{Value: "CLOUD"},
"company:hubname": auto.ConfigValue{Value: "MYHUB"},
}
createdStack.SetAllConfig(ctx, cfg)
_, err = createdStack.Refresh(ctx)
if err != nil {
fmt.Printf("Failed to refresh stack: %v\n", err)
os.Exit(1)
}
// wire up our update to stream progress to stdout
stdoutStreamer := optup.ProgressStreams(os.Stdout)
// run the update to deploy our EC2 Cluster
_, err = createdStack.Up(ctx, stdoutStreamer)
if err != nil {
fmt.Printf("Failed to update stack: %v\n\n", err)
os.Exit(1)
}
}`
I hope this Blog post is helpful to you and shows you how Auto PKG works with Go in Pulumi.
Till next time :)
Top comments (1)
Not sure why dev.to has formatted my stuff the way it has if you want a cleaner version check out my blog: theclouddude.co.uk/