Workers pool

In this episode we will create workers pool, to run workloads on "our" servers. This process requires a few steps, let's see how quickly and easily we can add new worker to the app.

During this episode I will heavily use Spacelift documentation.

Create private key

First step is to create the key which wil be used to authenticate worker with application.

I'll use WSL2 for it.

openssl req -new -newkey rsa:4096 -nodes -keyout spacelift.key -out spacelift.csr

These files we need to setup our pool.

Create workers pool

Navigate to Worker pools in the Spacelift app. You should see something similar like on the screen below.

Click New worker pool.

Add the csr file generated in previous step. Also, I changed space to root. Yes, you can have pools set by space.

Please remember, that one worker is able to run one execution. If this is good or bad, I leave it to you, however it doesn't sound very appealing if I will use virtual machines, am I right? But! We can use this VM as containers platform and it starts to look a little bit better.

Ok, we can create our pool.

My pool is created and the file with token is downloaded automatically. Well, I am not fully sure here. I almost missed this download, my browser didn't show anything. I knew there will be a token, so I was careful, but I think it will be nice if Spacelift improve the UX here.

Create worker

I decided to go easy way and I run the EC2 on AWS using Spacelift image.

I've changed the default volume only. The type I selected is gp3 and I increased the size to 20G.

The provided AMI is not completed. I mean, it contains everything what we need to download and configure the launcher. Let's do it then!

First, we need to encode the key we generated earlier

cat spacelift.key | base64 -w 0

Now we can login to the newly created machine and do

sudo -i
wget https://downloads.spacelift.io/spacelift-launcher
install spacelift-launcher /usr/bin/spacelift-launcher
chmod +x /usr/bin/spacelift-launcher
export SPACELIFT_TOKEN="<token_from_file>"
export SPACELIFT_POOL_PRIVATE_KEY="key_from_encode_operation>"
/usr/bin/spacelift-launcher

What I do not like - to make the activity quickly I had to use root account.

What we've just done is very unstable and should be treated as "let me check it" way only. When we do it properly, all these should be automated and run without any access from human side. In fact... Spacelift provides a repository with IaC to provision the infrastructure!

Let's see our workers pool

Yep, we are done!

The 'hmmm' moment

I need to dig deeper into the workers pool. I saw couple of things which made me think.

• What will happen if I stop my worker node?
• Logs for spacelift-launcher are set to debug by default (at least, this is what I see after quick check).
• In logs I saw information about S3. And this is something what I have to explore. What S3? Why? Who owns it? I have some ideas, what is behind, but I have to confirm it.

Takeways

Spacelift allows us to create private pools of workers which can be used for different workloads. Creation is simple and management is quite effective.