DEV Community: Paweł Piwosz

AWS Re:Invent announcement: Lambda SnapStart for .Net - let's try it!

Paweł Piwosz — Wed, 18 Dec 2024 23:53:25 +0000

AWS re:Invent 2024 announcements

In previous article I wrote about my top 10 announcements (very briefly). But there is one more which I found very important. However, I didn't place it on the top 10 list. Why?

Well, There are a few reasons.

SnapStart itself isn't new. We have this functionality in Java for around 1 year now.
It is yet another improvement for Lambda functions to minimize cold starts.

These two are the most important, I think.

Before we will discuss SnapStart, let's talk about the reason why AWS gave us this functionality.

What is the infamous cold start?

Everyone who knows AWS Lambda (and serverless compute in general) deeper, understands the concept of cold start. But let's go even deeper and explain it more thoroughly.

There is no magic. Serverless means nothing more than that you are not the one who manage the server. But there is a server. In order to run your code, AWS is using Firecracker virtual machines. It is solution built by AWS and uses KVM to run microVMs (how they call it). These microVMs are very lightweight virtual machines, run on Firecracer (which in fact is a virtualization technology).

The fact that this VM is lightweight gives it the needed performance and speed to spin up to server your requests. But you cannot beat the physics, not today, anyway. This VM needs some time to spin up.

When the Firecracker run your VM, it prepares the environment for you. Installs all dependencies for your runtime, downloads your code, and finally, starts the runtime. These elements are part of the cold start which is managed by AWS.

Another cold start is in our hands. It depends on how our code is written, how many dependencies we load, how we initialize elements in the code.

So, we have two cold starts. The one which we manage and we can try to decrease, and another, managed by AWS, which is not in our reach.

As we see on the picture above, the cold start can take a long time. When the customer tries to do something on our serverless app, for him it takes ages :) Second part of the picture shows something what we call a warm start. This happens when previous instance of microVM finished its work and is free to take another task. MicroVM is in this state for several minutes, and it that time it takes first incoming request and processes it, without any preparations.

It is not part of this article, but if you want to reuse warm functions, please remember about cache, stored data, etc :)

So, the SnapStart.

...or, well, not yet.

There is one more element, important in the case which we explore today.

If you look at the picture above carefully, you can see that run Lambda handler for warm start is shorter than during the cold start. And it makes perfect sense, at least in some languages, like .Net.

Disclaimer: I am not a developer. I don't know .Net. This article is not an anatomy of performance of .Net Lambda, but just my test for SnapStart. I'd like to make it clear. Clear? ;)

.Net uses dynamic compilation, what means that the part of the code is compiled only when and just when is executed by the runtime. As far as I know, this behavior can be changed, but I didn't try it. So my tests used only standard approach.

Another words, you run your function, this function calculates things, does many actions, and when it reaches the point where the data must be saved in the DynamoDB table, this part (module, or whatever we will call it) is dynamically compiled.

What means...

Yes, you're right. Let's call it the third type of cold start :) At this point the compilation takes time. More, or less, but it takes time. And we will see it soon on the screens.

This information is important for our further exploration.

SnapStart

Finally! Let's talk about SnapStart.

During the re:Invent 2024 AWS announced SnapStart for .Net and Python. SnapStart for Java is available for around one year.

How it works?

When you deploy the new function (or updated one), AWS is creating a snapshot of the microVM. This snapshot is done after all elements included in cold start process are completed. Every single time when the function is called and there is no warm instance available, the function is restored from snapshot, with all elements already prepared.

Who sees the catch here already? :)

The important prerequisite is to have enable versioning for the Lambda function. Without it SnapStart cannot be enabled.

What about the price? SnapStart itself is free. However, the storage for snapshots and the restore transfers are not. The pricing is on AWS page, you can easily calculate how much you will pay.

With SnapStart you have to remember about storing and caching data during the snapshot process. You have to ensure that you don't have any sensitive data stored.

Worth to remember is that the time of snapshot restore depends on many aspects. One of them is size of the memory you configured for your Lambda.

Enough of theory, let's see how it looks in practice.

Dotnet6

I started my experimentation with dotnet6. I have to confess, I didn't check, which runtimes work with SnapStart. It turned out, that dotnet6 doesn't, you have to use dotnet8. Anyway, I tested this version, to see the "clean" cold start and measure it.

The picture above shows two executions of my Lambda. It is extremally easy to see the cold start. And yes, it is unacceptable. Let's take a look into traces.

What can we see here? I think the second picture will help to better understand the runtime.

Initialization - this is our cold start. It took almost half of second. Is it a lot, or not? Well, for me - it is. But something else is more interesting. Do you remember when I wrote about dynamic compilation?

It is there. Compare the execution times for Invocation. 14 seconds (!!!) for cold function versus less than 300 ms.

It is unacceptable. Fortunately, we have a solution for it.

Before we go there, you have to know, you cannot enable SnapStart for this runtime. I told it a few paragraphs before.

Ready To Run and Dotnet8

Well, what I did is not a perfect test, but I don't care. The goal was to check the SnapStart, just that. So, I changed the runtime to dotnet8, Copilot had a lot of troubles with making the code workable again. I mentioned already, I don't know dotnet, but I had to clearly show Copilot where the error is, only then it was able to fix it.

Anyway, the code is available in this repository. I already direct you to tag v1.0, where you can find dotnet8 code prepared with Ready To Run functionality. And this is very important.

Ready To Run prepares the code to decrease sygnificantly the time needed to start your code. It doesn't, however, solve all problems with dynamic compilation, but helps a lot.

Let's see the picture:

We can see improvements in all executions. Warm functions were faster than before, but most importantly, cold one is much, much faster:

Although, we see longer time for initialization!

How it looks for warm execution?

This time it was so quick! the whole execution took 55 ms!

Yes, I know my instraumentation for Powertools lack a lot in this version, I forgot to add it, however, it doesn't change the conclusions!

Enable SnapStart

It is time to enable SnapStart. In order to do so, we need to change a little the SAM template. We need to enable versioning for Lambda functions. The code is under v2.0 tag.

This template enables versioning and also enables SnapStart.

If you would like to do it manually, you need to go to the configuration options.

Click Edit in General configuration and change the SnapStart from None to PublishedVersions.

Testing SnapStart!

Let's run our function!

I ensured that I execute function from cold state and I saw this:

We see clearly, there is no Init phase, it was replaced by Restore. This means that the function was restored, no Initialization was needed. The whole fnunction is a little bit longer (no worries, we will come back to it), and the Restore is very similar in duration like Init was. This needs some explanations, as it is extremally important, I cover it in Conlusions part.

For the record, below is the warm function execution:

No history here, the result is exactly as we exepcted.

Why I do not care about longer execution?

It is simple. One execution is not really measurable. In next section you will see the effect of multiple executions.

Next experiments

I run some "load test". 200 executions with concurency of 25. A few of the functions were throttled (well, my bad :) but it doesn't matter). What are the averages?

Cold start average duration time: 2.75s

Warm start average duration time: 0.07s

The times are quite good. Especially when the function is warm.

Let me provide some more numbers:

Cold start:

The longest execution: 8.5s (!!!) I believe this was just accident, however... It increased the average a little.

The longest repeatable execution time: 2.74s (without the 8s long run it is 2.5s)

The shortest execution time: 2.15

As we can see these times are quite close to each other (except one :) )

Warm start:

The longest execution: 0.18s

The shortest execution time: 0.027

Of course, there is still API Gateway to consider. But as I said, I do not do perfect performance tests :)

Conclusions

SnapStart works. That's for sure. Now, you may ask, what is the benefit? I didn't show any, right? Well, consider this:

I asked other Community Builders and what I heard confirmed my thoughts. If you check the code, you'll see, there is not much to do. It is simple. Collect record from DynamoDB, increment counter and store the value back in DynamoDB. That's all.

And this is the reason why SnapStart doesn't show the full potential. The time needed for all initializations is not long enough to give the reason to use snapStart. What's more, I used 512M of memory, if I tested 2048M, the restoration of snapshot was twice longer.

This means, with simple functions SnapStart is not necessary and will add something to your bill.

SnapStart didn't help with dynamic compilation. In fact, this is something what I expected. Lambda creates snapshot when the function is created / updated. Not after the execution.

It is time to test something more complicated, I already plan to make the function bigger, perform more operations. This should show improvements towards effectiveness of SnapStart. I will publish the second part in some time, stay tuned!

AWS re:Invent 2024 is a history now

Paweł Piwosz — Sun, 15 Dec 2024 20:06:33 +0000

It was my first re:Invent in person. Was it worth to go? This short text should answer to it :)

It is the major event in Cloud world. Organized by AWS, obviously. This year we had 13th edition of the event. 5 days in Las Vegas, fully packed with presentations, workshops, networking and expo. Re:Invent is also the place, where a lot of announcements happen. But in fact, AWS is announcing new products, updates, changes etc for a whole month before the event.

A few numbers

Thanks to work done by Wojtek Gawronski, Developer Advocate at AWS, who prepared this data, I can share a few numbers about the conference. These numbers can give you the glimpse about the scale. And scale is enormous.

60000 people joined the event in Las Vegas. It is a lot of people, right? Yes, you can feel the number of people. But imagine 5 big hotels in Las Vegas. There is a lot of space to accomodate this crowd!

3500 speakers and 1000 sessions (talks, panels and workshops). Believe me, it is hard and almost impossible to choose something for yourself. So many good sessions happens in the same time!

But there is a catch. This year we could experience how the hype works. I think more than 30% of sessions had “AI” in the title. The reality is simple. AI is a hype. However, I have a feeling that many of sessions had AI in the title for one reason - to have more chances to be selected into the conference. The main topics were hidden behind AI. I will explain this thesis deeper in the video on my channel soon.

Announcements

During the re:Invent we heard 123 announcements about new services, changes, updates, new functionaalities. In total, together with all announcements before and during the event we heard 545 announcements.

My top 10

In this first summary done by me I selected 10 announcements which I found most interesting
Aurora DSQL. Serverless distributed PostgreSQL compatible database. We can create distributed, multi-region, active-active database. Imagine the possibilities! I had the opportunity to work with DSQL during the workshops, and I have to say, it is really promissing
CloudWatch Database Insights. Together with other Insights services, we can monitor and analyze more deep data from databases.
S3 Tables. New type of S3 bucket which allows to store data in Apache Iceberg format.
Predictive scaling for ECS. Your infrastructure can scale proactively, based on ML learning and anticipating the changes in the workload
EKS Auto Mode. If you are not very familiar with AWS and / or Kubernetes, EKS Auto Mode helps you to easily provision cluster with all resources needed, like databases, storages, etc.
Elastic VMWare Service (EVS). With latest changes in licensing and purchasing options of VMWare, this solution helps client to migrate the workload from on-prem to AWS
Trainium2 (Trn2). New CPU produced by Amazon subsidiary - Annapurnalabs. This CPU is designed to provide better capabilities for ML learning, but also inference.
Ultraservers. Now you can build a huge cluster of 64 Trn2 CPUs. With internal Elastic Fabric Adapter (EFA), you can create a rack-wide cluster of servers with very fast internal network what is a huge advancement for machine learning processes.
Nova models in Bedrock. 4 text models (3 are available already), one for video and one for images. AWS claims these are the most multi-modal models, to the point where we talk about any-to-any.
Bedrock Marketplace. From now we are not limited to a few models given to us by AWS, but actually vendors can place their models in the marketplace and clients can easily start to use them with Amazon Bedrock.

I had some troubles to complete this list. There was so many great announcement! Maybe I'll write more about them, maybe in more detail way. Are you interested? Let me know in the comments!

More numbers

80 kilometers. This is the distance my watch measured during the event. It is a lot of walking! Re:Invent is truly the walkathon, not just a conference :)

3 Keynotes watched live

2 workshops. I attended two workshops (once without laptop :) But thanks to Johannes Koch, I was able to actively participate :) )

4 sessions attended. Someone can say here “what? Only 4?” Maybe. But consider this - the conference is not only sbout presentations. It is about networking, for mer, mainly about networking.

1 customer review session attended.

Not measured number of booths visited :)

A lot of people met, a lot of discussions and a lot of ideas for more videos.

2 re:Caps already planned. Re:Cap is an event where experts share their views on the conference, announcements etc. As for today, I participated as speaker in one event in Gdansk, we have another in Warsaw tomorrow. Quite possibly, this number will grow.

Flights rebooked 4 times. Yeah, my trip to Vegas wasn't easy. My trip there took almost 40 hours. I will record a video about it. And I am thinking about the title. It will be re:Mess, or re:Crap, I will see :D To be clear - it doesn't have anything wit the re:Invent itself!

Wrap up

It was a great event. My first, as I said, hopefully not last. Stay tuned for videos and potentially more posts about re:Invent!

Get your certificate easily! Knowledge? Who needs that?

Paweł Piwosz — Sun, 02 Apr 2023 13:23:37 +0000

Background

Let me first explain, how I see certificates today. I already published my text about it some time ago and honestly speaking, my opinion now is even stronger. You can find this text here. But shortly - I am not a fan of pass exams for just passing the exams. And potentially list numbers of certs in LinkedIn profile :) Certification should be a knowledge AND experience confirmation.

The story

This story begins some time ago, when I read a post on LinkedIn about offer to pass certification exam on behalf of someone. It was quite hard to believe, but I was not THAT surprised. In fact, I saw many "interesting" things during technical interviews when I did an evaluation of the candidates.

What makes me somehow happy (I know how it sounds) is the growing consiousness of the real value of certifications. Especially for those people who collects them. For example, I saw no so long ago someone who has more than 60 certificates. I mean, really???

Anyway, let's come back to the main line of the story.

An offer

I received an offer. An offer which can be treated as kind of scam or try to get my sensitive data, but I know that something like that might be genuine.

So, I get an offer to achieve any certificate I want. Easily. With 100% of success rate. No, not very extensive trainings provided by best of the bests. Just pass the exam with 100% success.

Great, right?

Sounds like a fraud? It should!

So, first of all - yeah, I really believe, you are the real person, Adria. Second... It doesn't smell that much yet... Right?

Oh, don't be naive :)

Anyway, yeah, of course, I WANT THEM ALL!

So, I decided to continue this game.

What is interesting though, I am AWS Community builder, I have (or should I say - I had :) ) a few of certificates and what you can easily see on my LinkedIn profile (at least, I hope it is :D) I have some knowledge about AWS.

So, I answered to "Adria"

Looks like Adria is very eager to help me, the answer was instant.

Considering what I told you just a few lines above, I asked about AWS certifications.

Now, of course there is nothing said that they will cheat on my behalf. But I believe this conclusion is more than obvious.

I said on the end that I report this wherever I can. I decided to leave the name and phone number uncovered as this is scam, fraud and something what brings the industry down. I have no wish to see something like this on the market.

I hope that AWS, Microsoft, GCP, to name the few here finally will find a solution to remove these cheaters from the landscape.

Sad bucket of cold water

I reported this person to LinkedIn. Sad thing is, LinkedIn doesn't see anything inapropriate here so far. I hope this will change too.

Drifts again

Paweł Piwosz — Sun, 05 Mar 2023 17:45:00 +0000

In the previous episodes we learned what is drift and how to detect it. We also learned how to configure the workers pools and use them to run our workloads. Finally, we know that drift detection can be run on workers pool only. So, let's put this knowledge together, and schedule the detection.

Change the behavior

First, we need to change the behavior for the whole stack. Navigate to Behavior tab and switch the Worker pool from default to newly created one.

Create the schedule

We have to repeat the steps we already try to done once.

Navigate to Settings then Scheduling and create new schedule for drift detection.

As we can see, there is no need to set the workers pool, it is already done on the stack level.

When done, we should see confirmation and details of scheduled task.

Drift detected!

After the check is done, we can see the visualised report about the drift. We can look on this report from a few places, but we do it now from Stack's Resources tab.

The drifted resources are marked with sign on it.

Reconcile

The configuration we did is to detect drift only. I mentioned briefly, that we can also act on it. When we setup the drift detection, we can switch Reconcile on. In this case, when drift will be detected, the corresponding stack will be triggered to fix it.

Summary

Clearing drifts in automatic way should be the goal of every team when use IaC. Spacelift is doing it well.

I think though, there are some points for improvements

Reload button. Even if the page is reloading periodically, I wish to have this option.
More details about drifted resources. Something like should be vs it is comparision for drifted resources.

Takeaways

We learned how to create drift detection and how to act on it. We also know how to check and visualise drifts.

Workers pool

Paweł Piwosz — Sat, 04 Mar 2023 20:28:04 +0000

In this episode we will create workers pool, to run workloads on "our" servers. This process requires a few steps, let's see how quickly and easily we can add new worker to the app.

During this episode I will heavily use Spacelift documentation.

Create private key

First step is to create the key which wil be used to authenticate worker with application.

I'll use WSL2 for it.

openssl req -new -newkey rsa:4096 -nodes -keyout spacelift.key -out spacelift.csr

These files we need to setup our pool.

Create workers pool

Navigate to Worker pools in the Spacelift app. You should see something similar like on the screen below.

Click New worker pool.

Add the csr file generated in previous step. Also, I changed space to root. Yes, you can have pools set by space.

Please remember, that one worker is able to run one execution. If this is good or bad, I leave it to you, however it doesn't sound very appealing if I will use virtual machines, am I right? But! We can use this VM as containers platform and it starts to look a little bit better.

Ok, we can create our pool.

My pool is created and the file with token is downloaded automatically. Well, I am not fully sure here. I almost missed this download, my browser didn't show anything. I knew there will be a token, so I was careful, but I think it will be nice if Spacelift improve the UX here.

Create worker

I decided to go easy way and I run the EC2 on AWS using Spacelift image.

I've changed the default volume only. The type I selected is gp3 and I increased the size to 20G.

The provided AMI is not completed. I mean, it contains everything what we need to download and configure the launcher. Let's do it then!

First, we need to encode the key we generated earlier

cat spacelift.key | base64 -w 0

Now we can login to the newly created machine and do

sudo -i
wget https://downloads.spacelift.io/spacelift-launcher
install spacelift-launcher /usr/bin/spacelift-launcher
chmod +x /usr/bin/spacelift-launcher
export SPACELIFT_TOKEN="<token_from_file>"
export SPACELIFT_POOL_PRIVATE_KEY="key_from_encode_operation>"
/usr/bin/spacelift-launcher

What I do not like - to make the activity quickly I had to use root account.

What we've just done is very unstable and should be treated as "let me check it" way only. When we do it properly, all these should be automated and run without any access from human side. In fact... Spacelift provides a repository with IaC to provision the infrastructure!

Let's see our workers pool

Yep, we are done!

The 'hmmm' moment

I need to dig deeper into the workers pool. I saw couple of things which made me think.

What will happen if I stop my worker node?
Logs for spacelift-launcher are set to debug by default (at least, this is what I see after quick check).
In logs I saw information about S3. And this is something what I have to explore. What S3? Why? Who owns it? I have some ideas, what is behind, but I have to confirm it.

Takeways

Spacelift allows us to create private pools of workers which can be used for different workloads. Creation is simple and management is quite effective.

Drift detection

Paweł Piwosz — Wed, 01 Mar 2023 22:31:55 +0000

This episode tells the story about what drift is and why it is important to understand how crucial this process is in order to ensure proper configuration and security of the infrastructure.

What is drift

I am sure many of you at least heard this term. Let's make it simple. Drift is unwanted by process, undocumented, most often manual, change in the configuration. Another words, someone logged into AWS (for example) and added something to Security Group, or IAM Policy. "For a moment, to check something". And this "small change" opens a lot of possibilities for attackers for long period of time until it is detected. If it is detected.

It is because of drifts that many teams is affraid to run their IaC templates to not "break something". Well, the main issues here are missing process, best practices are not used and IaC is used as "scripted Operator's hands".

So, let me be harsh. If anyone in your team says "do we really need to run this? I am not sure what will happen." Don't be angry on him. Be angry on his manager :)

In case of IaC we have two states, or phases, to care to. Those are quite distinct and easy to define - one is before deployment, second - the lifecycle after deployment.

Implement a proper mechanism based on best practices for the first phase is relatively easy. I spoke about it multiple times on different events (and probably I'll write something about it). Second one, however, is tricky. While first one is activated, or triggered by some event (like push to repo, for example), second must be executed continuously. Another factor here is the quality of drift detection, which is... questionable in many aspects.

But we have Spacelift! Let's see what it can do for us.

Preparations

Before we start to check the drifts, we add more resources to our template. Simple resources - Security Group, EC2 instance.

First, Security Group. Simple one.

data "aws_vpc" "default-vpc" {
  default = true
}

resource "aws_security_group" "my-sg" {
  name        = "test-sg"
  description = "test drifts"
  vpc_id      = data.aws_vpc.default-vpc.id

  ingress {
    description = "test entry"
    from_port   = 443
    to_port     = 443
    protocol    = "tcp"
    cidr_blocks = ["10.10.10.10/32"]
  }

  egress {
    description = "test entry"
    from_port   = 0
    to_port     = 0
    protocol    = "-1"
    cidr_blocks = ["0.0.0.0/0"]
  }
}

For those ofyou who are unfamiliar with Terraform - in first resource (well, data, in fact) we gather information about the default VPC in the Region and then we use this information to create Security Group in this VPC.

Now the time for EC2

data "aws_ami" "ubuntu-recent" {
  most_recent = true

  filter {
    name   = "name"
    values = ["ubuntu/images/hvm-ssd/ubuntu-jammy-22.04-amd64-server-*"]
  }

  filter {
    name   = "virtualization-type"
    values = ["hvm"]
  }

  owners = ["099720109477"] # Canonical
}

data "aws_subnet_ids" "list-of-subnets" {
  vpc_id = data.aws_vpc.default-vpc.id
}

locals {
  subnet_ids_list = tolist(data.aws_subnet_ids.list-of-subnets.ids)
}

resource "aws_network_interface" "ec2-eni" {
  subnet_id       = local.subnet_ids_list[0]
  security_groups = ["${aws_security_group.my-sg.id}"]
}

resource "aws_instance" "drift-test" {
  ami           = data.aws_ami.ubuntu-recent.id
  instance_type = "t2.micro"

  network_interface {
    network_interface_id = aws_network_interface.ec2-eni.id
    device_index         = 0
  }
}

What we've done here is as follows:

get the latest ID of the Ubuntu AMI
get the list of Subnets available in VPC (we had to manipulate this a little with locals and tolist)
create the ENI (network interface)
create EC2

Simple :)

When we apply the template through Spacelift, let's see our resources in Resources tab.

We should see 7 resources created.

Time to create a drift

Let's login to the console and do some mess around. What I've done:

Change the instance type
Change inbound rule in Security Group
Add another inbound rule
Remove outbound rule

Ok, we generated some drift.

Drift detection

Drift detection is a process which allows us to catch the drift. And now you should see the difficulty already. In fact, drift can happen anytime, so, the detection should be continuous. But how to achieve it? When we continuously detect drifts using Terraform, we will permamently lock the state. We don't want that.

Ah well, this is the topic for another story :)

Let's create our check using Spacelift.

Navigate to Settings and select the Scheduling tab in your stack. Click Add new schedule.

Well, there is a bad news. This is not working on public workers now. So, we need to setup the private worker, and we will do it in next episode and then we'll continue with drifts!

But let's explore the configuration now. We can add different types of achtions, we selected Drift detection. Reconcile option is very powerful, and provides the kind of self-healing approach. It meas, that if we enable the option, when drift will be detected, the stored template will be applied to recover the desired state of infrastructure. Desired means here stored in VCS, in the template.

Ignore state might be risky. Normally, we shouldn't run detection is our stack is in different state than Finished, as this might cause problems.

And finally Schedule is a cron-like expression to set the schedule.

Takeaways

We learn what is the main cause of drifts and what drifts are. We discussed how we can detect drifts and we checked if we can detect this problem using Spacelift. We learn, that on this day (3rd of March 2023) we have to do some additional work, which we will do in the next episode.

Look around the stack functionality

Paweł Piwosz — Tue, 28 Feb 2023 00:15:05 +0000

So, we already configured the Spacelift environment and we run our first template through it. It is time to look around the Spacelift console a little bit.

In this episode we will take a look into Stacks and what information we can collect.

But the preparations first.

I've created a new branch and added this part to main.tf

resource "aws_s3_bucket" "mysecondbucket" {
  bucket = "thisismysecondtestbucketforspacelift"
}

And then I created another branch (from the newly created one) ad added outputs.tf with this content

resource "aws_s3_bucket" "mysecondbucket" {
  bucket = "thisismysecondtestbucketforspacelift"
}

I published my branches, of course.

Now, I merged the third branch to second one through Pull Request. Why I did this? To prove that Spacelift is smart :) With the configuration we did earlier, nothing should happen. And that is true, indeed.

Now it is time to create PR to main branch. This time something happened.

Navigate to PRs tab in Stacks section (select your stack, of course).

We have our PR here, in Spacelift! Let's click on it, and here we can see quite interesting things.

On the screenshots above we see details of the run performed against the code in PR. Spacelift did a terraform plan for us.

More tabs

We already visited Settings, Runs and PRs tabs. Let's take a look on other things there.

Tasks

In this tab we can run some commands against the collected inventory. By this term I mean the information and data which are available for Spacelift.

Let's take a look.

Of course the command is terraform output, I realized that typo after first run :)

And the effect is below.

As we can see, the process is quite similar to normal run. In fact, we can call this functionality as 'ad-hoc execution'. So, if any one though how to deal with state file... Well, you can do it here. Let's try with terraform state list as an example.

Looks good, doesn't it? I miss one thing, though. Let me show you.

Imagine now even more crazy command. I would like to have an option to see these tasks by name. Why? Because we have possibility to re-run the task. With long and complicated command as title we need more "brain power" to locate and understand the task.

Environment

In this tab we see the default environment variables and we can define our own. Standard ones and secrets too. Nothing fancy, nothing different than typical behavior of CI/CD tool.

Resources

Here we can see graphical representation of resources created during the lifetime of the project. We can change the grouping and get details about each resource.

Outputs

Exactly the same like terraform output in case of our project. I do not like the layout, though. Seems to be incosistent with other pages of the service.

Dependencies and Notifications

At this point there is nothing much to say about them.

In Dependencies tab we can monitor, control and manage dependencies between stacks. In our case now, nothing is there.

We also do not have any notifications, so this tab is empty too.

Summary

We went through all functionalities in Stacks menu. We learn what information are there and how to get them and make useful. In next episode we will do something even more interesting :)

First configuration and execution

Paweł Piwosz — Sun, 26 Feb 2023 21:40:22 +0000

In this episode we will prepare our Spacelift account and we will look around the console.

Before we start

Before we start playing with Spacelift, we should create new repository (I called mine spacelift-demo) and put there two files.

providers.txt which is exactly the same like in first episode of this series
main.tf with this content:

provider "aws" {
  assume_role_with_web_identity {
    role_arn                = "arn:aws:iam::1234567890:role/spacelift-role"
    web_identity_token_file = "/mnt/workspace/spacelift.oidc"
  }

  default_tags {
    tags = {
      Environment = "Sandbox"
      Terraform   = "True"
      Repo        = "spacelift-prep"
      Project     = "Spacelift tutorial"
    }
  }
}

resource "aws_s3_bucket" "mybucket" {
  bucket = thisismytestbucketforspacelift
}

Simple stuff, we create one AWS S3 Bucket only.

Please remember, S3 Bucket name must be globally unique!

We configured the IAM Role which we want to use. Of course, again, it is hardcoded, what is not a good practice, but enough for the demo purposes.

And we need to commit this to main branch.

Finally we can go forward

After the login we can see the menu on the left side:

This menu should give us a glimpse of idea what Spacelift really is. As it is stated on the webpage

Spacelift is a sophisticated CI/CD platform for Terraform, CloudFormation, Pulumi, Kubernetes, and Ansible

Spacelift is a CI/CD tool. But is it only it? What we will explore is if Spacelift is more like an orchestrator for IaC.

So, let's get started!

First position in the menu is Stacks. Stacks are main entity of the projects. Stack uses repository of the code and manages the resources and states.

Let's create our first stack. Click Add stack button on top right side of the screen. It seems hidden a little bit, I think that can be improved :)

On the first screen we define basic information. Name of the stack, space (we will learn it more later), description, etc.

Ok, done, let's click continue.

And here we can be hit by some unexpected thing. Or should I say - expected? :) We need to provide the repository. And we can't just refresh the page if we create one now. That is exactly why we created the repo earlier. It wil be good to have the possibility to refresh the state, or even... create repo from here.

I provided the information like on screen below.

After we click Continue we go to very important, I would say the most important, part of the configuration. We deal with Terraform here, and this part allows us to configure the behavior of Terraform. It is not the place to explain what statefile is, so let me navigate through these fields shortly.

Backend in our case is Terraform. There are more of them. You may ask why there is no backend for Ansible, as Spacelift claims they can manage it too? Well, Ansible is serverless (oh wow, I just realized that synonym for this case :D), therefore there is no state to manage.
Smart sanitization - very nice function. Spacelift will try to determine if value is sensitive and should be hidden (useful for variables)
Version - quite obvious, isn't it?
Manage state - crucial thing. We can allow Spacelift to care about the statefile.
Import existing statefile - very cool feature. It means that we can migrate existing resources, or should I say, management of existing resources to Spacelift.

And now we go to the final step of configuration.

Nice thing - we can manage administrative stack. It means we can use Terraform to manage Spacelift! How cool is that? And what was first - chicken or egg? ;)

I set some advanced options too. First, Autodeploy. For this simple repo we don't need any additional actions, let's deploy it directly. Second, I enabled Local preview. This looks like very cool option, we can check the stack with spacectl command locally! We will test it soon.

Finally, we can decide about runners and their images and we can even customize the workflows.

Let's save the stack now.

Nicely done!

Burn the AWS!!

Or maybe not :) We have simple template prepared, so let's trigger it!

Let's click this small button:

And see what happened!

The process is clean and clear.

In case of the screen above you see the report after trigger through commit, however, it doesn't really matter, the info is very similar.

First, I need to say what I do not like. The order. Well, yes, this works, but somehow I prefer to see it in oposite direction. The oldest on top, newest on the bottom. A matter of habbit, I suppose.

We see couple of stages.

Queued - this is the time between trigger and actual start of the execution. Spacelift prepares some resources where our demo is executed.
Preparing - Actual preparation of the resources mentioned in step above
Initializing - in case of our demo - initialization of Terraform - terraform init.
Planning - well, terraform plan :)
Applying - ... you know, right? :) Do you remember, that we set immediate execution when we setup the stack? That is why apply was executed immediately after plan.
Finished - just info that run is ended.

What was done by our Terraform? We can check it from the level of Spacelift!

Do you see the ADD tab? We can see additionally the +1 note. This means... Yes, we added 1 resource!. When we go to this tab, we will be able too see all actions done by the execution. Sweet.

And we have (Space)lift (off)!

Paweł Piwosz — Sun, 26 Feb 2023 21:40:14 +0000

This time we will configure and run our first small template through Spacelift. Our first step will be to configure the connection between AWS and the service. I assume, that the Spacelift account is created and connected with GitHub.

In this short series we will learn how to configure, connect and start to use Spacelift.

First thing to do for each real developer is to switch the GUI to dark mode :D Go to your account settings on the bottom left corner of your screen and find the dark mode setting. Right, we are ready to go :)

AWS

Connect Spacelift with Cloud provider

To work with AWS we need to configure our connection to the vendor. The documentation provided by Spacelift is really rich and clear, however, I'd like to mention the security aspect. First way, the easier one, is to provide programmatically available user with role to assume. Is it a good solution? Well, good enough. However, I prefer to use another way, which also is provided by Spacelift - the OIDC connection. On the end of the day it does the same thing, but from the security standpoint - it is better.

OIDC needs some configuration (obviously) and part of it is a thumbprint. As I do this as a tutorial, not fully blown production ready stuff, I show you how to get this thumbprint using CLI approach.

First, let's collect url of our spacelift app. Simply, look on your browser :) In my case it it https://<subdomain>.app.spacelift.io/. We have it, so let's generate the thumbprint.

I use Ubuntu to get these information. In fact WSL2 on Windows :). Execute:

curl https://<subdomain>.app.spacelift.io/.well-known/openid-configuration|jq

Please note, I added jq on the end to have nicer output.

Find the line with jwks_uri. Copy from it the domain name only and use it in following command.

openssl s_client -servername <subdomain>.app.spacelift.io -showcerts -connect <subdomain>.app.spacelift.io:443

Ensure, you don't have https, etc. Just the domain.

Scroll the output and find the certificate. There will be something like

-----BEGIN CERTIFICATE-----
somestring
-----END CERTIFICATE-----

Create a file (for example certificate.crt) and copy this whole part there.

Now we are ready to generate thumbprint

openssl x509 -in certificate.crt -fingerprint -sha1 -noout |tr -d :

As you can see I used the tr command with pipe to get rid of :. If you are not familiar with pipes and redirections in Linux, no worries, here is my lab about it.

The string in output is a part which we need to use to complete our configuration of OIDC.

We can do it with Terraform, of course. In fact, if you plan to use it in the real project, I strongly recommend to do it with IaC. However, now you know how to do it from CLI :)

Let's terraform it

You know what? Creating all these resources on AWS from GUI is so old-fashioned :) Let's have a small Terraform template for it! We need to create:

OIDC itself
IAM Role to assume by Spacelift
IAM Policy which describes what Spacelift can do.

First, let's create the file providers.tf with this content

terraform {

  required_version = ">=1.3"

  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 4.0"
    }
  }
}

And we can forget about this file from now.

We know what is our thumbprint, so we can create first block in main.tf.

provider "aws" {
  default_tags {
    tags = {
      Environment = "Sandbox"
      Terraform   = "True"
      Repo        = "spacelift-prep"
      Project     = "Spacelift tutorial"
    }
  }
}

resource "aws_iam_openid_connect_provider" "spacelift" {
  url = "https://<subdomain>.app.spacelift.io"

  client_id_list = [
    "<subdomain>.app.spacelift.io",
  ]

  thumbprint_list = ["<thumbprint>"]
}

Please note, we also have the provider defined here.

Now, it is time to define the IAM Role. Within this definition, we will ensure that the Role can be assumend by Spacelift only. We want to build the trusted relation between this entity and Spacelift to secure our connection as much as possible.

The Role and condition inside is described well in Spacelift's documentation, so, I will not go into details. I just explain a few parts.

First, we use Federated access and we use for it the OIDC we defined earlier.

Second, please note the Condition section of the Role. This makes the connection more secure by narrowing the entities which can use this role. We can create even more precise boundary, all is in documentation. However, this one is enough for us at this moment.

And here is the Terraform code for our Role

resource "aws_iam_role" "spacelift-role" {
  depends_on = [
    aws_iam_openid_connect_provider.spacelift
  ]

  name        = "spacelift-role"
  description = "Role to assume by spacelift"

  assume_role_policy = <<ROLE
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Federated": "${aws_iam_openid_connect_provider.spacelift.arn}"
      },
      "Action": "sts:AssumeRoleWithWebIdentity",
      "Condition": {
        "StringEquals": {
          "<subdomain>.app.spacelift.io:aud": "<subdomain>.app.spacelift.io"
        }
      }
    }
  ]
}
ROLE
}

You might ask, why do you use this old-fashioned way with <<ROLE?. Well, good question :) For two reasons. I learned Terraform this way and this approach helps me to better see where Role or Policy document ends. Quite handy, especially for long documents.

Ok, finally, we will create "very secure" Policy and we will attach Policy to the Role. Please have in mind, that the Policy should by tailored to needs, not open like here. We do it for demo purposes, so we can live with it now.

resource "aws_iam_policy" "spacelift-policy" {
  name = "spacelift-policy"

  policy = <<POLICY
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "*",
      "Resource": ["*"]
    }
  ]
}
POLICY
}

resource "aws_iam_role_policy_attachment" "spacelift-iam-attachment" {
  role       = aws_iam_role.spacelift-role.name
  policy_arn = aws_iam_policy.spacelift-policy.arn
}

No magic here, I suppose.

Init

We have our template, we can execute it.

First, we need to initialize Terraform

terraform init

We can format and validate the template

terraform fmt

terraform validate

Well, I have to do one more thing. I use many AWS profiles, therefore I need to specify it. There are many ways to do so, I use the easier and less flexible one - I put it into template:

provider "aws" {
  profile = "demos"

Please remember, this is for demo purposes only, so, no issue with it.

Execute Terraform

We can deploy our stack.

First, let's see if all is ok

terraform plan

If all went ok, you should see something like

Plan: 4 to add, 0 to change, 0 to destroy.

Ok, so let's rock!

terraform apply -auto-approve

This Apply complete! Resources: 4 added, 0 changed, 0 destroyed. means success.

But wait... What Role should I use later? We can go to the GUI and... NO!

Outputs

Let's create one more file, called outputs.tf and put there

output "IAM-Role-to-assume" {
  description = "IAM Role to assume by Spacelift"
  value       = aws_iam_role.spacelift-role.arn
}

Now we can run terraform refresh and we already see the ARN of the Role. If this info is needed later, we can run terraform output.

Spacelift

Well, looks like this episode is more about Terraform than Spacelift :) It is important though, to have good connection created, so I believe this is not a big deal :)

Ok, let's do our work on Spacelift side!

Go to your dashboard, to the `Cloud integrations (bottom left of the screen)

And configure your AWS integration accordingly.

And... Yes, that's it :)

Key takeways

For now, we know how to connect the dots. In the next episode we will learn what Spacelift is and how to start with it.

Serverless Game

Paweł Piwosz — Mon, 09 Jan 2023 22:50:16 +0000

I started 2023 with something new. I was wondering, what could be interesting, fresh and exciting. And I found it.

The Game

I invented... the game. Well, no, I invented The Game :) The Game of Serverless. My idea was to make it very interactive, community driven and community building one. That is why as a platform I selected

This is the place where we can find so many brilliant and smart people. People eager t oshare their wast knowledge and experience.

First rule of The Game

Every week (most probably Sunday evening CEST) I publish new requirements for the application and infrastructure. The Players' task is to propose the solution.

Second rule of The Game

Important is that solution must utilize Serverless services of AWS.

Quite important disclaimer, I believe. This is the game, for fun. I do not expect and I do not ask for truly mature enterprise solutions. We, you all are paid for that. Here we share the tips and play the game for fun (I repeat myself, I know :) )

I play the role of "Game Master". My goal is to try to navigate you into the rocks. Another words, to force the situation, where Players will agree that Serverless is not an option. If this happen, I win :)

You, my dear Players, you receive the requirements and based on it you propose an architecture. You may even draw it, if you wish :)

The set of requirements is simple, at leas for now. In time the level should increase. For now we have business requirements (more or less). Soon we start to discuss non-functional requirements (or quality attributes, if you wish).

Third rule of The Game

The requirements. Well, the rule here is simple. If I didn't reveal something, you are free to propose whatever you wish (from Serverless area, of course). I collect proposals from all Players and base on it I create new set of requirements.

So, as you can see, it is indeed an interactive game. I have some requirements in mind. But... In a day when I wrote this article, we are in second week and I already had to modify the requirements I gave to Players :) This make it interesting, doesn't it?

I feel the thrill, how to join?

It is simple, really. I invite everyone to play.

Fourth rule of The Game

This is the game for fun (did I said that already?). Be nice, kind and happy :)

Resources

The best starting point is Github repository. I collect all data and progress there.

You can join directly in my LinkedIn profile

I wait for you there!

Let's have fun!

Deeper dive into SBOM

Paweł Piwosz — Fri, 09 Dec 2022 19:42:20 +0000

This episode is not about tools. This time we will take a look on the SBOM itself. We discussed basics of SBOMs, it is time to go a little bit deeper.

First thing which needs explanation is what is the difference between Software Composition Analysis (SCA) and SBOM? Both are analyzing the dependencies!

Well, not exactly.

SCA is an automated process to identify open source components in code base and evaluate these components against licenses, vulnerabilitites, security issues, etc. SBOM is the report generated by the SCA software. SBOM is highly defined and structured document and not all SCA tools generate their reports in the acceptable as SBOM way. These SBOMs are then compared with multiple up-to-dated databases to ensure the report quality.

So, in very simple way we can say that SCA is a tool where SBOM is a data. SBOM contains list of used components with some specific information. But SBOM itself doesn't care how its generation was done.

We talk here about different aspects of one approach to ensure security and quality which can be applied in C-SCRM in the organization.

I read a lot about it currently and I saw many reports where authors predict some numbers related to percentage of companies which will be use SCA and SBOMs in next few years. Honestly, I do not believe in these predictions. The awareness of these solutions is quite low now and implementation even lower. Yes, we are more aware of SCA, but SBOM, which looks like natural extension to the process, is not very well known. That is my personal opinion :)

How the SBOM should look is described, as I mentioned, in quite strict way in standard. If you like to go deeper, here is the link.

Let's take a look on three main perspectives (as it is called), where SBOM is very useful. The perspectives are

Produce
Choose
Operate

Produce software

The company who creates the software, by attaching the SBOM to their package will gain not only external (sell) benefits, but internal (development) as well. For example, by monitoring of the vulnerabilities in used packages, by knowing potential end date of lofe of specific component used in their software, by knowing all dependencies included in the code.

What is the benefit for external use? Well, simple - it gives the better possibilities to the partners to know "what is inside" and also creates, let's call it, "better picture" of the seller. Another words - "They know what they sell".

Below is a representation of areas of interest in Produce software perspective.

Choose software

Now, the ball flies to the second side of the field. The company interested in buying the product is able to (also) verify vulnerabilities, control and be aware of lifetime of the used components, control and understand licenses of the components, can target security analysis towards already defined targets.

Operate software

Final phase is when software is about to be bought and then operated. In this perspective the organization can use SBOM analysis as one of the decisive elements. During the operationa phase SBOM can help with independent mitigations (so the organization does not rely solely on the vendor), can better administer its assets and evaluate risks and usage.

That's conclude the second theory lesson in this series.

Cover image by Suzy from Pixabay

SBOM with Checkov

Paweł Piwosz — Fri, 25 Nov 2022 09:05:50 +0000

This episode might be quite surprising, at least for those of us who know IaC and did quality and security scans of IaC templates.

Well, yes, Checkov is a quality scanner, but from some time already it is more than that! Let's see on the frameworks which can be scanned by Checkov:

--framework {bitbucket_pipelines,circleci_pipelines,argo_workflows,arm,azure_pipelines,bicep,cloudformation,dockerfile,github_configuration,github_actions,gitlab_configuration,gitlab_ci,bitbucket_configuration,helm,json,yaml,kubernetes,kustomize,openapi,sca_package,sca_image,secrets,serverless,terraform,terraform_plan,all} [{bitbucket_pipelines,circleci_pipelines,argo_workflows,arm,azure_pipelines,bicep,cloudformation,dockerfile,github_configuration,github_actions,gitlab_configuration,gitlab_ci,bitbucket_configuration,helm,json,yaml,kubernetes,kustomize,openapi,sca_package,sca_image,secrets,serverless,terraform,terraform_plan,all} ...]

Quite a number, don't you think?

But... What about SBOMs? Can Checkov generate SBOM?

No. Well, not really.

But the report generated by Checkov can be exported in CDX format, what means, it can be consumed in the process!

Let's take a look. I install Checkov and download random repos from GitHub:

Terraform
CloudFormation
Dockerfile
Serverless
Kubernetes
Helm

$ pip install checkov

So, installation is not that hard, isn't it? ;P

$ git clone https://github.com/dwmkerr/terraform-consul-cluster.git
$ git clone https://github.com/splunk/splunk-aws-cloudformation.git
$ git clone https://github.com/webdevops/Dockerfile.git
$ git clone https://github.com/softprops/serverless-aws-rust-http.git
$ git clone https://github.com/kubernetes/examples.git
$ git clone https://github.com/prometheus-community/helm-charts.git

Ok. I'll generate a report for each repo with CycloneDX output. Also, I will not specify the framework, so it is a huge possibility, that some of these repos contain not only the "main" framework, but others as well. Will see.

checkov -d terraform-consul-cluster/ -o cyclonedx > tf.xml

For some reason, Checkov didn't save the report to the specified file, but created a folder. But it is not an issue, I used simple redirection and didn't spent time on it :)

Report is not very readable for human, but it doesn't matter, it should be (and it is) readable for machine. Checkov uses the newest version for CycloneDX - 1.4.

Let's take a look on details. In the "standard report" I found this issue:

Check: CKV2_AWS_12: "Ensure the default security group of every VPC restricts all traffic"
        FAILED for resource: module.consul-cluster.aws_vpc.consul-cluster
        File: /modules/consul/01-vpc.tf:2-10
        Guide: https://docs.bridgecrew.io/docs/networking_4

                2  | resource "aws_vpc" "consul-cluster" {
                3  |   cidr_block           = "${var.vpc_cidr}" // i.e. 10.0.0.0 to 10.0.255.255
                4  |   enable_dns_hostnames = true
                5  |
                6  |   tags {
                7  |     Name    = "Consul Cluster VPC"
                8  |     Project = "consul-cluster"
                9  |   }
                10 | }

What we have in generated SBOM?

<vulnerability bom-ref="070be6ca-0732-4cf3-b0c7-a423fc0f45be">
    <id>CKV2_AWS_12</id>
    <source>
        <name>checkov</name>
    </source>
    <ratings>
        <rating>
            <severity>unknown</severity>
        </rating>
    </ratings>
    <description>Resource: module.consul-cluster.aws_vpc.consul-cluster. Ensure the default security group of every VPC restricts all traffic</description>
    <advisories>
        <advisory>
            <url>https://docs.bridgecrew.io/docs/networking_4</url>
        </advisory>
    </advisories>
    <affects>
        <target>
            <ref>pkg:terraform/cli_repo/terraform-consul-cluster/modules/consul/01-vpc.tf/module.consul-cluster.aws_vpc.consul-cluster@sha1:26077595ad94ad61098ccc203af70aaf518a847b</ref>
        </target>
    </affects>
    </vulnerability>

Looks quite nice.

I generated SBOM reports from all repos I cloned. And I am really satisfied with results. Well done Bridgecrew!:)

Summary

I really like Checkov, and I say if for a few years now. It is more and more complex tool, even in the version available for free. I am really happy to see the SBOM option, as it becomes very important part of the process.

The great news is that SBOMs can cover also infrastructure as Code. Imagine, you buy a car. And you receive report where you see that every single component in this car passed verification and validation. Every single one, except the wheels. What can go wrong? These wheels here - it is IaC.

Why I said no on the beginning, when I asked myself if Checkov is a SBOM tool? Well, the point is that SBOM should contain all dependencies. Checkov's focus is on templates. Don't get me wrong, that is OK, there are other tools which should take care about code's dependencies. I said that to emphasize, Checkov cannot be only tool used in SBOM generation process.

So, to be correct, Checkov is not SCA tool but can generate SBOM report for its part.

Cover image by Suzy from Pixabay