Series Introduction
Welcome to Part 11 of this blog series that will go from the most basic example of a .net 5 webapi in C#, and the journey from development to production with a shift-left mindset. We will use Azure, Docker, GitHub, GitHub Actions for CI/C-Deployment and Infrastructure as Code using Pulumi.
In this post we will be looking at:
- Infrastructure as Code
- Pulumi
TL;DR
We got to grips with IaC and made use of our C# skills to build our own infrastructure - Go shift-left! Pulumi has been a joy to use, just so easy, the integration of GitHub Actions and Pulumi has equally been awesome.
We were able to preview our infrastructure changes from a Pull Request using our CI Workflow and achieved C-Deployment by deploying our infrastructure upon merging into main using our CD Workflow.
The Pulumi GitHub Bot works great and the Pulumi Console is simply invaluable. π€
GitHub Repository
peteking
/
Samples.WeatherForecast-Part-11
This repository is part of the blog post series, API's from Dev to Production - Part 11 on dev.to. Based on the standard .net standard Weather API sample.
Requirements
We will be picking-up where we left off in Part 10, which means youβll need the end-result from GitHub Repo - Part 10 to start with.
If you have followed this series all the way through, and I would encourage you to do so, but it isn't necessary if previous posts are knowledge to you already.
Don't forget to ensure you have setup Code Climate Quality with your repository.
If you want to follow along from the code from Part 10, that is really ideal, I have everything in order and it should all work fine. If however, you want to skip to the end and run it all, you can too, but there there are some things you'll need to do.
- Clone the GitHub Repo (Part-11) into your own GitHub
- Install/Obtain Access to Azure CLI
- Install Pulumi CLI
-
Navigate β
./src/Samples.Weatherforecast.Infra - Follow the instructions in this post about: Create a Service Principal
- Follow the instructions in this post about: Securely store Azure Service Principal credentials
-
Execute β
pulumi preview
This should create the Pulumi project and its stack.
Introduction
This post is all about Infrastructure as Code, we want to be more autonomous to build and manage our own infrastructure instead of being fully dependant on others. Infrastructure as Code has been around a while and there are many solutions, each cloud provider such as Microsoft Azure, AWS, GCP and more all have their own native solutions, and of course the number one name for some time has been Hashicorp's Terraform (OSS, Cloud, Enterprise). However, they all have something in common... they are all in their own languages, from declarative to DSL's and JSON. Using DSL's can be cumbersome as is learning a new language; however, a purpose built language could have some advantages. Pulumi is one of those solutions that takes a different angle, the ability to get to your end goal of having your infrastructure written as code but in a language you use all the time.
In this post we will dive head-first into, Pulumi!
Let's get started. π€
Microsoft Azure
Get access to Azure, this is kind of mandatory, we will be using Microsoft Azure - If however, you don't have an account, you can get one for FREE! π
Sign-up to Microsoft Azure.
Install/Obtain access to Azure CLI
OR
OR
OR
If you already have chocolately installed, you can simply execute:
choco install azure-cli
As long as you can get access, that's all we need, so use whichever method you prefer.
Pulumi
What is Pulumi?
Pulumi is a modern infrastructure as code platform built for engineers; developers and infrastructure.
The beauty of the product is you can build, deploy and manage your applications and infrastructure using languages and tools you are familiar with. Follow your existing engineering practices and guidelines, and take advantage of the same toolchain you use today.
This means if you major in C# for instance, you have all the power of the C# language and all your existing tools. It means you don't need to learn a bespoke language just to manage and provision your infrastructure = Learning curve is minimised.
When you deploy applications, everyone can work together, or you can choose to be isolated; maybe I'm trying something out or I have something very unique. You can work together and share infrastructure packages no matter what language it is written in, deploy with confidence by validating beforehand and automate it all!
If that wasn't enough, with Pulumi you can use policies to provide guardrails for teams, allowing them to feel empowered with autonomy (shift-left), and with the Pulumi Console you can see a full audit history. This is known as Policy as Code.
Pulumi is a breath of fresh air compared to other solutions, now I know what you're thinking, Terraform... Yes, I love it too, and I've worn that T-shirt, it's a good one, everyone has their go-to favourite T. However, I have a preference for Pulumi, its simplicity by using the same language I'm building API's in, whether that is C#, Go, JS/TS, Python etc. The Terraform T-shirt is now washed, ironed, and in the cupboard for a while. Luckily Pulumi can connect to Terraform also, so even if you have a centralised Platform team building foundational infrastructure, initial virtual networking topology, firewalls, SIEM etc., engineering teams can shift-left to build and manage their own components whilst using the outputs of Terraform. If for example I needed the output of a VNET from Terraform, Pulumi can get that and then I can provision my product using Pulumi as a pretty-well autonomous engineering team.
Sign-up
Pulumi has a free tier and they say it's free forever (for individual use) π
Sign-up to Pulumi.
Pulumi CLI
Pulumi is controlled primarily using its command line interface (CLI). It works with the Pulumi service to deploy changes to cloud apps and infrastructure. It also keeps a history of who updated what in your team and when; which is awesome! The CLI has been designed for productivity and ultrafast feedback, in addition to the ability to use it in CI/CD scenarios.
Install Pulumi CLI.
(Again) If you already have chocolately installed, you can simply execute:
choco install pulumi
Requirements check
β
Everything we've done previously, i.e. Part 10 etc.
β
Azure account
β
Azure CLI - Version used v2.24.0
β
Pulumi account (Console)
β
Pulumi CLI - Version used v2.14.0
Let's get started with Pulumi!
Fire-up your console of choice and navigate to the src directory of the WeatherAPI.
Create a folder called, Samples.WeatherForecast.Infra and navigate to it
mkdir Samples.WeatherForecast.Infra
cd Samples.WeatherForecast.Infra
Inside the infra directory - Create a new Pulumi Project.
pulumi new azure-csharp
From here, you'll be prompted by the Pulumi CLI for some values.
-
project name:
Samples.WeatherForecast.Infra -
project description:
Infra for WeatherForecast -
stack name:
dev -
azure-native:location: The Azure location to use:
uksouth
I've chosen UK South given my location, for you, please choose whatever suits your needs best.
βΉοΈ TIP
If you don't know what locations are available, below is a very quick & easy way to find out.
az login
# Once login successful, execute
az account list-locations --output table
Open VS Code by executing code .
You should see a nice, brand new Pulumi C# program! π
If you also login to the Pulumi Console, you should see your project there too!
Our API Infrastructure
Steps
Step 1
Open VS Code for the WeatherForecast-API if you haven't already, and Navigate β /src/WeatherForecastApi-Infra/
Step 2
Rename MyStack.cs to AzureStack.cs
Step 3
Inside AzureStack.cs, replace all code with the following
using Pulumi;
using Pulumi.AzureNative.Resources;
using Pulumi.AzureNative.Web;
using Pulumi.AzureNative.Web.Inputs;
class AzureStack : Stack
{
public AzureStack()
{
var config = new Pulumi.Config();
// Obtain our docker image from config
var dockerImage = config.Require("docker-image");
// Resource Group
var rg = new ResourceGroup("rg-weatherforecastapi-uks-");
// AppService Plan
var appServicePlan = new AppServicePlan("appplan-weatherforecastapi-uks-", new AppServicePlanArgs() {
ResourceGroupName = rg.Name,
Location = rg.Location,
Kind = "Linux",
Reserved = true,
Sku = new SkuDescriptionArgs
{
Name = "B1",
Tier = "BASIC"
},
});
// WebApp for Containers
var app = new WebApp("app-weatherforecastapi-uks-", new WebAppArgs()
{
ResourceGroupName = rg.Name,
ServerFarmId = appServicePlan.Id,
SiteConfig = new SiteConfigArgs
{
AppSettings = new[]
{
new NameValuePairArgs
{
Name = "WEBSITES_ENABLE_APP_SERVICE_STORAGE",
Value = "false"
},
new NameValuePairArgs
{
Name = "DOCKER_REGISTRY_SERVER_URL",
Value = "https://ghcr.io"
},
new NameValuePairArgs
{
Name = "WEBSITES_PORT",
Value = "8080" // Our custom image exposes port 8080. Adjust for your app as needed.
}
},
AlwaysOn = true,
LinuxFxVersion = $"DOCKER|{dockerImage}"
},
HttpsOnly = true
});
this.Endpoint = Output.Format($"https://{app.DefaultHostName}/weatherforecast");
this.HealthEndpoint = Output.Format($"https://{app.DefaultHostName}/health");
}
[Output] public Output<string> Endpoint { get; set; }
[Output] public Output<string> HealthEndpoint { get; set; }
}
I appreciate there is a chunk of code there, so let's break it down just a little.
The first couple of lines is simply getting the config from Pulumi. I have chosen the Require method because I would rather it fails miserably if the config setting is not present; without it, we shouldn't continue because we'll have no container image. π
Up next is the resource group creation, note that we do not actually specify its location - The location comes from the config file - Pulumi.dev.yaml, it's value is from, azure-native:location. π
There are a bunch of other Azure native settings, for more information on these, please see, Azure Native Configuration
Following-on from there is the creation of the Azure AppService Plan, this is essentially our underlying Virtual Machine (VM), I have picked the smallest available WebApp for Containers (Linux) I can; so if you are paying for this I recommend when you test, to destroy afterwards - Still, if you didn't, it would be a under GBPΒ£10 per month anyway.
Then we have the WebApp itself, this is the AppService, there are a few things we need to set here, mainly stating where the Docker registry is, in our case, GitHub Container Registry - Please note that the default is always DockerHub, you can easily point it to ACR (Azure Container Registry) also. We set AlwaysOn and HttpsOnly to true and we of course need to set the Docker Image, this is the, LinuxFxVersion property.
We need a way to output values, we want to output the address of the host as the end of the resource names will be dynamically generated. We set one for the weatherforecast and just to be helpful we set one for the health endpoint too.
Finally, the last 2 lines of the file are the declarations of the output properties, what denotes these as outputs is simply the 'Output' attribute.
Step 4
Modify the Pulumi.dev.yaml file.
Replace its contents with the below:
config:
azure-native:location: uksouth
Samples.WeatherForecast.Infra:docker-image: ghcr.io/peteking/samples-weatherforecast-part-11:3dae90cc47eff6861687444a6e91be89a65bed9f
We've added our own config here which is our docker image, therefore, it's configuration instead of C# code.
This should be the location of your Docker image.
If you're using the GitHub Container Registry, it should be a simple swap.
Step 5
We have enough to give it a good manual "interactive" test with the Pulumi CLI, just to make sure everything is working as we expect it to.
Open your terminal and Navigate β /src/WeatherForecastApi-Infra/
Execute:
pulumi up
Here you can see it has executed preview as part of the up command, it highlights what it plans to do, in this case, it will create 4 resources.
Select β yes
Once you've selected yes, Pulumi will go-ahead and create those resources, below is the response.
You can see the Pulumi
outputswe created, one that links to theweatherforecastendpoint and one to thehealthendpoint; we've done this because the resources are dynamic, i.e. at the end of our Azure Resources is a random string to ensure uniqueness. If this doesn't suit your use case, you can ensure consistency instead - For more information, see Pulumi - Autonaming
We now have the opportunity to test our cloud infrastructure as code deployment and see if our API works!
You can use the links from the output to save typing.
The weather forecast gets returned! π
Even the /health endpoint is reporting good news! π
If you want to minimise your costs π², you can of course destroy π₯ your resources, and with the powers of Pulumi that is also a breeze with the, destory command.
Execute:
pulumi destory
Select β yes
Once you've selected yes, Pulumi will go-ahead and destroy those resources, below is the output. π
βΉοΈ
You'll notice the output at the bottom stating the stack is still available - All this means is you have all the details, history and everything about your stack in the Pulumi Console; which is all rather helpful, don't you think?
GitHub Actions with Pulumi
All of what we have achieved is great, we have got everything we need infrastructure wise, built it all as code, we've even deployed and tested it for real. However, it's all rather interactive, we need to bake this into both our CI Workflow and our CD Workflow.
Our CI Workflow will execute pulumi preview and utilise the Pulumi GitHub Bot for instant feedback in the Pull Request; by doing this, the person reviewing the PR will be able to see what resources will be created/destroyed/updated - If anything is a miss then you have the opportunity to do something about it.
Our CD Workflow will execute pulumi up and deploy our resources to Azure; but only once all our tests pass, code coverage pass and our container scan is all good too.
In order to interact with Azure in GitHub Actions we will create a Service Principal.
Azure Service Principal
What is it?
A Service Principal is an object in a single tenant or directory in Azure AD. It is the identity of an application instance, it defines access for the application and what resources it can access. Instead of using a user identity, we can create a service identity and manage its security and access.
Create Azure Service Principal
Step 1
Create an, Azure Subscription (if you don't already have one).
Step 2
Obtain the subscription id:
az account list #It's the 'id' field.
Step 3
Set the subscription:
az account set --subscription=<id> #<id> is the 'id' field from above
Step 4
az ad sp create-for-rbac --name sp-weather-api
You should see a result from Step 4 with the following values
{
"appId": "WWWWWWWW-WWWW-WWWW-WWWW-WWWWWWWWWWWW",
"displayName": "ServicePrincipalName",
"name": "http://ServicePrincipalName",
"password": "XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX",
"tenant": "YYYYYYYY-YYYY-YYYY-YYYY-YYYYYYYYYYYY"
}
βΉοΈ TIP
If you'd rather it in another format to save time later, you can use the--sdk-authparameter.
Securely store Azure Service Principal credentials
Now we have the Service Principal, we need to be able to store the details securely.
We can store these in GitHub Secrets which is totally fine right, but that means they are not linked to our Pulumi Stack, so another way is to store them inside Pulumi; which can also store secrets
From the output of our Service Principal creation command, the following values can be mapped like so:
| Service Principal Value | SDK Name |
|---|---|
| appId | clientId |
| password | clientSecret |
| tenant | tenantId |
This is where the
--sdk-authparameter can help, it will automatically map these for you.
Pulumi Secrets γοΈ
By storing these in Pulumi we get multi-user access for deployment which includes using it interactively; if we stored them in GitHub Secrets, we can only use them directly from the pipeline itself; there are pros and cons this of course - Choose what suites you best.
pulumi config set azure-native:clientId "WWWWWWWW-WWWW-WWWW-WWWW-WWWWWWWWWWWW"
pulumi config set azure-native:clientSecret "XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX" --secret
β Important
Note the--secretat the end of the command, this is very important; otherwise, the value will be stored in plane text, whereas we want it to be encrypted and stored as a secret.
pulumi config set azure-native:tenantId "YYYYYYYY-YYYY-YYYY-YYYY-YYYYYYYYYYYY"
pulumi config set azure-native:subscriptionId "ZZZZZZZZ-ZZZZ-ZZZZ-ZZZZ-ZZZZZZZZZZZZ"
Once you have executed these commands, you'll notice your Pulumi.dev.yml file has been updated accordingly.
CI Workflow
Let's modify our CI Workflow ./github/workflows/ci-pull-request.yml
Add a new job at the bottom of the Workflow.
infra:
needs: ci
runs-on: ubuntu-latest
steps:
- name: Checkout repo
uses: actions/checkout@v2
with:
# Disabling shallow clone is recommended for improving relevancy of reporting
fetch-depth: 0
- name: Install Pulumi CLI
uses: pulumi/action-install-pulumi-cli@v1.1.0
- name: Set Pulumi config values
working-directory: ./src/Samples.WeatherForecast.Infra
env:
PULUMI_ACCESS_TOKEN: ${{ secrets.PULUMI_ACCESS_TOKEN }}
run: |
pulumi stack select dev
pulumi config set Samples.WeatherForecast.Infra:docker-image ${{ env.image-name }}
- name: Pulumi Preview
uses: pulumi/actions@v3.1.0
with:
command: preview
stack-name: dev
work-dir: ./src/Samples.WeatherForecast.Infra
github-token: ${{ secrets.GITHUB_TOKEN }}
env:
PULUMI_ACCESS_TOKEN: ${{ secrets.PULUMI_ACCESS_TOKEN }}
Where do I get the Pulumi Personal Access Token from?
- Navigate β Pulumi Console - https://app.pulumi.com
- Navigate β Your profile (top-right)
- Click β Access Tokens - Quick link here: https://app.pulumi.com/YOUR_PULUMI_USER_NAME/settings/tokens
- Click β Create token
- Copy the secret generated
- Safely store the secret in GitHub Actions Secrets with the name of
PULUMI_ACCESS_TOKEN
Install the Pulumi GitHub Bot
Unless you want to dive into the Actions log to find out how the pulumi preview went, you better install the Pulumi GitHub Bot
Once you have the bot installed, when you initiate a PR, you'll see something like the below:
Even better though, is you can view more details in the Pulumi Console like so:
CD Workflow
Let's modify our CD Workflow ./github/workflows/build-and-publish.yml
Add a new job at the bottom of the Workflow.
infra:
needs: build-and-push
runs-on: ubuntu-latest
steps:
- name: Checkout repo
uses: actions/checkout@v2
with:
# Disabling shallow clone is recommended for improving relevancy of reporting
fetch-depth: 0
- name: Install Pulumi CLI
uses: pulumi/action-install-pulumi-cli@v1.1.0
- name: Set Pulumi config values
working-directory: ./src/Samples.WeatherForecast.Infra
env:
PULUMI_ACCESS_TOKEN: ${{ secrets.PULUMI_ACCESS_TOKEN }}
run: |
pulumi stack select dev
pulumi config set Samples.WeatherForecast.Infra:docker-image ${{ env.image-name }}
- name: Pulumi Up
uses: pulumi/actions@v3.1.0
with:
command: up
stack-name: dev
work-dir: ./src/Samples.WeatherForecast.Infra
github-token: ${{ secrets.GITHUB_TOKEN }}
env:
PULUMI_ACCESS_TOKEN: ${{ secrets.PULUMI_ACCESS_TOKEN }}
Commit your code
It's time to commit you code, but before you do, let's update our branch protection rule - Now we have an infra job in our Workflow which does a pulumi preview, we most likely want this to be mandatory and pass.
Now that we have our branch protection rules updated and CD Workflow file completed, let's ensure it all works.
- Raise your PR and see the CI Workflow kick-off.
- Verify everything is fine by looking at the Pulumi GitHub Bot or the Pulumi Console.
- Wait until all GitHub Status Checks pass and merge your PR into
main. - Verify your Workflow is successful.
- Either:
- Go to the Pulumi Console and see your Activity.
- OR see the output from the
infrajob in the CD Workflow.
- Try the output links to verify your API is running.
I prefer the Pulumi Console, see below, you can see the merge of my branch and the creation of the infrastructure once it was merged into main.
I was then able to validate that both the weatherforecast and health endpoints was functional.
What have we learned? π€
We have learned all about IaC - Infrastructure as Code, and utilised Pulumi to achieve this - We have leveraged our existing C# skills to shift-left on our infrastructure; therefore, we are now more autonomous.
We've been able to test (preview) and verify our infrastructure changes during Pull Requests as part of CI, and we have achieved C-Deployment by releasing our infrastructure in our CD Workflow.
Relief - That's it! π
If you've made it this far, congratulations and thank you π€, I know it's a long post and a lot to take in, but I hope you have found it worthwhile.
There is unfortunately or fortunately, depending on how you look at it, still a lot to do π So there will be a more coming soon! π€ͺ
UPDATE AUGUST 27TH 2021
The below GitHub Issue has now been resolved.
π Note
Adding Pulumi to your repo in GitHub will break CodeQL,there is an active GitHub issue open for this.
peteking posted onHi,
I have failure in the codeql-action, it has failed with an exit code of 100, looking at the docs it says it's likely a bug based on this error code.
This is an open source sample project, so all open.
The recent addition made as you can see from the PR is I've added Pulumi for its Azure infrastructure provisioning.
I think it start to look at Pulumi here:
Running TRAP import for CodeQL database at /home/runner/work/_temp/codeql_databases/csharp... Running TRAP import for CodeQL database at /home/runner/work/_temp/codeql_databases/csharp... Pulumi.AzureNative.dll.trap.gz, 9885761: java.lang.OutOfMemoryError: Java heap spaceand then slowly goes wrong and fails, out of memory.
Here is the codeql workflow run here: https://github.com/peteking/Samples.WeatherForecast-Part-11/pull/1/checks?check_run_id=2738620747
Part of the log is here for quick reference:
Finalizing csharp /opt/hostedtoolcache/CodeQL/0.0.0-20210517/x64/codeql/codeql database finalize --threads=2 /home/runner/work/_temp/codeql_databases/csharp Running pre-finalize script /opt/hostedtoolcache/CodeQL/0.0.0-20210517/x64/codeql/csharp/tools/pre-finalize.sh in /home/runner/work/Samples.WeatherForecast-Part-11/Samples.WeatherForecast-Part-11. Running pre-finalize script /opt/hostedtoolcache/CodeQL/0.0.0-20210517/x64/codeql/csharp/tools/pre-finalize.sh in /home/runner/work/Samples.WeatherForecast-Part-11/Samples.WeatherForecast-Part-11. [2021-06-03 15:44:36] [build-stderr] Scanning for files in /home/runner/work/Samples.WeatherForecast-Part-11/Samples.WeatherForecast-Part-11... [2021-06-03 15:44:36] [build-stderr] Scanning for files in /home/runner/work/Samples.WeatherForecast-Part-11/Samples.WeatherForecast-Part-11... [2021-06-03 15:44:36] [build-stderr] /home/runner/work/_temp/codeql_databases/csharp: Indexing files in in /home/runner/work/Samples.WeatherForecast-Part-11/Samples.WeatherForecast-Part-11... [2021-06-03 15:44:36] [build-stderr] /home/runner/work/_temp/codeql_databases/csharp: Indexing files in in /home/runner/work/Samples.WeatherForecast-Part-11/Samples.WeatherForecast-Part-11... [2021-06-03 15:44:36] [build-stderr] /home/runner/work/_temp/codeql_databases/csharp: Running in /home/runner/work/Samples.WeatherForecast-Part-11/Samples.WeatherForecast-Part-11: [/opt/hostedtoolcache/CodeQL/0.0.0-20210517/x64/codeql/xml/tools/index-files.sh, /home/runner/work/_temp/codeql_databases/csharp/working/files-to-index16194344738812882790.list] [2021-06-03 15:44:36] [build-stderr] /home/runner/work/_temp/codeql_databases/csharp: Running in /home/runner/work/Samples.WeatherForecast-Part-11/Samples.WeatherForecast-Part-11: [/opt/hostedtoolcache/CodeQL/0.0.0-20210517/x64/codeql/xml/tools/index-files.sh, /home/runner/work/_temp/codeql_databases/csharp/working/files-to-index16194344738812882790.list] Running TRAP import for CodeQL database at /home/runner/work/_temp/codeql_databases/csharp... Running TRAP import for CodeQL database at /home/runner/work/_temp/codeql_databases/csharp... Pulumi.AzureNative.dll.trap.gz, 9885761: java.lang.OutOfMemoryError: Java heap space com.semmle.inmemory.trap.TrapScanner.getTokenString(TrapScanner.java:521) com.semmle.inmemory.trap.TrapScanner.getLabelValue(TrapScanner.java:578) com.semmle.inmemory.trap.TRAPReader.scanOneField(TRAPReader.java:754) Pulumi.AzureNative.dll.trap.gz, 9885761: java.lang.OutOfMemoryError: Java heap space com.semmle.inmemory.trap.TRAPReader.scanTuple(TRAPReader.java:495) com.semmle.inmemory.trap.TrapScanner.getTokenString(TrapScanner.java:521) com.semmle.inmemory.trap.TRAPReader.scanTuplesAndLabels(TRAPReader.java:439) com.semmle.inmemory.trap.TrapScanner.getLabelValue(TrapScanner.java:578) com.semmle.inmemory.trap.TRAPReader.importTuples(TRAPReader.java:376) com.semmle.inmemory.trap.TRAPReader.scanOneField(TRAPReader.java:754) com.semmle.inmemory.trap.ImportTasksProcessor.process(ImportTasksProcessor.java:172) com.semmle.inmemory.trap.TRAPReader.scanTuple(TRAPReader.java:495) com.semmle.inmemory.trap.ImportTasksProcessor.lambda$null$1(ImportTasksProcessor.java:129) com.semmle.inmemory.trap.TRAPReader.scanTuplesAndLabels(TRAPReader.java:439) com.semmle.inmemory.trap.ImportTasksProcessor$$Lambda$234/0x000000010031fc40.accept(Unknown Source) com.semmle.util.concurrent.FutureUtils.lambda$null$8(FutureUtils.java:136) com.semmle.util.concurrent.FutureUtils$$Lambda$236/0x000000010031f440.get(Unknown Source) java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(Unknown Source) java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) java.base/java.lang.Thread.run(Unknown Source) com.semmle.inmemory.trap.TRAPReader.importTuples(TRAPReader.java:376) com.semmle.inmemory.trap.ImportTasksProcessor.process(ImportTasksProcessor.java:172) com.semmle.inmemory.trap.ImportTasksProcessor.lambda$null$1(ImportTasksProcessor.java:129) com.semmle.inmemory.trap.ImportTasksProcessor$$Lambda$234/0x000000010031fc40.accept(Unknown Source) com.semmle.util.concurrent.FutureUtils.lambda$null$8(FutureUtils.java:136) com.semmle.util.concurrent.FutureUtils$$Lambda$236/0x000000010031f440.get(Unknown Source) java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(Unknown Source) java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) java.base/java.lang.Thread.run(Unknown Source) Oops! A fatal internal error occurred. Oops! A fatal internal error occurred. java.lang.RuntimeException: java.lang.OutOfMemoryError: Java heap space java.lang.RuntimeException: java.lang.OutOfMemoryError: Java heap space at com.semmle.util.exception.Exceptions.asUnchecked(Exceptions.java:103) at com.semmle.inmemory.trap.TrapImporter.run(TrapImporter.java:105) at com.semmle.cli2.ql.dataset.ImportCommand.executeSubcommand(ImportCommand.java:98) at com.semmle.cli2.picocli.PlumbingRunner.run(PlumbingRunner.java:110) at com.semmle.cli2.picocli.SubcommandCommon.runPlumbingInProcess(SubcommandCommon.java:159) at com.semmle.cli2.database.FinalizeCommand.executeSubcommand(FinalizeCommand.java:110) at com.semmle.cli2.picocli.SubcommandCommon.call(SubcommandCommon.java:448) at com.semmle.util.exception.Exceptions.asUnchecked(Exceptions.java:103) at com.semmle.inmemory.trap.TrapImporter.run(TrapImporter.java:105) at com.semmle.cli2.ql.dataset.ImportCommand.executeSubcommand(ImportCommand.java:98) at com.semmle.cli2.picocli.PlumbingRunner.run(PlumbingRunner.java:110) at com.semmle.cli2.picocli.SubcommandCommon.runPlumbingInProcess(SubcommandCommon.java:159) at com.semmle.cli2.database.FinalizeCommand.executeSubcommand(FinalizeCommand.java:110) at com.semmle.cli2.picocli.SubcommandCommon.call(SubcommandCommon.java:448) at com.semmle.cli2.picocli.SubcommandMaker.runMain(SubcommandMaker.java:201) at com.semmle.cli2.picocli.SubcommandMaker.runMain(SubcommandMaker.java:209) at com.semmle.cli2.CodeQL.main(CodeQL.java:96) Caused by: java.lang.OutOfMemoryError: Java heap space at com.semmle.cli2.picocli.SubcommandMaker.runMain(SubcommandMaker.java:201) at com.semmle.cli2.picocli.SubcommandMaker.runMain(SubcommandMaker.java:209) at com.semmle.cli2.CodeQL.main(CodeQL.java:96) Caused by: java.lang.OutOfMemoryError: Java heap space Error: The process '/opt/hostedtoolcache/CodeQL/0.0.0-20210517/x64/codeql/codeql' failed with exit code 100 Error: The process '/opt/hostedtoolcache/CodeQL/0.0.0-20210517/x64/codeql/codeql' failed with exit code 100 at Object.toolrunnerErrorCatcher (/home/runner/work/_actions/github/codeql-action/v1/lib/toolrunner-error-catcher.js:79:19) at processTicksAndRejections (internal/process/task_queues.js:93:5) at async Object.finalizeDatabase (/home/runner/work/_actions/github/codeql-action/v1/lib/codeql.js:414:13) at async finalizeDatabaseCreation (/home/runner/work/_actions/github/codeql-action/v1/lib/analyze.js:74:9) at async Object.runAnalyze (/home/runner/work/_actions/github/codeql-action/v1/lib/analyze.js:172:5) at async run (/home/runner/work/_actions/github/codeql-action/v1/lib/analyze-action.js:50:30) at async runWrapper (/home/runner/work/_actions/github/codeql-action/v1/lib/analyze-action.js:96:9)
Next up
Part 12 in this series will be about:
-
More Infrastructure as Code with Pulumi
- Environmental support (multiple Stacks)
- Security
- What can we do on the cheap in Azure?
- What should we do in a more real scenario?
- GitHub Actions
- Clean-up a little and add Environment capabilities
Top comments (0)