DEV Community: Polar Squad

When LLMs struggle: Architecture, context, and hidden complexity

Jari Haikonen — Tue, 31 Mar 2026 06:44:35 +0000

The obvious LLM failures are easy to catch. Syntax errors, broken configs, a pipeline that refuses to run. You see the problem immediately and fix it. Those are not the ones that should worry you.

The ones that should worry you are the ones that look completely fine. The code runs. The config is valid. The output looks reasonable. And yet, when someone with more experience takes a look, the problems become obvious immediately. They were just invisible to you.

The knowledge mirror

There is a pattern I noticed pretty quickly when working on tasks outside my main area.

When I work in areas I know well, like Terraform or CI/CD pipelines, I can evaluate the model's output almost automatically. I know what good looks like, I know the common failure patterns, and I catch mistakes fast. The feedback loop is tight.

But when I work on something I know less about, that feedback loop breaks. And the problem is that the model does not help you here at all. It does not become more cautious in unfamiliar territory. It does not tell you when it is guessing. It produces the same confident, well-formatted output regardless of whether it is right or wrong.

Anthropic's engineering team documented the same behavior when building multi-agent coding harnesses. Their published observation: when asked to evaluate their own output, agents reliably respond by confidently praising it, even when the quality is obviously mediocre to a human observer. The confidence is not correlated with the actual quality of the work.

So what you end up with is a mirror: the model reflects your level of knowledge back at you. If you know the domain, you catch the mistakes. If you do not, you miss them. And the less you know, the more you are at the mercy of output you cannot properly evaluate.

In practice, for me this showed up most clearly in development tasks. There are usually several valid ways to implement the same thing in code, and the right choice depends on context, team conventions, performance requirements, and a lot of other things the model cannot know. If you are not familiar enough with those trade-offs yourself, the model will just pick one. And if you do not notice, it gets built on.

The problem with architectural decisions

LLMs are actually quite good at implementing things. Give the model a clear approach and it will execute it well. The problem is when you ask it to choose the approach.

Architectural decisions involve context the model simply does not have: your team's skill level, how much complexity the team can realistically own and maintain at once, which parts of your system are already overengineered, the operational cost of what it is about to build, your future plans. Without that, it defaults to what it has seen most in training data, which is usually the textbook approach or the most complex one, not necessarily the most appropriate one for your situation.

In DevOps this matters more than it might seem, because infrastructure decisions have long tails. A bad pattern in a Terraform module layout, a poorly thought-out pipeline structure, a dependency that should not be there, these things propagate. Fixing them later is far more expensive than catching them early.

There is also the consistency problem. The model has absorbed a lot of old documentation and outdated best practices, and it has no sense of what is current or what fits your context. So it might solve the same kind of problem in two different places in your codebase using two completely different approaches. Both technically valid, but inconsistent in ways that make things harder to maintain over time.

The practical answer is documentation: write down your decisions and conventions and feed them to the model. This genuinely helps. But the model does not always follow the rules you set for it, especially on longer tasks. You still need to review what it produces. Documentation reduces the drift, it does not eliminate it.

A concrete example: a colleague was working on changes that touched multiple Terragrunt module levels and wanted to structure it as a single PR. The constraint is straightforward: you cannot use outputs from one level in another before applying the first one, so the rule is one PR per level. The fix: use git checkout origin/main -- path_to_file to revert the level-two file back to main in your current branch, open the first PR, merge and apply it, then create a second PR with the level-two changes. She had already asked Copilot. It had given her something much longer and more complicated.

The one thing to watch: if the reverted file had significant changes in it, save or stash them before running the checkout, because it will wipe them from your branch.

A model that produces something technically correct but architecturally wrong is in some ways more dangerous than one that produces something broken, because at least broken things announce themselves.

When the model starts looping

The other failure mode that shows up regularly is looping. You give the model a problem, it gives you an answer, the answer is wrong, you tell it so, it gives you a variation, that is also wrong, and so on. Anthropic's engineering team describes the same failure in the same terms for longer agentic tasks: on complex work, the agent tends to go off the rails over time, producing increasingly elaborate answers that are no more correct than the first one.

A good example of this from my own work: I was building a GitLab CI pipeline and wanted to keep it DRY. The changes: blocks that control when jobs run were being repeated across every rule, so I asked the model to clean it up using YAML anchors. It produced something that looked completely reasonable, valid YAML, clean structure. The pipeline failed. I fed the error back. It adjusted. Still failed. A few iterations in, the suggestions were getting more elaborate but the pipeline kept breaking in the same way.

The next article in this series gets into exactly why this happens with GitLab specifically, and what the pattern looks like in practice. The signs are pretty recognizable once you have seen it a few times: the answers are getting longer but not more correct, the error is not actually changing between iterations, and you are spending more time explaining the problem than it would take to just fix it.

The right move at that point is to stop, step back, figure out the root cause yourself, and either fix it directly or come back to the model with a much more specific prompt that contains the missing context. What does not work is adding more context and hoping the next attempt will break the pattern. Usually it does not.

Senior knowledge is more important, not less

In practice I have found the opposite of what the "AI replaces engineers" conversation suggests.

With LLMs, experienced engineers spend less time writing code and config and more time reviewing output, catching bad patterns and making architectural decisions. The volume of output goes up significantly, which means the demand for quality review goes up with it. And here is the uncomfortable part: a junior engineer can now generate code faster than a senior engineer can critically audit it. The rate-limiting factor that used to keep review meaningful has been removed.

If you do not have the experience to evaluate what the model produces, you are not doing less work. You are just accumulating a gap between how much exists in your codebase and how much anyone genuinely understands. Addy Osmani calls this comprehension debt, and the pattern he describes maps closely to what I have been seeing in practice.

The value of experience has not disappeared. It has moved to a different place in the workflow.

The takeaway

LLMs struggle most where human expertise matters most: architectural decisions, trade-off reasoning, domain-specific behavior. That is not a reason to avoid using them in those areas. It is a reason to stay engaged as the expert when you do.

One of the most concrete examples of this I have seen is GitLab pipelines, where the model is technically correct about YAML and completely wrong about GitLab's implementation at the same time. If that sounds familiar, that is exactly what the next article in this series is about.

LLMs in DevOps: Why They Work Best as a "Very Fast Junior Engineer"

Jari Haikonen — Thu, 26 Mar 2026 11:37:46 +0000

I was staring at roughly 10,000 lines of network rules spread across a live cloud environment. Two environments, dev and prod, two regions each, all handled by their own separate configuration files. The task was to cross-check what had already been imported into Terraform and what hadn't, and then split the rules correctly across all those files. That kind of task could easily take weeks to do carefully by hand. With an LLM doing the heavy lifting, I was done in three hours.

That was the moment the mental model clicked for me.

And yes, these were network security rules. But here is the thing: in a Terraform import workflow, the tooling itself is the safety net. The goal is a 1:1 match between your IaC and the actual state of the environment. If the AI-generated configuration has any drift from reality, Terraform tells you immediately when you run the import. You are not trusting the AI blindly, you are using it to do the repetitive work and then letting Terraform verify the result. That is a very different risk profile from asking an LLM to design your network security from scratch.

I have been using AI tools as a regular part of my DevOps work for about a year now, not just occasionally but daily, across hobby projects, volunteer work and professional infrastructure and development work. I come at this as a lead DevOps consultant with over 20 years in IT, so I have a pretty good baseline for what good looks like and what bad looks like.

After a year of this, I have some clear opinions about what these tools are actually good for, where they fall apart, and what way of thinking about them actually helps in practice.

This is not a model comparison and not a benchmark. Those exist already. This is just what I have noticed from using LLMs in real DevOps work.

A year of real use

The work has covered:

coding in JavaScript, TypeScript, Golang and Java
IaC with Terraform and Terragrunt
configuration management with Ansible
CI/CD work on GitLab, GitHub, Docker Compose and Helm

I have tried several models, including Claude (Sonnet, Opus, Haiku), ChatGPT, Gemini and Grok, and different IDEs like VS Code and Cursor. The models do have different strengths and there are clear gaps between them, but I am not going to get into that here. What I want to talk about is what using all of them has taught me about AI-assisted DevOps work in general.

The pattern that kept repeating

Across all that work, one pattern kept showing up.

When I gave the model clear context, a well-scoped task and some constraints to work within, the output was fast and impressively good. When I gave it an open-ended problem or let things run without much correction, the quality dropped quickly. Dead code started accumulating, inconsistent patterns appeared and the model started looping through variations of the same wrong answer.

The difference was not which model I was using. The difference was how much structure I brought to the interaction.

And that structure comes directly from your own maturity and experience in the domain. The more you know, the more precisely you can specify what you want, and the better the output gets. This is probably the most underappreciated factor in how well LLMs actually perform in practice.

Compare these two prompts for the same task:

"Create pipeline that deploys my nodejs app"

versus:

"Create CI/CD pipelines for pull requests and deploying on main branch. Add quality gates to the PR pipeline: format, lint, security, build and docker build. In the main pipeline do docker builds and use the registry for cached images to make builds faster. On the Dockerfiles use multi-stage builds where possible to keep the final image small, and make sure we are not running as root. Make the pipelines DRY on the sections that overlap"

The second prompt does not just describe what to build. It reflects years of experience with CI/CD, Docker best practices and security thinking. Someone without that background would not even know to ask for those things. The model cannot supply that knowledge from its own side, it can only work with what you give it.

It is not that the model is bad. It just has no stake in the outcome and no experience to fall back on. It will produce output either way. The quality of that output depends almost entirely on the quality of the guidance behind it.

A very fast junior engineer

The mental model that finally made this click for me: an LLM behaves like a very fast junior engineer.

A good junior can produce a lot of work quickly and they follow clear instructions well. But they struggle with architectural decisions, tend to go with the most obvious approach rather than the most appropriate one, and need supervision.

They act this way not because they are useless but because they lack the context and experience to make the right call on their own. Leave them unsupervised long enough and small decisions start to compound into bigger problems.

LLMs behave exactly like this, just at roughly ten times the speed. The speed is real and genuinely useful, but it does not change the underlying dynamic.

There is an important flip side to this that is worth saying directly: the analogy only works if you actually are the senior. If you jump into a domain you know nothing about, the dynamic inverts. The model becomes the one with more apparent knowledge and you have no real basis to supervise it. You cannot catch the bad architectural decisions because you do not recognise them. That is when you get the worst outcomes: confident-sounding output that is quietly wrong in ways that take a long time to find and fix.

When you accept this framing, a few things shift in how you work:

Your job becomes that of an architect instead of a typist. You define the structure, the constraints, the approach. The model handles the execution.
Structuring the problem well matters more than prompting well. A well-defined task with clear context will beat a cleverly worded prompt for an undefined problem every time.
You still need to know your domain. The better you understand the area you are working in, the better you can guide the model and catch its mistakes. Domain expertise is not optional, it is what makes the supervision possible in the first place.

What this looks like in practice

One thing that has helped quite a lot is writing documentation and conventions that both humans and the model can use. Not AI-specific memory tools or special prompting tricks, but actual documentation that would exist for your team anyway. Things like guidelines in a Terraform modules folder, pipeline conventions, naming rules.

When that structure exists and you give the model access to it, it follows the established patterns instead of inventing new ones. The corrections get smaller and the output actually fits the system you are building.

The other thing is knowing when to stop iterating with the model and just fix something yourself. Sometimes two or three rounds of back and forth are not making progress and the model is just looping. At that point the fastest path forward is usually to step in, fix the specific issue yourself, and re-engage the model for the work around it.

The takeaway

AI is already genuinely useful in DevOps workflows. The value you get out of it scales with the quality of the supervision and structure you bring as the engineer. The model is the junior. You are the senior. That dynamic does not disappear as the tools get faster or more capable.

The rest of this series goes into the details. The next piece looks at where LLMs consistently struggle and why the failures are harder to catch than they look. After that, a concrete GitLab pipeline example that most DevOps engineers will recognise. And then the positive story: why importing existing infrastructure into Terraform is one of the best use cases for LLMs I have found.

If you have been using LLMs in DevOps or platform engineering work, I am curious what mental model you have settled on. Does the junior engineer analogy match your experience or have you found a better way to think about it?

AI/ML Platforms: Pros and Cons

Jani Ranta — Mon, 28 Oct 2024 11:33:47 +0000

When choosing between AI/ML platforms, each provider has its strengths and trade-offs:

Azure Machine Learning excels in seamless integration with Microsoft services, strong security features, and ease of use through AutoML tools, but can become costly with complex pricing and requires expertise within the Azure ecosystem.
AWS SageMaker offers extensive flexibility, scalability, and strong AWS service integration, making it ideal for large-scale applications, though it can be challenging for beginners due to its steep learning curve and potentially high costs.
Google Vertex AI is known for its user-friendly interface and superior AutoML capabilities, particularly for data-heavy operations, but has fewer pre-built models compared to AWS and can be expensive with larger datasets.
On the other hand, open-source solutions like Hugging Face offer cost savings and high customization, but demand significant technical expertise and manual setup, making them harder to scale without the right resources.

Machine Learning and AI Model Development

Category	Microsoft Azure	Amazon Web Services	Google Cloud Platform	Open Source
Machine Learning and AI Model Development	Azure Machine Learning	Amazon SageMaker	Vertex AI	MLflow, Kubeflow, Ray Serve
Image Recognition and Computer Vision	Computer Vision	Amazon Rekognition	Vision AI	DeepStack AI Server
Natural Language Processing (NLP)	Text Analytics, Language Understanding (LUIS)	Amazon Comprehend	Natural Language API	Haystack, Rasa, spaCy REST API, Hugging Face
Speech Recognition and Text-to-Speech	Azure AI Speech Service	Amazon Transcribe, Amazon Polly	Speech-to-Text, Text-to-Speech	Vosk (for Speech Recognition), Coqui TTS (for Text-to-Speech)
Chatbots and Interactive Applications	Azure Bot Services, Microsoft Copilot Studio	Amazon Lex	Dialogflow	Rasa, Botpress
Automated Text Processing and Analysis	Azure AI Document Intelligence	Amazon Comprehend	Document AI	Tesseract OCR, Apache Tika
Generative Large Language Models (LLMs)	Azure OpenAI Service	Amazon Bedrock	Vertex AI, Gemini	DeepSpeed, Haystack, Hugging Face Transformers, GPT-J/NeoX Playground, Ollama

Azure Machine Learning: A cloud-based service designed for the entire machine learning lifecycle, enabling data scientists and engineers to build, train, and deploy models at scale. It supports various frameworks and offers tools for MLOps, data preparation, and model management.

Amazon SageMaker: A fully managed service that provides tools for building, training, and deploying machine learning models quickly. It includes features like built-in algorithms, Jupyter notebooks, and model monitoring capabilities

Vertex AI: A unified platform that simplifies the machine learning workflow by integrating various tools and services for data preparation, model training, and deployment. It supports AutoML and custom training with TensorFlow and PyTorch.

MLflow: An open-source platform for managing the machine learning lifecycle, including experimentation, reproducibility, and deployment. It provides a tracking server, projects, and a model registry.

Kubeflow: An open-source machine learning toolkit for Kubernetes, designed to facilitate the deployment, orchestration, and management of machine learning workflows on Kubernetes clusters.

Ray Serve: A scalable model serving library that allows users to deploy machine learning models in production with minimal latency. It integrates seamlessly with Ray, a distributed computing framework.

Image Recognition and Computer Vision

Computer Vision: A service that provides algorithms for image analysis, including object detection, image classification, and optical character recognition (OCR) capabilities.

Amazon Rekognition: A service that makes it easy to add image and video analysis to applications, offering features like facial recognition and object detection.

Vision AI: A Google Cloud service that provides powerful image analysis capabilities through pre-trained models and custom model training

DeepStack AI Server: An open-source platform for implementing AI capabilities in applications, including image recognition and face detection.

Natural Language Processing (NLP)

Text Analytics, Language Understanding (LUIS): Azure services that provide capabilities for sentiment analysis, key phrase extraction, and language understanding for building conversational applications.

Amazon Comprehend: A natural language processing service that uses machine learning to find insights and relationships in text, such as sentiment and entity recognition.

Natural Language API: A Google Cloud service that allows developers to analyze and understand text through features like entity recognition and sentiment analysis.

Haystack, Rasa, spaCy REST API, Hugging Face: Open-source frameworks and libraries for building NLP applications, offering capabilities for intent recognition, dialogue management, and text processing.

Speech Recognition and Text-to-Speech

Azure AI Speech Service: A service that provides speech recognition and text-to-speech capabilities, enabling applications to convert speech to text and vice versa.

Amazon Transcribe, Amazon Polly: Services for automatic speech recognition and text-to-speech, allowing developers to add voice capabilities to applications easily.

Speech-to-Text, Text-to-Speech: Google Cloud services that enable audio transcription and speech synthesis, providing high-quality voice outputs for applications.

Vosk, Coqui TTS: Open-source tools for speech recognition and text-to-speech, allowing developers to integrate voice capabilities into their applications without relying on cloud services.

Chatbots and Interactive Applications

Azure Bot Services: A platform for building and deploying intelligent chatbots that can interact with users across various channels. With Microsoft Copilot Studio on Azure this service becomes very powerful.

Amazon Lex: A service for building conversational interfaces using voice and text, powered by the same technology as Alexa.

Dialogflow: A Google Cloud service for building conversational agents, offering natural language understanding and integration with various messaging platforms.

Rasa, Botpress: Open-source frameworks for developing conversational AI applications, providing tools for building, training, and deploying chatbots.

Automated Text Processing and Analysis

Azure AI Document Intelligence: A service that helps automate the extraction of information from documents, enhancing data processing workflows.

Amazon Comprehend: Also mentioned under NLP, it provides capabilities for analyzing text and extracting insights from documents.

Document AI: A Google Cloud service that automates the extraction of structured data from unstructured documents, improving data processing efficiency.

Tesseract OCR, Apache Tika: Open-source tools for optical character recognition and document parsing, enabling automated text extraction from images and documents.

Last but not least, Generative Large Language Models (LLMs)

AWS SageMaker: A fully managed machine learning (ML) service. Data scientists and developers can quickly and confidently build, train, and deploy ML models into a production-ready hosted environment. It provides a UI experience for running ML workflows that makes SageMaker ML tools available across multiple integrated development environments (IDEs).

Azure OpenAI Service: A service that provides access to OpenAI's powerful language models, enabling developers to build applications that require natural language understanding and generation.

Amazon Bedrock: A managed service that allows developers to build and scale generative AI applications using foundation models from various providers.

Vertex AI, Gemini: Google Cloud services that facilitate the development of generative AI applications, offering access to advanced language models for various use cases.

DeepSpeed, Haystack, Hugging Face Transformers, GPT-J/NeoX Playground, Ollama: Open-source tools and frameworks for building and deploying generative AI applications, providing capabilities for training and fine-tuning large language models.

Feature/Capability	Azure Machine Learning	AWS SageMaker	Google Vertex AI	Open Source Solutions
Management Type	Fully managed	Fully managed	Fully managed	Self-managed
Ease of Use	High, with visual tools and AutoML	Moderate, with no-code options available	High, with integrated tools	Varies by tool
Model Training	Supports various frameworks, AutoML	Supports various frameworks, AutoML	Supports various frameworks, AutoML	MLflow, Kubeflow, Ray Serve
Model Deployment	Easy endpoint configuration	Easy endpoint configuration	Easy endpoint configuration	Requires manual setup
Pre-built Models	Yes, through Azure Model Gallery	Yes, through SageMaker Model Zoo	Yes, through Model Garden	Depends on the specific tool
Integration with Other Services	Strong integration with Azure services	Strong integration with AWS services	Strong integration with Google services	Varies by tool
Generative AI Support	Yes, through Azure OpenAI Service	Yes, through SageMaker	Yes, through GenAI	DeepSpeed, Hugging Face Transformers
NLP Capabilities	Comprehensive (Text Analytics, LUIS)	Comprehensive (Comprehend)	Comprehensive (Natural Language API)	Haystack, Rasa, spaCy REST API
Computer Vision Capabilities	Yes, through Computer Vision	Yes, through Amazon Rekognition	Yes, through Vision AI	DeepStack AI Server
Speech Recognition	Yes, through Azure AI Speech Service	Yes, through Amazon Transcribe	Yes, through Speech-to-Text	Vosk
Text-to-Speech	Yes, through Azure AI Speech Service	Yes, through Amazon Polly	Yes, through Text-to-Speech	Coqui TTS
Collaboration Tools	Azure ML Workspaces	SageMaker Studio for team collaboration	Vertex AI Workbench	Varies by tool
Cost Structure	Pay-as-you-go, pricing varies by usage	Pay-as-you-go, pricing varies by usage	Pay-as-you-go, pricing varies by usage	Free, but requires infrastructure
Customization	High customization options available	High customization options available	High customization options available	High, depending on the framework

How to use AWS Roles Anywhere

Janne Pohjolainen — Wed, 21 Feb 2024 09:27:20 +0000

What is AWS Roles Anywhere?

AWS Roles Anywhere enables you to use AWS Policies and AWS Roles for workloads such as servers, containers and applications that are running outside AWS without having to create long-term credentials.

AWS Roles Anywhere uses SSL certificates to manage workload access through IAM Roles and handing over short-living session keys.

How does AWS Roles Anywhere work?

A workload needs to have an SSL certificate (and a private key) that is signed by a Certificate Authority configured in AWS Roles Anywhere as a Trust Anchor. It is possible to configure external CAs or to use an AWS Private CA.

AWS Roles Anywhere Profiles map which IAM Roles are possible to be assumed by the workload. The workload is then given a short-living session keys which grants access to AWS services according to the role policies.

This is more secure than if user credentials or other long-term credentials are used, as they usually have much more access than a single role. It's also possible to set boundaries for the roles added to the Profile which override the role's policies.

What use-cases does AWS Roles Anywhere have?

These roles can be used in any kind of workload running outside AWS that needs access to the resources in the cloud. For example,

On-premises databases servers could connect to S3 to fetch data.
If you use Kubernetes in a private cloud, you can create a role that gives the application container running in the cluster permissions to specific AWS resources only.
If you already have an existing CA and PKI (Public-Key Infrastructure) system, another use case is to use AWS Roles Anywhere to use the existing certificates to grant access to AWS services.

Let's dive in deeper...

Some infrastructure is needed first

First, a Certificate Authority (CA) is needed. For this example, we will create an AWS Private CA and then create an AWS Roles Anywhere trust anchor for it.
After that, we will create an IAM Role with S3 read-only access and a Roles Anywhere Profile that links the IAM role to the trust anchor and CA.

We will not dive deep into these and concentrate more on the workload side. Terraform code used to create CA, trust anchor, profile, role and an S3 bucket for this example can be found in https://github.com/jpohjolainen/aws_roles_anywhere

Once run, Terraform will output the ARNs of the newly created CA, trust anchor, role and profile, and also a S3 bucket name.

These will be needed later on for creating a certificate for signing in to AWS and when revoking a certificate.

Speaking of certificates, let's create a new certificate request and get a certificate from the CA.

Note: Only the first private CA will be free for 30 days. If you create one and then delete it and create another, you will need to pay for the CA immediately. A private CA costs 300€ ($400) a month.

Note 2: It is also possible to create your own CA with an OpenSSL command and configure it using external CA. Here is how to do that: https://aws.amazon.com/blogs/security/iam-roles-anywhere-with-an-external-certificate-authority/

Creating a certificate and private key for our application

Create Certificate Signing Request (CSR) and a new Private Key.

Create CSR app-cert.csr and private key app-private-key. Change the Subject to your liking. C=Country, ST=State, OU=Organization Unit, O=Organization and CN=Common Name (name of app, or hostname/domain)

openssl req -new -newkey rsa:2048 \
 -out "app-cert.csr" \
 -keyout "app-private.key" \
 -subj "/C=DE/ST=Berlin/OU=DevOps/CN=app1"

The previous command prompted for password for the private key, but we need to remove it as this is for an application

openssl rsa -in "app-private.key" -out "app-private-nopass.key"

Request a certificate from the CA based on the CSR.

aws acm-pca issue-certificate \
 --certificate-authority "arn:aws:acm-pca:eu-west-1:xxxxxx:certificate-authority/zzzzzzzzz" \
 --csr "fileb://app-cert.csr" \
 --signing-algorithm "SHA256WITHRSA" \
 --validity Value=365,Type="DAYS"

The --csr option really has fileb:// instead of file://. It is used to read files in binary format in the AWS CLI.

Note: The following signing algorithms are available: SHA256WITHECDSA, SHA384WITHECDSA, SHA512WITHECDSA, SHA256WITHRSA, SHA384WITHRSA and SHA512WITHRSA.
The specified signing algorithm family (RSA or ECDSA) must match the algorithm family of the CA's secret key.

The validity type can be YEARS, MONTHS, DAYS and the number as the value, END_DATE with value of YYYYMMDDHHMMSS or ABSOLUTE with a unix timestamp as the value.

Get the CertificateARN from the reply. It is used in next section.

Download the issued certificate.

Use the AWS CLI to download the certificate from the private CA

aws acm-pca get-certificate \
 --certificate-arn "arn:aws:acm-pca:eu-west-1:xxxxxx:certificate-authority/zzzzzzzzz/certificate/af7d3bf5c562a7d91f9310da8ae6ea8d" \
 --certificate-authority-arn "arn:aws:acm-pca:eu-west-1:xxxxxx:certificate-authority/zzzzzzzzz" \
 --output json \
 |jq -r '.Certificate, .CertificateChain' > app-cert.pem

Note: This uses jq to get the certificate in JSON format. It's possible to use --output text and direct that to a file, but then you need to edit the file to move the second -----BEGIN CERTIFICATE----- to its own line.

Show the certificate

openssl x509 -in app-cert.pem -noout -text

Creating a container with an application

Once we have the certificate and the private key created, we need to have a configuration for AWS.
$HOME/.aws/config is used by AWS SDK and CLI for getting access to AWS by signing in with the certificate and assume the role.

AWS provides a tool to help with the sign-in called AWS Signing Helper. In this example, it is downloaded inside the Dockerfile when building the image.
https://docs.aws.amazon.com/rolesanywhere/latest/userguide/credential-helper.html

.aws/config

The helper tool can then be used in the file $HOME/.aws/config to login to the AWS when SDK or CLI is used. Here we need the ARNs that the Terraform code above returns. Save this to a file called aws-config. The Dockerfile expects this name:

[default]
    credential_process = /usr/local/bin/aws_signing_helper credential-process --certificate /app/app-cert.pem --private-key /app/app-private-nopass.key --trust-anchor-arn arn:aws:rolesanywhere:eu-west-1:xxxxxx:trust-anchor/yyyyyyyy --profile-arn arn:aws:rolesanywhere:eu-west-1:xxxxxx:profile/ccccccc --role-arn arn:aws:iam::xxxxxx:role/RolesAnywhere

Note: Change the --trust-anchor-arn, --profile-arn and --role-arn to values gotten from Terraform code.

Small application

Here is a small Python code to print S3 buckets. Save this to a file gets3buckets.py

import boto3
import time

def hello_s3():
    s3_resource = boto3.resource("s3")
    print("Hello, Amazon S3! Let's list your buckets:")
    for bucket in s3_resource.buckets.all():
        print(f"\t{bucket.name}")


if __name__ == "__main__":
    while True:
        hello_s3()
        time.sleep(5)

Copy the certificate, private key, aws-config and the above python code into a directory.

Dockerfile

Create a Dockerfile in the same directory as the certificates and the other files previously created:

FROM debian:stable-slim

ARG homedir=/app

RUN DEBIAN_FRONTEND=noninteractive apt-get update \
    && apt-get upgrade \
    && apt-get install --no-install-recommends -y \
        awscli \
        curl \
        python3-boto3 \
    && rm -rf /var/lib/apt/lists/*

# Download AWS Signing Helper
RUN cd /usr/local/bin \
    && curl -LO https://rolesanywhere.amazonaws.com/releases/1.1.1/X86_64/Linux/aws_signing_helper \
    && chmod 0755 aws_signing_helper

# Create user to run the app
RUN adduser --system --home "$homedir" --no-create-home --shell /bin/false userapp
RUN mkdir "$homedir" && chown userapp "$homedir"

# After this everything is run under the user
USER userapp

COPY --chown=userapp --chmod=0600 ./app-cert.pem "$homedir"
# This should never be copied inside the image. It should be mounted from outside
COPY --chown=userapp --chmod=0600 ./app-private-nopass.key "$homedir"


RUN mkdir "$homedir"/.aws \
    && chmod 0700 "$homedir"/.aws

# Copy the aws-config 
COPY --chown=userapp --chmod=0644 ./aws-config "$homedir"/.aws/config

COPY --chown=userapp --chmod=0755 gets3buckets.py "$homedir"

WORKDIR "$homedir"

CMD python3 /app/gets3buckets.py

Note: The private key should never be baked into the container except when testing locally. The private key should be in a secret store like AWS Secrets Manager and then mounted from there when using inside Kubernetes.

Build the container

docker build -t s3rolesanywhere:test .

When running the container, it will use the certificate and the private key with the aws_signing_helper tool to request AWS secrets and session keys, and assume the role. It will then print the buckets from S3 every 5 seconds.

docker run -ti --rm s3rolesanywhere:test

The output should be something like this:

Hello, Amazon S3! Let's list your buckets:
    private-ca-crl-xxxxxxxx
    ...

Revoking certificate

Get certificate serial number

To revoke a certificate, you need to have its Serial number. You can get it from the certificate with openssl command

openssl x509 -in app-cert.pem -noout -serial

Revoke certificate in AWS Private CA

Then, you can revoke the certficate in AWS Private CA with the following command

aws acm-pca revoke-certificate \
 --certificate-authority-arn <ARN of CA> \
 --certificate-serial <Serial of the cert to revoke> \
 --revocation-reason "<Reason for revoking>"

Note: Only these are valid values for --revocation-reason: AFFILIATION_CHANGED, CESSATION_OF_OPERATION, A_A_COMPROMISE, PRIVILEGE_WITHDRAWN, SUPERSEDED, UNSPECIFIED, KEY_COMPROMISE, CERTIFICATE_AUTHORITY_COMPROMISE

It may take some time for the CRL file to appear.

Download AWS Private CA CRL file

You can then download the CRL from the S3 bucket configured for CRL in the AWS Private CA.

List CRL files in the Private CA S3 Bucket (private_ca_s3_bucket is the output from Terraform)

aws s3 ls s3://<private_ca_s3_bucket>/crl

Get the CRL file from the S3 bucket

aws s3 cp s3://<private_ca_s3_bucket>/crl/xxxxxxxxx.crl .

This file is in DER format, and it needs to be in PEM format for AWS Roles Anywhere to accept.

openssl crl -inform DER -in xxxxxxxx.crl -outform PEM -out privateca.crl

Import CRL to AWS Roles Anywhere

A CRL file in PEM format needs to be then uploaded/imported to the AWS Roles Anywhere for it to revoke the access to the certificate

aws rolesanywhere import-crl \
 --crl-data privateca.crl \
 --name <give some name> \
 --trust-anchor-arn <ARN of the Turst Anchor> --enabled

Note: You can only import 2 CRLs. Even with same names it doesn't overwrite them, so you need to manually remove them. Check aws rolesanywhere list-crls and aws rolesanywhere delete-crl AWS CLI commands.

After some time, the access with that certificate should not work anymore. The error will be something like

botocore.exceptions.CredentialRetrievalError: Error when 
retrieving credentials from custom-process: 
2024/02/16 11:11:34 AccessDeniedException: Certificate revoked

Conclusion

It is a pretty neat way of giving AWS access to applications and servers outside of AWS. It is more secure than creating normal, long-term user credentials and using them for applications. Compared to normal user credentials, if the session is hijacked in transit, then it is only valid for a short while instead of potentially being live for even months or years. And if the private key is compromised, then it can only access the services configured with policies to the role.

But revoking certificates is not all that straightforward. The CA keeps up a list called Certificate Revocation List (CRL). This needs to be imported to AWS Roles Anywhere seperately. AWS Roles Anywhere doesn’t have any automated update system of the CRL, even with AWS Private CA that can push the CRL to S3. It can only be imported through AWS CLI or API, but for example Terraform does not yet have that capability.

I feel that unless a company is already heavily invested in certificates and PKI systems, the AWS Roles Anywhere brings complications not needed on top of all other user management, especially as Terraform doesn't support import-crl now, and playing with AWS Private CA or OpenSSL requires more manual work than I think is necessary.

Crossplane: How do providers work

Joonas Venäläinen — Tue, 17 Oct 2023 07:16:35 +0000

Providers are the meat around Crossplane’s bones, and they are used to extend the capabilities of Crossplane. When Crossplane is installed, it doesn't have any capabilities to interact with external systems. A core Crossplane pod will only watch the following resources.

compositeresourcedefinitions.apiextensions.crossplane.io
compositionrevisions.apiextensions.crossplane.io         
compositions.apiextensions.crossplane.io                 
configurationrevisions.pkg.crossplane.io                 
configurations.pkg.crossplane.io                         
controllerconfigs.pkg.crossplane.io                      
locks.pkg.crossplane.io                                  
providerrevisions.pkg.crossplane.io                      
providers.pkg.crossplane.io                              
storeconfigs.secrets.crossplane.io

When you install a provider, a new pod is created to Crossplane's installation namespace. This pod is a Kubernetes Controller that watches the CRDs that are also installed as part of the provider package.

To find out what different kinds of providers are available, you can check the Upbound Marketplace and crossplane-contrib repository. For this series, we are going to work with the following providers:

Those GCP providers are installed from the provider-family-gcp package. These provider-family packages are special packages that allow you to install only the provider packages you need instead of installing everything, which would mean 343 CRDs if you install the provider-gcp package instead. Crossplane also states:

On average, 30 CRDs are used from Provider packages.

Looking at the average number, you would still have ~313 CRDs in the cluster that aren't used 🤯.

Install the providers

cat <<EOF | kubectl apply --filename=-
apiVersion: pkg.crossplane.io/v1
kind: Provider
metadata:
  name: provider-gcp-storage
spec:
  package: xpkg.upbound.io/upbound/provider-gcp-storage:v0.36.0
---
apiVersion: pkg.crossplane.io/v1
kind: Provider
metadata:
  name: provider-gcp-cloudplatform
spec:
  package: xpkg.upbound.io/upbound/provider-gcp-cloudplatform:v0.36.0
---
apiVersion: pkg.crossplane.io/v1
kind: Provider
metadata:
  name: provider-terraform
spec:
  package: xpkg.upbound.io/upbound/provider-terraform:v0.10.0
EOF

After a little while, you should see the providers installed and in a healthy state

kubectl get provider
---
NAME                          INSTALLED   HEALTHY   PACKAGE                                                      AGE
provider-gcp-cloudplatform    True        True      xpkg.upbound.io/upbound/provider-gcp-cloudplatform:v0.36.0   116s
provider-gcp-storage          True        True      xpkg.upbound.io/upbound/provider-gcp-storage:v0.36.0         116s
provider-terraform            True        True      xpkg.upbound.io/upbound/provider-terraform:v0.10.0           116s
upbound-provider-family-gcp   True        True      xpkg.upbound.io/upbound/provider-family-gcp:v0.37.0          107s

Now the providers are installed and ready, we need to set up the ProviderConfig, which configures the credentials for the provider to be able to interact with external systems, in this case, with Google Cloud. You can have multiple ProviderConfigs and reference them in managed resources using providerConfigRef. ProviderConfigs are cluster-scoped resources.

You can set up a ProviderConfig per tenant when you have a multi-tenant cluster. When creating compositions, you could patch the value of providerConfigRef in managed resources with a value of spec.claimRef.namespace, which points to the namespace where the XRC was created.

Every provider has their own individual settings available when it comes to ProviderConfig. For the GCP provider, you can find all the available configuration options here and for Terraform provider here. If you need to override Controller related settings eg. ServiceAccount you can use ControllerConfig for that.

In upcoming chapters, we will create resources in Google Cloud that involve creating a bucket, serviceaccount, iam-binding, and serviceaccountkey. Use the following to configure a new service account with needed permissions in GCP:

# GCP Project ID
PROJECT_ID=""

gcloud iam service-accounts create crossplane-sa-demo --display-name "Crossplane Service Account Demo"

gcloud projects add-iam-policy-binding $PROJECT_ID --member serviceAccount:crossplane-sa-demo@$PROJECT_ID.iam.gserviceaccount.com --role roles/storage.admin
gcloud projects add-iam-policy-binding $PROJECT_ID --member serviceAccount:crossplane-sa-demo@$PROJECT_ID.iam.gserviceaccount.com --role roles/iam.serviceAccountAdmin
gcloud projects add-iam-policy-binding $PROJECT_ID --member serviceAccount:crossplane-sa-demo@$PROJECT_ID.iam.gserviceaccount.com --role roles/iam.serviceAccountKeyAdmin
gcloud projects add-iam-policy-binding $PROJECT_ID --member serviceAccount:crossplane-sa-demo@$PROJECT_ID.iam.gserviceaccount.com --role roles/storage.iamMember

Create a service account key:

gcloud iam service-accounts keys create credentials.json --iam-account=crossplane-sa-demo@$PROJECT_ID.iam.gserviceaccount.com

Create a Kubernetes secret in crossplane-system namespace that contains the previously created credentials:

kubectl create secret generic gcp-creds --from-file=creds=./credentials.json -n crossplane-system

Create ProviderConfig that uses these credentials:

cat <<EOF | kubectl apply --filename=-
apiVersion: gcp.upbound.io/v1beta1
kind: ProviderConfig
metadata:
  name: default
spec:
  projectID: $PROJECT_ID
  credentials:
    source: Secret
    secretRef:
    name: gcp-creds
    namespace: crossplane-system
    key: creds
EOF

If you run this inside GKE, using the Workload Identity for authentication is much better. You can find detailed instructions for it here.

You can also read the secret from the filesystem using fs. This might come in handy in cases where you are leveraging, for example, Hashicorp Vault with Vault Agent sidecar to inject secrets to pods. Here is a quick example of how you would configure it without going into too much detail about how to work with Vault Agent Injector:

apiVersion: pkg.crossplane.io/v1alpha1
kind: ControllerConfig
metadata:
  name: gcp-config
spec:
  metadata:
    annotations:
    vault.hashicorp.com/agent-inject: "true"
    vault.hashicorp.com/role: "crossplane-providers"
    ...
---
apiVersion: pkg.crossplane.io/v1
kind: Provider
metadata:
  name: provider-gcp-storage
spec:
  package: xpkg.upbound.io/upbound/provider-gcp-storage:v0.36.0
  controllerConfigRef:
    name: gcp-config
---
apiVersion: gcp.upbound.io/v1beta1
kind: ProviderConfig
metadata:
  name: default
spec:
  projectID: $PROJECT_ID
  credentials:
    source: Filesystem
    fs:
      path: /vault/secrets/gcp-creds

Now we can quickly test that everything is working by creating a Bucket resource:

cat <<EOF | kubectl apply --filename=-
apiVersion: storage.gcp.upbound.io/v1beta1
kind: Bucket
metadata:
  name: ps-bucket-${RANDOM}
spec:
  forProvider:
    location: US
EOF

After a little while, you should see the bucket resource ready and synced:

kubectl get bucket
---
NAME                            READY   SYNCED   EXTERNAL-NAME                  AGE
bucket-crossplane-demo-30855    True    True     bucket-crossplane-demo-30855   15m

At this point, we are ready to start working with GCP using Crossplane. I will go through setting up the Terraform provider configs later in the series when it's time to start working with it.

Remember to delete the test bucket resource:

kubectl delete bucket <bucket_name>

The next chapter quickly reviews available configuration options for managed resources.

Crossplane: Streamline your infrastructure provisioning & management

Joonas Venäläinen — Tue, 17 Oct 2023 07:16:15 +0000

Crossplane is an extension to Kubernetes, which transforms your Kubernetes cluster into a universal control plane. It allows you to manage anything that has an API available with the help of provider packages. It's also fully extendable, so you can build your own providers to support your APIs. Everybody likes 🍕 so here is a post providers-101-ordering-pizza-with-kubernetes-and-crossplane which goes through on a high level how to build a provider that is capable of ordering pizza for you inside Kubernetes cluster.

Crossplane is often used to interact with cloud providers like Azure, AWS, and GCP. By using the Crossplane, you can bring your infrastructure management to Kubernetes. Another significant benefit of Crossplane is that it acts as a Kubernetes Controller, constantly monitoring the state of external resources. So, if someone would modify/delete the resources outside of Kubernetes, Crossplane would reconcile the resources.

By connecting Kubernetes and the cloud provider APIs with Kubernetes Custom Resource Definitions (CRDs), Crossplane can be used to enable a Kubernetes-native approach to managing cloud resources. We can for example define a Database custom resource which will then provision a database in the cloud provider that we have hooked Crossplane into. This makes it easier for developers to start consuming infrastructure resources by hiding the complexity behind Crossplane compositions. The Ops team can create and maintain these compositions and the XRDs which define how the resource will look like for the consumers. Without going into too much detail, we could have a custom resource Database that the developers can consume.

apiVersion: storage.polarsquad.com/v1alpha1
kind: Database
metadata:
  name: ps-demo-db-mysql
spec:
  parameters:
    name: ps-demo-db-mysql
    size: "small"

Here, we define the size as small. The Ops team could set small, medium, and large options for the size. Then, in compositions, patch these values to specific instance types depending on the cloud platform. This also enhances the experience for the developers as they don't have to know the specific instance types.

As the resources are native Kubernetes manifests, you can bundle them with the other manifests you use to deploy the application to the cluster. When the resources are created, Crossplane will create the connection secrets to the application namespace for pods to consume.

Overall, the Crossplane allows you to form your own Cloud Platform inside Kubernetes. With the help of compositions, you can build a self-service platform where developers can easily create resources on-demand when they need them without having to jump through hoops to get additional backing services.

Throughout the series, I will be using the terms XRD,XR, and XRS. Here is a quick overview of what they stand for.

XRD - Composite Resource Definition
XR - Composite Resource
XRC - Claim

Prerequisites:

Access to Google Cloud and gcloud cli
Kubernetes cluster eg. minikube
Helm
Access to Aiven

Aiven offers a free tier so you can create an account and use it to get through the tutorial.

Install Crossplane

helm repo add crossplane-stable https://charts.crossplane.io/stable

helm install crossplane --namespace crossplane-system --create-namespace crossplane-stable/crossplane

At this point, the Crossplane is ready, and the next step is to install the needed Providers that give the Crossplane capabilities to provision managed resources to external systems.

In this series, we will provision resources to Google Cloud using the Google Cloud providers and, later in the series, leverage Terraform provider to manage resources that don't have a Crossplane native provider available yet.

Links to each part of this series:

Crossplane: How do providers work

Prometheus Observability Platform: Grafana

Aleksi Waldén — Thu, 14 Sep 2023 10:28:01 +0000

Grafana is the industry standard open-source product for visualising metrics stored in a TSDB format, or a variety of other data sources. With Grafana, we can create dashboards, queries, and alerts from the data that we have. With all our metrics in long-term storage, we can use a single data source to access all the metrics from all our infrastructure that uses the metrics platform. This enables easily creating dashboards that aggregate data from multiple different Kubernetes clusters, and enable drilling down to a single resource easily.

Demo

Next, we will set up a Grafana instance into our minikube and use Promxy as the default data source. This example assumes that you have completed the following steps, as the components from those are needed:

Prerequisites:

base64

First we start with adding the Grafana Helm chart repository, and installing its contents into the Grafana namespace:

helm repo add grafana https://grafana.github.io/helm-charts

Next, we define Promxy as the data source. In the Helm values file, we need the following block to do this:

datasources.yaml:
  apiVersion: 1
  datasources:
  - name: Promxy
    type: prometheus
    url: "http://promxy.promxy.svc.cluster.local:8082"
    isDefault: true

We are using the svc.cluster.local address for the Promxy service, because all our services are inside the cluster.

I have converted the above into json so that it can be passed to Helm:

helm install grafana grafana/grafana --create-namespace --namespace grafana --set-json 'datasources={"datasources.yaml":{"apiVersion":1,"datasources":[{"name":"Promxy","type":"prometheus","url":"http://promxy.promxy.svc.cluster.local:8082","isDefault":true}]}}'

Next we need to get the admin password for the admin user:

kubectl get secret --namespace grafana grafana -o jsonpath="{.data.admin-password}" | base64 --decode ; echo

Now we can port-forward the Grafana service:

kubectl port-forward -n grafana services/grafana 9090:80

Navigate to http://localhost:9090 to access the web UI, and log in with the username admin, and the password acquired in the previous step. From here you can verify that Promxy is set up and acting as the default data source by navigating to Administration -> Data sources -> Promxy, and clicking the Test button at the bottom of the page.

Assuming the test was successful, we can then navigate to the “Explore” item in the menu

and check that we have metrics available in the “metrics explorer” section. Alternatively, we can use the following query to check that metrics are available:

sum(kube_pod_container_status_restarts_total) by (namespace, container)

N.B. you might have to change the time range for the query to get results:

We have now achieved setting up Grafana as the metrics visualisation tool for our metrics platform. This enables us to create dashboards and Grafana alerts for metrics from all sources sending metrics to our long-term storage cluster (or clusters if we have multiple regions) that are queried using Promxy.

Prometheus Observability Platform: Application metrics

Aleksi Waldén — Thu, 14 Sep 2023 10:27:31 +0000

When creating our own applications, we need to use a metrics library to generate the metrics and then inside our application functions increment said metrics. With Go for example, we can use the Prometheus library. The metrics will then be exposed to the /metrics endpoint. If our application is inside a Kubernetes cluster with a prometheus-operator, we can use a ServiceMonitor to scrape its metrics. If we don’t have such a possibility, we can instead set up an application to send metrics straight to our long-term storage solution. For VictoriaMetrics, we can use the github.com/VictoriaMetrics/metrics library to send the metrics to VictoriaMetrics. Remember to add authentication logic into the section pushing the metrics to the long-term storage, if necessary.

Demo

This example assumes that you have completed the following steps as the components from those are needed:

Prometheus Observability Platform: Prometheus

Let's set up a hello-world Golang application in our cluster, and use ServiceMonitor to send its metrics to Prometheus.

First, we need to update our kube-prometheus-stack Helm deployment to pick up ServiceMonitor resources with a certain label attached. We need to pass the following value to our Helm chart:

prometheus:
  prometheusSpec:
    serviceMonitorSelector:
      matchExpressions:
      - key: app
        operator: Exists

I have converted that into json so that it can be passed to Helm:

helm upgrade kube-prometheus-stack prometheus-community/kube-prometheus-stack --namespace prometheus --reuse-values --set-json 'prometheus.prometheusSpec.serviceMonitorSelector={"matchExpressions":[{"key":"app","operator":"Exists"}]}'

This will update our kube-prometheus-stack to pick up ServiceMonitor resources from any namespace, as long as they have an app label attached.

Next, we are going to create a namespace for our hello-world application which is a simple Golang application exposing metrics via the Prometheus module. We will borrow this already-made application, which has logic defined to increment a metric called hello_processed_total each time the page is loaded.

To create a namespace and a pod, we use the following commands:

kubectl create namespace hello-world

kubectl run hello-world --namespace=hello-world --image='okteto/hello-world:golang-metrics' --labels app=hello-world

Next, we need to create a service for the new pod:

cat <<'EOF' | kubectl create -f -
apiVersion: v1
kind: Service
metadata:
  labels:
    app: hello-world
  name: hello-world
  namespace: hello-world
spec:
  ports:
  - name: http
    port: 8080
  selector:
    app: hello-world
  type: ClusterIP
EOF

Now we can test that our application is working by port-forwarding it. We can also check what the hello_processed_total metric looks like:

kubectl port-forward -n hello-world services/hello-world 9090:8080

Now navigate to http://localhost:9090 and http://localhost:9090/metrics. You should see a metric called hello_processed_total with a number attached. Each reload of the page will increment this number.

Next, we need to set up a ServiceMonitor to send these metrics to Prometheus:

cat <<'EOF' | kubectl create -f -
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: hello-world
  namespace: hello-world
  labels:
    app: hello-world
spec:
  selector:
    matchLabels:
      app: hello-world
  endpoints:
    - port: http
EOF

This ServiceMonitor will target services matching the label selector (app=hello-world) and will scrape the port called “http”.

Now, if we port-forward our Prometheus service, we should see a new service in the service discovery section:

kubectl port-forward -n prometheus services/kube-prometheus-stack-prometheus 9090:9090

Navigate to http://localhost:9090/service-discovery and you should see that there is a new service discovered with the name serviceMonitor/hello-world/hello-world/0 and it should show 1/1 active targets.

We can now query the hello_processed_total metric:

We have now achieved sending metrics from our custom app running in its own namespace into Prometheus.

Next part: Prometheus Observability Platform: Grafana

Prometheus Observability Platform: Handling multiple regions

Aleksi Waldén — Thu, 14 Sep 2023 10:27:13 +0000

When we have multiple regions such as the EU and US, we need to have a long-term storage solution running in both of those. If we want to combine the resources into a single query we need to use a query layer that can query both endpoints. One such component we can use is Promxy.

Promxy uses the same PromQL syntax as Prometheus and we can define server groups with multiple endpoints. In our case, we would define our EU and US long-term storage endpoints under one server group. We can then use the single Promxy endpoint to query both the EU and the US.

Demo

This example assumes that you have completed the following steps, as the components from those are needed:

We can use the Helm chart offered in the Promxy repository to deploy a proxy to our Kubernetes cluster.

First we clone the repository, because the Helm chart is not published to a public registry:

git clone https://github.com/jacksontj/promxy.git

Then we navigate to the folder containing the Helm chart:

cd promxy/deploy/k8s/helm-charts/promxy/

We set up Promxy with the following server_groups:

server_groups:
  - static_configs:
    - targets:
      - vmcluster-victoria-metrics-cluster-vmselect.victoriametrics.svc.cluster.local:8481
      labels:
        region: eu
    scheme: http
    path_prefix: /select/0/prometheus

I have converted this to json so we can pass it to the helm chart:

{"server_groups":[{"static_configs":[{"targets":["vmcluster-victoria-metrics-cluster-vmselect.victoriametrics.svc.cluster.local:8481"],"labels":{"region":"eu"}}],"scheme":"http","path_prefix":"/select/0/prometheus"}]}

To install Promxy from the local Helm chart we use the following:

helm install promxy . --create-namespace --namespace promxy --set 'image.tag=latest' --set-json 'config.promxy={"server_groups":[{"static_configs":[{"targets":["vmcluster-victoria-metrics-cluster-vmselect.victoriametrics.svc.cluster.local:8481"],"labels":{"region":"eu"}}],"scheme":"http","path_prefix":"/select/0/prometheus"}]}'

We can now port-forward the Promxy service and access the web UI from http://localhost:9090:

kubectl port-forward -n promxy services/promxy 9090:8082

Here we can perform the same query for the kube_pod_container_status_restarts_total to verify that Promxy is able to reach the VictoriaMetrics data:

If we have more regions than just the US, we can add them under the server_groups and query multiple VictoriaMetrics instances from a single Promxy source.

We have now achieved setting up Promxy with server_groups to use for querying VictoriaMetrics instances.

Next part: Prometheus Observability Platform: Application metrics

Prometheus Observability Platform: Alert routing

Aleksi Waldén — Thu, 14 Sep 2023 10:26:37 +0000

Alertmanager is a component usually bundled with Prometheus to handle routing the alerts to receivers such as Slack, e-mail, and PagerDuty. It uses a routing tree to send alerts to one or multiple receivers.

Routes define which receivers each alert should be sent to. You can define rules for the routes. The rules are evaluated from top to bottom, and alerts are sent to matching receivers. Usually, the match block is used to match the label name and value for a certain receiver. Notification integrations are configured for each receiver. There are multiple different options available, such as email_configs, slack_configs, and webhook_configs.

Alertmanager has a web UI that can be used to view current alerts and silence them if needed.

With a platform setup, we usually don’t want to use multiple Alertmanagers, so we disable the provisioning of additional alertmanagers for Prometheus deployments that include them automatically. Instead, we use one centralised Alertmanager inside, for example, a Kubernetes cluster which is aimed at monitoring platform usage.

Demo

This example assumes that you have completed the following steps, as the components from those are needed:

Prerequisites:

Amtool (https://github.com/prometheus/alertmanager#install-1)

Now that we have an alert defined and deployed to vmalert we can add Alertmanager to our platform. Because we are creating this with a platform aspect in mind, we will install Alertmanager as a separate resource, and not as a part of the kube-platform-stack. We will use a tool called amtool which is bundled with alertmanager to run unit tests on our alert rules

We can install the Alertmanager with the following Helm chart:

helm install alertmanager prometheus-community/alertmanager --create-namespace --namespace alertmanager

We can now port-forward the alertmanager service to access the alertmanager web UI from http://localhost:9090:

kubectl port-forward -n alertmanager services/alertmanager 9090:9093

To trigger a test alert, we can use the following command from another terminal tab while keeping the port-forwarding on:

curl -H "Content-Type: application/json" -d '[{"labels":{"alertname":"TestAlert"}}]' localhost:9090/api/v1/alerts

We can now use amtool to list the currently firing alerts:

amtool alert query --alertmanager.url=http://localhost:9090
---
Alertname   Starts At                Summary  State   
TestAlert   2023-07-07 07:23:55 UTC           active

Let's add a test receiver and routing for it. Below is an example of the configuration we want to pass to Alertmanager in Helm values format.

config:
  receivers:
    - name: default-receiver
    - name: test-team-receiver

  route:
    receiver: 'default-receiver'
    group_wait: 30s
    group_interval: 5m
    repeat_interval: 4h
    routes:
      - receiver: 'test-team-receiver'
        matchers:
        - team="test-team"

I have converted the above into a json one-liner so we can pass it into Helm without having to create an intermediate file.

helm upgrade alertmanager prometheus-community/alertmanager --namespace alertmanager --set-json 'config.receivers=[{"name":"default-receiver"},{"name":"test-team-receiver"}]' --set-json 'config.route={"receiver":"default-receiver","group_wait":"30s","group_interval":"5m","repeat_interval":"4h","routes":[{"receiver":"test-team-receiver","matchers":["team=\"test-team\""]}]}'

We can now use amtool to test that an alert that has the label team=test-team gets routed to the test-team-receiver:

amtool config routes test --alertmanager.url=http://localhost:9090 team=test-team
---
test-team-receiver

amtool config routes test --alertmanager.url=http://localhost:9090 team=test     
---
default-receiver

We have now set up an Alertmanager which can route alerts depending on team label value.

Next, we need to update vmalert to route alerts into the Alertmanager using the cluster local address of the alertmanager service:

helm upgrade vmalert vm/victoria-metrics-alert --namespace victoriametrics --reuse-values --set server.notifier.alertmanager.url="http://alertmanager.alertmanager.svc.cluster.local:9093"

Now we can run a pod that will be crashing to increment the kube_pod_container_status_restarts_total metric by creating a pod that has a typo in the sleep command:

kubectl run crashpod --image busybox:latest --command -- slep 1d

Next we port-forward the alertmanager service. We should see an alert in there when we navigate to http://localhost:9090:

kubectl port-forward -n alertmanager services/alertmanager 9090:9093

We have now achieved setting up Alertmanager as our tool for routing alerts from the vmalert component.

Next part: Prometheus Observability Platform: Handling multiple regions

Prometheus Observability Platform: Alerts

Aleksi Waldén — Thu, 14 Sep 2023 10:25:36 +0000

With Prometheus, we can use PromQL to write alert rules and evaluate them using the given evaluation rules and intervals. Alerts have an evaluation period: if an alert is active for the duration of the evaluation period, then it will fire. Prometheus is usually bundled with a component called Alertmanager, which is used to route alerts to different receivers such as Slack and email. Once an alert fires, it is sent to the Alertmanager which uses a routing table to find out if the alert is to be sent to a receiver, and how to route it.

Prometheus alerts are evaluated against the local storage. With VictoriaMetrics, we can use the vmalert component to evaluate alert rules against the VictoriaMetrics long-term storage using the same PromQL syntax as with Prometheus. It is tempting to write all the alerting rules in VictoriaMetrics, but depending on the size of the infrastructure we might want to evaluate some rules on the Prometheus servers where the data originates from, to avoid overloading VictoriaMetrics.

Alert rules can be very complex, and it is best to validate them before deploying them to Prometheus. Promtool can be used to validate Prometheus alerting rules and run unit tests on them. You can implement these simple validation and unit testing steps in your continuous integration (CI) system.

A good monitoring platform enables teams to write their own alerts against the metrics stored in the long-term storage. We can do this in a mono-repository or multi-repository fashion. With a mono-repository, we have all the infrastructure and the alerting defined in the same repository and pipelines delivering them to servers. A multi-repository approach would set up a separate repository for the alerts, where we define the alerting rules using PromQL, and add validation and unit tests.

The main benefit of the multi-repository approach is reduced cognitive load. The contributors do not see or need to be aware of anything else than the alert rules. This also eliminates the possibility of introducing bugs into the underlying infrastructure. The downside of this approach is tying the separated alerting configuration back to the Prometheus server.

Terraform can be used to set up the repository used for alerting as a remote module and thus pull the alerting rules into the server when deploying the server. With a mono-repository, we can more easily tie the alerts into the Prometheus server, but if we are using Terraform then we need to either split the state of the alerts or accept that the contributors might affect more resources than just the alerts which might also cause more anxiety to the contributors.

Demo

This example assumes that you have completed the following steps as the components from those are needed:

Prerequisites:

Promtool (https://github.com/prometheus/prometheus/tree/main#building-from-source)
yq (optional)
jq (optional)

Figuring out suitable metrics for alerts can be hard. The awesome-prometheus-alerts website is an excellent source for inspiration for this. It has a collection of pre-made alerts using the PromQL syntax. For example, we can set up an alert for crash-looping Kubernetes pods, with the alert named KubernetesPodCrashLooping.

Below there is an example unit test for the KubernetesPodCrashLooping alert. First, we want to simplify the alert a little and add some required blocks for promtool to be able to validate the rule. This file is saved as kube-alerts.rules.yml:

groups:
  - name: kube-alerts
    rules:
    - alert: KubernetesPodCrashLooping
      expr: increase(kube_pod_container_status_restarts_total[5m]) > 2
      for: 10m
      labels:
        severity: warning
      annotations:
        summary: Pod {{$labels.namespace}}/{{$labels.pod}} is crash looping

We can use the command promtool check rules kube-alert.rules.yml to validate the rule. If everything is OK, the response looks like this:

promtool check rules kube-alert.rules.yml
---
Checking kube-alert.rules.yml
  SUCCESS: 1 rules found

To write a unit test for this alert, we create a file called kube-alert.test.yml:

rule_files:
  - kube-alert.rules.yml

evaluation_interval: 1m

tests:
  - interval: 1m

    input_series:
      - series: kube_pod_container_status_restarts_total{namespace="test-namespace",pod="test-pod"}
        values: '1+2x15'

    alert_rule_test:
      - alertname: KubernetesPodCrashLooping
        eval_time: 15m
        exp_alerts:
          - exp_labels:
              severity: warning
              namespace: test-namespace
              pod: test-pod
            exp_annotations:
              summary: Pod test-namespace/test-pod is crash looping

So we are expecting an increase of over 2 in the kube_pod_container_status_restarts_total time series within 5 minutes, and that this increase is active for at least 10 minutes. In the summary field, we expect to receive namespace and pod labels and to a severity label with the value “warning”.

To write a test for this rule we need to create an input series that can trigger this rule and pass all the labels needed for the summary field. Because our evaluation time is 15 minutes we need at least 15 entries into our series. The syntax ‘1+2x15’ adds 2 to the previous number 15 times to create a time series. We also pass the required namespace and pod labels and write the expected summary field response.

To run the unit test we use the command promtool test rules kube-alert.test.yml which will return the following response if all went well:

promtool test rules kube-alert.test.yml
---
Unit Testing:  kube-alert.test.yml
  SUCCESS

Next we need to deploy vmalert so that we can evaluate alert rules against the data in the long-term storage.

First we have to convert our alert rule into a format that works with Helm. The problem with promtool and Helm charts is that the groups: section is required in both of them, so we need to remove it from the alert we have created, but if we remove it then promtool no longer works. There are multiple ways to handle this, for example the Terraform trimprefix() function which can be used to remove the groups: section from the alert rules. For this use case we are going to use a monstrous one-liner to remove the groups: section, convert the output into json and then output it into a single line so we can pass it to Helm:

cat kube-alert.rules.yml | sed '/groups:/d' | yq -o=json | jq -c

This will get us the following json one line string:

[{"name":"kube-alerts","rules":[{"alert":"KubernetesPodCrashLooping","expr":"increase(kube_pod_container_status_restarts_total[5m]) > 2","for":"10m","labels":{"severity":"warning"},"annotations":{"summary":"Pod {{$labels.namespace}}/{{$labels.pod}} is crash looping"}}]}]

Now we can deploy the vmalert Helm chart:

helm install vmalert vm/victoria-metrics-alert --namespace victoriametrics --set 'server.notifier.alertmanager.url=http://localhost:9093' --set 'server.datasource.url=http://vmcluster-victoria-metrics-cluster-vmselect:8481/select/0/prometheus' --set 'server.remote.write.url=http://vmcluster-victoria-metrics-cluster-vminsert:8480/insert/0/prometheus' --set 'server.remote.read.url=http://vmcluster-victoria-metrics-cluster-vmselect:8481/select/0/prometheus' --set-json 'server.config.alerts.groups=[{"name":"kube-alerts","rules":[{"alert":"KubernetesPodCrashLooping","expr":"increase(kube_pod_container_status_restarts_total[5m]) > 2","for":"10m","labels":{"severity":"warning"},"annotations":{"summary":"Pod {{$labels.namespace}}/{{$labels.pod}} is crash looping"}}]}]'

server.notifier.alertmanager: We are using a placeholder value here for now, as we cannot install the chart without providing some value here
server.datasource.url: Prometheus HTTP API compatible datasource
server.remote.write.url: Remote write url for storing rules and alert states
server.remote.read.url: URL to restore the alert states from

We can now port-forward the vmalert service and navigate to the web UI in http://localhost:9090

We have now achieved creating an alert rule, writing a unit test for it, and setting up vmalert with the alert rule defined.

Next part: Prometheus Observability Platform: Alert routing

Prometheus Observability Platform: Long-term storage

Aleksi Waldén — Thu, 14 Sep 2023 10:25:05 +0000

As Prometheus is not so well designed for persisting data, a long-term storage solution is called for. Multiple different products can handle long-term storage for Prometheus metrics for example VictoriaMetrics, Grafana Mimir, Thanos, and M3.

With some of these options, we get the capability to store the data into object storage which is ideal for modern workloads running in Kubernetes as we don’t want to store any persistent data inside our cluster. Object storage can be for example Azure Blob storage or AWS S3. This option however has downsides on performance (compared to block storage), so if you have high performance requirements, you might have to look into block storage options.

In this document, we will be focusing on VictoriaMetrics. It was chosen because it is open-source, highly performant, and all its crucial components are free. VictoriaMetrics can only handle block storage, but it is also very fast due to its simple architecture designed only for local storage. It can be run in single mode or clustered. The central part of the architecture consists of:

The vmstorage component, which stores the time series data;
vmselect, used to fetch and merge data from vmstorage; and
vminsert, which inserts the data into vmstorage nodes.

In the clustered version, data is distributed evenly across the vmstorage nodes by the vminsert component, and the distributed data is then fetched and merged by the vmselect component. In Kubernetes, each of these components will have its own pod and the vmselect and vminsert components will have a service to load balance the traffic. All the vmstorage endpoints (pods) will be connected to the vminsert and vmselect pods.

VictoriaMetrics also has multiple additional features, such as:

the vmalert component which can be used for alerts about the data, and
vmagent which can be used as a data ingestion point, and for filtering and re-labelling metrics.
the vmauth component for simple authentication, which uses credentials from the Authorization header. (You can also use some other component in front of vminsert or vmagent, such as oauth2-proxy to handle authentication.)

We can set up Prometheus to write the data it receives into the long-term storage using the remote_write block in the configuration. If authentication is set up, it also needs to be defined in the remote_write block.

Demo

We will now set up the following architecture with minikube, Prometheus, and VictoriaMetrics. This example assumes that you have completed the steps from Prometheus Observability Platform: Prometheus

First we add the VictoriaMetrics helm chart and install it into the VictoriaMetrics namespace:

helm repo add vm https://victoriametrics.github.io/helm-charts/

helm install vmcluster vm/victoria-metrics-cluster --create-namespace --namespace victoriametrics

We should now see six pods running in the victoriametrics namespace:

kubectl get pods -n victoriametrics
---
NAME                                                           READY   STATUS    RESTARTS   AGE
vmcluster-victoria-metrics-cluster-vminsert-f8d48695c-gqx25    1/1     Running   0          58s
vmcluster-victoria-metrics-cluster-vminsert-f8d48695c-t8kcn    1/1     Running   0          58s
vmcluster-victoria-metrics-cluster-vmselect-77465fb479-42wjs   1/1     Running   0          58s
vmcluster-victoria-metrics-cluster-vmselect-77465fb479-t2jhp   1/1     Running   0          58s
vmcluster-victoria-metrics-cluster-vmstorage-0                 1/1     Running   0          58s
vmcluster-victoria-metrics-cluster-vmstorage-1                 1/1     Running   0          58s

To access the vmselect web UI we need to port forward the vmselect service:

kubectl port-forward -n victoriametrics services/vmcluster-victoria-metrics-cluster-vmselect 9090:8481

You can then navigate to http://localhost:9090/select/0/prometheus/vmui to access the vmselect VMUI. The URL is a clustered url with the 0 representing the accountid of the cluster.

To set up remote writing from Prometheus into the VictoriaMetrics we need to update our kube-prometheus-stack deployment with the following:

helm upgrade kube-prometheus-stack prometheus-community/kube-prometheus-stack --namespace prometheus --reuse-values --set 'prometheus.prometheusSpec.remoteWrite[0].url=http://vmcluster-victoria-metrics-cluster-vminsert.victoriametrics.svc.cluster.local:8480/insert/0/prometheus/'

Let’s break down the URL provided above. First, we have the service name for vminsert, which you can find with the following command:

kubectl get svc -n victoriametrics 
---
NAME                                           TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)                      AGE
vmcluster-victoria-metrics-cluster-vminsert    ClusterIP   10.100.63.152   <none>        8480/TCP                     3h32m
vmcluster-victoria-metrics-cluster-vmselect    ClusterIP   10.99.12.151    <none>        8481/TCP                     3h32m
vmcluster-victoria-metrics-cluster-vmstorage   ClusterIP   None            <none>        8482/TCP,8401/TCP,8400/TCP   3h32m

Next we have the namespace victoriametrics. Since Prometheus and VictoriaMetrics are in different namespaces, we have svc.cluster.local followed by the port number of the vminsert service and finally, we have the Prometheus enabled write endpoint with the accountid 0 in it, because we are using the clustered version of VictoriaMetrics.

On the VMUI (http://localhost:9090/select/0/prometheus/vmui) we can now verify that metrics are coming from Prometheus.

kubectl port-forward -n victoriametrics services/vmcluster-victoria-metrics-cluster-vmselect 9090:8481

Navigate to the Query section and insert kube_pod_container_status_restarts_total into the Query field. You should now see approximately the same output as you did from Prometheus.

We have now set up a simple Prometheus and VictoriaMetrics integration

Next part: Prometheus Observability Platform: Alerts