DEV Community: Arseny Zinchenko

AWS: ALB and Cloudflare - Configuring mTLS and AWS Security Rules

Arseny Zinchenko — Thu, 18 Jun 2026 08:23:32 +0000

AWS: ALB and Cloudflare - Configuring mTLS and AWS Security Rules

While preparing the infrastructure for migrating RTFM from the DigitalOcean server to AWS (see AWS: basic infrastructure setup for WordPress) I decided to also try AWS ALB mutual authentication (for some reason I thought this feature launched at the last re:Invent, at the end of 2024, but it’s actually been around since late 2023 - see Mutual authentication for Application Load Balancer reliably verifies certificate-based client identities).

The idea is to allow connections to the ALB only from clients that pass authentication, and in the Cloudflare+AWS ALB case, Cloudflare will sign all its requests with a TLS certificate (see Cloudflare Authenticated Origin Pulls), and AWS ALB will verify them. - This way, access to the Load Balancer URL will only be possible for Cloudflare.

The inspiration for this idea came from the recent story about the TCP/IP: SYN flood attack on the RTFM server and Hacker News hug of death - when TCP connections were being opened directly to port 443 on the DigitalOcean server, bypassing Cloudflare.

In addition to mTLS I later decided to also add IP restrictions - configured ALB access only for Cloudflare addresses defined in AWS Security Rules, because mTLS is about authentication, while Security Rules are about network-level protection.

Although initially I was planning to use pure mTLS - it was interesting to see how it works in ALB, and I was too lazy to build automation for updating AWS Security Rules.

The example here was done on a test domain - so we’ll also add it to Cloudflare along the way, but both solutions are already working for rtfm.co.ua itself.

Why restrict access to the AWS Application Load Balancer in the first place:

Cloudflare has many Security rules that block unwanted traffic - if requests go directly to the ALB, they’ll reach NGINX, and then some rules need to be duplicated there as well (which was my situation)
cost during DDoS: when the blog lived on a DigitalOcean server with just a public IP for the droplet, requests to that IP didn’t affect cost (only extra traffic), but in AWS the number of connections to an ALB affects cost

P.S. And once again you try to write concisely - and end up with a lot of text :-(

AWS ALB cost and LCU

The cost of an AWS Load Balancer includes an hourly charge for the instance itself, the standard Data Transfer Charge, charges for allocated Public IPs, and separately - Load Balancer Capacity Units (LCU).

For Network Load Balancer there’s its own unit - Network Load Balancer Capacity Unit (NLCU), and for Gateway Load Balancer, accordingly, GLCU.

See Elastic Load Balancing pricing.

LCU is not a very intuitive unit, so a few words about it.

1 LCU equals:

New connections : new connections to the ALB - 25 connections per second consume 1 LCU
Active connections : every 3,000 active connections per minute (or 1,500 when using mTLS)
Processed bytes : 1 LCU covers 1 GB of traffic per hour (or 0.4 GB for Lambda targets) - counted as ingress + egress combined
Rule evaluations : checking 1,000 rules per second is one LCU

Rule evaluations refer to Listener rules:

And if you have many rules like “IF path is /api/* THEN forward to target-group-api" - you pay for each evaluation for every new HTTP request.

When determining which of the above parameters to use for LCU billing, AWS takes whichever one hits its threshold first: meaning as soon as you have 26 requests per second, even if Processed bytes were only 10 megabytes - you’ll be charged 1 LCU based on New connections.

So in the event of a DDoS (and RTFM has had a few, though not severe ones) - we can easily rack up unexpected costs.

There’s a calculator - Load Balancer Capacity Unit Reservation Calculator - where you can estimate how much LCU a given load will cost.

AWS Load Balancer mTLS use cases

Actually, if you’re using ALB access restrictions via a Security Group with a list of only allowed IPs - then mTLS isn’t needed at all, because you’re already cutting connections at the AWS network level, before the request even reaches the Load Balancer.

What can genuinely be useful:

service-to-service authentication:
for example, we have an Internal ALB behind which various monitoring services live, like VictoriaLogs
logs may contain sensitive information like tokens or even passwords, so access to this endpoint needs to be restricted
VMAuth with password authentication is good, but mTLS adds another layer of protection (see VictoriaMetrics: VMAuth - proxy, authentication and authorization)
mobile client authentication to an API:
certificates are embedded in the clients (certificate pinning), which they use to access the API
but keep in mind potential issues with certificate rotation - either in case of compromise or simply when the certificate expires
IoT: we don’t have that (yet), but in general - a very useful use case

Configuring ALB mutual TLS with Cloudflare

First let’s look at the mTLS access restriction option, since it’s an interesting and useful feature.

Then we’ll add the AWS Security Rules approach with automated rule updates.

Configuring Cloudflare

This post was written before the RTFM migration, using a test domain, so let’s quickly go through the process of adding and configuring a domain in Cloudflare.

Adding a new domain to Cloudflare

Go to Domains, and click Onboard a domain:

Enter the name, the rest of the settings can be left at defaults:

When importing records from AWS DNS, it missed a few Let’s Encrypt validation records, but that’s fine for now:

Get the list of new Name Servers:

Update them at the domain registrar:

Wait for the update - time depends on the DNS zone administrator:

For the kiev.ua DNS zone this took about half an hour:

Go to SSL/TLS and enable the Authenticated Origin Pulls option.

Once we enable this option, Cloudflare will start attaching its client certificate to every new request to the origin - in our case, AWS ALB.

I really wanted to go into more detail about SSL/TLS handshake and keys/certificates here - but decided it would be out of scope for this post, especially since I once worked through the details in the post What is: SSL/TLS in detail (in Rus).

Although I might write a new post on the topic, especially now that TLS 1.3 is current, while my post covered TLS 1.2, which was relevant at the time.

Or you can read the already-mentioned Introducing mTLS for Application Load Balancer - it covers the basic TLS concepts pretty well.

So Cloudflare just passes the client certificate, and from there it’s up to the origin what to do with it - and in the next step we’ll configure the ALB to verify it.

For now just enable the option - it doesn’t cost extra (except that the LCU cost per Active connections will be 1,500 instead of 3,000), won’t add significant load to the ALB or extra traffic:

Add a CNAME record for the domain with the value == ALB URL:

Verify that everything works now, because even though we enabled Authenticated Origin Pulls on Cloudflare, no verification is being performed on the ALB yet:

And a request directly to the ALB URL still works, just with an SSL error:

Configuring the Application Load Balancer

Now we need to configure client certificate verification from Cloudflare on our ALB.

Getting the Cloudflare Certificate Authority

The ALB will use the public Certificate Authority certificate from Cloudflare for verification - we need to add it to the Load Balancer’s Trust Store and then attach it to the ALB Listener.

Note that using a Trust Store also costs money - “$0.0056 per hour per Trust Store Associated with Application Load Balancer when using Mutual TLS”, which is ~$4 per month.

Download the certificate:

$ curl -o cloudflare-origin-pull-ca.pem https://developers.cloudflare.com/ssl/static/authenticated_origin_pull_ca.pem

The certificate is shared across all of Cloudflare - though there’s an option to upload your own and only pass verification with that one.

But for the case where we just want to block random Chinese bots, the Cloudflare certificate itself is sufficient.

Check the downloaded file:

$ file cloudflare-origin-pull-ca.pem
cloudflare-origin-pull-ca.pem: PEM certificate

Or peek inside it:

$ openssl x509 -in cloudflare-origin-pull-ca.pem -text -noout
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number: 6310029703491235425 (0x5791ba9556c22e61)
        Signature Algorithm: sha512WithRSAEncryption
        Issuer: C=US, O=CloudFlare, Inc., OU=Origin Pull, L=San Francisco, ST=California, CN=origin-pull.cloudflare.net
        Validity
            Not Before: Oct 10 18:45:00 2019 GMT
            Not After : Nov 1 17:00:00 2029 GMT
        Subject: C=US, O=CloudFlare, Inc., OU=Origin Pull, L=San Francisco, ST=California, CN=origin-pull.cloudflare.net
...

The important part here - Not After : Nov 1 17:00:00 2029:

you can enable the “Allow expired client certificates” option in the ALB mTLS settings
or simply add the new certificate to the Trust Store and remove the old one at the end of 2029

Creating a Trust Store

To add the certificate to the ALB, we need to create a Trust Store, but certificates can only be added to a Trust Store from AWS S3, so first upload it to some bucket of yours:

Go to EC2 > Load Balancing > Trust Stores, create a new one:

Specify the path to the certificate file in S3:

Configuring mTLS for the ALB Listener

Edit the HTTPS Listener - since mTLS logically happens there:

Enable the mTLS option, and there are two sub-options:

Passthrough : pass the certificate to the targets in the Target Group - validation is done by the backend
Verify with trust store : perform verification on the ALB itself using the certificate from the Trust Store - our case

Switch to “Verify with trust store”:

Now try a direct connection to the ALB URL:

$ curl -kI https://rtfm-alb-1984146384.eu-west-1.elb.amazonaws.com
curl: (56) Recv failure: Connection reset by peer

And in the browser:

But through Cloudflare everything works:

$ curl -I https://test-tls.setevoy.kiev.ua
HTTP/2 200

And rtfm.co.ua itself is already on this ALB with mTLS enabled.

Protection with AWS Security Group and Cloudflare IP ranges

But mTLS alone isn’t quite enough for full ALB protection:

during a DDoS we’ll still be spending LCU on connections
requests that don’t go through Cloudflare need to be filtered at the backend - in my case NGINX

So the most reliable approach is to restrict traffic at the AWS Security Group level attached to the Load Balancer: then no LCU will be spent at all, because packets won’t even reach the ALB.

There are ready-made Terraform modules, for example - cloudflare-security-group, but it only works with a single Security Group. There’s a more advanced option - cloudflare-sg-updater, which finds Security Groups by tags.

That said, personally I’m not a big fan of AWS Lambda, and at least for now I’m not using Terraform for RTFM, so I did it simply with a shell script using the AWS CLI.

Plenty of recent practice for remembering how to write scripts - see FreeBSD: Home NAS, part 15: backup automation - scripts, rsync, rclone.

The AWS CLI on Amazon Linux comes pre-installed, you just need to either configure an AWS profile or attach an EC2 Instance Profile with an IAM policy that grants permission to modify the Security Group.

One nuance to keep in mind if you’re writing the script yourself: IPs do change occasionally, even if rarely, and they need to be removed from the Security Group. But simply deleting all old entries and then adding new ones is not an option, because at that moment the ALB will lose its connection to Cloudflare.

So the script performs a comparison of old and new addresses and only removes those that are no longer in the latest update from Cloudflare.

Creating an IAM Role for the EC2 Instance profile

Eventually an IAM Role will need to be added to the instance anyway, so instead of using an AWS CLI profile with keys, it’s better to set up access to edit the Security Group through an EC2 Instance profile from the start.

The role needs rights to run ec2:DescribeSecurityGroups, ec2:AuthorizeSecurityGroupIngress, and ec2:RevokeSecurityGroupIngress - let's create a separate IAM Policy.

Create a new Security Group for the ALB - leave the Inbound rules empty, and in Outbound - keep the default “All to All”:

Create a new IAM Policy, restrict access to only this Security Group.

For ec2:DescribeSecurityGroups you can't set a resource restriction since the rule is global for all SGs, so we describe it in two separate Statements - one with Resource: "*", and the second with the specific SG.

Apply a Resource restriction: specify access limited to a specific group, because the script will perform destructive actions - deleting rules, and for such actions it's always better to limit the blast radius in case "something goes wrong":

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": "ec2:DescribeSecurityGroups",
            "Resource": "*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "ec2:AuthorizeSecurityGroupIngress",
                "ec2:RevokeSecurityGroupIngress"
            ],
            "Resource": "arn:aws:ec2:eu-west-1:ACCOUNT_ID:security-group/sg-029ffc5f56be700ea"
        }
    ]
}

Set the policy name and save:

Create an IAM Role, in Use Case select EC2:

Attach the policy created above:

Set the role name and save (though the name chosen here isn’t ideal - because 100% more permissions will be added to this role down the line):

In Security > Modify IAM Role, attach the role to the EC2 - can be done without stopping the instance:

Attach our new role:

SSH into the instance, and verify that it now has access to, for example, ec2:DescribeSecurityGroups:

[ec2-user@ip-10-0-3-146 ~]$ aws ec2 describe-security-groups --group-ids sg-029ffc5f56be700ea
{
    "SecurityGroups": [
        {
            "GroupId": "sg-029ffc5f56be700ea",
            "IpPermissionsEgress": [
                {
                    "IpProtocol": "-1",
                    "UserIdGroupPairs": [],
                    "IpRanges": [
                        {
                            "CidrIp": "0.0.0.0/0"
                        }
                    ],
                    "Ipv6Ranges": [],
                    "PrefixListIds": []
                }
            ],
...

Shell script for updating the ALB Security Group

Add a script that will:

fetch the current list of Cloudflare IP CIDRs (both IPv4 and IPv6)
use the AWS CLI to get the current list of rules in the Security Group
compare the new addresses with the Security Group values and add only those addresses not currently present
same logic for removal - if an IP is no longer in Cloudflare’s current list, remove it from the Security Group

There’s only one Security Group, so it’s simply set at the top in the $SG_ID variable:

#!/bin/bash

# Update ALB security group with current Cloudflare IP ranges
SG_ID="sg-029ffc5f56be700ea"
PORT=443

# Fetch current Cloudflare IP ranges
CF_IPS=$(curl -s https://www.cloudflare.com/ips-v4; echo; curl -s https://www.cloudflare.com/ips-v6)

# Get current IPs from security group
CURRENT_IPS=$(aws ec2 describe-security-groups --group-ids $SG_ID --query "SecurityGroups[0].IpPermissions[?FromPort==\`$PORT\`].[IpRanges[].CidrIp, Ipv6Ranges[].CidrIpv6]" --output text)

# Add missing IPs
for IP in $CF_IPS; do
    if ! echo "$CURRENT_IPS" | grep -q "$IP"; then
        echo "Adding $IP"
        if echo "$IP" | grep -q ":"; then
            aws ec2 authorize-security-group-ingress --group-id $SG_ID --ip-permissions "[{\"IpProtocol\":\"tcp\",\"FromPort\":$PORT,\"ToPort\":$PORT,\"Ipv6Ranges\":[{\"CidrIpv6\":\"$IP\"}]}]"
        else
            aws ec2 authorize-security-group-ingress --group-id $SG_ID --protocol tcp --port $PORT --cidr $IP
        fi
    fi
done

# Remove IPs no longer in Cloudflare list
for IP in $CURRENT_IPS; do
    if ! echo "$CF_IPS" | grep -q "$IP"; then
        echo "Removing $IP"
        if echo "$IP" | grep -q ":"; then
            aws ec2 revoke-security-group-ingress --group-id $SG_ID --ip-permissions "[{\"IpProtocol\":\"tcp\",\"FromPort\":$PORT,\"ToPort\":$PORT,\"Ipv6Ranges\":[{\"CidrIpv6\":\"$IP\"}]}]"
        else
            aws ec2 revoke-security-group-ingress --group-id $SG_ID --protocol tcp --port $PORT --cidr $IP
        fi
    fi
done

echo "Done"

Set the script’s execute permissions:

[root@ip-10-0-3-146 ~]# chmod +x /opt/update-alb-sg/update-alb-sg-from-cloudflare.sh

Run it manually to verify:

Check the Security Group:

Replacing the Security Group on the ALB

Edit the Security Groups list for the Load Balancer:

Add the new one, remove the old one:

Verify everything works:

$ curl -I https://rtfm.co.ua/
HTTP/2 200

But from home, direct access to the ALB is now blocked:

$ curl --connect-timeout 5 -kI rtfm-alb-1984146384.eu-west-1.elb.amazonaws.com
curl: (28) Connection timed out after 5002 milliseconds

Add it to cron.

Running from crontab

In Amazon Linux AL2023, the standard cron was removed - install it:

# yum install -y cronie
# systemctl enable crond
# systemctl start crond

Schedule it to run once a day, at 3am for example:

0 3 * * * /opt/update-alb-sg/update-alb-sg-from-cloudflare.sh >> /var/log/update-cf-sg.log 2>&1

You can also subscribe to the Cloudflare changelog/RSS to receive notifications when IP ranges change.

But IP range changes are very rare - the last time was in 2023, see IP Ranges:

And that’s basically it.

Now we can sleep soundly: our “frontend” is Cloudflare with its Security rules, WAF, and DoS/DDoS protection, and on the “backend” we’re protected at the network level, L3/L4 - and the Load Balancer and EC2 are living peacefully.

See my post on TCP/IP: OSI and TCP/IP models, TCP packets, Linux sockets and ports and AWS’s own documentation - Network security.

P.S. AWS documentation is one of the things I genuinely love about it. I once had to work extensively with Azure - and this is a real problem there.

Originally published at RTFM: Linux, DevOps, and system administration.

AWS: Monitoring AWS OpenSearch Service Cluster with CloudWatch

Arseny Zinchenko — Wed, 31 Dec 2025 10:00:00 +0000

Let’s continue our journey with AWS OpenSearch Service.

What we have is a small AWS OpenSearch Service cluster with three data nodes, used as a vector store for AWS Bedrock Knowledge Bases.

Previous parts:

We already had our first production incident :-)

We launched a search without filters, and our t3.small.search died due to CPU overload.

So let’s take a look at what we have in terms of monitoring all this happiness.

CloudWatch metrics
Memory monitoring
kNN Memory usage
JVM Memory usage
Collecting metrics to VictoriaMetrics
Creating a Grafana dashboard
VictoriaMetrics/Prometheus sum(), avg() та max()
Cluster status
Nodes status
CPUUtilization: Stats
CPUUtilization: Graph
JVMMemoryPressure: Graph
JVMGCYoungCollectionCount and JVMGCOldCollectionCount
KNNHitCount vs KNNMissCount
Final result
t3.small.search vs t3.medium.search on graphs
Creating Alerts

Now let’s do something basic, just with CloudWatch metrics, but there are several solutions for monitoring OpenSearch:

CloudWatch metrics from OpenSearchService itself — data on CPU, memory, and JVM, which we can collect in VictoriaMetrics and generate alerts or use in the Grafana dashboard, see Monitoring OpenSearch cluster metrics with Amazon CloudWatch
CloudWatch Events generated by OpenSearch Service — see Monitoring OpenSearch Service events with Amazon EventBridge — can be sent via SNS to Opsgenie, and from there to Slack.
Logs in CloudWatch Logs — we can collect them in VictoriaLogs and generate some metrics and alerts, but I didn’t see anything interesting in the logs during our production incident, see Monitoring OpenSearch logs with Amazon CloudWatch Logs.
Monitors of OpenSearch itself — capable of anomaly detection and custom alerting, there is even a Terraform resource opensearch_monitor, see also Configuring alerts in Amazon OpenSearch Service
There is also the Prometheus Exporter Plugin, which opens an endpoint for collecting metrics from Prometheus/VictoriaMetrics (but it cannot be added to AWS OpenSearch Managed, although support promises that there is a feature request — maybe it will be added someday).

CloudWatch metrics

There are quite a few metrics, but the ones that may be of interest to us are those that take into account the fact that we do not have dedicated master and coordinator nodes, and we do not use ultra-warm and cold instances.

Cluster metrics :

ClusterStatus: green/yellow/red - the main indicator of cluster status, control of data shard activity
Shards: active/unassigned/delayedUnassigned/activePrimary/initializing/relocating - more detailed information on the status of shards, but here is just the total number, without details on specific indexes
Nodes: the number of nodes in the cluster - knowing how many live nodes there should be - we can alert when a node goes down
SearchableDocuments: not that it's particularly interesting to us, but it might be useful later on to see what's going on in the indexes in general.
CPUUtilization: the percentage of CPU usage across all nodes, and this is a must-have
FreeStorageSpace: also useful to monitor
ClusterIndexWritesBlocked: Is everything OK with index writes?
JVMMemoryPressure and OldGenJVMMemoryPressure: percentage of JVM heap memory usage - we'll dig into JVM monitoring separately later, because it's a whole other headache.
AutomatedSnapshotFailure: probably good to know if the backup fails
CPUCreditBalance: useful for us because we are on t3 instances (but we don't have it in CloudWatch)
2xx, 3xx, 4xx, 5xx: data on HTTP requests and errors; I only collect 5xx for alerts here
ThroughputThrottle and IopsThrottle: we encountered disk access issues in RDS, so it is worth monitoring here as well, see PostgreSQL: AWS RDS Performance and monitoring; here you may need to look at the metrics from EBS volume metrics, but for start, you can simply add alerts to Throttle in general
HighSwapUsage: similar to the previous metrics - we once had a problem with RDS, so it's better to monitor this as well.

EBS volume metrics — these are basically standard EBS metrics, as for EC2 or RDS:

ReadLatency and WriteLatency: read/write delays
- sometimes there are spikes, so you can add
ReadThroughput and WriteThroughput: total disk load, let's say this way
DiskQueueDepth: I/O operations queue
- is empty in CloudWatch (for now?), so we’ll skip it
ReadIOPS and WriteIOPS: number of read/write operations per second

Instance metrics — here are the metrics for each OpenSearch instance (not the server, EC2, but OpenSearch itself) on each node:

FetchLatency and FetchRate: how quickly we get data from shards (but I couldn't find it in CloudWatch either)
ThreadCount: the number of threads in the operating system that were created by the JVM (Garbage Collector threads, search threads, write/index threads, etc.)
The value is stable in CloudWatch, but for now, we can add it to Grafana for the overall picture and see if there is anything interesting there
ShardReactivateCount: how often shards are transferred from cold/inactive states to active ones, which requires operating system resources, CPU, and memory; Well... maybe we should check if it has any significance for us at all.
- But there is nothing in CloudWatch either — “did not match any metrics”
ConcurrentSearchRate and ConcurrentSearchLatency: the number and speed of simultaneous search requests - this can be interesting if there are many parallel requests hanging for a long time
but for us (yet?), these values are constantly at zero, so we skip them
SearchRate: number of search queries per minute, useful for the overall picture
SearchLatency: search query execution speed, probably very useful, you can even set up an alert
IndexingRate and IndexingLatency: similar, but for indexing new documents
SysMemoryUtilization: percentage of memory usage on the data node, but this does not give a complete picture; you need to look at the JVM memory.
JVMGCYoungCollectionCount and JVMGCOldCollectionCount: the number of Garbage Collector runs, useful in conjunction with JVM memory data, which we will discuss in more detail later.
SearchTaskCancelled and SearchShardTaskCancelled: bad news :-) if tasks are canceled, something is clearly wrong (either the user interrupted the request, or there was an HTTP connection reset, or timeouts, or cluster load)
- but we always have zeroes, even when the cluster went down, so I don’t see the point in collecting these metrics yet
ThreadpoolIndexQueue and ThreadpoolSearchQueue: the number of tasks for indexing and searching in the queue; when there are too many of them, we get ThreadpoolIndexRejected and ThreadpoolSearchRejected
ThreadpoolIndexQueue is not available in CloudWatch at all, and ThreadpoolSearchQueue is there, but it's also constantly at zero, so we're skipping it for now
ThreadpoolIndexRejected and ThreadpoolSearchRejected: actually, above
in CloudWatch, the picture is similar — ThreadpoolIndexRejected is not present at all, ThreadpoolSearchRejected is zero
ThreadpoolIndexThreads and ThreadpoolSearchThreads: the maximum number of operating system threads for indexing and searching; if all are busy, requests will go to ThreadpoolIndexQueue/ThreadpoolSearchQueue
- OpenSearch has several types of pools for threads — search, index, write, etc., and each pool has a threads indicator (how many are allocated), see OpenSearch Threadpool.
- The Node Stats API (GET _nodes/stats/thread_pool) has an active threads metric, but I don't see it in CloudWatch.
ThreadpoolIndexThreads is not available in CloudWatch at all, and ThreadpoolSearchThreads is static, so I think we can skip monitoring them for now.
PrimaryWriteRejected: rejected write operations in primary shards due to issues in the thread pool write or index, or load on the data node
- CloudWatch is empty for now, but we will add collection and alerts
ReplicaWriteRejected: rejected write operations in replica shards - added to the primary document, but cannot be written to the replica
- CloudWatch is empty for now, but we will add collection and alerts

k-NN metrics — useful for us because we have a vector store with k-NN:

KNNCacheCapacityReached: when the cache is full (see below)
KNNEvictionCount: how often data is removed from the cache - a sign that there is not enough memory
KNNGraphMemoryUsage: off-heap memory usage for the vector graph itself
KNNGraphQueryErrors: number of errors when searching in vectors
- in CloudWatch are empty for now, but we will add collection and alert
KNNGraphQueryRequests: total number of queries to k-NN graphs
KNNHitCount and KNNMissCount: how many results were returned from the cache, and how many had to be read from the disk
KNNTotalLoadTime: speed of loading from disk to cache (large graphs or loaded EBS - time will increase)

Memory monitoring

Let’s think about how we can monitor the main indicators, starting with memory, because, well, this is Java.

What do we have about memory metrics?

SysMemoryUtilization: percentage of memory usage on the server (data node) in general
JVMMemoryPressure: total percentage of JVM Heap usage; JVM Heap is allocated by default to 50% of the server's memory, but no more than 32 gigabytes.
OldGenJVMMemoryPressure: see below
KNNGraphMemoryUsage: this was discussed in the first post - AWS: introduction to OpenSearch Service as a vector store
CloudWatch also has a metric called KNNGraphMemoryUsagePercentage, but it is not included in the documentation

kNN Memory usage

First, a brief overview of k-NN memory.

So, on EC2, we allocate memory for the JVM Heap (50% of what is available on the server) and separately for the off-heap for the OpenSearch vector store, where it keeps graphs and cache. For vector store, see Approximate k-NN search, plus the operating system itself and its file cache.

We don’t have a metric like “KNNGraphMemoryAvailable,” but with KNNGraphMemoryUsagePercentage and KNNGraphMemoryUsage, we can calculate it:

KNNGraphMemoryUsage: we currently have 662 megabytes
KNNGraphMemoryUsagePercentage: 60%

This means that 1 gigabyte is allocated outside the JVM Heap memory for k-NN graphs (this is on t3.medium.search).

From the documentation k-Nearest Neighbor (k-NN) search in Amazon OpenSearch Service:

OpenSearch Service uses half of an instance’s RAM for the Java heap (up to a heap size of 32 GiB). By default, k-NN uses up to 50% of the remaining half

Knowing that we currently have t3.medium.search, which provides 4 gigabytes of memory - 2 GB goes to the JVM Heap, and 1 gigabyte goes to the k-NN graph.

The main part of KNNGraphMemory is used by the k-NN cache, i.e., the part of the system's RAM where OpenSearch keeps HNSW graphs from vector indexes so that they do not have to be read from disk each time (see k-NN clear cache).

Therefore, it is useful to have graphs for EBS IOPS and k-NN cache usage.

JVM Memory usage

Okay, let’s review what’s going on in Java in general. See What Is Java Heap Memory?, OpenSearch Heap Size Usage and JVM Garbage Collection, and Understanding the JVMMemoryPressure metric changes in Amazon OpenSearch Service.

To put it simply:

Stack Memory: in addition to the JVM Heap, we have a Stack, which is allocated to each thread, where it keeps its variables, references, and startup parameters
- set via -Xss, default value from 256 kilobytes to 1 megabyte, see Understanding Threads and Locks (couldn't find how to view in OpenSearch Service)
- if we have many threads, there will be a lot of memory for their stacks
cleared when the thread dies
Heap Space :
used to allocate memory that is available to all threads
managed by Garbage Collectors (GC)
in the context of OpenSearch, we will have search and indexation caches here

In Heap memory, we have:

Young Generation : fresh data, all new objects
data from here is either deleted completely or moved to Old Generation
Old Generation : the OpenSearch process code itself, caches, Lucene index structures, large arrays

If OldGenJVMMemoryPressure is full, it means that the Garbage Collector cannot clean it up because there are references to the data, and then we have a problem - because there is no space in the Heap for new data, and the JVM may crash with an OutOfMemoryError.

In general, “heap pressure” is when there is little free memory in Young Gen and Old Gen, and there is nowhere to place new data to respond to clients.

This leads to frequent Garbage Collector runs, which take up time and system resources — instead of processing requests from clients.

As a result, latency increases, indexing of new documents slows down, or we get ClusterIndexWritesBlocked - to avoid Java OutOfMemoryError , because when indexing, OpenSearch first writes data to the Heap and then "dumps" it to disk.

See Key JVM Metrics to Monitor for Peak Java Application Performance.

So, to get a picture of memory usage, we monitor:

SysMemoryUtilization - for an overall picture of the EC2 status
- in our case, it will be consistently around 90%, but that’s OK
JVMMemoryPressure - for an overall picture of the JVM
- should be cleaned regularly with Garbage Collector (GC)
- if it is constantly above 80–90%, there are problems with running GC
OldGenJVMMemoryPressure - for Old Generation Heap data
- should be at 30–40%; if it is higher and is not being cleared, then there are problems either with the code or with GC
KNNGraphMemoryUsage - in our case, this is necessary for the overall picture

It is worth adding alerts for HighSwapUsage - we already had active swapping when we launched on t3.small.search, and this is an indication that there is not enough memory.

Collecting metrics to VictoriaMetrics

So, how do you choose metrics?

First, we look for them in CloudWatch Metrics and see if the metric exists at all and if it returns any interesting data.

For example, SysMemoryUtilization provides information.

Here we had a spike on t3.small.search, after which the cluster crashed:

But the HighSwapUsage metric also needs to be moved to t3.medium.search:

ClusterStatus is present:

Shards exist, but they are indexed by all criteria, and there is no way to filter by individual criteria:

It is also important to note that collecting metrics from CloudWatch also costs money for API requests, so it is not advisable to collect everything indiscriminately.

In general, we use YACE (Yet Another CloudWatch Exporter) to collect metrics from CloudWatch, but it does not support OpenSearch Managed cluster, see Features.

Therefore, we will use a standard exporter — CloudWatch Exporter.

We deploy it from the Helm monitoring chart (see VictoriaMetrics: creating a Kubernetes monitoring stack with your own Helm chart), add a new config to it:

...

prometheus-cloudwatch-exporter:
  enabled: true
  serviceAccount:
    name: "cloudwatch-sa"
    annotations:
      eks.amazonaws.com/sts-regional-endpoints: "true"
  serviceMonitor:
    enabled: true
  config: |-
    region: us-east-1
    metrics:

    - aws_namespace: AWS/ES
      aws_metric_name: KNNGraphMemoryUsage
      aws_dimensions: [ClientId, DomainName, NodeId]
      aws_statistics: [Average]

    - aws_namespace: AWS/ES
      aws_metric_name: SysMemoryUtilization
      aws_dimensions: [ClientId, DomainName, NodeId]
      aws_statistics: [Average]

    - aws_namespace: AWS/ES
      aws_metric_name: JVMMemoryPressure
      aws_dimensions: [ClientId, DomainName, NodeId]
      aws_statistics: [Average]

    - aws_namespace: AWS/ES
      aws_metric_name: OldGenJVMMemoryPressure
      aws_dimensions: [ClientId, DomainName, NodeId]
      aws_statistics: [Average]

Please note that different metrics may have different Dimensions - check them in CloudWatch:

Deploy, check:

And even the numbers turned out to be as we calculated in the first post — we have ~130,000 documents in the production index, according to the formula num_vectors * 1.1 * (4*1024 + 8*16), which equals 604032000 bytes, or 604.032 megabytes.

And on the graph we have 662,261 kilobytes — that’s 662 megabytes, but across all indexes combined.

Now we have metrics in VictoriaMetrics : aws_es_knngraph_memory_usage_average, aws_es_sys_memory_utilization_average, aws_es_jvmmemory_pressure_average, aws_es_old_gen_jvmmemory_pressure_average.

Add the rest in the same way.

To find out what metrics are called in VictoriaMetrics/Prometheus, open the port to CloudWatch Exporter:

$ kk port-forward svc/atlas-victoriametrics-prometheus-cloudwatch-exporter 9106

And search for metrics with curl and grep:

$ curl -s localhost:9106/metrics | grep aws_es
# HELP aws_es_cluster_status_green_maximum CloudWatch metric AWS/ES ClusterStatus.green Dimensions: [ClientId, DomainName] Statistic: Maximum Unit: Count
# TYPE aws_es_cluster_status_green_maximum gauge
aws_es_cluster_status_green_maximum{job="aws_es",instance="",domain_name="atlas-kb-prod-cluster",client_id="492***148",} 1.0 1758014700000
# HELP aws_es_cluster_status_yellow_maximum CloudWatch metric AWS/ES ClusterStatus.yellow Dimensions: [ClientId, DomainName] Statistic: Maximum Unit: Count
# TYPE aws_es_cluster_status_yellow_maximum gauge
aws_es_cluster_status_yellow_maximum{job="aws_es",instance="",domain_name="atlas-kb-prod-cluster",client_id="492***148",} 0.0 1758014700000
# HELP aws_es_cluster_status_red_maximum CloudWatch metric AWS/ES ClusterStatus.red Dimensions: [ClientId, DomainName] Statistic: Maximum Unit: Count
# TYPE aws_es_cluster_status_red_maximum gauge
aws_es_cluster_status_red_maximum{job="aws_es",instance="",domain_name="atlas-kb-prod-cluster",client_id="492***148",} 0.0 1758014700000
...

Creating a Grafana dashboard

OK, we have metrics from CloudWatch — that’s enough for now.

Let’s think about what we want to see in Grafana.

The general idea is to create a kind of dashboard overview, where all the key data for the cluster will be displayed on a single board.

What metrics are currently available, and how can we use them in Grafana? I wrote them down here so as not to get confused, because there are quite a few of them:

aws_es_cluster_status_green_maximum, aws_es_cluster_status_yellow_maximum, aws_es_cluster_status_red_maximum: you can create a single Stats panel
aws_es_nodes_maximum: also some kind of stats panel - we know how many there should be, and we'll mark it red when there are fewer Data Nodes than there should be.
aws_es_searchable_documents_maximum: just for fun, we will show the number of documents in all indexes together in a graph
aws_es_cpuutilization_average: one graph per node, and some Stats with general information and different colors
aws_es_free_storage_space_maximum: just Stats
aws_es_cluster_index_writes_blocked_maximum: did not add to Grafana, only alert
aws_es_jvmmemory_pressure_average: graph and stats
aws_es_old_gen_jvmmemory_pressure_average: somewhere nearby, also graph + Stats
aws_es_automated_snapshot_failure_maximum: this is just for alerting
aws_es_5xx_maximum: both graph and Stats
aws_es_iops_throttle_maximum: graph to see in comparison with other data such as CPU/Mem usage
aws_es_throughput_throttle_maximum: graph
aws_es_high_swap_usage_maximum: both graph and Stats - graph, to see in comparison with CPU/disks
aws_es_read_latency_average: graph
aws_es_write_latency_average: graph
aws_es_read_throughput_average: I didn't add it because there are too many graphs.
aws_es_write_throughput_average: I didn't add it because there are too many graphs.
aws_es_read_iops_average: a graph that is useful for understanding how the k-NN cache works - if there is not enough of it (and we tested on t3.small.search with 2 gigabytes of total memory), then there will be a lot of reading from the disk.
aws_es_write_iops_average: similarly
aws_es_thread_count_average: I didn't add it because it's pretty static and I didn't see any particularly useful information in it.
aws_es_search_rate_average: also just a graph
aws_es_search_latency: similarly, somewhere nearby
aws_es_sys_memory_utilization_average: Well, it will constantly be around 90% until I remove it from Grafana, but I added it to alerts.
aws_es_jvmgcyoung_collection_count_average: graph showing how often it is called
aws_es_jvmgcold_collection_count_average: graph showing how often it is called
aws_es_primary_write_rejected_average: graph, but I haven't added it yet because there are too many graphs - only alerts
aws_es_replica_write_rejected_average: graph, but I haven't added it yet because there are too many graphs - only alerts

k-NN:

aws_es_knncache_capacity_reached_maximum: only for warning alerts
aws_es_knneviction_count_average: did not add, although it may be interesting
`aws_es_knngraph_memory_usage_average: did not add
aws_es_knngraph_memory_usage_percentage_maximum: graph instead of aws_es_knngraph_memory_usage_average
aws_es_knngraph_query_errors_maximum: alert only
aws_es_knngraph_query_requests_sum: graph
aws_es_knnhit_count_maximum: graph
aws_es_knnmiss_count_maximum: graph
`aws_es_knntotal_load_time_sum: it would be nice to have a graph, but there is no space on the board

VictoriaMetrics/Prometheus `sum()`, `avg()` and max()`

First, let’s recall what functions we have for data aggregation.

With CloudWatch for OpenSearch, we will receive two main types: counter and gauge:

`
$ curl -s localhost:9106/metrics | grep cpuutil

HELP aws_es_cpuutilization_average CloudWatch metric AWS/ES CPUUtilization Dimensions: [ClientId, DomainName, NodeId] Statistic: Average Unit: Percent

TYPE aws_es_cpuutilization_average gauge

aws_es_cpuutilization_average{job="aws_es",instance="",domain_name="atlas-kb-prod-cluster",node_id="BzX51PLwSRCJ7GrbgB4VyA",client_id="492***148",} 10.0 1758099600000
...
`

The difference between them:

counter : the value can only increase the value
gauge : the value can increase and decrease

Here we have “TYPE aws_es_cpuutilization_average gauge", because CPU usage can both increase and decrease.

See the excellent documentation VictoriaMetrics — Prometheus Metrics Explained: Counters, Gauges, Histograms & Summaries:

How can we use it in graphs?

If we just look at the values, we have a set of labels here, each forming its own time series:

aws_es_cpuutilization_average{node_id="BzX51PLwSRCJ7GrbgB4VyA"} == 9
aws_es_cpuutilization_average{node_id="IIEcajw5SfmWCXe_AZMIpA"} == 28
aws_es_cpuutilization_average{node_id="lrsnwK1CQgumpiXfhGq06g"} == 8

With sum() without a label, we simply get the sum of all values:

If we do sum by (node_id), we will get the value for a specific time series, which will coincide with the sample without sum by ():

(the meaning changes as I write and make inquiries)

With `max() without filters, we simply obtain the maximum value selected from all the time series received:

And with avg() - the average value of all values, i.e., the sum of all values divided by the number of time series:

Let’s calculate it ourselves:

(41+46+12)/3
33

Actually, the reason I decided to write about this separately is because even with sum() and by (node_id), you can sometimes get the following results:

Although without sum() there are none:

And they happened because Pod was being recreated from CloudWatch Exporter at that moment:

And at that moment, we were receiving data from the old pod and the new one.

Therefore, the options here are to use either max() or just avg(). Although⁣ max() is probably better, because we are interested in the "worst" indicators.

Okay, now that we’ve figured that out, let’s get started on the dashboard.

Cluster status

Here, I would like to see all three values — Green, Yellow, and Red — on a single Stats panel.

But since we don’t have if/else in Grafana, let’s make a workaround.

We collect all three metrics and multiply the result of each by 1, 2, or 3:

sum(aws_es_cluster_status_green_maximum) by (domain_name) * 1 +
sum(aws_es_cluster_status_yellow_maximum) by (domain_name) * 2 +
sum(aws_es_cluster_status_red_maximum) by (domain_name) * 3

Accordingly, if aws_es_cluster_status_green_maximum == 1, then 1 * 1 == 1, and aws_es_cluster_status_yellow_maximum == 0 and aws_es_cluster_status_red_maximum will be == 0, then the multiplication will return 0.

And if aws_es_cluster_status_green_maximum becomes 0, but aws_es_cluster_status_red_maximum is 1, then 1 * 2 equals 3, and based on the value 3, we will change the indicator in the Stats panel.

And add Value mappings with text and colors:

Get the following result:

Nodes status

It’s simple here — we know the required number, and we get the current one from aws_es_nodes_maximum:

sum(aws_es_nodes_maximum) by (domain_name)

And again, using Value mappings, we set the values and colors:

In case we ever increase the number of nodes and forget to update the value for “OK” here, we add a third status, ERR:

CPUUtilization: Stats

Here, we will make a cross-tabulation with the Gauge visualization type:

avg(aws_es_cpuutilization_average) by (domain_name)

Set Text size and Unit:

And Thresholds:

Description ChatGPT generates pretty well — useful for developers and for us in six months, or we can just take the description from AWS documentation:

The percentage of CPU usage for data nodes in the cluster. Maximum shows the node with the highest CPU usage. Average represents all nodes in the cluster.

Add the rest of the stats:

CPUUtilization: Graph

Here we will display a graph for the CPU of each node — the average over 5 minutes:

max(avg_over_time(aws_es_cpuutilization_average[5m])) by (node_id)

And here is another example of how sum() created spikes that did not actually exist:

Therefore, we do max().

Set Gradient mode == Opacity, and Unit == percent:

Set Color scheme and Thresholds, enable Show thresholds:

In Data links, you can set a link to the DataNode Health page in the AWS Console:

https://us-east-1.console.aws.amazon.com/aos/home?region=us-east-1#opensearch/domains/atlas-kb-prod-cluster/data_Node/${__field.labels.node_id}

All available fields — Ctrl+Space:

Actions seems to have appeared not so long ago. I haven’t used it yet, but it looks interesting — you can push something:

JVMMemoryPressure: Graph

Here, we are interested in seeing whether memory usage “sticks” and how often the Garbage Collector is launched.

The query is simple — you can do max by (node_id), but I just made a general picture for the cluster:

max(aws_es_jvmmemory_pressure_average)

And the schedule is similar to the previous one:

In Description, add the explanation “when to worry”:

Represents the percentage of JVM heap in use (young + old generation).

Values below 75% are normal. Sustained pressure above 80% indicates frequent GC and potential performance degradation.

Values consistently > 85–90% mean heap exhaustion risk and may trigger ClusterIndexWritesBlocked — investigate immediately.

JVMGCYoungCollectionCount and JVMGCOldCollectionCount

A very useful graph to see how often Garbage Collects are triggered.

In the query, we will use increase[1m] to see how the value has changed in a minute:

max(increase(aws_es_jvmgcyoung_collection_count_average[1m])) by (domain_name)

And for Old Gen:

max(increase(aws_es_jvmgcold_collection_count_average[1m])) by (domain_name)

Unit — ops/sec, Decimals set to 0 to have only integer values:

KNNHitCount vs KNNMissCount

Here, we will generate data for a second — rate():

sum(rate(aws_es_knnhit_count_average[5m]))

And for Cache Miss:

sum(rate(aws_es_knnmiss_count_average[5m]))

Unit ops/s, colors can be set via Overrides:

The statistics here, by the way, are very mediocre — there are consistently a lot of cache misses, but we haven’t figured out why yet.

Final result

We collect all the graphs and get something like this:

`t3.small.search` vs `t3.medium.search` on graphs

And here’s an example of how a lack of resources, primarily memory, is reflected in the graphs: we had t3.medium.search, then we switched back to t3.small.search to see how it would affect performance.

t3.small.search is only 2 gigabytes of memory and 2 CPU cores.

Of these 2 gigabytes of memory, 1 gigabyte was allocated to JVM Heap, 500 megabytes to k-NN memory, and 500 remained for other processes.

Well, the results are quite expected:

Garbage Collectors started running constantly because it was necessary to clean up the memory that was lacking.
Read IOPS increased because data was constantly being loaded from the disk to the JVM Heap Young and k-NN.
Search Latency increased because not all data was in the cache, and I/O operations from the disk were pending.
and CPU utilization jumped — because the CPU was loaded with Garbage Collectors and reading from the disk

Creating Alerts

You can also check out the recommendations from AWS — Recommended CloudWatch alarms for Amazon OpenSearch Service.

OpenSearch ClusterStatus Yellow and OpenSearch ClusterStatus Red: here, simply if more than 0:

...
      - alert: OpenSearch ClusterStatus Yellow
        expr: sum(aws_es_cluster_status_yellow_maximum) by (domain_name, node_id) > 0
        for: 1s
        labels:
          severity: warning
          component: backend
          environment: prod
        annotations:
          summary: 'OpenSearch ClusterStatus Yellow status detected'
          description: |-
            The primary shards for all indexes are allocated to nodes in the cluster, but replica shards for at least one index are not
            *OpenSearch Doman*: `{{ "{{" }} $labels.domain_name }}`
          grafana_opensearch_overview_url: 'https://{{ .Values.monitoring.root_url }}/d/b2d2dabd-a6b4-4a8a-b795-270b3e200a2e/aws-opensearch-cluster-cloudwatch'

      - alert: OpenSearch ClusterStatus Red
        expr: sum(aws_es_cluster_status_red_maximum) by (domain_name, node_id) > 0
        for: 1s
        labels:
          severity: critical
          component: backend
          environment: prod
        annotations:
          summary: 'OpenSearch ClusterStatus RED status detected!'
          description: |-
            The primary and replica shards for at least one index are not allocated to nodes in the cluster
            *OpenSearch Doman*: `{{ "{{" }} $labels.domain_name }}`
          grafana_opensearch_overview_url: 'https://{{ .Values.monitoring.root_url }}/d/b2d2dabd-a6b4-4a8a-b795-270b3e200a2e/aws-opensearch-cluster-cloudwatch'
...

Through labels, we have implemented alert routing in Opsgenie to the necessary Slack channels, and the annotation grafana_opensearch_overview_url is used to add a link to Grafana in a Slack message:

OpenSearch CPUHigh — if more than 20% for 10 minutes:

- alert: OpenSearch CPUHigh
        expr: sum(aws_es_cpuutilization_average) by (domain_name, node_id) > 20
        for: 10m
...

OpenSearch Data Node down — if the node is down:

- alert: OpenSearch Data Node down
        expr: sum(aws_es_nodes_maximum) by (domain_name) < 3
        for: 1s
        labels:
          severity: critical
...

aws_es_free_storage_space_maximum - we don't need it yet.

OpenSearch Blocking Write — alert us if write blocks have started:

...
      - alert: OpenSearch Blocking Write
        expr: sum(aws_es_cluster_index_writes_blocked_maximum) by (domain_name) >= 1
        for: 1s
        labels:
          severity: critical
...

And the rest of the alerts I’ve added so far:

...
      - alert: OpenSearch AutomatedSnapshotFailure 
        expr: sum(aws_es_automated_snapshot_failure_maximum) by (domain_name) >= 1
        for: 1s
        labels:
          severity: critical
...
      - alert: OpenSearch 5xx Errors 
        expr: sum(aws_es_5xx_maximum) by (domain_name) >= 1
        for: 1s
        labels:
          severity: critical
...
      - alert: OpenSearch IopsThrottled
        expr: sum(aws_es_iops_throttle_maximum) by (domain_name) >= 1
        for: 1s
        labels:
          severity: warning
...
      - alert: OpenSearch ThroughputThrottled
        expr: sum(aws_es_throughput_throttle_maximum) by (domain_name) >= 1
        for: 1s
        labels:
          severity: warning
...
      - alert: OpenSearch SysMemoryUtilization High Warning
        expr: avg(aws_es_sys_memory_utilization_average) by (domain_name) >= 95
        for: 5m
        labels:
          severity: warning
...
      - alert: OpenSearch PrimaryWriteRejected High
        expr: sum(aws_es_primary_write_rejected_maximum) by (domain_name) >= 1
        for: 1s
        labels:
          severity: critical
...
      - alert: OpenSearch KNNGraphQueryErrors High
        expr: sum(aws_es_knngraph_query_errors_maximum) by (domain_name) >= 1
        for: 1s
        labels:
          severity: critical
...
      - alert: OpenSearch KNNCacheCapacityReached
        expr: sum(aws_es_knngraph_query_errors_maximum) by (domain_name) >= 1
        for: 1s
        labels:
          severity: warning
...

As we use it, we’ll see what else we can add.

Originally published at RTFM: Linux, DevOps, and system administration.

Terraform: creating an AWS OpenSearch Service cluster and users

Arseny Zinchenko — Tue, 30 Dec 2025 10:00:00 +0000

In the first part, we covered the basics of AWS OpenSearch Service in general and the types of instances for Data Nodes — AWS: Getting Started with OpenSearch Service as a Vector Store.

In the second part, we covered access, AWS: Creating an OpenSearch Service Cluster and Configuring Authentication and Authorization.

Now let’s write Terraform code to create a cluster, users, and indexes.

We will create the cluster in VPC and use the internal user database for authentication.

But in VPC, you can’t… Because — surprise! — AWS Bedrock requires OpenSearch Managed Cluster to be public, not in VPC.

The OpenSearch Managed Cluster you provided is not supported because it is VPC protected. Your cluster must be behind a public network.

I wrote to the AWS tech. support, and they said:

However, there is an ongoing product feature request (PFR) to have Bedrock KnowledgeBases support provisioned Open Search clusters in VPC.

And they suggest using Amazon OpenSearch Serverless, which we are actually running away from because the prices are ridiculous.

The second problem that arose when I started writing resources bedrockagent_knowledge_base is that it does not support storage_configuration with typeOPENSEARCH_MANAGED`, only Serverless.

But Pull Request for this already exists, maybe someday they will approve it.
(UPD: this was already merged)

So, we will create an OpenSearch Managed Service cluster with three indexes — Dev/Staging/Prod.

The cluster will have three small data nodes, and each index will have 1 primary shard and 1 replica, because the project is small, and the data in our Production index on AWS OpenSearch Serverless, from which we want to migrate to AWS OpenSearch Service, is currently only 2 GiB, and is unlikely to grow significantly in the future.

It would be good to create the cluster in our own Terraform module to make it easier to create some test environments, as I did for AWS EKS, but there isn’t much time for that right now, so we’ll just use tf files with a separate prod.tfvars for variables.

Maybe later I’ll write separately about transferring it to our own module, because it’s really convenient.

In the next part, we’ll talk about monitoring, because our Production has already crashed once :-)

Terraform files structure
Project planning
Creating a cluster
Custom endpoint configuration
Terraform Outputs
Creating OpenSearch Users
Error: elastic: Error 403 (Forbidden)
Creating Internal Users
Internal database users
Adding IAM Users
Creating AWS Bedrock IAM Roles and OpenSearch Role mappings
Creating OpenSearch indexes

Terraform files structure

The initial file and directory structure of the project is as follows:

$ tree . . ├── README.md └── terraform ├── Makefile ├── backend.tf ├── data.tf ├── envs │ └── prod │ └── prod.tfvars ├── locals.tf ├── outputs.tf ├── providers.tf ├── variables.tf └── versions.tf

In the providers.tf - provider settings, currently only AWS, and through it we set the default tags:

provider "aws" { region = var.aws_region default_tags { tags = { component = var.component created-by = "terraform" environment = var.environment } } }

In the data.tf, we collect AWS Account ID, Availability Zones, VPC, and private subnets in which we will create a cluster in which we will eventually create a cluster:

`
data "aws_caller_identity" "current" {}

data "aws_availability_zones" "available" {
state = "available"
}

data "aws_vpc" "eks_vpc" {
id = var.vpc_id
}

data "aws_subnets" "private" {
filter {
name = "vpc-id"
values = [var.vpc_id]
}

tags = {
subnet-type = "private"
}
}
`

File variables.tf with our default variables, then we will add new ones:

`
variable "aws_region" {
type = string
}

variable "project_name" {
description = "A project name to be used in resources"
type = string
}

variable "component" {
description = "A team using this project (backend, web, ios, data, devops)"
type = string
}

variable "environment" {
description = "Dev/Prod, will be used in AWS resources Name tag, and resources names"
type = string
}

variable "vpc_id" {
type = string
description = "A VPC ID to be used to create OpenSearch cluster and its Nodes"
}
`

Pass the values of variables through a separate prod.tfvars file, then, if necessary, we can create a new environment through a file of the type envs/test/test.tfvars:

aws_region = "us-east-1" project_name = "atlas-kb" component = "backend" environment = "prod" vpc_id = "vpc-0fbaffe234c0d81ea" dns_zone = "prod.example.co"

In the Makefile, we simplify our local life:

PROD

init-prod:
terraform init -reconfigure -backend-config="key=prod/atlas-knowledge-base-prod.tfstate"

plan-prod:
terraform plan -var-file=envs/prod/prod.tfvars

apply-prod:
terraform apply -var-file=envs/prod/prod.tfvars

destroy-prod:

terraform destroy -var-file=envs/prod/prod.tfvars

What files will be next?

We will also have AWS Bedrock, which will need to be configured for access — we will do this through its IAM Role, and I will not write about Bedrock here — because it is a separate topic, and Terraform does not yet support OPENSEARCH_MANAGED, so we did it manually, and then we will execute terraform import.

We will create indexes, users for our Backend API, and Bedrock IAM Role mappings in OpenSearch’s internal database through Terraform OpenSearch Provider to simplify OpenSearch Dashboards access.

Project planning

We can create a cluster from the Terraform resource aws_opensearch_domain, or we can use ready-made modules, such as the opensearch from @Anton Babenko.

Let’s take Anton’s module, because I use his modules a lot, and everything works great.

Creating a cluster

Examples — terraform-aws-opensearch/tree/master/examples.

Add a variable with cluster parameters to the variables.tf:

`
...

variable "cluser_options" {
description = "A map of options to configure the OpenSearch cluster"
type = object({
instance_type = string
instance_count = number
volume_size = number
volume_type = string
engine_version = string
auto_software_update_enabled = bool
})
}
`

And a value in prod.tfvars:

`
...

cluser_options = {
instance_type = "t3.small.search"
instance_count = 3
volume_size = 50
volume_type = "gp3"
engine_version = "OpenSearch_2.19"
auto_software_update_enabled = true
}
`

t3.small.search instances are the most minimal and sufficient for us at this time, although there are limitations for t3, such as the AWS OpenSearch Auto-tune feature not being supported.

In general, t3 is not intended for production use cases. See also Operational best practices for Amazon OpenSearch Service, Current generation instance types, and Amazon OpenSearch Service quotas.

I set the version here to 2.9, but 3.1 was added just a few days ago — see Supported versions of Elasticsearch and OpenSearch.

We take three nodes so that the cluster can select a cluster manager node if one node fails, see Dedicated master node distribution, Learning OpenSearch from scratch, part 2: Digging deeper, and Enhance stability with dedicated cluster manager nodes using Amazon OpenSearch Service.

Contents of the locals.tf:

locals { # 'atlas-kb-prod' env_name = "${var.project_name}-${var.environment}" }

Most of the locals will be right here, but some that are very "local" to a particular code will be in the resource code files.

Add the file opensearcth_users.tf - for now, there is only a root user here, and the password is stored in AWS Parameter Store (instead of AWS Secrets Manager - "that's just how it happened historically“):

ROOT

generate root password

waiting for write-only: https://github.com/hashicorp/terraform-provider-aws/pull/43621

then will update it with the ephemeral type

resource "random_password" "os_master_password" {
length = 16
special = true
}

store the root password in AWS Parameter Store

resource "aws_ssm_parameter" "os_master_password" {
name = "/${var.environment}/${local.env_name}-root-password"
description = "OpenSearch cluster master password"
type = "SecureString"
value = random_password.os_master_password.result
overwrite = true
tier = "Standard"

lifecycle {
ignore_changes = [value] # to prevent diff every time password is regenerated
}
}

data "aws_ssm_parameter" "os_master_password" {
name = "/${var.environment}/${local.env_name}-root-password"
with_decryption = true

depends_on = [aws_ssm_parameter.os_master_password]
}
`

Let’s write the opensearch_cluster.tf file.

I left the config for VPC here for future reference and just as an example, although it will not be possible to transfer an already created cluster to VPC — you will have to create a new one, see Limitations in the documentation Launching your Amazon OpenSearch Service domains within a VPC:

`
module "opensearch" {
source = "terraform-aws-modules/opensearch/aws"
version = "~> 2.0.0"

# enable Fine-grained access control
# by using the internal user database, we'll simply access to the Dashboards
# for backend API Kubernetes Pods, will use Kubernetes Secrets with username:password from AWS Parameter Store
advanced_security_options = {
enabled = true
anonymous_auth_enabled = false
internal_user_database_enabled = true

master_user_options = {
  master_user_name = "os_root"
  master_user_password = data.aws_ssm_parameter.os_master_password.value
}

}

# can't be used with t3 instances
auto_tune_options = {
desired_state = "DISABLED"
}

# have three data nodes - t3.small.search nodes in two AZs
# will use 3 indexes - dev/stage/prod with 1 shard and 1 replica each
cluster_config = {
instance_count = var.cluser_options.instance_count
dedicated_master_enabled = false
instance_type = var.cluser_options.instance_type

# put both data-nodes in different AZs
zone_awareness_config = {
  availability_zone_count = 2
}

zone_awareness_enabled = true

}

# the cluster's name
# 'atlas-kb-prod'
domain_name = "${local.env_name}-cluster"

# 50 GiB for each Data Node
ebs_options = {
ebs_enabled = true
volume_type = var.cluser_options.volume_type
volume_size = var.cluser_options.volume_size
}

encrypt_at_rest = {
enabled = true
}

# latest for today:
# https://docs.aws.amazon.com/opensearch-service/latest/developerguide/what-is.html#choosing-version
engine_version = var.cluser_options.engine_version

# enable CloudWatch logs for Index and Search slow logs
# TODO: collect to VictoriaLogs or Loki, and create metrics and alerts
log_publishing_options = [
{ log_type = "INDEX_SLOW_LOGS" },
{ log_type = "SEARCH_SLOW_LOGS" },
]

ip_address_type = "ipv4"

node_to_node_encryption = {
enabled = true
}

# allow minor version updates automatically
# will be performed during off-peak windows
software_update_options = {
auto_software_update_enabled = var.cluser_options.auto_software_update_enabled
}

# DO NOT use 'atlas-vpc-ops' VPC and its private subnets
# > "The OpenSearch Managed Cluster you provided is not supported because it is VPC protected. Your cluster must be behind a public network."
# vpc_options = {
# subnet_ids = data.aws_subnets.private.ids
# }

# # VPC endpoint to access from Kubernetes Pods
# vpc_endpoints = {
# one = {
# subnet_ids = data.aws_subnets.private.ids
# }
# }

# Security Group rules to allow access from the VPC only
# security_group_rules = {
# ingress_443 = {
# type = "ingress"
# description = "HTTPS access from VPC"
# from_port = 443
# to_port = 443
# ip_protocol = "tcp"
# cidr_ipv4 = data.aws_vpc.ops_vpc.cidr_block
# }
# }

# Access policy
# necessary to allow access for AWS user to the Dashboards
access_policy_statements = [
{
effect = "Allow"

  principals = [{
    type = "*"
    identifiers = ["*"]
  }]

  actions = ["es:*"]
}

]

# 'atlas-kb-ops-os-cluster'
tags = {
Name = "${var.project_name}-${var.environment}-os-cluster"
}
}
`

Basically, everything is described in the comments, but in short:

enable fine-grained access control and a local user database
three data nodes, each with 50 gigabytes of disk space, in different Availability Zones
enable logs in CloudWatch
create a cluster in private subnets
allow access for everyone in the Domain Access Policy
well, that’s it for now… we can’t use Security Groups because we’re not in VPC, but how do we create an IP-based policy? We don’t know CIDR Bedrock
or in the principals.identifiers we could add a limit on our IAM Users + Bedrock AIM Role

Run creating the cluster and go to have some tea, as this process will take around 20 minutes.

Custom endpoint configuration

After creating the cluster, check access to the Dashboards. If everything is OK, add a custom endpoint.

Note : Custom endpoints have their own quirks: in Terraform OpenSearch Provider, you need to use the custom endpoint URL, but in AWS Bedrock Knowledge Base, you need to use the default cluster URL.

To do this, we need to create a certificate in AWS Certificate Manager, and add a new record in Route53.

I expected a possible chicken-and-egg problem here, because Custom Endpoint settings depend on AWS ACM and a record in AWS Route53, and the record in AWS Route53 will depend on the cluster because it uses its endpoint.

But no, if you create a new cluster with the settings described below, everything is created correctly: first, the certificate in AWS ACM, then the cluster with Custom Endpoint, then the record in Route53 with CNAME to the cluster default URL.

Add a new local - os_custom_domain_name:

locals { # 'atlas-kb-prod' env_name = "${var.project_name}-${var.environment}" # 'opensearch.prod.example.co' os_custom_domain_name = "opensearch.${var.dns_zone}" }

Add the Route53 zone data retrieval to the data.tf:

`
...

data "aws_route53_zone" "zone" {
name = var.dns_zone
}
`

Add certificate creation and Route53 entry to the opensearch_cluster.tf:

TLS for the Custom Domain

module "prod_opensearch_acm" {
source = "terraform-aws-modules/acm/aws"
version = "~> 6.0"

# 'opensearch.example.co'
domain_name = local.os_custom_domain_name
zone_id = data.aws_route53_zone.zone.zone_id

validation_method = "DNS"
wait_for_validation = true

tags = {
Name = local.os_custom_domain_name
}
}

resource "aws_route53_record" "opensearch_domain_endpoint" {
zone_id = data.aws_route53_zone.zone.zone_id
name = local.os_custom_domain_name
type = "CNAME"
ttl = 300
records = [module.opensearch.domain_endpoint]
}

...
`

And in the module "opensearch", add the custom endpoint settings:

... domain_endpoint_options = { custom_endpoint_certificate_arn = module.prod_opensearch_acm.acm_certificate_arn custom_endpoint_enabled = true custom_endpoint = local.os_custom_domain_name tls_security_policy = "Policy-Min-TLS-1-2-2019-07" } ...

Run terraform init and terraform apply, check the settings:

And check access to the Dashboards.

Terraform Outputs

Let’s add some outputs.

For now, just for ourselves, but later we may use them in imports from other projects, see Terraform: terraform_remote_state — getting outputs from other state files:

`
output "vpc_id" {
value = var.vpc_id
}

output "cluster_arn" {
value = module.opensearch.domain_arn
}

output "opensearch_domain_endpoint_cluster" {
value = "https://${module.opensearch.domain_endpoint}"
}

output "opensearch_domain_endpoint_custom" {
value = "https://${local.os_custom_domain_name}"
}

output "opensearch_root_username" {
value = "os_root"
}

output "opensearch_root_user_password_secret_name" {
value = "/${var.environment}/${local.env_name}-root-password"
}
`

Creating OpenSearch Users

All that’s left now are users and indexes.

We will have two types of users:

regular users from the OpenSearch internal database — for our Backend API in Kubernetes (actually, we later switched to IAM Roles, which are mapped to the Backend via EKS Pod Identities)
and users (IAM Role) for Bedrock — there will be three Knowledge Bases, each with its own IAM Role, for which we will need to add an OpenSearch Role and map it to IAM roles

Let’s start with regular users.

Add a provider, in my case it is in the versions.tf file:

`
terraform {

required_version = "~> 1.6"

required_providers {
aws = {
source = "hashicorp/aws"
version = "~> 6.0"
}
opensearch = {
source = "opensearch-project/opensearch"
version = "~> 2.3"
}
}
}
`

In the providers.tf file, describe access to the cluster:

`
...

provider "opensearch" {
url = "https://${local.os_custom_domain_name}"
username = "os_root"
password = data.aws_ssm_parameter.os_master_password.value
healthcheck = false
}
`

Error: elastic: Error 403 (Forbidden)

Here is an important point about the url value in the provider configuration. I wrote about it above, and now I will show you how it looks.

First, in the provider.url, I set it as outputs of the module, i.e. module.opensearch.domain_endpoint.

Because of this, I got a 403 error when I tried to create users:

... opensearch_user.os_kraken_dev_user: Creating... opensearch_role.os_kraken_dev_role: Creating... ╷ │ Error: elastic: Error 403 (Forbidden) │ │ with opensearch_user.os_kraken_dev_user, │ on opensearch_users.tf line 23, in resource "opensearch_user" "os_kraken_dev_user": │ 23: resource "opensearch_user" "os_kraken_dev_user" { │ ╵ ╷ │ Error: elastic: Error 403 (Forbidden) │ │ with opensearch_role.os_kraken_dev_role, │ on opensearch_users.tf line 30, in resource "opensearch_role" "os_kraken_dev_role": │ 30: resource "opensearch_role" "os_kraken_dev_role" {

Thus, set the URL in the form of FQDN, which we did for Custom Endpoint, something like "url = https://opensearch.exmaple.com" - and everything works well.

Creating Internal Users

Now for the users themselves.

There will be three of them — dev, staging, prod, each with access to the corresponding index.

Here we will use opensearch_user.

If the cluster is created in VPC, a VPN connection is required so that the provider can connect to the cluster.

Add list() to the variables.tf with a list of environments:

`
...

variable "app_environments" {
type = list(string)
description = "The Application's environments, to be used to created Dev/Staging/Prod DynamoDB tables, etc"
}
`

And the value in prod.tfvars:

`
...

app_environments = [
"dev",
"staging",
"prod"
]
`

Internal database users

At first, I planned to just use local users, and wrote this option in this post — let it be. Next, I will show how we did it in the end — with IAM Users and IAM Roles.

In the file opensearch_users.tf, add three passwords, three users, and three roles to which we map users in loops - each role with access to its own index:

`
...

KRAKEN

resource "random_password" "os_kraken_password" {
for_each = toset(var.app_environments)
length = 16
special = true
}

store the root password in AWS Parameter Store

resource "aws_ssm_parameter" "os_kraken_password" {
for_each = toset(var.app_environments)

name = "/${var.environment}/${local.env_name}-kraken-${each.key}-password"
description = "OpenSearch cluster Backend Dev password"
type = "SecureString"
value = random_password.os_kraken_password[each.key].result
overwrite = true
tier = "Standard"

lifecycle {
ignore_changes = [value] # to prevent diff every time password is regenerated
}
}

Create a user

resource "opensearch_user" "os_kraken_user" {
for_each = toset(var.app_environments)

username = "os_kraken_${each.key}"
password = random_password.os_kraken_password[each.key].result
description = "Backend EKS ${each.key} user"

depends_on = [module.opensearch]
}

And a full user, role and role mapping example:

resource "opensearch_role" "os_kraken_role" {
for_each = toset(var.app_environments)

role_name = "os_kraken_${each.key}_role"
description = "Backend EKS ${each.key} role"

cluster_permissions = [
"indices:data/read/msearch",
"indices:data/write/bulk*",
"indices:data/read/mget*"
]
index_permissions {
index_patterns = ["kraken-kb-index-${each.key}"]
allowed_actions = ["*"]
}

depends_on = [module.opensearch]
}
`

In cluster_permissions, we add permissions that are required for both the index level and the cluster level, because Bedrock did not work without them, see Cluster wide index permissions.

Deploy and check in Dashboards:

Adding IAM Users

The idea here is the same, except that instead of regular users with a login:password for authentication, IAM and its Users && Roles are used.

More on the role for Bedrock later, but for now, let’s add user mapping.

What we need to do is take a list of our Backend team users, give them an IAM Policy with access to OpenSearch, and then add mapping to a local role in the OpenSearch internal users database.

For now, we can use the local role all_access, although it would be better to write our own later. See Predefined roles and About the master user.

Add a new variable to the variables.tf:

`
...

variable "backend_team_users_arns" {
type = list(string)
}
`

Its value in the prod.tfvars:

`
...

backend_team_users_arns = [
"arn:aws:iam::492*148:user/arseny",
"arn:aws:iam::492148:user/misha",
"arn:aws:iam::492148:user/oleksii",
"arn:aws:iam::492*148:user/vladimir",
"os_root"
]
`

Here, we had to mess around with the user os_root, because otherwise it would be removed from the role.

So, it’s better to make normal roles — but for MVP, it’s okay.

And we add the mapping of these IAM Users to the role all_access:

`
...

BACKEND TEAM

resource "opensearch_roles_mapping" "all_access_mapping" {
role_name = "all_access"

users = var.backend_team_users_arns
}
`

Deploy, check the all_access role:

Note : ChatGPT stubbornly insisted on adding IAM Users to Backend Roles, but no, and this is clearly stated in the documentation — you need to add them to Users, see Additional master users.

And for all the IAM Users we need to add an IAM policy with access.

Again, for MVP, we can simply take the AWS managed policy AmazonOpenSearchServiceFullAccess, which is connected to the IAM Group:

Creating AWS Bedrock IAM Roles and OpenSearch Role mappings

We already have Bedrock, now just need to create new IAM Roles and map them to OpenSearch Roles.

Add the iam.tf file - describe the IAM Role and IAM Policy (Identity-based Policy for access to OpenSearch), also in a loop for each of the var.app_environments:

MAIN ROLE FOR KNOWLEDGE BASE

grants permissions for AWS Bedrock to interact with other AWS services

resource "aws_iam_role" "knowledge_base_role" {
for_each = toset(var.app_environments)
name = "${var.project_name}-role-${each.key}-managed"
assume_role_policy = jsonencode({
Version = "2012-10-17"
Statement = [
{
Action = "sts:AssumeRole"
Effect = "Allow"
Principal = {
Service = "bedrock.amazonaws.com"
}
Condition = {
StringEquals = {
"aws:SourceAccount" = data.aws_caller_identity.current.account_id
}
ArnLike = {
# restricts the role to be assumed only by Bedrock knowledge base in the specified region
"aws:SourceArn" = "arn:aws:bedrock:${var.aws_region}:${data.aws_caller_identity.current.account_id}:knowledge-base/*"
}
}
}
]
})
}

IAM policy for Knowledge Base to access OpenSearch Managed

resource "aws_iam_policy" "knowledge_base_opensearch_policy" {
for_each = toset(var.app_environments)
name = "${var.project_name}-kb-opensearch-policy-${each.key}-managed"
policy = jsonencode({
Version = "2012-10-17"
Statement = [
{
Effect = "Allow"
Action = [
"es:",
]
Resource = [
module.opensearch.domain_arn,
"${module.opensearch.domain_arn}/"
]
}
]
})
}

resource "aws_iam_role_policy_attachment" "knowledge_base_opensearch" {
for_each = toset(var.app_environments)
role = aws_iam_role.knowledge_base_role[each.key].name
policy_arn = aws_iam_policy.knowledge_base_opensearch_policy[each.key].arn
}
`

Next, in the opensearch_users.tf, let's create:

opensearch_role: with cluster_permissions and index_permissions for each index
locals with all the IAM Roles we created above
and opensearch_roles_mapping for each opensearch_role.os_bedrock_roles, which we add to each opensearch_rolevia backend_roles.

It looks something like this:

`
...

BEDROCK

resource "opensearch_role" "os_bedrock_roles" {
for_each = toset(var.app_environments)
role_name = "os_bedrock_${each.key}_role"
description = "Backend Bedrock KB ${each.key} role"

cluster_permissions = [
"indices:data/read/msearch",
"indices:data/write/bulk*",
"indices:data/read/mget*"
]

index_permissions {
index_patterns = ["kraken-kb-index-${each.key}"]
allowed_actions = ["*"]
}

depends_on = [module.opensearch]
}

'aws_iam_role' is defined in iam.tf

locals {
knowledge_base_role_arns = {
for env, role in aws_iam_role.knowledge_base_role :
env => role.arn
}
}

resource "opensearch_roles_mapping" "os_bedrock_role_mappings" {
for_each = toset(var.app_environments)
role_name = opensearch_role.os_bedrock_roles[each.key].role_name

backend_roles = [
local.knowledge_base_role_arns[each.key]
]

depends_on = [module.opensearch]
}
`

Actually, this is where I encountered Bedrock access errors, which forced me to add cluster_permissions:

The knowledge base storage configuration provided is invalid… Request failed: [security_exception] no permissions for [indices:data/read/msearch] and User [name=arn:aws:iam::492*148:role/kraken-kb-role-dev, backend_roles=[arn:aws:iam::492*148:role/kraken-kb-role-dev], requestedTenant=null]

Deploy, check:

Creating OpenSearch indexes

The provider already exists, so we’ll take the opensearch_index resource.

In locals, we write the index template - I just took it from the developers from the old configuration:

`
locals {
# 'atlas-kb-prod'
env_name = "${var.project_name}-${var.environment}"
# 'opensearch.prod.example.co'
os_custom_domain_name = "opensearch.${var.dns_zone}"

# index mappings

os_index_mappings = <<-EOF
{
"dynamic_templates": [
{
"strings": {
"match_mapping_type": "string",
"mapping": {
"fields": {
"keyword": {
"ignore_above": 8192,
"type": "keyword"
}
},
"type": "text"
}
}
}
],
"properties": {
"bedrock-knowledge-base-default-vector": {
"type": "knn_vector",
"dimension": 1024,
"method": {
"name": "hnsw",
"engine": "faiss",
"parameters": {
"m": 16,
"ef_construction": 512
},
"space_type": "l2"
}
},
"AMAZON_BEDROCK_METADATA": {
"type": "text",
"index": false
},
"AMAZON_BEDROCK_TEXT_CHUNK": {
"type": "text",
"index": true
}
}
}
EOF
}
`

Create a file named opensearch_indexes.tf. Add the indexes themselves - here, I decided not to use a loop, but to create separate Dev/Staging/Prod files directly:

Dev Index

resource "opensearch_index" "kb_vector_index_dev" {
name = "kraken-kb-index-dev"

# enable approximate nearest neighbor search by setting index_knn to true
index_knn = true
index_knn_algo_param_ef_search = "512"
number_of_shards = "1"
number_of_replicas = "1"
mappings = local.os_index_mappings

# When new documents are ingested into the Knowledge Base,
# OpenSearch automatically creates field mappings for new metadata fields under
# AMAZON_BEDROCK_METADATA. Since these fields are created outside of TF resource definitions,
# TF detects them as configuration drift and attempts to recreate the index to match its
# known state.
#
# This lifecycle rule prevents unnecessary index recreation by ignoring mapping changes
# that occur after initial deployment.
lifecycle {
ignore_changes = [mappings]
}
}

...
`

Deploy, check:

That’s basically it.

Bedrock is already connected, everything is working.

But it took a little bit of effort.

And I’m sure it won’t be the last time :-)

Originally published at RTFM: Linux, DevOps, and system administration.

Terraform: using Ephemeral Resources and Write-Only Attributes

Arseny Zinchenko — Mon, 29 Dec 2025 10:00:00 +0000

Ephemeral resources and write-only arguments appeared in Terraform a long time ago, back in version 1.10, but there was no opportunity to write about them in detail.

The main idea behind them is not to leave “traces” in the state file, which is especially useful for passwords or tokens, because the data only exists during the execution of Terraform itself in its memory.

However, there are certain limitations to their use — we’ll look at those later, but first, let’s see everything in action.

Example without ephemeral values and write-only arguments
Using Write-Only Attributes
Using Ephemeral Resources
The “This output value is not declared as returning an ephemeral value” error
The “Ephemeral outputs are not allowed in context of a root module” error
Using values from Ephemeral resources
Using Ephemeral Outputs
Useful links

Example without ephemeral values and write-only arguments

Let’s start with the old scheme, without using ephemeral resources and write-only arguments — we will create a random password, the resource aws_secretsmanager_secret, store this password in it, and get it from data:

provider "aws" {
  region = "us-east-1"
  default_tags {
    tags = {
      component = "devops"
      created-by = "terraform"
      environment = "test"
    }
  }
}

### RESOURCES ###

# generate a random password
resource "random_password" "test_random_password" {
   length = 8
   special = false
}

# create an AWS Secret resource
resource "aws_secretsmanager_secret" "test_aws_secret" {
  name = "db_password"
  description = "database passsword"
  recovery_window_in_days = 0
}

# create an AWS Secret value
resource "aws_secretsmanager_secret_version" "test_aws_secret_version" {
  secret_id = aws_secretsmanager_secret.test_aws_secret.id
  secret_string = random_password.test_random_password.result
}

### DATA SOURCES ###

# retrieve the AWS Secret value
data "aws_secretsmanager_secret_version" "test_aws_secret_data" {
  secret_id = aws_secretsmanager_secret.test_aws_secret.id

  depends_on = [aws_secretsmanager_secret_version.test_aws_secret_version]
}

### OUTPUTS ###

# get the random password value
output test_random_password {
  value = random_password.test_random_password.result
  sensitive = true
}

# get the AWS Secret value
output "test_aws_secret" {
  value = data.aws_secretsmanager_secret_version.test_aws_secret_data.secret_string
  sensitive = true
}

Here we are:

resource "random_password": generate the password itself
resource "aws_secretsmanager_secret": create a new entry in AWS Secrets Manager
resource "aws_secretsmanager_secret_version": write the value from resource "random_password" to this Secret
data “aws_secretsmanager_secret_version”: get the value from AWS Secrets Manager
output “test_random_password”: output the value from resource ‘random_password’
output “test_aws_secret”: output the value obtained from AWS Secrets Manager

Execute terraform init and terraform apply:

...
Apply complete! Resources: 3 added, 0 changed, 0 destroyed.

Outputs:

test_aws_secret = <sensitive>
test_random_password = <sensitive>

Looks OK — in outputs, thanks to the sensitive = true, nothing is displayed.

But the password is in the state file:

$ cat terraform.tfstate
{
  ...
  "outputs": {
    "test_aws_secret": {
      "value": "1atcZYGR",
      "type": "string",
      "sensitive": true
    },
    "test_random_password": {
      "value": "1atcZYGR",
      "type": "string",
      "sensitive": true
    }
  },
...
  "resources": [
    {
      "mode": "data",
      "type": "aws_secretsmanager_secret_version",
      "name": "test_aws_secret_data",
      ...
            "secret_string": "1atcZYGR",
...
    {
      "mode": "managed",
      "type": "aws_secretsmanager_secret_version",
      "name": "test_aws_secret_version",
      ...
            "secret_string": "1atcZYGR",
...
    {
      "mode": "managed",
      "type": "random_password",
      "name": "test_random_password",
      ...
            "result": "1atcZYGR",

Now let’s start hiding this data from the state.

Using Write-Only Attributes

Resource attributes with the suffix _wo are "write-only" data, meaning that Terraform keeps them in memory during operations but does not store them anywhere.

However, not all resources support these attributes. For example, in AWS RDS, you can pass a password via the password_wo attribute of the aws_db_instance resource, but in aws_opensearch_domain and its master_user_password attribute to create a root user in the internal user database - not yet.

Official documentation — Use write-only arguments.

aws_secretsmanager_secret_version also supports write-only attributes - secret_string_wo instead of secret_string, and secret_string_wo_version instead of secret_string_version.

The use of secret_string_wo_version is mandatory for secret_string_wo, because since Terraform does not store password information, it will not know when to update it. To do this, we set a version that we increment each time we want to update the password.

Edit the code, change the only resource “aws_secretsmanager_secret_version” - set secret_string_wo and secret_string_wo_version, leaving the rest unchanged:

...
# create an AWS Secret value
resource "aws_secretsmanager_secret_version" "test_aws_secret_version" {
  secret_id = aws_secretsmanager_secret.test_aws_secret.id
  #secret_string = random_password.test_random_password.result
  secret_string_wo = random_password.test_random_password.result
  secret_string_wo_version = 1
}
...

Run terraform apply, and check the state now:

$ cat terraform.tfstate
{
  ...
  "outputs": {
    "test_aws_secret": {
      "value": "1atcZYGR",
      "type": "string",
      "sensitive": true
    },
    "test_random_password": {
      "value": "1atcZYGR",
      "type": "string",
      "sensitive": true
    }
  },
...
  "resources": [
    {
      "mode": "data",
      "type": "aws_secretsmanager_secret_version",
      "name": "test_aws_secret_data",
      ...
            "secret_string": "1atcZYGR",
...
    {
      "mode": "managed",
      "type": "aws_secretsmanager_secret_version",
      "name": "test_aws_secret_version",
      ...
            "secret_string": "",
            "secret_string_wo": null,
            "secret_string_wo_version": 1,

...
    {
      "mode": "managed",
      "type": "random_password",
      "name": "test_random_password",
      ...
            "result": "1atcZYGR",

Now we have managed.aws_secretsmanager_secret_version.test_aws_secret_version with no values for secret_string and secret_string_wo.

Using Ephemeral Resources

The idea behind “ephemeral” resources is the same as with write-only arguments — these resources only exist in Terraform’s memory during the execution of terraform apply and are not stored in the state file.

However, the use of such resources is limited:

you can refer to them in write-only arguments
in other ephemeral resources
in locals
in ephemeral variables
in providers, provisioners, and connections

Documentation — Ephemeral block reference.

Let’s edit our code and change resource “random_password” to ephemeral "random_password", leave resource “aws_secretsmanager_secret_version” - it will write the password to AWS Secrets Manager but will not store the value in state, and add a new resource - ephemeral “aws_secretsmanager_secret_version”, through which we will get this password back in Terraform.

At the same time, in the secret_string_wo and in output “test_random_password” we now refer to the password through ephemeral - ephemeral.random_password.test_random_password.result.

And in the output “test_aws_secret” we also use ephemeral.aws_secretsmanager_secret_version.test_aws_secret_data.secret_string.

The data "aws_secretsmanager_secret_version" can be removed, because we will now get the password from the ephemeral “aws_secretsmanager_secret_version”:

...

### RESOURCES ###

# generate a random password
ephemeral "random_password" "test_random_password" {
   length = 8
   special = false
}

# create an AWS Secret resource
resource "aws_secretsmanager_secret" "test_aws_secret" {
  name = "db_password"
  description = "database passsword"
  recovery_window_in_days = 0
}

# create an AWS Secret value
resource "aws_secretsmanager_secret_version" "test_aws_secret_version" {
  secret_id = aws_secretsmanager_secret.test_aws_secret.id
  #secret_string = random_password.test_random_password.result
  secret_string_wo = ephemeral.random_password.test_random_password.result
  secret_string_wo_version = 1
}

### DATA SOURCES ###

# Retrieve the password from Secrets Manager (ephemeral)
ephemeral "aws_secretsmanager_secret_version" "test_aws_secret_version_ephemeral" {
  secret_id = aws_secretsmanager_secret.test_aws_secret.id
}

# retrieve the AWS Secret value
# data "aws_secretsmanager_secret_version" "test_aws_secret_data" {
# secret_id = aws_secretsmanager_secret.test_aws_secret.id

# depends_on = [aws_secretsmanager_secret_version.test_aws_secret_version]
# }

### OUTPUTS ###

# get the random password value
output test_random_password {
  value = ephemeral.random_password.test_random_password.result
  sensitive = true
}

# get the AWS Secret value
output "test_aws_secret" {
  value = ephemeral.aws_secretsmanager_secret_version.test_aws_secret_version_ephemeral.secret_string
  sensitive = true
}

The “This output value is not declared as returning an ephemeral value” error

Execute terraform apply and catch the first error:

...
│ Error: Ephemeral value not allowed
│ 
│ on main.tf line 53, in output "test_random_password":
│ 53: value = ephemeral.random_password.test_random_password.result
│ 
│ This output value is not declared as returning an ephemeral value, so it cannot be set to a result derived from an ephemeral value.
╵
╷
│ Error: Ephemeral value not allowed
│ 
│ on main.tf line 59, in output "test_aws_secret":
│ 59: value = ephemeral.aws_secretsmanager_secret_version.test_aws_secret_version_ephemeral.secret_string
│ 
│ This output value is not declared as returning an ephemeral value, so it cannot be set to a result derived from an ephemeral value.

But even if we add the parameter ephemeral = true:

...
### OUTPUTS ###

# get the random password value
output test_random_password {
  value = ephemeral.random_password.test_random_password.result
  sensitive = true
  ephemeral = true
}

# get the AWS Secret value
output "test_aws_secret" {
  value = ephemeral.aws_secretsmanager_secret_version.test_aws_secret_version_ephemeral.secret_string
  sensitive = true
  ephemeral = true
}

It still won’t work.

The “Ephemeral outputs are not allowed in context of a root module” error

Now the error will look like this:

...
╷
│ Error: Ephemeral output not allowed
│ 
│ on main.tf line 52:
│ 52: output test_random_password {
│ 
│ Ephemeral outputs are not allowed in context of a root module
╵
╷
│ Error: Ephemeral output not allowed
│ 
│ on main.tf line 59:
│ 59: output "test_aws_secret" {
│ 
│ Ephemeral outputs are not allowed in context of a root module

Because Ephemeral outputs can only be used in modules — we’ll see how later.

OK — for now, let’s just remove Outputs, and now terraform apply runs without any problems:

$ terraform apply
...
random_password.test_random_password: Refreshing state... [id=none]
ephemeral.random_password.test_random_password: Opening...
ephemeral.random_password.test_random_password: Opening complete after 0s
...
ephemeral.aws_secretsmanager_secret_version.test_aws_secret_version_ephemeral: Opening...
...
ephemeral.random_password.test_random_password: Closing...
ephemeral.random_password.test_random_password: Closing complete after 0s
...

Please note that for ephemeral resources, Terraform now performs Opening and Closing operations instead of Reading and Refreshing state. That is, it simply creates an object in memory, reads the resource into it, and then “closes” and removes it from memory.

Let’s check the state file now:

...
    {
      "mode": "managed",
      "type": "aws_secretsmanager_secret_version",
      "name": "test_aws_secret_version",
      ...
            "secret_string": "",
            "secret_string_wo": null,
            "secret_string_wo_version": 1,
...

Now we have:

resources ephemeral “random_password” and ephemeral “aws_secretsmanager_secret_version” are not in the state at all,
and managed.aws_secretsmanager_secret_version.test_aws_secret_version still has an empty field in secret_string_wo because we made it write-only earlier

OK, but how do we use the password now? Because we removed data “aws_secretsmanager_secret_version”.

Using values from Ephemeral resources

We have already seen an example of referencing Ephemeral resources above when we did secret_string_wo = ephemeral.random_password.test_random_password.result.

Similarly, we can use ephemeral.aws_secretsmanager_secret_version.db_password_wo_ephemeral.secret_string.

As mentioned above, we cannot do this everywhere, but it is allowed in providers.

To verify this, let’s run PostgreSQL with our password (we’ll take it directly from AWS Console > AWS Secrets Manager):

Launch a container, to which we pass the variable POSTGRES_PASSWORD="1atcZYGR":

$ docker run --rm --name some-postgres -e POSTGRES_PASSWORD="1atcZYGR" -p 5432:5432 postgres

Add the provider to our code and use it to connect to the container, where we will create a test database.

In the provider’s password field we will use a value from the ephemeral.aws_secretsmanager_secret_version.test_aws_secret_version_ephemeral.secret_string:

...

### PostgreSQL Configuration

terraform {
  required_providers {
    postgresql = {
      source = "cyrilgdn/postgresql"
      version = "~> 1.20"
    }
  }
}

provider "postgresql" {
  host = "localhost"
  port = 5432
  username = "postgres"
  password = ephemeral.aws_secretsmanager_secret_version.test_aws_secret_version_ephemeral.secret_string
  sslmode = "disable"
}

resource "postgresql_database" "demo_db" {
  name = "demo_db"
  template = "template0"
  connection_limit = -1
  allow_connections = true
}

Run terraform init and terraform apply:

$ terraform init && terraform apply
...
ephemeral.aws_secretsmanager_secret_version.test_aws_secret_version_ephemeral: Opening...
ephemeral.aws_secretsmanager_secret_version.test_aws_secret_version_ephemeral: Opening complete after 1s
postgresql_database.demo_db: Creating...
postgresql_database.demo_db: Creation complete after 0s [id=demo_db]
ephemeral.aws_secretsmanager_secret_version.test_aws_secret_version_ephemeral: Closing...
ephemeral.aws_secretsmanager_secret_version.test_aws_secret_version_ephemeral: Closing complete after 0s

Apply complete! Resources: 1 added, 0 changed, 0 destroyed.

Check the database:

$ export PGPASSWORD="1atcZYGR"
$ psql -h localhost -U postgres -c "\l"
                                                    List of databases
   Name | Owner | Encoding | Locale Provider | Collate | Ctype | Locale | ICU Rules | Access privileges   
-----------+----------+----------+-----------------+------------+------------+--------+-----------+-----------------------
 demo_db | postgres | UTF8 | libc | en_US.utf8 | en_US.utf8 | | | 
...

In the same way, we could use an ephemeral resource via locals:

...
locals {
  db_password_local = ephemeral.aws_secretsmanager_secret_version.test_aws_secret_version_ephemeral.secret_string
}

provider "postgresql" {
  host = "localhost"
  port = 5432
  username = "postgres"
  password = local.db_password_local
  #password = ephemeral.aws_secretsmanager_secret_version.test_aws_secret_version_ephemeral.secret_string
  sslmode = "disable"
}

resource "postgresql_database" "demo_db" {
  name = "demo_db_via_local"
  template = "template0"
  connection_limit = -1
  allow_connections = true
}

Check:

$ terraform apply
...
  # postgresql_database.demo_db will be updated in-place
  ~ resource "postgresql_database" "demo_db" {
        id = "demo_db"
      ~ name = "demo_db" -> "demo_db_via_local"
        # (10 unchanged attributes hidden)
    }
...
Apply complete! Resources: 0 added, 1 changed, 0 destroyed.

And in the state file, the password is not visible anywhere:

$ cat terraform.tfstate | grep 1atcZYGR | echo $?
127

Using Ephemeral Outputs

Above, we tried to use output “test_aws_secret” with ephemeral = true, but got the error "Ephemeral outputs are not allowed in context of a root module".

Let’s try using it in our own module.

Documentation — ephemeral — Avoid storing values in state or plan files.

Let’s create a module modules/secret_ephemeral, in which we will generate a password and save it in AWS Secrets Manager, and add Ephemeral Output.

And in the root module, we will use outputs of this module to get ephemeral “aws_secretsmanager_secret_version”, as we did above.

Let’s write the file modules/secret_ephemeral/secret.tf:

### RESOURCES ###

# generate a random password
ephemeral "random_password" "test_random_password" {
   length = 8
   special = false
}

# create an AWS Secret resource
resource "aws_secretsmanager_secret" "test_aws_secret" {
  name = "db_password_via_module"
  description = "database passsword"
  recovery_window_in_days = 0
}

# create an AWS Secret value
resource "aws_secretsmanager_secret_version" "test_aws_secret_version" {
  secret_id = aws_secretsmanager_secret.test_aws_secret.id
  #secret_string = random_password.test_random_password.result
  secret_string_wo = ephemeral.random_password.test_random_password.result
  secret_string_wo_version = 1
}

# Retrieve the password from Secrets Manager (ephemeral)
ephemeral "aws_secretsmanager_secret_version" "test_aws_secret_version_ephemeral" {
  secret_id = aws_secretsmanager_secret.test_aws_secret.id
}

output "password_ephemeral" {
  value = ephemeral.aws_secretsmanager_secret_version.test_aws_secret_version_ephemeral.secret_string
  ephemeral = true
}

In the main file main.tf, remove everything related to the password, add a module call, and in locals use its output:

...

### PostgreSQL Configuration

terraform {
  required_providers {
    postgresql = {
      source = "cyrilgdn/postgresql"
      version = "~> 1.20"
    }
  }
}

module "secret_ephemeral" {
  source = "./modules/secret_ephemeral"
}

locals {
  db_password_local = module.secret_ephemeral.password_ephemeral
}

provider "postgresql" {
  host = "localhost"
  port = 5432
  username = "postgres"
  password = local.db_password_local
  #password = ephemeral.aws_secretsmanager_secret_version.test_aws_secret_version_ephemeral.secret_string
  sslmode = "disable"
}

resource "postgresql_database" "demo_db" {
  name = "demo_db_via"
  template = "template0"
  connection_limit = -1
  allow_connections = true
}

First, you need to create a password — run terraform apply without resource “postgresql_database”, and update the container launch with the new password:

$ docker run --rm --name some-postgres -e POSTGRES_PASSWORD="PHsfzcIx" -p 5432:5432 postgres

Now our provider uses a password from the Ephemeral Output module modules/secret_ephemeral:

...
module.secret_ephemeral.ephemeral.aws_secretsmanager_secret_version.test_aws_secret_version_ephemeral: Opening...
module.secret_ephemeral.ephemeral.aws_secretsmanager_secret_version.test_aws_secret_version_ephemeral: Opening complete after 1s
postgresql_database.demo_db: Creating...
postgresql_database.demo_db: Creation complete after 0s [id=demo_db_via]
module.secret_ephemeral.ephemeral.aws_secretsmanager_secret_version.test_aws_secret_version_ephemeral: Closing...
module.secret_ephemeral.ephemeral.aws_secretsmanager_secret_version.test_aws_secret_version_ephemeral: Closing complete after 0s

Apply complete! Resources: 1 added, 0 changed, 0 destroyed.

In the state, we still don’t have a password:

$ cat terraform.tfstate | grep PHsfzcIx | echo $?
127

That’s basically it.

It’s a shame that aws_opensearch_domain doesn't support write-only. I wanted to use it for the root password :-(

But there is already an issue on GitHub Support ephemeral “write-only” argument for aws_opensearch_domain, and even a comment saying “I have started working on this issue, and will submit a PR shortly”.

And in the pull request itself, you can even see how it’s implemented.

Useful links

Originally published at RTFM: Linux, DevOps, and system administration.

Golang: interfaces - the “magic” of calling methods through interface

Arseny Zinchenko — Thu, 04 Dec 2025 09:25:13 +0000

Golang: interfaces - the “magic” of calling methods through interface

Interfaces in Go allow you to describe access to data or behavior without providing concrete implementations inside the interface itself.

In this way, we create a “common bus” that we can then use to “connect” external “systems.”

In other words, an interface is an abstraction that defines a contract. The contract describes what can be done, but the concrete type decides how it is done.

Originally, I planned to write a post about using interfaces in practice, but it naturally evolved into a more in-depth look at how they are implemented internally and what actually happens during a method call through an interface.

It would probably make sense to start with pointers and methods in Go, since this topic heavily relies on them, but that is material for another post.

See also the following related article Golang: interfaces, types, and methods using the example of io.Copy() (TODO: add translation, in Ukrainian for now).

Empty interfaces

The simplest form of an interface is one that does not specify any methods.

Using such an interface, we can create a function that accepts a value of any type. Otherwise, we would have to explicitly specify the parameter type:

package main

import "fmt"

// define an empty interface
// it can hold a value of any type
type Any interface{}

func printValue(v Any) {
  // print the value
  fmt.Println("Value:", v)
}

func main() {
  // pass int
  printValue(42)
  // pass string
  printValue("hello")
  // pass float
  printValue(3.14)
  // pass slice
  printValue([]int{1, 2, 3})
}

Without an interface, we would have to create separate functions for each type or use generics instead.

Another option is to use the any type, which is just an alias for interface{}:

...
type MyAny any

func printValue(v Any) {
  // print the value
  fmt.Println("Value:", v)
}
...

Interfaces and methods

If the empty interface any says "I accept any value", then a classic interface says "I am only interested in specific behavior".

This behavior is described via a set of method signatures that a type must implement in order to satisfy the interface.

When a value is assigned to an interface variable, or passed to a function that expects an interface, Go checks whether the underlying type implements all required methods.

From that point on, we can call the methods through the interface.

In other words, an interface acts as an intermediary that allows you to call a method regardless of the concrete type.

For example:

package main

import "fmt"

// define an interface 'MyInterface' with a single method 'MyMethod' returning a string
type MyInterface interface {
  MyMethod() string
}

// define 'MyStruct' struct with a 'MyField' field
type MyStruct struct {
  MyField string
}

// define 'MyMethod' method for the 'MyStruct' struct
// this makes 'MyStruct' implicitly implement 'MyInterface'
// 'MyMethod' method uses 'MyStruct' as the receiver, so this method is tied to the 'MyStruct' type
func (receiver MyStruct) MyMethod() string {
  return "Executing " + receiver.MyField
}

// define a function 'sayHello()' which accepts any type that implements 'MyInterface'
// and prints the value returned by its 'MyMethod'
func sayHello(g MyInterface) {
  fmt.Println(g.MyMethod())
}

func main() {
  // create an instance of MyStruct
  myObj := MyStruct{MyField: "Hello, Interface!"}

  // pass the MyStruct instance to the function.
  // this works because MyStruct implements MyInterface.
  sayHello(myObj)
}

In this example:

declare our own interface type with the name MyInterface
this interface describes one method signature - MyMethod(), and this method should return data of type string
create your own data type MyStruct with type struct, which has one field MyField with type string
“bind” the MyMethod() function to the struct by specifying a receiver (receiver MyStruct), which makes MyStruct implement MyInterface
describe our “main working” function sayHello(), which takes the interface as an argument and calls the method MyMethod(), which is in this interface
create an instance of our data type MyStruct, in which we write the value "Hello, Interface!" in the field MyField
and call our working function, passing this structure as an argument

I tried to make the relationships between all the parts more explicit, since they are not always obvious:

create an object myObj with type MyStruct
call sayHello(), passing the argument myObj, which inside the function sayHello () becomes the variable g, which is associated with our interface MyInterface, which provides access to the method MyMethod()
in the sayHello() function, by calling g.MyMethod(), we refer to the MyInterface interface, saying "I need your MyMethod() method"
MyInterface interface "sees" that a MyStruct value is currently stored inside it, and redirects the call to that concrete implementation

Okay, now the picture is becoming clearer.

Except for one thing - how does the interface “see” that it contains the object myObj with the method MyMethod()?

The interface “magic”: the `iface` structure

To understand this, let’s look at how Go represents an interface in memory and recreate a simplified version of that structure.

Then, using pointers, we can inspect the actual memory addresses and stored values.

Let’s “slightly” modify the original code:

package main

import (
  "fmt"
  "unsafe"
)

// define MyInterface interface
// (same as before)
type MyInterface interface {
  MyMethod() string
}

// define MyStruct struct
// (same as before)
type MyStruct struct {
  MyField string
}

// define MyMethod with a POINTER receiver
// - before: func (p MyStruct) MyMethod() ... // by value
// - now: func (p *MyStruct) MyMethod() ... // by pointer
// this means the method operates on the original data in memory
func (p *MyStruct) MyMethod() string {
  return "Executing " + p.MyField
}

// this helper struct is used to inspect an interface value
// it represents the internal memory layout of an interface variable
// 'tab' points to a table with information about the interface and its methods
//
// type iface struct {
// // pointer to the 'itab' struct, see below
// tab *itab
// // here will be a pointer to the concrete value (myObj)
// data unsafe.Pointer
// }
//
// type itab struct {
// // pointer to the 'type MyInterface interface'
// inter *interfacetype
// // pointer to the 'type MyStruct struct'
// typ *rtype
// // in our case we have 1 method, thus '[N]uintptr' == [1]uintptr
// // fun[0] will contain the address of the 'MyMethod' implementation
// fun [N]uintptr // will have '[1]uintptr', and
// }
type ifaceStruct struct {
  // Pointer to type/method info table
  tab unsafe.Pointer
  // Pointer to the actual data
  // in our case, here will be a pointer to the 'myObj' struct
  data unsafe.Pointer
}

// HERE IS THE "MAGIC"
// We modify sayHello to inspect the `g` it receives.
//
// 'g' is a new, local variable of the 'MyInterface' type.
// when the function is called, `myObj` is assigned to `g`.
//
// because 'g' is an interface value, it internally consists of two pointers:
// 1. tab: A pointer to the "interface table" (itab) that links
// the interface type (MyInterface) to the concrete type (*MyStruct)
// and stores pointers to the methods that satisfy the interface
// 2. data: A pointer to the actual data. In our case, this will be
// the pointer we passed in (`myObj`).
func sayHello(g MyInterface) {
  fmt.Println("Inside sayHello()")

  // Get the address of `g` and cast it to our helper struct 'ifaceStruct'

  // This line does three things in one go:
  // 1. &g - takes the memory address of our interface variable `g`
  // 2. unsafe.Pointer(&g) - casts that address to a raw, untyped pointer
  // 3. (*ifaceStruct)(...) - re-interprets that raw pointer as a pointer to our helper struct

  // As a result, `g_internal` is now a `*ifaceStruct` that points to
  // the exact same memory location as `g`, letting us access its .tab and .data fields.
  g_internal := (*ifaceStruct)(unsafe.Pointer(&g))

  fmt.Printf("Internal 'Type' pointer (tab): %p\n", g_internal.tab)
  fmt.Printf("Internal 'Data' pointer (data): %p\n", g_internal.data)
  fmt.Println("Result:", g.MyMethod())
}

func main() {
  // Create the object and get a pointer to it
  // 'myObj' now holds a pointer to a MyStruct instance in memory
  myObj := &MyStruct{MyField: "Hello, Interface!"}

  // Print location of the 'myObj' struct
  fmt.Println("Inside main()")
  fmt.Printf("Address of the original 'myObj' in main(): %p\n", myObj)

  // Pass the pointer to the function
  // i.e. we pass an address of the 'myObj' struct location
  sayHello(myObj)
}

Note: in modern Go, itab was moved to the ABI package, see src/internal/abi/iface.go.

Run the code:

$ go run interface-details.go
Inside main()
Address of the original 'myObj' in main(): 0xc000014070
Inside sayHello()
Internal 'Type' pointer (tab): 0x4e5a28
Internal 'Data' pointer (data): 0xc000014070
Result: Executing Hello, Interface!

The comments above describe most of the details, but in short:

in the sayHello(myObj) call we pass the address "0xc000014070" to the function sayHello() - a reference to the structure MyStruct with the field MyField, which contains the value "Hello, Interface! "
sayHello() function takes an argument of type interface, and the variable g contains two pointers: tab (to the structure itab, which stores information about the type and methods), and data (to the value of type MyStruct)

We can try to visualize it like this:

MyInterface (variable 'g')
+----------------------------------------+
| tab => itab(MyInterface, *MyStruct)
| data => &MyStruct{"Hello, Interface!"}
+----------------------------------------+

And the most interesting “magic” happens at compile time, when the itab structure is created:

Go checks the methods in the code, finds the structure MyStruct with the method MyMethod()
checks the interfaces and finds MyInterface, which requires the method MyMethod() string
checks that MyStruct.MyMethod() and MyInterface.MyMethod() match
creates an interface table (itab - interface table) that links MyStruct to MyInterface and stores the addresses of the methods that implement the interface

And then, when executing the program during the call sayHello(myObj), Go creates a new variable g of type iface, in which these two pointers (tab and data) are combined:

the pointer to itab (which the compiler created for the pair MyStruct + MyInterface) will be placed in g.tab.
a pointer to myObj (i.e., an address of type "0xc000014070") will be placed in g.data.

As a result, in g.tab we will have the structure itab, and in the field fun[0] will be the address of the function MyMethod(), and in g.data will contain a pointer to an instance of MyStruct with the field MyField.

And then when calling:

...
fmt.Println("\nResult:", g.MyMethod())
...

We are actually making a call like:

...
return "Executing " + *MyStruct.MyField
...

Finally, we can also inspect the itab itself by defining a helper our own structure itabStruct, similar to how we inspected the interface value:

package main

import (
  "fmt"
  "unsafe"
)

// define MyInterface interface
type MyInterface interface {
  MyMethod() string
}

// define MyStruct struct
type MyStruct struct {
  MyField string
}

// define MyMethod method with a POINTER receiver
func (p *MyStruct) MyMethod() string {
  return "Executing " + p.MyField
}

// This helper represents the interface value itself (the 2-word struct)
type ifaceStruct struct {
  // Pointer to the 'itab' (interface table)
  tab unsafe.Pointer
  // Pointer to the actual data (our *MyStruct)
  data unsafe.Pointer
}

// NEW
// this helper represents the internal 'runtime.itab' structure
type itabStruct struct {
  // inter: Pointer to the interface type's definition (MyInterface)
  inter unsafe.Pointer
  // typ: Pointer to the concrete type's definition (*MyStruct)
  typ unsafe.Pointer
  // hash: hash of the concrete type, used internally for lookups
  hash uint32
  // _ [4]byte: Padding (on 64-bit systems)
  _ [4]byte
  // fun: method dispatch table - an array of function pointers
  // each entry corresponds to a method defined in the interface
  // here it stores the address of the concrete MyMethod implementation
  fun [1]uintptr
}

// HERE IS THE "MAGIC"
func sayHello(g MyInterface) {
  fmt.Println("--- Inside sayHello() ---")

  // 1. Get the address of `g` and cast it to our helper struct
  g_internal := (*ifaceStruct)(unsafe.Pointer(&g))

  // Print the two main pointers
  fmt.Printf("g.tab (pointer to itab): %p\n", g_internal.tab)
  fmt.Printf("g.data (pointer to myObj): %p\n", g_internal.data)

  // NEW - 2. DE-REFERENCE THE 'tab' POINTER
  // Cast the 'tab' pointer to our itabStruct pointer
  itab_ptr := (*itabStruct)(g_internal.tab)

  // NEW - 3. PRINT THE CONTENTS OF THE 'itab'
  fmt.Println("\n--- Inspecting the 'itab' (at address g.tab) ---")
  fmt.Printf("itab.inter (ptr to MyInterface info): %p\n", itab_ptr.inter)
  fmt.Printf("itab.typ (ptr to *MyStruct info): %p\n", itab_ptr.typ)
  fmt.Printf("itab.hash (hash of *MyStruct type): %x\n", itab_ptr.hash)

  // This is the final link in the chain: 
  // the actual memory address of the method that will be invoked when g.MyMethod() is called.
  fmt.Printf("itab.fun[0] (ADDRESS OF THE METHOD): 0x%x\n", itab_ptr.fun[0])

  // 4. Call the method as usual
  fmt.Println("\nResult:", g.MyMethod())
}

func main() {
  // Create the object and get a pointer to it
  myObj := &MyStruct{MyField: "Hello, Interface!"}

  fmt.Println("--- Inside main() ---")
  fmt.Printf("Address of original 'myObj' in main(): %p\n", myObj)

  // Pass the pointer to the function - Go will create an 'iface' value
  // linking the interface 'MyInterface' with the concrete type *MyStruct.
  sayHello(myObj)
}

Result:

$ go run interface-details-3.go
--- Inside main() ---
Address of original 'myObj' in main(): 0xc00019a020
--- Inside sayHello() ---
g.tab (pointer to itab): 0x4e6c08
g.data (pointer to myObj): 0xc00019a020

--- Inspecting the 'itab' (at address g.tab) ---
itab.inter (ptr to MyInterface info): 0x4a9d80
itab.typ (ptr to *MyStruct info): 0x4a86e0
itab.hash (hash of *MyStruct type): 1ac3179f
itab.fun[0] (ADDRESS OF THE METHOD): 0x499c40

Result: Executing Hello, Interface!

That is, when we call g.MyMethod(), Go takes the function address from itab.fun[0] and invokes it, passing it a pointer from g.data as an argument - that's all there is to the "magic" of dynamic method calls via an interface.

Now you can use the interfaces, already having an idea of how they work.

Useful links

Originally published at RTFM: Linux, DevOps, and system administration.

Golang: creating OpenAI Exporter for VictoriaMetrics

Arseny Zinchenko — Sun, 23 Nov 2025 11:27:50 +0000

Got a new task to monitor costs on OpenAI - to see how much each project spends per day, and send alerts to Slack if costs exceed a set threshold.

I tried several existing exporters for the OpenAI API, but none provided cost metrics per project - so we’ll build one ourselves.

To create the exporter, let’s use Golang. The idea is simple: get data from the OpenAI API, generate metrics, and send them to VictoriaMetrics instance.

I last used Go in 2019 (just a single time), so we’ll refresh how things work and occasionally dive into the internals of some libraries.

So, the primary goal of this post is to demonstrate how to build an API client in Go, work with Go structs and JSON, and use the VictoriaMetrics Go client - including a look at how it works under the hood - to generate metrics and push them into a VictoriaMetrics instance for further use in VMAlert.

Also, it’s worth checking out the VictoriaMetrics blog - their team has been working with Go for many years and offers a lot of in-depth, practical Go content.

Let’s go.

OpenAI API

OpenAI API documentation - Costs, and its returned value - Costs object.

To access Costs, you need a separate key - create it at platform.openai.com in Admin keys:

To obtain Costs, you need to set the start_time parameter in Unix format. Let's set it to a variable:

$ TODAY=$(date -d "$(date +%Y-%m-%d) 00:00:00" +%s)
$ echo $TODAY
1762898400

And check access with curl:

$ curl -s -H "Content-Type: application/json" -H "Authorization: Bearer $OPENAI_ADMIN_KEY" "https://api.openai.com/v1/organization/costs?start_time=$TODAY"
{
  "object": "page",
  "has_more": false,
  "next_page": null,
  "data": [
    {
      "object": "bucket",
      "start_time": 1762819200,
      "end_time": 1762905600,
      "results": [
        {
          "object": "organization.costs.result",
          "amount": {
            "value": 5.65750295,
            "currency": "usd"
...

OK, access is working - let’s go to Golang.

Golang gore REPL

I prefer working in the console, and since my primary language is Python - I really like being able to run quick experiments using terminal-based tools.

For Go, we can use the gore package:

$ go install github.com/x-motemen/gore/cmd/gore@latest

Run (don’t forget to add $GOPATH/bin to the $PATH) and check that you are getting the current date and time:

$ gore
gore version 0.6.1 :help for help
gore> :import time
gore> time.Now()
%!t(int64=1763025027)2025-11-13 11:10:27 Local

Or just use Go Playground.

Creating a Golang API client

First, we will write a client that will call the OpenAI API and display the result on the console, and then we will add VictoriaMetrics metric generation.

What we need for API requests:

have a URL
have time
for now, we’ll just output the result to the console

Create a project directory and perform initialization:

$ mkdir ~/Work/atlas-monitoring/exporters/openai-exporter
$ go mod init openai-exporter

For the API client, we can use the standard library net/http, or more specialized types such as resty or sling.

I decided to try resty because it's interesting, the code looks nicer, and it has a nice way to pass parameters.

Documentation on resty - here>>> and here>>>.

Restyu version 3 is already available, but it is still in beta, so we will take the second one.

Let’s try with resty first in gore:

gore> :import "github.com/go-resty/resty/v2"
gore> client := resty.New()
...
gore> resp, err := client.R().Get("https://httpbin.org/get")
...
gore> fmt.Println(resp, err)
...
313
nil

To execute API requests, first call the New() method to create an object of the type Client struct, and then use the R() (request) method to make calls.

Documentation for New() here>>>, its code here>>>.

In Go, types don’t explicitly list their methods in the source, but all associated methods are clearly visible through go doc:

$ go doc github.com/go-resty/resty/v2.Client | grep "New()\|R()"
func New() *Client
func (c *Client) R() *Request

The Usage section contains the following example:

...
resp, err := client.R().
    EnableTrace().
    Get("https://httpbin.org/get")
...

Where resty uses method chaining, when methods of a certain type return the same type.

How it looks:

with the resty.New() function, we create a client - New() returns *Client struct with its associated methods.
for Client struct, there is a method R() that returns *Request struct.
for the Request structure, we have the EnableTrace() method, which also returns Request.
and for the same Request we have the Get() method, which also returns Request plus error

This allows us to build chained calls such as: Client => R() => Request => EnableTrace() => Request => Get().

Okay, let’s get to the code.

Creating a `resty` client

Create a main.go file:

package main

import (
  "fmt"

  "github.com/go-resty/resty/v2"
)

// set global const as ay be used in other packages
const (
  baseURL = "https://api.openai.com/v1"
  costsPath = "/organization/costs"
)

func main() {
  client := resty.New()

  // build 'https://api.openai.com/v1/organization/costs'
  response, err := client.R().Get(baseURL + costsPath)
  if err != nil {
    panic(err)
  }

  fmt.Println(response)
}

Run it to test:

$ go run main.go
{
  "error": {
    "message": "You didn't provide an API key. You need to provide your API key in an Authorization header using Bearer auth (i.e. Authorization: Bearer YOUR_KEY). You can obtain an API key from https://platform.openai.com/account/api-keys."
...

Great, it works.

Now let’s add getting the API key from a variable.

Use os:

...
import (
  "fmt"
  "os"
...
func main() {
  client := resty.New()

  apiKey = os.Getenv("OPENAI_ADMIN_KEY")
...

Next, we need to add an auth header to our request, this can be done using the func (*Client) SetAuthToken method, which simply adds a value to the Token field in the Client.

There is also a separate method func (r *Request) SetAuthToken, which sets the token for specific requests rather than for the entire client, but in our case, we make it simpler by using the general Client.

Let’s do method chaining from the example above - for Client we call SetAuthToken(), which sets the token, then call R() to create a request, and then call Get(), to which we pass the URL:

...
...
  apiKey := os.Getenv("OPENAI_ADMIN_KEY")

  // build 'https://api.openai.com/v1/organization/costs'
  response, err := client.SetAuthToken(apiKey).R().Get(baseURL + costsPath)
...

Let’s check:

$ go run main.go
{
  "error": {
    "message": "Missing query parameter 'start_time'",
...

OK, we’ve passed authentication, now we need to add the parameters.

Here we have four options:

func (r *Request) SetQueryParam(param, value string) *Request: sets one parameter key=value, can be used if there are only 1–2 parameters in total
func (r *Request) SetQueryParams(params map[string]string) *Request: similar, but accepts a map with a list of parameters
func (r *Request) SetQueryParamsFromValues(params url.Values) *Request: if the http library is used, parameters can be passed via the url.Values type
func (r *Request) SetQueryString(query string) *Request: passes a ready list of parameters in a single string, for example - SetQueryString("a=1&b=2")

Right now, we only need start_time, but we will be adding more parameters later, so we can write them all to a map, which we will then pass to a SetQueryParams() call.

For start_time, we need to pass the time - do this with time.Now().

The time to the OpenAI API must be provided in the Unix format, so add the Unix() function.

Let’s check how it will look in the gore session:

gore> :import time
gore> timeNow := time.Now().Unix()
1762956432

Now add a code to create the timeNow variable with the time, create setQueryParams map of strings with a list of parameters also in strings, and add the SetQueryParams() call to the client:

...
  timeNow := time.Now().Unix()

  setQueryParams := map[string]string{
    "start_time": timeNow,
  }

  // build 'https://api.openai.com/v1/organization/costs'
  response, err := client.SetAuthToken(apiKey).
    R().SetQueryParams(setQueryParams).
    Get(baseURL + costsPath)
...

But if you call this code now, you’ll get an error because timeNow := time.Now().Unix() returns an int64:

gore> fmt.Printf("%t", timeNow)
%!t(int64=1762957173)21
nil

And in the setQueryParams(), we need to pass a string because SetQueryParams() accepts a map with a string:

func (r *Request) SetQueryParams(params map[string]string) *Request

Therefore, convert our variable timeNow to a string using strconv.FormatInt():

gore> :import strconv
gore> s := strconv.FormatInt(timeNow, 10)
gore> fmt.Printf("%t", s)
%!t(string=1763371451)22

And now, our variable timeNow looks like this:

...
timeNow := strconv.FormatInt(time.Now().Unix(), 10)
...

Run, and check the result:

$ go run main.go
{
  "object": "page",
  "has_more": false,
  "next_page": null,
  "data": [
    {
      "object": "bucket",
      "start_time": 1762905600,
      "end_time": 1762992000,
      "results": [
        {
          "object": "organization.costs.result",
          "amount": {
            "value": 6.442440250000003,
            "currency": "usd"
          },
          "line_item": null,
          "project_id": null,
          "organization_id": "org-ORG"
...

Great, we have the data we need.

Now we have to add one more parameter - group_by=project_id:

...
  setQueryParams := map[string]string{
    "start_time": timeNow,
    "group_by": "project_id",
  }
...

And after this, we have data for each project_id in the results:

$ go run main.go
{
  "object": "page",
  "has_more": false,
  "next_page": null,
  "data": [
    {
      "object": "bucket",
      "start_time": 1762905600,
      "end_time": 1762992000,
      "results": [
        {
          "object": "organization.costs.result",
          "amount": {
            "value": 1.76643575,
            "currency": "usd"
          },
          "line_item": null,
          "project_id": "proj_1",
          "organization_id": "org-ORG"
        },
        {
          "object": "organization.costs.result",
          "amount": {
            "value": 0.47790999999999995,
            "currency": "usd"
          },
          "line_item": null,
          "project_id": "proj_2",
          "organization_id": "org-ORG"
        },
...

Next, we need to store the result in a variable for further work.

`resty` JSON Unmarshall

resty supports automatic JSON unmarshalling via SetResult() method:

func (r *Request) SetResult(res interface{}) *Request {
  if res != nil {
    r.Result = getPointer(res)
  }
  return r
}

It accepts the argument type any (interface{}), passes it to its function getPointer(), where it checks whether it is a pointer type:

func getPointer(v interface{}) interface{} {
  vv := valueOf(v)
  if vv.Kind() == reflect.Ptr {
    return v
  }
  ...
}

Then, SetResult() calls parseResponseBody() to write the value from Request.Result to the object that was passed as an argument to SetResult():

...
  // default after response middlewares
  c.afterResponse = []ResponseMiddleware{
    parseResponseBody,
    saveResponseIntoFile,
  }
...

And in the function parseResponseBody() the method Unmarshalc is called, which then calls Client.JSONUnmarshal(), and the field JSONUnmarshal contains the function json.Unmarshal():

...
func createClient(hc *http.Client) *Client {
  if hc.Transport == nil {
    hc.Transport = createTransport(nil)
  }

  c := &Client{ // not setting lang default values
                ...
    JSONUnmarshal: json.Unmarshal,
...

See source code of the resty/v2/client.go.

So, we get the result in JSON, and by using SetResult() we can save the necessary fields in some object.

Creating a Go struct for JSON Unmarshall

Let’s think about how we want to structure the data.

We have project_id and amount - how much this project has spent, which we get from the OpenAI API /organization/costs.

We also have Project Names, which we can get from /organization/projects, but more on that later.

As a result, we can build something like this:

[
  {
    "project_id": "Id1",
    "project_name": "Name1",
    "project_spend": 100
  },
  {
    "project_id": "Id2",
    "project_name": "Name2",
    "project_spend": 200
  }
]

What does Go offer for this?

array , array: fixed length, indexed type, all objects of the same type - [3]int{1,2,3}
slice : similar to array, but not fixed length - []int{1,2,3}
maps : a set of key:value elements of variable length of the same type - map[string]string{"key_name": "value_value"}
structs : a complex type that can include other types - struct{ Name string; Age int }{ Name: "Nino", Age: 35 }

Since we know what types we get from the API and all the fields in them, a slice of structs will work for us, where each element of the slice will be a structure with fields in which we will store project_id, amount, and project_name.

Go struct for the Project ID and Amount

The structure for us may look like this:

type ProjectSpend struct {
  ProjectID string
  ProjectSpend int
}

Then, let’s create a slice with this structure:

data := []ProjectSpend{}

Now let’s take a look at what the OpenAI API returns.

The /organization/costs response is:

{
  "object": "page",
  "has_more": false,
  "next_page": null,
  "data": [
    {
      "object": "bucket",
      "start_time": 1763078400,
      "end_time": 1763164800,
      "results": [
        {
          "object": "organization.costs.result",
          "amount": {
            "value": 2.16911625,
            "currency": "usd"
          },
          "line_item": null,
          "project_id": "proj_1",
          "organization_id": "org-ORG"
        },
        {
          "object": "organization.costs.result",
          "amount": {
            "value": 0.1846203,
            "currency": "usd"
          },
          "line_item": null,
          "project_id": "proj_2",
          "organization_id": "org-ORG"
        },
        ...
      ]
    }
  ]
}

Here we have the following structure:

begins with JSON object {}
has several JSON properties - "object": "page", etc
followed by an array data []
which contains another object {}
which starts with properties "object": "bucket", etc
and in which there is another array results []
which includes another object {}
which starts with property "object": "organization.costs.result"
followed by property amount, which contains a nested object {}
with two properties - value and value

If we want to reflect this to a Go struct, we need to create several structures that will transfer data to each other:

the first structure “captures” the fitst data[]
the second structure receives results[]
the third receives the value of the field project_id
and the fourth reads amount

How this might look in code, using struct composition, when one struct contains a field whose type is another struct:

type ResponceAmount struct {
  Value float64
}

type ResponceProjectID struct {
  ProjectID string `json:"project_id"`
  Amount ResponceAmount
}

type ResponseResults struct {
  Results []ResponceProjectID
}

type ResponseData struct {
  Data []ResponseResults
}

res := &ResponseData{}

And now we can execute json.Unmarshall by calling SetResult(), to which we pass a pointer - res := &ResponseData{}:

...
  _, err := client.SetAuthToken(apiKey).
    R().SetQueryParams(setQueryParams).
    SetResult(res).
    Get(baseURL + costsPath)

fmt.Println("Result: ", res)
...

The result is:

$ go run main.go
...
Result: &{[{[{proj_1 {2.16911625}} {proj_Agtar0XzJdXXLhGt8YCRNZMY {0.1846203}} {proj_2 {0.1531728}} {proj_3 {0.19788874999999997}}]}]}

Or we can make it more concise by using nested anonymous structs:

...
  // catch data[] and pass to nested struct
  // catch results[] and pass to next nested struct
  // catch 'project_id' property to the 'ProjectID' field, and pass to next nested struct
  // catch 'amount' property to the 'Amount' field, and pass to next nested struct
  // finally, catch 'value' property to the 'Value' field
  type ResponseData struct {
    Data []struct {
      Results []struct {
        ProjectID string `json:"project_id"`
        Amount struct {
          Value float64
        }
      }
    }
  }
...

And we will get the same result.

Next, we will need to generate metrics with labels.

We do this in two for loops, in which we iterate through the fields of each structure:

...
  // catch each item from the 'Response.Data[]'
  for _, dataItem := range res.Data {
    // catch each iteam from the 'Response.Data[].Results[]'
    for _, result := range dataItem.Results {

      project := result.ProjectID
      amount := result.Amount.Value

      // print in VictoriaMetrics gauge format
      fmt.Printf("openai_stats{type=\"costs\", project=\"%s\"} %f\n", project, amount)
    }
  }
...

Result now:

$ go run main.go
openai_stats{type="costs", project="proj_1"} 2.170784
openai_stats{type="costs", project="proj_2"} 0.241411
openai_stats{type="costs", project="proj_3"} 0.213558
openai_stats{type="costs", project="proj_4"} 0.198619

Now let’s do the same for project names, because using values like “proj_123” in metric labels isn’t very useful - it’s better to display human-readable project names instead.

Go struct for the Project Names

Add a second endpoint, see the documentation List projects:

...
const (
  baseURL = "https://api.openai.com/v1"
  costsPath = "/organization/costs"
  projectsPath = "/organization/projects"
)
...

And move the execution of requests to OpenAI to a dedicated function:

...
func getOpenAi(client *resty.Client, path string, out any) error {
  _, err := client.R().
    SetResult(out).
    Get(path)
  return err
}
...

Move setting the OPENAI_ADMIN_KEY key and parameters to the client creation call with the resty.New().

Next, call our function getOpenAi(), and pass the created and configured client as the first argument:

...
func main() {
  //client := resty.New()

  apiKey := os.Getenv("OPENAI_ADMIN_KEY")
  timeNow := strconv.FormatInt(time.Now().Unix(), 10)

  setQueryParams := map[string]string{
    "start_time": timeNow,
    "group_by": "project_id",
  }

  // use pointer to ResponseData struct
  // as 'json.Unmarshal' requires a pointer to write results
  costsRes := &CostsResponseData{}

  client := resty.New().
    SetAuthToken(apiKey).
    SetQueryParams(setQueryParams)

  getOpenAi(client, baseURL+costsPath, costsRes)

  fmt.Println("Result: ", costsRes)
...

Run to check:

$ go run main.go
Result: &{[{[{proj_1 {2.1707842499999996}} {proj_2 {0.24141089999999998}} {proj_3 {0.21355799999999994}} {proj_4 {0.46123659999999994}}]}]}
openai_stats{type="costs", project="proj_1"} 2.170784
openai_stats{type="costs", project="proj_2"} 0.241411
...

Now let’s move on to obtaining project names.

A request to the api.openai.com/v1/organization/projects endpoint will return data in the following format:

{
    "object": "list",
    "data": [
        {
            "id": "proj_abc",
            "object": "organization.project",
            "name": "Project example",
            "created_at": 1711471533,
            "archived_at": null,
            "status": "active"
        }
    ],
    "first_id": "proj-abc",
    "last_id": "proj-xyz",
    "has_more": false
}

We do the same as when obtaining costs - add a structure:

...
type ProjectsResponse struct {
  Data []struct {
    ID string
    Name string
  }
}
...

And in the main(), add a second call to the getOpenAi() and errors handling:

...
  // use pointer to ResponseData struct
  // as 'json.Unmarshal' requires a pointer to write results
  costsRes := &CostsResponseData{}
  if err := getOpenAi(client, baseURL+costsPath, costsRes); err != nil {
    panic(err)
  }

  projectsRes := &ProjectsResponse{}
  if err := getOpenAi(client, baseURL+projectsPath, projectsRes); err != nil {
    panic(err)
  }

  fmt.Println("Costs Result: ", costsRes)
  fmt.Println("Projects Result: ", projectsRes)
...

And now, we have the following result:

$ go run main.go
Costs Result: &{[{[{proj_1 {2.1707842499999996}} {proj_2 {0.24141089999999998}} {proj_3 {0.21355799999999994}} {proj_4 {0.46123659999999994}}]}]}
Projects Result: &{[{proj_1 Default project} {proj_2 Assistant Test/Eval} {proj_3 Kraken Production} {proj_4 Knowledge Base}]}
...

sanitizing names - formatting values with `strings.Replace()`

But our names contain spaces and “/” symbols, and project names contain capital letters - and we want our metric labels to look like “my_project_name”.

Let’s add a function that will perform normalization using the ToLower() and ReplaceAll() methods from the strings package:

...
func normalizeLabel(s string) string {
  s = strings.ToLower(s)
  s = strings.ReplaceAll(s, " ", "_")
  s = strings.ReplaceAll(s, "/", "_")
  return s
}
...

The next step is to build a map in which we will have project_id and project_names:

...
  projectNames := make(map[string]string)

  // get each 'ProjectsResponse.Data[].ID'
  // get each 'ProjectsResponse.Data[].Name'
  // populate the projectNames map with:
  // 'project_id' = 'project_name'
  for _, p := range projectsRes.Data {
    projectNames[p.ID] = normalizeLabel(p.Name)
  }

  fmt.Println("Projects Names: ", projectNames)
...

As a result, we have:

$ go run main.go
Projects Names: map[proj_1:kraken_production proj_2:assistant_test_eval proj_3:knowledge_base proj_4:default_project]

Now let’s update our two loops - let’s use names instead of IDs in the label:

...
  // catch each item from the 'Response.Data[]'
  for _, dataItem := range costsRes.Data {
    // catch each item from the 'Response.Data[].Results[]'
    for _, result := range dataItem.Results {

      // get ''Response.Data[].Results[].ProjectID'
      id := result.ProjectID

      // get ''Response.Data[].Results[].Amount.Value'
      amount := result.Amount.Value

      // use the 'id' to get the project name from the projectNames map
      project := projectNames[id]
      if project == "" {
        project = "unknown"
      }

      // print in VictoriaMetrics gauge format
      fmt.Printf("openai_stats{type=\"costs\", project=\"%s\"} %f\n", project, amount)
    }
  }
...

And the result:

$ go run main.go
openai_stats{type="costs", project="knowledge_base"} 2.170784
openai_stats{type="costs", project="kraken_production"} 0.241411
openai_stats{type="costs", project="assistant_test_eval"} 1.083077
openai_stats{type="costs", project="default_project"} 0.461237

Now we have the following code:

package main

import (
  "fmt"
  "os"
  "strconv"
  "strings"
  "time"

  "github.com/go-resty/resty/v2"
)

// set global const as ay be used in other packages
const (
  baseURL = "https://api.openai.com/v1"
  costsPath = "/organization/costs"
  projectsPath = "/organization/projects"
)

// catch data[] and pass to nested struct
// catch results[] and pass to next nested struct
// catch 'project_id' property to the 'ProjectID' field, and pass to next nested struct
// catch 'amount' property to the 'Amount' field, and pass to next nested struct
// finally, catch 'value' property to the 'Value' field
type CostsResponseData struct {
  Data []struct {
    Results []struct {
      ProjectID string `json:"project_id"`
      Amount struct {
        Value float64
      }
    }
  }
}

type ProjectsResponse struct {
  Data []struct {
    ID string
    Name string
  }
}

func getOpenAi(client *resty.Client, path string, out any) error {
  _, err := client.R().
    SetResult(out).
    Get(path)
  return err
}

func normalizeLabel(s string) string {
  s = strings.ToLower(s)
  s = strings.ReplaceAll(s, " ", "_")
  s = strings.ReplaceAll(s, "/", "_")
  return s
}

func main() {
  //client := resty.New()

  apiKey := os.Getenv("OPENAI_ADMIN_KEY")
  timeNow := strconv.FormatInt(time.Now().Unix(), 10)

  setQueryParams := map[string]string{
    "start_time": timeNow,
    "group_by": "project_id",
  }

  client := resty.New().
    SetAuthToken(apiKey).
    SetQueryParams(setQueryParams)

  // use pointer to ResponseData struct
  // as 'json.Unmarshal' requires a pointer to write results
  costsRes := &CostsResponseData{}
  if err := getOpenAi(client, baseURL+costsPath, costsRes); err != nil {
    panic(err)
  }

  projectsRes := &ProjectsResponse{}
  if err := getOpenAi(client, baseURL+projectsPath, projectsRes); err != nil {
    panic(err)
  }

  projectNames := make(map[string]string)

  // get each 'ProjectsResponse.Data[].ID'
  // get each 'ProjectsResponse.Data[].Name'
  // populate the projectNames map with:
  // 'project_id' = 'project_name'
  for _, p := range projectsRes.Data {
    projectNames[p.ID] = normalizeLabel(p.Name)
  }

  // catch each item from the 'Response.Data[]'
  for _, dataItem := range costsRes.Data {
    // catch each item from the 'Response.Data[].Results[]'
    for _, result := range dataItem.Results {

      // get ''Response.Data[].Results[].ProjectID'
      id := result.ProjectID

      // get ''Response.Data[].Results[].Amount.Value'
      amount := result.Amount.Value

      // use the 'id' to get the project name from the projectNames map
      project := projectNames[id]
      if project == "" {
        project = "unknown"
      }

      // print in VictoriaMetrics gauge format
      fmt.Printf("openai_stats{type=\"costs\", project=\"%s\"} %f\n", project, amount)
    }
  }
}

Now we can move on to creating real metrics and recording them in VictoriaMetrics.

Planning metrics for VictoriaMetrics

So, our metrics will be in the form of openai_stats{type="costs", project="prodject_id"} 5.55.

And what does the problem say you need to do as a result?

if daily spending on OpenAI exceeds the average for the last few days (with a certain threshold) - shout in Slack

So, we will need the daily amount, and once we have it, we can make comparisons with previous periods.

And what do we get back in the API?

Let’s check the Costs object:

The aggregated costs details of the specific time bucket.

And what do we get when we make a request only with⁣ start_time without end_time?

Let’s examine the time in the response received:

...
      "start_time": 1762905600,
      "end_time": 1762992000,
...

Here, the start_time will be:

$ date -d @1762905600
Wed Nov 12 02:00:00 EET 2025

А end_time:

$ date -d @1762992000
Thu Nov 13 02:00:00 EET 2025

It is 00:00 UTC.

That is, it returns the amount spent for today, for the current day, because this text was written on the 12th Nov.

So here it is:

...
      "start_time": 1762905600,
      "end_time": 1762992000,
      ...
            "value": 1.76643575,
            "currency": "usd"
          },
          "line_item": null,
          "project_id": "proj_1",
...

We can see that today the project with ID “proj_1” spent 1.76643575 bucks.

Okay…

How can we store this in metrics? Create a Counter type that constantly increases and update it every minute or hour?

Then the time series (see What is a Time Series?) for this metric will look something like this:

openai_stats{type="costs", project="prodject_id"}
  1762960223 1.76
  1762960237 1.80
  1762960249 1.95

And then we can create a request for the alert, something like this:

if 
avg_over_time(openai_stats{type="costs", project="prodject_id"}[1d)
> 
avg_over_time(openai_stats{type="costs", project="prodject_id"}[3d)
then send alert

But there is an important caveat with a Counter: it resets its value if the exporter restarts, see counter reset.

In addition, if we receive data starting at 00:00, then from the next day the value will start at 0.00 USD.

This means the metric value can go both up and down, so we need a Gauge rather than a Counter.

VictoriaMetrics Go client

There is a Prometheus client library for Go, but for our purposes we’ll use the VictoriaMetrics package, which also includes a PushMetrics() function to send metrics directly to a VictoriaMetrics endpoint. without needing to have a dedicated scrape job.

Create metrics with `NewGauge()`

Let’s look at the documentation for type Gauge, where there is an example of creating a metric object.

The NewGauge() function takes two arguments: the name of the metric with labels and the function that updates the value for this metric, see gauge.go:

func NewGauge(name string, f func() float64) *Gauge {
  return defaultSet.NewGauge(name, f)
}

But if we want to set the value ourselves, instead of passing the second argument f func(), we can simply pass nil and then use the Set() method.

Let’s try how it works with nil and Set():

gore> :import "github.com/VictoriaMetrics/metrics"
gore> g := metrics.NewGauge(`test_gauge`, nil)
gore> g.Set(9.00)
gore> :import fmt
gore> fmt.Println(g.Get())
9

Great.

Now let’s think about how we’re going to do all this.

We need:

create a new metrics.NewGauge() for each metric
then, once an minute or hour, receive data from the API
for each metric, execute Set()

That is, we generate metrics, each with its own label value project:

project_1 := metrics.NewGauge(openai_stats{type="costs", project="prodject_1"}) 
project_2 := metrics.NewGauge(openai_stats{type="costs", project="prodject_2"}) 
project_3 := metrics.NewGauge(openai_stats{type="costs", project="prodject_3"})

Then, for each project_N, we execute Set().

We now have a loop that fills in fmt.Printf("openai_stats{type=\"costs\", project=\"%s\"} %f\n", project, amount).

Let’s first add metric generation directly to it and see how it might look.

To output to the console, use the metrics.WritePrometheus() function, which writes in Prometheus format to the channel specified by the first argument.

After the loops, add:

...
      // print in VictoriaMetrics gauge format
      //fmt.Printf("openai_stats{type=\"costs\", project=\"%s\"} %f\n", project, amount)
      metricName := fmt.Sprintf(`test_openai_stats{type="costs", project="%s"}`, project)
      gauge := metrics.NewGauge(metricName, nil)
      gauge.Set(amount)

    }
  }

  metrics.WritePrometheus(os.Stdout, false)
...

As a result, we have the following data:

$ go run main.go
test_openai_stats{type="costs", project="assistant_test_eval"} 4.9838991
test_openai_stats{type="costs", project="default_project"} 0.5281144000000001
test_openai_stats{type="costs", project="knowledge_base"} 2.17244425
test_openai_stats{type="costs", project="kraken_production"} 0.5510669499999999

Great.

Now let’s think about the whole logic of execution.

What we have now:

creation of resty.Client
initialization of the structure costsRes := &CostsResponseData{}
call getOpenAi() with arguments (client, baseURL+costsPath, costsRes), where we fill in the data in the CostsResponseData structure
initialization of the projectsRes := &ProjectsResponse{}
call getOpenAi() with arguments (client, baseURL+projectsPath, projectsRes), where we fill in the data in the ProjectsResponse structure
initialization of the projectNames map
fill it in with the data "project_id": "project_name"
further loops, in which:
get project_id
get amount
by project_id we get the project name, write it to the variable project
generate the name of the metric and label from the project into the metricName
with metrics.NewGauge() generate a new metric
with gauge.Set(amount) write a value into it
with metrics.WritePrometheus(), all generated metrics are output to the console

And all this is now executed when the main() is called.

Instead, when calling main(), i.e., when starting the exporter, we need to:

create resty.Client
then periodically update the data and write it to VictoriaMetrics:
call getOpenAi() to fill in the ProjectsResponse
with getOpenAi() fill in the structure CostsResponseData
fill in projectNames
run loops to generate metrics and execute Set()
at the end of the loops, execute WritePrometheus()

However, with this approach, we will be rewriting the fields in ProjectsResponse, CostsResponseData, and projectNames every hour, which is not very good from a performance point of view.

. But if we have a new project, we will immediately "catch" it and add a new metric for it.

So, what we need to do is move our logic into a dedicated function, call it periodically, and then execute WritePrometheus().

Write this function, just replace the NewGauge() with GetOrCreateGauge(), because the next time our function is called, the metrics will already have been created:

...
...
func fetchAndPush(client *resty.Client, costsRes *CostsResponseData, projectsRes *ProjectsResponse, projectNames map[string]string) {
  if err := getOpenAi(client, baseURL+costsPath, costsRes); err != nil {
    panic(err)
  }
  if err := getOpenAi(client, baseURL+projectsPath, projectsRes); err != nil {
    panic(err)
  }
  // get each 'ProjectsResponse.Data[].ID'
  // get each 'ProjectsResponse.Data[].Name'
  // populate the projectNames map with:
  // 'project_id' = 'project_name'
  for _, p := range projectsRes.Data {
    projectNames[p.ID] = normalizeLabel(p.Name)
  }

  // catch each item from the 'Response.Data[]'
  for _, dataItem := range costsRes.Data {
    // catch each item from the 'Response.Data[].Results[]'
    for _, result := range dataItem.Results {

      // get 'Response.Data[].Results[].ProjectID'
      // i.e. 'proj_123'
      id := result.ProjectID

      // get 'Response.Data[].Results[].Amount.Value'
      amount := result.Amount.Value

      // use the 'id' to get the project name from the projectNames map
      project := projectNames[id]
      if project == "" {
        project = "unknown"
      }

      // print in VictoriaMetrics gauge format
      //fmt.Printf("openai_stats{type=\"costs\", project=\"%s\"} %f\n", project, amount)
      metricName := fmt.Sprintf(`test_openai_stats{type="costs", project="%s"}`, project)
      gauge := metrics.GetOrCreateGauge(metricName, nil)
      gauge.Set(amount)

    }
  }

  metrics.WritePrometheus(os.Stdout, false)
}
...

Now in the main() we have only this:

...
func main() {
  //client := resty.New()

  apiKey := os.Getenv("OPENAI_ADMIN_KEY")
  timeNow := strconv.FormatInt(time.Now().Unix(), 10)

  setQueryParams := map[string]string{
    "start_time": timeNow,
    "group_by": "project_id",
  }

  client := resty.New().
    SetAuthToken(apiKey).
    SetQueryParams(setQueryParams)

  // use pointer to ResponseData struct
  // as 'json.Unmarshal' requires a pointer to write results
  costsRes := &CostsResponseData{}

  projectsRes := &ProjectsResponse{}

  // will be populated with key:value pairs:
  // 'proj_123' = 'kraken_production'
  projectNames := make(map[string]string)

  fetchAndPush(client, costsRes, projectsRes, projectNames)
}

Let’s run it to check:

$ go run main.go
test_openai_stats{type="costs", project="assistant_test_eval"} 6.3417053
test_openai_stats{type="costs", project="default_project"} 0.6592560500000001
test_openai_stats{type="costs", project="knowledge_base"} 2.17244425
test_openai_stats{type="costs", project="kraken_production"} 0.6170747

Now, instead of simply printing to the console, we need to write the data to VictoriaMetrics.

Recording metrics to VictoriaMetrics with `InitPush()` and `PushMetrics()`

To record metrics in VictoriaMetrics, we have two main functions: InitPush() and PushMetrics().

Inside the `InitPush()` implementation

The InitPush() function allows you to perform periodic recordings with a specified interval, while PushMetrics() allows you to simply record all metrics that are in Set struct once. More about Set below.

Now, just for the sake of interest, let’s take a look at how the VictoriaMetrics client performs recording.

We find the code InitPush():

func InitPush(pushURL string, interval time.Duration, extraLabels string, pushProcessMetrics bool) error {
  writeMetrics := func(w io.Writer) {
    WritePrometheus(w, pushProcessMetrics)
  }
  return InitPushExt(pushURL, interval, extraLabels, writeMetrics)
}

Here:

in our code, we call InitPush(), pass the URL and interval to this function
InitPush() creates a variable writeMetrics - an anonymous function that takes an argument of type io. Writer type, which will then call the WritePrometheus() function, to which this io.Writer is passed
next, the InitPushExt() function is called with the pushURL, interval, and the writeMetrics object as arguments

Let’s look at InitPushExt():

func InitPushExt(pushURL string, interval time.Duration, extraLabels string, writeMetrics func(w io.Writer)) error {
  opts := &PushOptions{
    ExtraLabels: extraLabels,
  }
  return InitPushExtWithOptions(context.Background(), pushURL, interval, writeMetrics, opts)
}

Here, parameters are simply added from the PushOptions, to which we can pass parameters of type extraLabels, and then InitPushExtWithOptions() is called, to which our writeMetrics is passed.

Let’s look at InitPushExtWithOptions(): here, a goroutine is created that calls pushMetrics() with a specified interval, to which our writeMetrics object is passed (i.e., the anonymous function that will call the WritePrometheus()):

func InitPushExtWithOptions(ctx context.Context, pushURL string, interval time.Duration, writeMetrics func(w io.Writer), opts *PushOptions) error {
  pc, err := newPushContext(pushURL, opts)
        ...
  go func() {
    ticker := time.NewTicker(interval)
                ...
        ctxLocal, cancel := context.WithTimeout(ctx, interval+time.Second)
        err := pc.pushMetrics(ctxLocal, writeMetrics)

Next, pushMetrics() creates a buffer bytes.Buffer, passes it to the writeMetrics(), and writeMetrics() calls WritePrometheus(), which receives this buffer:

func (pc *pushContext) pushMetrics(ctx context.Context, writeMetrics func(w io.Writer)) error {
  bb := getBytesBuffer()
  defer putBytesBuffer(bb)

  writeMetrics(bb)
...

And then WritePrometheus() writes the collected metrics to this buffer:

// WritePrometheus writes all the metrics from s to w in Prometheus format.
func (s *Set) WritePrometheus(w io.Writer) {
...

Next, from this buffer (still in pushMetrics()), a request body is created and headers are set:

And then the data is sent to the specified URL:

VictoriaMetrics and the Set struct

Now let’s return to “WritePrometheus() writes the collected metrics to this buffer”.

WritePrometheus() is a method of the Set structure:

func (s *Set) WritePrometheus(w io.Writer) {
  ...
}

And Set is created when we call NetGauge():

func NewGauge(name string, f func() float64) *Gauge {
  return defaultSet.NewGauge(name, f)
}

defaultSet is a call to NewSet():

where defaultSet = NewSet()

And NewSet() fills the Set structure:

// NewSet creates new set of metrics.
//
// Pass the set to RegisterSet() function in order to export its metrics via global WritePrometheus() call.
func NewSet() *Set {
  return &Set{
    m: make(map[string]*namedMetric),
  }
}

That is, when calling NetGauge(), we pass an argument with the name of the metric, NetGauge() calls NewSet(), passes this metric, and NewSet() initializes the Set structure, setting our metric in the namedMetric field.

Inside the `PushMetrics()` function implementation

With PushMetrics(), the flow is almost the same - writeMetrics is created, PushMetricsExt() is called:

func PushMetrics(ctx context.Context, pushURL string, pushProcessMetrics bool, opts *PushOptions) error {
  writeMetrics := func(w io.Writer) {
    WritePrometheus(w, pushProcessMetrics)
  }
  return PushMetricsExt(ctx, pushURL, writeMetrics, opts)
}

And PushMetricsExt() calls pushMetrics(), but only once, not in a loop:

func PushMetricsExt(ctx context.Context, pushURL string, writeMetrics func(w io.Writer), opts *PushOptions) error {
  pc, err := newPushContext(pushURL, opts)
  if err != nil {
    return err
  }
  return pc.pushMetrics(ctx, writeMetrics)
}

Okay, let’s return to our code.

So, what we need to do now is call PushMetrics() instead of WritePrometheus().

Creating `context` and calling `PushMetrics()`

For PushMetrics(), we need to pass a context that manages goroutines and terminates them either on a timeout or if the program itself receives SIGTERM or SIGKILL signals from the system.

More details about context will follow below, but for now, let’s just add import "context" and create an empty context with Background() in main():

...
import (
  "context"
...
func main() {
        ...
  // will be populated with key:value pairs:
  // 'proj_123' = 'kraken_production'
  projectNames := make(map[string]string)

  ctx := context.Background()
...

In our function fetchAndPush(), add a parameter with type context.Context:

...
func fetchAndPush(ctx context.Context, ...) {
   ...
}

Add context passing to the fetchAndPush() call:

...
       fetchAndPush(ctx, client, costsRes, projectsRes, projectNames)
...

Set a variable with the VictoriaMetrics instance URL, change metrics.WritePrometheus() to metrics.PushMetrics(), to which we pass the context received from main():

...

  //metrics.WritePrometheus(os.Stdout, false)
  pushURL := "http://localhost:8428/api/v1/import/prometheus"

  if err := metrics.PushMetrics(ctx, pushURL, false, nil); err != nil {
    panic(err)
  }
}

In the pushURL, I'm using localhost where have kubectl port-forward:

$ kubectl -n ops-monitoring-ns port-forward svc/vmsingle-vm-k8s-stack 8428

When we launch the exporter in Kubernetes, we will add a new environment variable passed from a Helm chart values.

At this point, the main thing left to do is to run our function on a schedule.

Using `gocron` for tasks scheduling

There is a nice package called gocron. Let's add it and set it to run our fetchAndPush() function every minute:

import (
        ...
  "github.com/go-co-op/gocron"
        ...
)

...
func main() {
  s := gocron.NewScheduler(time.Local)

  s.Every(1).Minute().Do(func() {
    fetchAndPush(client, costsRes, projectsRes, projectNames)
  })

  s.StartBlocking()
}

Then we can change it to call once an hour - s.Every(1).Hour().Do( ... ), or at the beginning of each hour - s.Cron("0 * * * *").Do( ... ).

And finally, launch the cron with StartBlocking(), which blocks the finishing of the main() function itself.

Open access to VictoriaMetrics in Kubernetes, if not done yet:

$ kk -n ops-monitoring-ns port-forward svc/vmsingle-vm-k8s-stack 8428

Let’s launch our exporter:

$ go run main.go
test_openai_stats{type="costs", project="assistant_test_eval"} 6.501765299999999
test_openai_stats{type="costs", project="default_project"} 0.6592560500000001
test_openai_stats{type="costs", project="knowledge_base"} 2.17411225
test_openai_stats{type="costs", project="kraken_production"} 0.6471627999999999
^Csignal: interrupt

And check the data in VictoriaMetrics:

However, some “unknown” project has appeared here, so logging will need to be added.

What else needs to be fixed:

currently, the initialization of the CostsResponseData and ProjectsResponse structures is performed in main(), and then data is written to them each time fetchAndPush() is called
if a project is deleted from OpenAI, it will remain in the structures, and we will continue to write metrics for a project that no longer exists
thus, this needs to be moved to the fetchAndPush() and simply filled in from scratch each time
the same for the projectNames - move initialization to fetchAndPush() itself
SetQueryParams - currently passed identically for both getOpenAi() calls, but there is no group_by parameter for the /organization/projects OpenAI endpoint
in the label metric, it is better to replace type="" with category=""
need to add external labels - something like “job="openai-exporter""
instead of using panic(err), return an error to the calling function, handle it there, and log messages
add correct handling of SIGTERM and SIGINT signals
resty.client can perform retries in case of errors, so we can add SetRetryCount() and SetRetryWaitTime()
and add execution and error logs

Creating a Golang context

While working in our code, several simultaneous operations are launched - with gocron.NewScheduler() we launch the execution of our function fetchAndPush() which makes HTTP requests with resty.Client.Get(), and VictoriaMetrics launches its operations to write to the VictoriaMetrics endpoint.

To shut the application down gracefully instead of simply “killing” it when receiving SIGINT or SIGTERM, Go allows us to control the termination process of our functions and goroutines through the context of execution.

Another example of when we need to control the execution of an operation is to set a time limit for execution, as it is done, for example, in the VictoriaMetrics Go client for the InitPushExtWithOptions() function:

...
  go func() {
    ticker := time.NewTicker(interval)
    defer ticker.Stop()
    stopCh := ctx.Done()
    for {
      select {
      case <-ticker.C:
        ctxLocal, cancel := context.WithTimeout(ctx, interval+time.Second)
        err := pc.pushMetrics(ctxLocal, writeMetrics)
...

Here, the execution of pc.pushMetrics() is limited by interval, which is passed when calling InitPush().

At the same time, the execution context includes not only signal handling and lifecycle management of functions and goroutines, but also other information related to this execution:

Package context defines the Context type, which carries deadlines, cancellation signals, and other request-scoped values across API boundaries and between processes

I moved the details about context to a dedicated section - Bonus: How execution control works with Go context below - because it is a very interesting mechanism, and now let’s just add it to our code.

So, what do we need:

create a context
create a “signal interceptor” SIGINT (Ctrl+C) and SIGTERM (a signal from the operating system when the program execution ends, for example, when kubelet stops the container)
send a stop signal to all child functions and goroutines
complete execution of main()

To do this, instead of calling context.Background() in main(), we can use signal.NotifyContext() which receives the necessary system calls and sends a stop signal to all related tasks:

...
  rootCtx, rootCancel := signal.NotifyContext(
    context.Background(),
    os.Interrupt,
    syscall.SIGTERM,
  )
  defer rootCancel()
...

Next, we have the call of the gocron.NewScheduler(), and at the end of the main() we launch the creation and reading from the channel:

...
  // block until Ctrl+C cancels rootCtx
  <-rootCtx.Done()
}

As soon as NotifyContext() receives SIGTERM, it closes the channel rootCtx.Done() channel, after which all child context channels will be closed in cascade, then all child goroutines listening to these contexts will terminate, and main() will be able to terminate correctly.

resty.client can also work with context via SetContext() to which we pass our rootCtx when calling if err := getOpenAI(ctx, ... ) {...}.

The final result of the OpenAI Exporter code

After all the edits, the entire exporter code now looks like this:

package main

import (
 "context"
 "fmt"
 "log"
 "os"
 "os/signal"
 "strconv"
 "strings"
 "syscall"
 "time"

 "github.com/VictoriaMetrics/metrics"
 "github.com/go-co-op/gocron"
 "github.com/go-resty/resty/v2"
)

const (
 // base URL of the OpenAI Admin API
 baseURL = "https://api.openai.com/v1"

 // endpoints that we call
 costsPath = "/organization/costs"
 projectsPath = "/organization/projects"

 // VictoriaMetrics push endpoint (Prometheus remote write format)
 //pushURL = "http://localhost:8428/api/v1/import/prometheus"
)

// structure describing the JSON for costs API
// resty will unmarshal into this struct automatically
type CostsResponseData struct {
 Data []struct {
  Results []struct {
   ProjectID string `json:"project_id"`
   Amount struct {
    Value float64 `json:"value"`
   } `json:"amount"`
  } `json:"results"`
 } `json:"data"`
}

// structure describing the JSON for projects API
// used to map project_id → readable project name
type ProjectsResponse struct {
 Data []struct {
  ID string `json:"id"`
  Name string `json:"name"`
 } `json:"data"`
}

// normalizeLabel converts a project name into a Prometheus-safe label
// - lowercases
// - replaces spaces with underscores
// - replaces slashes to avoid label parser issues
func normalizeLabel(s string) string {
 s = strings.ToLower(s)
 s = strings.ReplaceAll(s, " ", "_")
 s = strings.ReplaceAll(s, "/", "_")
 return s
}

// getOpenAI performs a GET request to the OpenAI Admin API
// and unmarshals the returned JSON into the 'out' structure.
//
// ctx: allows cancellation (we pass rootCtx so Ctrl+C cancels requests)
// client: the resty client with authentication
// path: "/organization/costs" or "/organization/projects"
// params: optional query parameters
func getOpenAI(ctx context.Context, client *resty.Client, path string, params map[string]string, out any) error {
 // create HTTP request object
 req := client.R().
  SetContext(ctx). // attach context so cancellation works
  SetResult(out) // register target structure for unmarshalling JSON

 // set optional query parameters
 if params != nil {
  req.SetQueryParams(params)
 }

 // perform request
 resp, err := req.Get(baseURL + path)
 if err != nil {
  return fmt.Errorf("http transport error calling %s: %w", path, err)
 }

 // check HTTP status codes
 if !resp.IsSuccess() {
  return fmt.Errorf(
   "OpenAI API error: path=%s status=%d body=%s",
   path,
   resp.StatusCode(),
   resp.String(),
  )
 }

 return nil
}

// fetchAndPush performs one exporter cycle:
//
// 1. fetch costs grouped by project_id
// 2. fetch readable project names
// 3. build project_id → normalized_name map
// 4. create/update Prometheus gauges
// 5. push all metrics to VictoriaMetrics
//
// ctx: the root context (cancelled when Ctrl+C is pressed)
func fetchAndPush(ctx context.Context, client *resty.Client, vmUrl string) error {
 // create fresh response holders for every iteration
 costsRes := &CostsResponseData{}
 projectsRes := &ProjectsResponse{}
 projectNames := make(map[string]string)

 // build query parameters for costs API
 // start_time: current timestamp (Unix)
 // group_by: instruct API to group costs per project_id
 timeNow := strconv.FormatInt(time.Now().Unix(), 10)
 costParams := map[string]string{
  "start_time": timeNow,
  "group_by": "project_id",
 }

 // fetch costs data
 if err := getOpenAI(ctx, client, costsPath, costParams, costsRes); err != nil {
  return fmt.Errorf("fetch costs: %w", err)
 }

 // fetch project definitions
 if err := getOpenAI(ctx, client, projectsPath, nil, projectsRes); err != nil {
  return fmt.Errorf("fetch projects: %w", err)
 }

 // fill map: project_id → normalized_label
 for _, p := range projectsRes.Data {
  projectNames[p.ID] = normalizeLabel(p.Name)
 }

 // process returned costs
 for _, dataItem := range costsRes.Data {
  for _, result := range dataItem.Results {
   id := result.ProjectID
   amount := result.Amount.Value

   // resolve project readable name
   project := projectNames[id]
   if project == "" {
    project = "unknown"
   }

   metricName := fmt.Sprintf(
    `openai_stats{project="%s",category="costs"}`,
    project,
   )

   // get or create gauge
   gauge := metrics.GetOrCreateGauge(metricName, nil)

   // update gauge value
   gauge.Set(amount)

   // log written metric
   // explanation: this log helps to debug what exactly was pushed
   log.Printf("metric updated: name=%s value=%f", metricName, amount)
  }
 }

 // push metrics with job="openai_exporter"
 pushOpts := &metrics.PushOptions{
  ExtraLabels: `job="openai_exporter"`,
 }

 // push all collected metrics
 if err := metrics.PushMetrics(ctx, vmUrl, false, pushOpts); err != nil {
  return fmt.Errorf("push metrics: %w", err)
 }

 return nil
}

func main() {
 // create a context that automatically cancels on OS signals (Ctrl+C, kill, SIGTERM)
 //
 // how it works:
 // - signal.NotifyContext wraps the parent context and subscribes it to OS signals
 // - when the program receives Ctrl+C (SIGINT) or SIGTERM:
 // Go internally calls rootCancel()
 // the context's Done() channel is closed
 // - all goroutines waiting on <-rootCtx.Done() are instantly unblocked
 // - any operation bound to this context (HTTP requests, timeouts, jobs)
 // receives ctx.Err()==context.Canceled and stops gracefully
 //
 // practically:
 // - main goroutine waits for <-rootCtx.Done()
 // - when Ctrl+C arrives => rootCtx.Done() closes => program starts graceful shutdown
 //
 // 'defer rootCancel()' is used to clean up internal signal resources when main() exits normally
 rootCtx, rootCancel := signal.NotifyContext(
  context.Background(),
  os.Interrupt,
  syscall.SIGTERM,
 )
 defer rootCancel()

 // load OpenAI admin API key
 apiKey := os.Getenv("OPENAI_ADMIN_KEY")
 if apiKey == "" {
  log.Fatal("OPENAI_ADMIN_KEY is not set")
 }

 // load VictoriaMetrics URL admin API key
 vmUrl := os.Getenv("VM_URL")
 if vmUrl == "" {
  log.Fatal("VM_URL is not set")
 }

 // create resty client with:
 // - bearer token
 // - automatic retries (3 attempts)
 client := resty.New().
  SetAuthToken(apiKey).
  SetRetryCount(3).
  SetRetryWaitTime(2 * time.Second)

 // create scheduler using local timezone
 s := gocron.NewScheduler(time.Local)

 // register a job that runs every 1 minute
 s.Every(1).Minute().Do(func() {
  start := time.Now()

  log.Println("starting fetch-and-push cycle")

  // run our exporter cycle
  if err := fetchAndPush(rootCtx, client, vmUrl); err != nil {
   log.Println("ERROR during fetchAndPush:", err)
   return
  }

  log.Println("fetch-and-push completed in", time.Since(start))
 })

 log.Println("starting scheduler...")

 // run scheduler in background goroutine
 s.StartAsync()

 // block until Ctrl+C cancels rootCtx
 <-rootCtx.Done()

 log.Println("received Ctrl+C, stopping scheduler...")

 // shutdown scheduler gracefully
 s.Stop()

 log.Println("scheduler stopped, exiting")
}

Let’s check it out on VictoriaMetrics:

And let’s compare this with the data on OpenAI’s own website at platform.openai.com/settings/organization/usage:

The same $6.95 that we see in VictoriaMetrics from our exporter.

The code could be improved further, for example, by breaking down the large function fetchAndPush(), and adding URL transfer to VictoriaMetrics from environment variables, but for now, we'll live with this version.

Bonus: How execution control works with Go `context`

In our fetchAndPush() function we call metrics.PushMetrics(), passing a context to it.

To make things clearer, let’s take another look at InitPush(), since the use of context is most visible there.

So, InitPush() calls InitPushExt(), and InitPushExt() calls InitPushExtWithOptions(), passing it an empty context.Background() - return InitPushExtWithOptions(context.Background() ...).

So, InitPush() calls InitPushExt(), and InitPushExt() in its turn calls InitPushExtWithOptions(), passing it an empty context.Background() - the return InitPushExtWithOptions(context.Background(), ...) part.

Inside InitPushExtWithOptions(), a goroutine (go func() {}) is launched, where a local context is created:

...
        ctxLocal, cancel := context.WithTimeout(ctx, interval+time.Second)
        err := pc.pushMetrics(ctxLocal, writeMetrics)
...

Closing the channel with `cancel()`

When context.WithTimeout() is called, the following sequence occurs:

WithTimeout() calls WithDeadline()
WithDeadline() calls WithDeadlineCause():
where an object is created - c := &timerCtx{}
the timerCtx struct embeds cancelCtx
which means timerCtx now has access to all methods of cancelCtx
next, WithDeadlineCause() checks if dur <= 0, meaning that the deadline has already passed:
calls c.cancel(true, DeadlineExceeded, cause)
returns “return c, func() { c.cancel(false, Canceled, nil) }", which is returned to InitPushExtWithOptions() in the part ctxLocal, cancel := context. WithTimeout() and "func() { c.cancel() }" and becomes cancel()
c.cancel() is a method of the timerCtx - "func (c *timerCtx) cancel()" - and it internally calls c.cancelCtx.cancel()
and c.cancelCtx.cancel() is a method of the cancelCtx structure - func (c *cancelCtx) cancel(), which calls d, _ := c.done.Load().(chan struct{})
and calls close(d)

So here:

func (c *cancelCtx) cancel(removeFromParent bool, err, cause error) {
...
  d, _ := c.done.Load().(chan struct{})
  if d == nil {
    ...
  } else {
    close(d)
  }
...

c.done.Load() is called from the done field of the cancelCtx structure:

type cancelCtx struct {
        ...
  done atomic.Value // of chan struct{}, created lazily, closed by first cancel call
        ...
}

Where Load() is a method of the Value structure:

func (v *Value) Load() (val any)

That is, in the d, _ := c.done.Load().(chan struct{}), the Load() method is executed, and then a type assertion to chan struct{} is performed. After that, d becomes a channel of type chan struct{}, and close(d) is called.

And close() is a built-in Go function that closes the channel passed as an argument.

As soon as the Done() channel is closed, all goroutines waiting on <-ctx.Done() are unblocked and can finish their work properly.

In the InitPushExtWithOptions(), this is done here:

go func() {
    ...
    stopCh := ctx.Done()
    ...
    case <-stopCh:
                              ...
      return
    }
  }
}()

Closing the channel reads a zero value, which triggers the condition case =>, which ends the loop by calling return =>, which ends the entire go func() {}.

Okay.

And where did the channel come from?

How the `context` channel is created

In order for a function or goroutine to continuously “listen” for the channel to be closed, we call Done():

...
  go func() {
    ticker := time.NewTicker(interval)
    defer ticker.Stop()
    stopCh := ctx.Done()
      ...
      case <-stopCh:
        if wg != nil {
          wg.Done()
        }
        return
      }
...

And ctx.Done() is a method of the Context interface:

type Context interface {
...
  Done() <-chan struct{}
}

Where the channel itself is created:

...
func (c *cancelCtx) Done() <-chan struct{} {
        ...
  if d == nil {
    d = make(chan struct{})
    c.done.Store(d)
  }
  return d.(chan struct{})
}
...

So, when called:

context.WithTimeout() =>
WithDeadline() =>
WithDeadlineCause() =>
c := &timerCtx {} which has cancelCtx
а cancelCtx {} which has Done()

And when we call:

...
  rootCtx, rootCancel := signal.NotifyContext(
    context.Background(),
    os.Interrupt,
    syscall.SIGTERM,
  )
...

So, in the signal.NotifyContext(), we pass an empty parent context, and signal.NotifyContext() creates and returns its own context using context.WithCancel():

...
func NotifyContext(parent context.Context, signals ...os.Signal) (ctx context.Context, stop context.CancelFunc) {
  ctx, cancel := context.WithCancel(parent)
  c := &signalCtx{
           ...
  }
        ...
  return c, c.stop
}
...

Therefore, at the end of our main(), we can call reading from the channel:

...
  // block until Ctrl+C cancels rootCtx
  <-rootCtx.Done()
...

As soon as the channel is closed, control returns to the main(), where gocron.Stop() is executed and the program terminates.

Originally published at RTFM: Linux, DevOps, and system administration.

VictoriaMetrics: Churn Rate, High cardinality, metrics, and IndexDB

Arseny Zinchenko — Tue, 04 Nov 2025 11:34:21 +0000

One day I received one of the default VictoriaMetrics alerts that are generated during the deployment of the Helm chart victoria-metrics-k8s-stack:

I thought about writing a short post like “What is Churn Rate and how to fix it,” but in the end, I ended up diving deep into how VictoriaMetrics works with data in general — and it turned out to be a very interesting topic.

Let’s first briefly discuss what “metrics” and time series are, and then see how they affect system resources — CPU, memory, and disk.

Metric vs Time Series vs Sample

We all deal with metrics in monitoring — whether it’s Prometheus, VictoriaMetrics, or InfluxDB — and we then use these metrics in our Grafana dashboards or VMAlert alert rules.

But what exactly is a “metric”? And what are time series, samples, and data points? How does the number of different values of a single label for a metric affect disk and memory usage?

For example, in my blog, I usually just use the word “metric” because in 99% of cases, that’s enough to describe the object in question.

However, to work effectively with monitoring systems, it is necessary to understand the difference between these concepts.

Metric vs Time Series vs Sample
What is Metric?
What is the Time Series?
What are Sample and Data Points?
High Cardinality vs High Churn rate
“Life of a Metric”
“Write-path”: vminsert та vmstorage
vmstorage, IndexDB, and TSID
vmstorage and data storage on disk
“Read-path”: search for data from vmselect and vmstorage
Practical example: recording 10,000 metrics and 10,000 labels

What is Metric?

Metric: what is measured

For example — cpu\_usage, memory\_free, http\_requests\_total, database\_connections.

The VictoriaMetrics documentation contains a very precise idea — it’s like the variable names through which we transfer data, see Structure of a metric.

A metric has its own name and, optionally, a set of labels (or tags) that allow you to add more context to a specific measurement.

In addition, labels affect how data for this metric will be stored and searched.

In other words, a metric is a “scheme” that describes what we are measuring and the characteristics (labels) by which we can group data.

Example:

Metric: "cpu_usage{server, core}"

Here:

metric name: cpu_usage
label name: server
label name: core

What is the Time Series?

Time Series: a sequence of data

This is the complete sequence of records grouped for a specific metric and its labels with values — that is, a set of metric\_name{label\_name="label\_value"} - and sorted by time.

Example:

Metric: "cpu_usage{server, core}"
├── Time series: cpu_usage{server="web01", core="0"}
│ ├── 1753857852, 75.5
│ ├── 1753857912, 76.2
│ ├── 1753857972, 74.8
│ └── 1753858032, 73.1
├── Time series: cpu_usage{server="web01", core="1"}
│ ├── 1753857852, 82.3
│ ├── 1753857912, 81.7
│ └── ...
└── Time series: cpu_usage{server="web02", core="0"}
    ├── 1753857852, 45.2
    ├── 1753857912, 47.8
    └── ...

Here, for the metric cpu\_usage{server, core}, we have three different time series:

cpu\_usage{server="web01", core="0"}

At 1753857852 (Wed Jul 30 2025 06:44:12 GMT), the value was 75.5.
At 1753857912 (Wed Jul 30 2025 06:45:12 GMT), the value was 76.2.
…

cpu\_usage{server="web01", core="1"}
at time 1753857852, the value was 82.3
…
cpu\_usage{server="web02", core="0"}
at time 1753857852, the value was 45.2
…

What are Sample and Data Points?

Sample: a specific record in a data sequence (time series).

Sample and Data Point are synonyms and represent a single metric value at a specific point in time.

It looks like (timestamp, value), for example, “1753857852 75.5” — that is, in Unix timestamp 1753857852 the value was 75.5%.

Example:

Metric: "cpu_usage{server, core}"
├── Time series: cpu_usage{server="web01", core="0"}
│ ├── Sample: 1753857852, 75.5
│ ├── Sample: 1753857912, 76.2
│ ├── Sample: 1753857972, 74.8
│ └── Sample: 1753858032, 73.1
├── Time series: cpu_usage{server="web01", core="1"}
│ ├── Sample: 1753857852, 82.3
│ ├── Sample: 1753857912, 81.7
│ └── ...
└── Time series: cpu_usage{server="web02", core="0"}
    ├── Sample: 1753857852, 45.2
    ├── Sample: 1753857912, 47.8
    └── ...

Here:

For the time series cpu\_usage{server="web01", core="0"}, we have four samples:
1753857852, 75.5
1753857912, 76.2
1753857972, 74.8
1753858032, 73.1

And the data for the entire observation period for each unique combination cpu\_usage{server="some\_server", core="some\_core"} will form the same time series, even if this data is collected for years, until the value in either server or core changes.

High Cardinality vs High Churn rate

Both problems have the same “origin,” but differ slightly in essence.

High cardinality is a “persistent problem” that affects data storage, indexing, and search.

It occurs when we have many unique combinations of labels, even if the values of the metrics themselves are infrequent or cease to arrive.

This leads to a large number of live and inactive series, which increases the size of IndexDB, memory usage, and search time. We will discuss IndexDB in more detail later.

See Cardinality explorer in VictoriaMetrics blogs.

High churn rate is an “online problem” where we constantly create new time series due to changes in label values, especially short-lived or dynamic ones (as in Kubernetes — pod\_name, container\_id, job\_id, or something like client\_ip).

This creates a large stream of new entries in IndexDB, loading the CPU, memory, and disk.

“Life of a Metric”

There is a very cool video that I saw many years ago, The Inner Life of the Cell, for some reason it came to mind here.

To understand how the number of labels (more precisely, their values) affects the amount of data in the system and CPU and memory usage, let’s take a look at how the whole process in VictoriaMetrics works “under the hood”.

A wonderful series of posts by Phuong Le will help us with this: How vmagent Collects and Ships Metrics Fast with Aggregation, Deduplication, and More.

There are seven parts, and for a truly “deep dive” into the internal architecture of VictoriaMetrics I highly recommend reading them.

But now we will quickly go through the process of adding new data and searching for it, and focus more on the Churn Rate issue.

“Write-path”: `vminsert` and `vmstorage`

So, let’s start from the beginning: vmagent collects metrics from exporters, and then this data must be written to vmstorage via vminsert.

In the case of vmsingle, all components work in a single process, but for a better picture, let's separate them.

vminsert collects data into its memory, then sends it to vmstorage in blocks of up to 100 megabytes.

At the beginning of each block, vminsert sets the total size of the block, after which vmstorage begins to read the data in it in blocks of 24+n bytes, rows:

The first 8 bytes indicate the size n — the size of the next sector, which contains the name of the metric and its labels.
the second sector — these n bytes with the metric name and labels
The third sector, 8 bytes in size, contains the sample value (“75.5” from the examples above).
The fourth contains a timestamp, another 8 bytes

As a result, a row of 8*3 bytes (24) + n bytes is formed, where n is the length of the metric name and its label.

vmstorage forms its own blocks, with a maximum of 10,000 lines in each:

`vmstorage`, IndexDB, and TSID

Then the most interesting magic begins — Time Series ID, or TSID.

For each unique combination of metric+label+label value, VictoriaMetrics has its own unique ID, which is used to store data and for subsequent data retrieval.

TSID itself is an identifier (see type TSID struct), a purely internal mechanism of VictoriaMetrics itself, which values, unfortunately, we cannot see anywhere in logs:

// TSID is unique id for a time series.
//
// Time series blocks are sorted by TSID.
type TSID struct {
  MetricGroupID uint64

  JobID uint32

  InstanceID uint32

  // MetricID is the unique id of the metric (time series).
  //
  // All the other TSID fields may be obtained by MetricID.
  MetricID uint64
}

With a set of metric names and their tags (labels), vmstorage first checks its TSID Cache. If we already have a generated TSID for this combination, we use it.

If the data is not in the cache (the value of vm\_slow\_row\_inserts\_total increases), vmstorage refers to the IndexDB and starts searching for TSID there.

If TSID is found in IndexDB, it is added to the vmstorage cache, and the process continues:

If these are entirely new metric names and labels with their values, a new TSID is generated and registered in the vmstorage cache.

IndexDB stores two indexes, each containing several mappings between fields and IDs, as described in the section How IndexDB is Structured:

1 — Tag to metric IDs (Global index): each tag (label) is mapped to a metric name (its ID)
2 — Metric ID to TSID (Global index): The ID of each metric is mapped to TSID.
3 — Metric ID to metric name (Global index): mapping the actual name of the metric to its ID
4 — Deleted metric ID: tracker of deleted metric IDs.
5 — Date to metric ID (Per-day index): mapping dates to metric IDs for quick date-based searches (“is there data for this metric on this date?”)
6 — Date with tag to metric IDs (Per-day index): similar to the first Tag to metric IDs mapping, but by date
7 — Date with metric name to TSID (Per-day index): similar to the second Metric ID to TSID mapping, but by metric names and dates

These indexes are stored in memory and periodically flushed to persistent storage IndexDB in the indexdb/ directory, where, as in the data/ directory where the time series themselves are stored, data is merged to optimize storage and search.

For more details, see Part 3 of VictoriaMetrics’ blogs — How vmstorage Processes Data: Retention, Merging, Deduplication.

Returning to the issue of Churn Rate and High cardinality , each individual metric + label creates separate TSIDs, mappings are created in indexes for each label, and with a large amount of new data constantly being written from memory to disk, disk operations are called more often, resulting in a load on the CPU, memory, and disk I/O operations.

`vmstorage` and data storage on disk

Actually, we have already seen the most interesting part — the roles of IndexDB and TSID — but let’s go through the rest of the process.

From the data received from vminsert, we read the data and form our own blocks from rows.

Each row vmstorage no longer stores the name of the metric but its TSID, and for each TSID it contains records with values and time (actually, time series):

They are then stored in memory in “raw-row shards,” after which they form in-memory LSM parts (see Log-structured merge-tree and LSM tree and Sorted string tables (SST)):

Which are then recorded on a disk:

And on the disk, as with IndexDB data, Merge Process, Deduplication, and Downsampling occur in a similar manner.

But what interests us is how it looks on the disk:

$ kk exec -ti vmsingle-vm-k8s-stack-ff6f9bf4c-qt2mj -- tree victoria-metrics-data/data
victoria-metrics-data/data
├── big
│ ├── 2025_09
│ │ └── 18688A4D78E7FBFB
│ │ ├── index.bin
│ │ ├── metadata.json
│ │ ├── metaindex.bin
│ │ ├── timestamps.bin
│ │ └── values.bin
│ ├── 2025_10
│ │ ├── 186A34EE1061F960
│ │ │ ├── index.bin
│ │ │ ├── metadata.json
│ │ │ ├── metaindex.bin
│ │ │ ├── timestamps.bin
│ │ │ └── values.bin
│ │ ├── 186CDDD43EA4892F...

── small
    ├── 2025_09
    │ ├── 18688A4D78E8044E
    │ │ ├── index.bin
    │ │ ├── metadata.json
    │ │ ├── metaindex.bin
    │ │ ├── timestamps.bin
    │ │ └── values.bin
    │ ├── 18688A4D78E80B8F
    │ │ ├── index.bin
    │ │ ├── metadata.json
    │ │ ├── metaindex.bin
    │ │ ├── timestamps.bin
    │ │ └── values.bin
...

Here, data from in-memory parts is “dumped” into small parts, and small parts are then merged into big parts.

Each part contains its own index, which is responsible for mapping data to timestamps and values:

“Read-path”: search for data from `vmselect` and `vmstorage`

When we search for data, vmselect sends a request to vmstorage with metrics, labels (tags), and the date for which the search should be performed.

vmstorage in IndexDB finds the corresponding MetricIDs for all metrics that have this tag using the tag to metric IDs.

Next, based on the Metric ID IndexDB, it finds the corresponding TSIDs in the metric ID to TSID records and returns them to vmstorage.

With TSID — vmtorage checks in-memory, small, and big parts, searching for the required TSID in the metaindex.bin files.

And once it finds the right metadata.bin it reads the corresponding index.bin, which already tells you in which lines of timestamp.bin and values.bin to find the necessary data, which is then returned to the vmselect.

Practical example: recording 10,000 metrics and 10,000 labels

It’s all interesting to read about in theory, but let’s see some practical examples, because it’s always interesting to see how it looks in reality.

What we will do:

will launch two containers with VictoriaMetrics
will write 10,000 metrics to each via API, but:
in one instance, all metrics of the label will have the same value
in the second instance, the label value will constantly change

And then we’ll see how this affected the data size.

Create directories:

$ mkdir vm-data-light
$ mkdir vm-data-heavy

Launch two containers — vm-light and vm-heavy, connect each to the corresponding directory - ./vm-data-light and ./ vm-data-heavy, each listening to its own TCP port:

$ docker run --rm --name vm-light -p 8428:8428 -v ./vm-data-light:/victoria-metrics-data victoriametrics/victoria-metrics
$ docker run --rm --name vm-heavy -p 8429:8428 -v ./vm-data-heavy:/victoria-metrics-data victoriametrics/victoria-metrics

Let’s check the size of the directories now:

$ du -sh vm-data-light/
76K vm-data-light/

$ du -sh vm-data-heavy/
76K vm-data-heavy/

And the number of files in them:

$ find vm-data-light/ -type f | wc -l
5

$ find vm-data-heavy/ -type f | wc -l
5

Everywhere it’s the same.

Now we write two scripts — “light” and “heavy”.

First, the “light” version:

#!/usr/bin/env bash

for i in $(seq 1 10000); do
  echo "my_metric{label=\"value-1\"} $i" | curl -s \
    --data-binary @- \
    http://localhost:8428/api/v1/import/prometheus
done

echo "DONE: stable series sent"

Here, in a loop from 1 to 10,000, we record the metric my\_metric{label="value-1"}, but each time we simply increase the only value we store.

The second script is the “heavy” version:

#!/usr/bin/env bash

for i in $(seq 1 10000); do
  echo "my_metric{label=\"value-$i\"} $i" | curl -s \
    --data-binary @- \
    http://localhost:8429/api/v1/import/prometheus
done

echo "DONE: high churn series sent"

It is similar, but here we also use the value of the variable $i to change the value in the label - my\_metric{label="value-$i"} $i.

Run tests:

$ bash light.sh

$ bash heavy.sh

And compare the data.

Data size in the data/ directory:

$ du -sh vm-data-light/data/
152K vm-data-light/data/

$ du -sh vm-data-heavy/data/
372K vm-data-heavy/data/

Data size in the indexdb/ directory:

$ du -sh vm-data-light/indexdb/
56K vm-data-light/indexdb/

$ du -sh vm-data-heavy/indexdb/
764K vm-data-heavy/indexdb/

Number of files in the data/:

$ find vm-data-light/data/ -type f | wc -l
26

$ find vm-data-heavy/data/ -type f | wc -l
26

Number of files in the indexdb/:

$ find vm-data-light/indexdb/ -type f | wc -l
8

$ find vm-data-heavy/indexdb/ -type f | wc -l
53

8 vs 53!

The directories and files tree in vm-data-light/data/ and vm-data-heavy/data/ will be the same, but let's take a look at IndexDB.

The vm-data-light/indexdb/:

$ tree vm-data-light/indexdb/
vm-data-light/indexdb/
├── 1872FB055ACC4FF8
│ └── parts.json
├── 1872FB055ACC4FF9
│ ├── 1872FB055C5E523F
│ │ ├── index.bin
│ │ ├── items.bin
│ │ ├── lens.bin
│ │ ├── metadata.json
│ │ └── metaindex.bin
│ └── parts.json
├── 1872FB055ACC4FFA
│ └── parts.json
└── snapshots

6 directories, 8 files

Whereas in the vm-data-heavy/indexdb/ the picture is absolutely different:

$ tree vm-data-heavy/indexdb/
vm-data-heavy/indexdb/
├── 1872FB05F8C559B2
│ └── parts.json
├── 1872FB05F8C559B3
│ ├── 1872FB05FA9633D4
│ │ ├── index.bin
│ │ ├── items.bin
│ │ ├── lens.bin
│ │ ├── metadata.json
│ │ └── metaindex.bin
│ ├── 1872FB05FA9633D5
│ │ ├── index.bin
│ │ ├── items.bin
│ │ ├── lens.bin
│ │ ├── metadata.json
│ │ └── metaindex.bin
│ ├── 1872FB05FA9633D6
│ │ ├── index.bin
│ │ ├── items.bin
│ │ ├── lens.bin
│ │ ├── metadata.json
│ │ └── metaindex.bin
│ ├── 1872FB05FA9633D8
│ │ ├── index.bin
│ │ ├── items.bin
│ │ ├── lens.bin
│ │ ├── metadata.json
│ │ └── metaindex.bin
│ ├── 1872FB05FA9633DA
│ │ ├── index.bin
│ │ ├── items.bin
│ │ ├── lens.bin
│ │ ├── metadata.json
│ │ └── metaindex.bin
│ ├── 1872FB05FA9633DB
│ │ ├── index.bin
│ │ ├── items.bin
│ │ ├── lens.bin
│ │ ├── metadata.json
│ │ └── metaindex.bin
│ ├── 1872FB05FA9633DC
│ │ ├── index.bin
│ │ ├── items.bin
│ │ ├── lens.bin
│ │ ├── metadata.json
│ │ └── metaindex.bin
│ ├── 1872FB05FA9633DD
│ │ ├── index.bin
│ │ ├── items.bin
│ │ ├── lens.bin
│ │ ├── metadata.json
│ │ └── metaindex.bin
│ ├── 1872FB05FA9633DE
│ │ ├── index.bin
│ │ ├── items.bin
│ │ ├── lens.bin
│ │ ├── metadata.json
│ │ └── metaindex.bin
│ ├── 1872FB05FA9633DF
│ │ ├── index.bin
│ │ ├── items.bin
│ │ ├── lens.bin
│ │ ├── metadata.json
│ │ └── metaindex.bin
│ └── parts.json
├── 1872FB05F8C559B4
│ └── parts.json
└── snapshots

15 directories, 53 files

That is:

vm-data-light/indexdb: 6 directories, 8 files
vm-data-heavy/indexdb: 15 directories, 53 files

In addition, we can compare the statistics with /api/v1/status/tsdb.

Light version:

$ curl -s http://localhost:8428/prometheus/api/v1/status/tsdb | jq
{
  "status": "success",
  "data": {
    "totalSeries": 1,
    "totalLabelValuePairs": 2,
    "seriesCountByMetricName": [
      {
        "name": "my_metric",
        "value": 1,
        "requestsCount": 0,
        "lastRequestTimestamp": 0
      }
    ],
    "seriesCountByLabelName": [
      {
        "name": " __name__",
        "value": 1
      },
      {
        "name": "label",
        "value": 1
      }
    ],
    "seriesCountByFocusLabelValue": [],
    "seriesCountByLabelValuePair": [
      {
        "name": " __name__ =my_metric",
        "value": 1
      },
      {
        "name": "label=value-1",
        "value": 1
      }
    ],
    "labelValueCountByLabelName": [
      {
        "name": " __name__",
        "value": 1
      },
      {
        "name": "label",
        "value": 1
      }
    ]
  }
}

Whereas in the “heavy version” there are just more numbers of everything:

$ curl -s http://localhost:8429/prometheus/api/v1/status/tsdb | jq
{
  "status": "success",
  "data": {
    "totalSeries": 10000,
    "totalLabelValuePairs": 20000,
    "seriesCountByMetricName": [
      {
        "name": "my_metric",
        "value": 10000,
        "requestsCount": 0,
        "lastRequestTimestamp": 0
      }
    ],
    "seriesCountByLabelName": [
      {
        "name": " __name__",
        "value": 10000
      },
      {
        "name": "label",
        "value": 10000
      }
    ],
    "seriesCountByFocusLabelValue": [],
    "seriesCountByLabelValuePair": [
      {
        "name": " __name__ =my_metric",
        "value": 10000
      },
      ...
      {
        "name": "label=value-1003",
        "value": 1
      },
      {
        "name": "label=value-1004",
        "value": 1
      }
    ],
    "labelValueCountByLabelName": [
      {
        "name": "label",
        "value": 10000
      },
      {
        "name": " __name__",
        "value": 1
      }
    ]
  }
}

That’s all, actually.

I’m going to rewrite the configs for vmagent to drop some of the labels, especially from Karpenter (see Karpenter: monitoring and Grafana dashboard for Kubernetes WorkerNodes) - because there are dozens of them for each metric.

For relabeling in VictoriaMetrics, see Relabeling cookbook. And I hope to create another post about that soon.

Originally published at RTFM: Linux, DevOps, and system administration.

Kubernetes: monitoring processes with process-exporter

Arseny Zinchenko — Mon, 03 Nov 2025 11:21:30 +0000

We are debugging one issue with memory usage in Kubernetes Pods, and decided to look at the memory and number of processes on the nodes.

The problem is that a Kubernetes Pod with Livekit usually consumes about 2 gigabytes of memory, but sometimes there are spikes of up to 10–11 gigabytes, which causes the Pod to crash:

What we want to determine is whether it is one process that starts to “eat” so much memory, or whether many processes are simply being created in the container?

The simplest option here is to use Prometheus Process Exporter, which runs as a DaemonSet, creates its own container on each WorkerNode, and collects statistics from /proc for all or selected processes on EC2.

There is a good (and working) Helm chart kir4h/process-exporter, let’s take it.

Starting Process Exporter
Creating Name Groups
Name Group Template variables and information about each process
The result of our debate

Starting Process Exporter

Add the repository, install:

$ helm repo add kir4h https://kir4h.github.io/charts
$ helm install my-process-exporter kir4h/process-exporter

Or, in our case, we install it via Helm dependency and add the chart to Chart.yaml of our monitoring stack chart:

...
- name: process-exporter
  version: ~1.0
  repository: https://kir4h.github.io/charts
  condition: process-exporter.enabled

Add values for it:

...
process-exporter:
  enabled: true
  tolerations:
  - effect: NoSchedule
    operator: Exists
  - key: CriticalAddonsOnly
    operator: Exists
    effect: NoSchedule
  - key: CriticalAddonsOnly

Deploy and check DaemonSet:

$ kk get ds
NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE
atlas-victoriametrics-process-exporter 9 9 9 9 9 <none> 76m
...

And check ServiceMonitor:

$ kk get serviceMonitor | grep process
atlas-victoriametrics-process-exporter 3d3h

For VictoriaMetrics, VMServiceScrape is automatically created:

$ kk get VMServiceScrape | grep process
atlas-victoriametrics-process-exporter 3d3h operational

We check whether there are metrics, for example, for namedprocess_namegroup_memory_bytes:

Creating Name Groups

We now have data on all processes — we don’t need it.

Specifically, in our case, we are interested in statistics on our Backend API processes, Python processes.

We have three main ones: the Backend API itself, Celery Workers, and Livekit itself, and each service runs in its own Pods from separate Deployments.

Find processes in the submissions and see how they are launched.

Backend API:

root@backend-api-deployment-5695989cb5-rjhv9:/app# ps aux
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 1 0.0 0.2 40348 34712 ? Ss 07:59 0:02 /usr/local/bin/python /usr/local/bin/gunicorn challenge_backend.run_api:app [...]
root 7 1.2 2.5 2075368 414564 ? Sl 07:59 1:32 /usr/local/bin/python /usr/local/bin/gunicorn challenge_backend.run_api:app [...]
root 8 1.1 2.6 1999384 422228 ? Sl 07:59 1:23 /usr/local/bin/python /usr/local/bin/gunicorn challenge_backend.run_api:app [...]
root 9 1.2 2.6 2002492 429192 ? Sl 07:59 1:30 /usr/local/bin/python /usr/local/bin/gunicorn challenge_backend.run_api:app [...]
...

Celery workers:

root@backend-celery-workers-deployment-5bc64557c8-zbq2j:/app# ps aux
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 1 0.2 1.4 544832 236720 ? Ss 07:27 0:24 /usr/local/bin/python /usr/local/bin/celery -A celery_app.app worker [...]
...

Ta Livekit:

root@backend-livekit-agent-deployment-7d9bf86564-qgjzb:/app# ps aux
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 1 0.4 1.8 2112944 294772 ? Ssl 07:06 0:46 python -m cortex.livekit_agent.main start
root 24 0.0 0.0 15788 12860 ? S 07:06 0:00 /usr/local/bin/python -c from multiprocessing.resource_tracker import main;main (34)
root 25 0.0 0.6 342976 102852 ? S 07:06 0:02 /usr/local/bin/python -c from multiprocessing.forkserver import main [...]
...

Add the configuration for process-exporter — describe nameMatchers:

...
process-exporter:
  enabled: true
  tolerations:
    operator: Exists
    effect: NoSchedule
  - key: CriticalAddonsOnly
  config:
    # metrics will be broken down by thread name as well as group name
    threads: true
    # any process that otherwise isn't part of its own group becomes part of the first group found (if any) when walking the process tree upwards
    children: true
    # means that on each scrape the process names are re-evaluated
    recheck: false
    # remove_empty_groups drop empty groups if no processes found
    remove_empty_groups: true
    nameMatchers: 
      # gunicorn (python + uvicorn workers)
      - name: "gunicorn"
        exe:
          - /usr/local/bin/python
        cmdline:
          - ".*gunicorn.*"

      # celery worker
      - name: "celery-worker"
        exe:
          - /usr/local/bin/python
        cmdline:
          - ".*celery.*worker.*"

      # livekit agent
      - name: "livekit-agent"
        exe:
          - python
          - /usr/local/bin/python
        cmdline:
          - ".*cortex.livekit_agent.main.*"

      # livekit multiprocessing helpers
      - name: "livekit-multiproc"
        exe:
          - /usr/local/bin/python
        cmdline:
          - ".*multiprocessing.*"

Here, the exe is a list of executables (there can be several), and in cmdline are the arguments with which the process is launched.

That is, for Livekit, we have exe — “/usr/local/bin/python", and cmdline - "-c from multiprocessing.resource_tracker [...] " or "-c from multiprocessing.forkserver [...]".

Let’s deploy, and now there are only three groups left:

But there are nuances.

First, statistics are collected from each node across the entire group of processes.

That is, if we do the following:

sum(namedprocess_namegroup_memory_bytes{memtype="resident", groupname="celery-worker"}) by (groupname, instance, pod)

This will give us the sum of all RSSs of all Celery workers on the node where the corresponding process-exporter Pod is running:

The second problem is that Process Exporter does not have a label named WorkerNode from which metrics are collected.

Therefore, we can only search manually here — by Pod IP (label instance) we can find its Node:

$ kk get pod -o wide | grep 10.0.45.166
atlas-victoriametrics-process-exporter-4zdzl 1/1 Running 0 6m51s 10.0.45.166 ip-10-0-40-195.ec2.internal <none> <none>

And then see what kind of holes there are on this node:

$ kk describe node ip-10-0-40-195.ec2.internal | grep celery
  dev-backend-api-ns backend-celery-workers-deployment-5bc64557c8-hqhz4 200m (5%) 0 (0%) 1500Mi (10%) 0 (0%) 3h28m
  dev-backend-api-ns backend-celery-workers-long-running-deployment-57d7cb9984-nlfs4 200m (5%) 0 (0%) 1500Mi (10%) 0 (0%) 3h12m
  prod-backend-api-ns backend-celery-workers-deployment-5597dfd875-m7c2n 500m (12%) 0 (0%) 1500Mi (10%) 0 (0%) 99m
  staging-backend-api-ns backend-celery-workers-long-running-deployment-5bb44795b7-pcmj2 200m (5%) 0 (0%) 1500Mi (10%) 0 (0%) 103m

And now let’s take a look at the processes and their RSS:

[root@ip-10-0-40-195 ec2-user]# ps -eo rss,cmd | grep celery
232888 /usr/local/bin/python /usr/local/bin/celery -A celery_app.app worker --loglevel=info -Q default
241656 /usr/local/bin/python /usr/local/bin/celery -A celery_app.app worker --loglevel=info -Q default
...
239232 /usr/local/bin/python /usr/local/bin/celery -A celery_app.app worker --loglevel=info -Q default
252240 /usr/local/bin/python /usr/local/bin/celery -A celery_app.app worker --loglevel=info -Q default
 2416 grep --color=auto celery

On the graph, we have 4,604,280,832 bytes here:

Let’s calculate it ourselves:

[root@ip-10-0-40-195 ec2-user]# ps -eo rss,cmd | grep celery | grep -v grep | awk '{sum += $1} END {print sum*1024 " bytes"}'
4608430080 bytes

Returning to the issue of there being no information on each process: we can obtain an average value for each one, because we have the metric namedprocess_namegroup_num_procs:

Let’s check again on the node itself:

[root@ip-10-0-40-195 ec2-user]# ps aux | grep celery | grep -v grep | wc -l
20

And we can make such a request:

sum(namedprocess_namegroup_memory_bytes{memtype="resident", groupname="celery-worker", instance="10.0.45.166:9256"}) by (groupname, instance, pod)
/
sum(namedprocess_namegroup_num_procs{groupname="celery-worker", instance="10.0.45.166:9256"}) by (groupname, instance, pod)

Result ~230 MB:

As we saw in the ps -eo rss,cmd output.

Name Group Template variables and information about each process

Or, if we really want to see statistics for each process, we can use dynamic names for groupname with {{.PID}} - then a separate group will be formed for each process, see Using a config file: group name:

...
    nameMatchers: 
      # gunicorn (python + uvicorn workers)
      - name: "gunicorn-{{.Comm}}-{{.PID}}"
        exe:
          - python
          - /usr/bin/python
          - /usr/local/bin/python
        cmdline:
          - ".*gunicorn.*"

      # celery worker
      - name: "celery-worker-{{.Comm}}-{{.PID}}"
        exe:
          - python
          - /usr/bin/python
          - /usr/local/bin/python
        cmdline:
          - ".*celery.*worker.*"

      # livekit agent
      - name: "livekit-agent-{{.Comm}}-{{.PID}}"
        exe:
          - python
          - /usr/bin/python
          - /usr/local/bin/python
        cmdline:
          - ".*livekit_agent.*"

      # livekit multiprocessing helpers
      - name: "livekit-multiproc-{{.Comm}}-{{.PID}}"
        exe:
          - python
          - /usr/bin/python
          - /usr/local/bin/python
        cmdline:
          - ".*multiprocessing.*"

As a result, we have the following groups:

But this option is only OK if you need to debug something and disable it, because it will lead to a High cardinality issue.

The result of our debate

Actually, what we needed to find out was whether memory was “leaking” in a single process, or whether multiple processes were simply being created in a single Pod.

To do this, Grafana created a graph with the following query:

sum(namedprocess_namegroup_memory_bytes{memtype="resident", groupname=~"livekit-multiproc-.*"}) by (groupname, instance)

We added graphs with Livekit metrics — lk_agents_active_job_count and lk_agents_child_process_count, and separately - a graph from VictoriaLogs, where we display the number of API requests for each user by the field token_email:

namespace: "prod-backend-api-ns" "GET /cortex/livekit-token" | unpack_json fields (token_email) | stats by (token_email) count()

And as a result, we have the following picture:

Here we see that the same user starts making a bunch of requests to connect to Livekit, which creates a bunch of processes in Livekit Pod (a new Livekit Job for each request), and as a result, the total amount of memory in the Pod goes through the roof, because 40 processes at ~380 MB each is ~15 gigabytes of memory.

However, in each specific process, memory is maintained at a level of 300–400 megabytes.

It remains to figure out why the processes are crashing, but that’s a task for the developers.

Originally published at RTFM: Linux, DevOps, and system administration.

Terraform: AWS EKS Terraform module update from version 20.x to version 21.x

Arseny Zinchenko — Thu, 18 Sep 2025 10:02:45 +0000

AWS EKS Terraform module version v21.0.0 added support for the AWS Provider Version 6.

Documentation — here>>>.

The main changes in the AWS EKS module are the replacement of IRSA with EKS Pod Identity for the Karpenter sub-module:

Native support for IAM roles for service accounts (IRSA) has been removed; EKS Pod Identity is now enabled by default

Also, “The aws-auth sub-module has been removed”, but I personally removed it a long time ago.

Some variables have also been renamed.

I wrote about upgrading from version 19 to 20 in Terraform: EKS and Karpenter — upgrade module version from 19.21 to 20.0, and this time we will follow the same path — change the module versions and see what breaks.

I have a separate “Testing” environment for this, which I first roll out with the current versions of modules/providers, then update the code, deploy the upgrade, and when everything is fixed, I upgrade EKS Production (because we have one cluster on dev/staging/prod).

In Karpenter’s own Helm chart, there seem to be no significant changes, although version 1.6 has already been released. You can update it at the same time, but that’s for another time.

Overall, the upgrade went smoothly, but there were two issues that required some debugging: a problem with the EC2 metadata for AWS Load Balancer Controller during the upgrade, and a problem with EKS Add-ons when creating a new cluster with AWS EKS Terraform module v21.x.

Upgrade AWS EKS Terraform module

Upgrade AWS Provider Version 6

First, change the AWS Provider version — finally, because the open pool requests from Renovate were annoying, and I couldn’t close them.

It’s simple — just change the version to 6:

...
  required_providers {
    aws = {
      source = "hashicorp/aws"
      version = "~> 6.0"
    }
  }
...

Use the pessimistic constraint operator to allow upgrades of all minor versions.

This will be considered both by Renovate, and when executing terraform init -upgrade.

Upgrade `terraform-aws-modules/eks/aws`

Let’s upgrade the EKS module version — change 20 to 21, also with the “~>":

...
module "eks" {
  source = "terraform-aws-modules/eks/aws"
  version = "~> v21.0"
  ...

And Karpenter too, I have it as a separate module:

module "karpenter" {
  source = "terraform-aws-modules/eks/aws//modules/karpenter"
  version = "~> v21.0"
  ...

Run terraform init, and get the " does not match configured version constraint" error, I've already described it in the Terraform: “no available releases match the given constraints post:

$ terraform init
...
registry.terraform.io/hashicorp/aws 5.100.0 does not match configured version constraint >= 4.0.0, >= 4.36.0, >= 4.47.0, >= 5.0.0, ~> 5.14, >= 6.0.0
...

Because .terraform.lock.hcl still contains the old version of the AWS provider:

$ cat envs/test-1-33/.terraform.lock.hcl | grep -A 5 5.100
  version = "5.100.0"
  constraints = ">= 4.0.0, >= 4.33.0, >= 4.36.0, >= 4.47.0, >= 5.0.0, ~> 5.14, >= 5.95.0"

You can drop the file and run terraform init again, or you can run terraform init -upgrade to pull all upgrades at once:

$ terraform init -upgrade

Check .terraform.lock.hcl again - now everything is OK:

$ git diff .terraform.lock.hcl
diff --git a/terraform/envs/test-1-33/.terraform.lock.hcl b/terraform/envs/test-1-33/.terraform.lock.hcl
index bd44714..cb2eace 100644
--- a/terraform/envs/test-1-33/.terraform.lock.hcl
+++ b/terraform/envs/test-1-33/.terraform.lock.hcl
@@ -24,98 +24,85 @@ provider "registry.terraform.io/alekc/kubectl" {
 }

 provider "registry.terraform.io/hashicorp/aws" {
- version = "5.100.0"
- constraints = ">= 4.0.0, >= 4.33.0, >= 4.36.0, >= 4.47.0, >= 5.0.0, ~> 5.14, >= 5.95.0"
+ version = "6.7.0"
+ constraints = ">= 4.0.0, >= 4.36.0, >= 4.47.0, >= 5.0.0, >= 6.0.0, ~> 6.0"
   hashes = [
...

Let’s run terraform plan and see what breaks.

Renamed variables в `terraform-aws-modules/eks/aws`

The first, as expected, were errors about missing variables, because they had been renamed in the module:

$ terraform plan -var-file=test-1-33.tfvars
...
│ Error: Unsupported argument
│ 
│ on ../../modules/atlas-eks/eks.tf line 34, in module "eks":
│ 34: cluster_name = "${var.env_name}-cluster"
│ 
│ An argument named "cluster_name" is not expected here.
╵
╷
│ Error: Unsupported argument
│ 
│ on ../../modules/atlas-eks/eks.tf line 38, in module "eks":
│ 38: cluster_version = var.eks_version
│ 
│ An argument named "cluster_version" is not expected here.
╵
╷
│ Error: Unsupported argument
│ 
│ on ../../modules/atlas-eks/eks.tf line 42, in module "eks":
│ 42: cluster_endpoint_public_access = var.eks_params.cluster_endpoint_public_access
│ 
│ An argument named "cluster_endpoint_public_access" is not expected here.
╵
╷
│ Error: Unsupported argument
│ 
│ on ../../modules/atlas-eks/eks.tf line 46, in module "eks":
│ 46: cluster_enabled_log_types = var.eks_params.cluster_enabled_log_types
│ 
│ An argument named "cluster_enabled_log_types" is not expected here.
╵
╷
│ Error: Unsupported argument
│ 
│ on ../../modules/atlas-eks/eks.tf line 50, in module "eks":
│ 50: cluster_addons = {
│ 
│ An argument named "cluster_addons" is not expected here.
╵
╷
│ Error: Unsupported argument
│ 
│ on ../../modules/atlas-eks/eks.tf line 148, in module "eks":
│ 148: cluster_security_group_name = "${var.env_name}-cluster-sg"
│ 
│ An argument named "cluster_security_group_name" is not expected here.
...

Let’s go to the upgrade documentation and find out what the variables are now called:

cluster_name => name
cluster_version => kubernetes_version
cluster_endpoint_public_access => endpoint_public_access
cluster_enabled_log_types => enabled_log_types
cluster_addons -> addons
cluster_security_group_name -> security_group_name

Although, in my opinion, the prefix cluster_* would have been better, because we have node_security_group_name, and there was cluster_security_group_name - it is clear which parameter is for what.

And now there is node_security_group_name and "some" security_group_name.

Removed variables в terraform-aws-`modules/eks/aws//modules/karpenter`

OK, edit the variable names in the main module code, run terraform plan again - now we have errors for changes in the karpenter module:

...
 Error: Unsupported argument
│ 
│ on ../../modules/atlas-eks/karpenter.tf line 7, in module "karpenter":
│ 7: irsa_oidc_provider_arn = module.eks.oidc_provider_arn
│ 
│ An argument named "irsa_oidc_provider_arn" is not expected here.
╵
╷
│ Error: Unsupported argument
│ 
│ on ../../modules/atlas-eks/karpenter.tf line 8, in module "karpenter":
│ 8: irsa_namespace_service_accounts = ["karpenter:karpenter"]
│ 
│ An argument named "irsa_namespace_service_accounts" is not expected here.
╵
╷
│ Error: Unsupported argument
│ 
│ on ../../modules/atlas-eks/karpenter.tf line 14, in module "karpenter":
│ 14: enable_irsa = true
│ 
│ An argument named "enable_irsa" is not expected here.
...

They were removed because IRSA no longer exists — an EKS Pod Identity will now be created for Karpenter, see [main.tf#L92](https://github.com/terraform-aws-modules/terraform-aws-eks/blob/master/modules/karpenter/main.tf#L92).

I wrote about EKS Pod Identities in AWS: EKS Pod Identities — a replacement for IRSA? Simplifying IAM access management and in Terraform: managing EKS Access Entries and EKS Pod Identities.

Let’s remove them:

...
  #irsa_oidc_provider_arn = module.eks.oidc_provider_arn
  #irsa_namespace_service_accounts = ["karpenter:karpenter"]
  #enable_irsa = true
...

Run terraform plan again.

Important: Karpenter’s EKS Identity Provider Namespace

And here is an important point:

...
  # module.atlas_eks.module.karpenter.aws_eks_pod_identity_association.karpenter[0] will be created
  + resource "aws_eks_pod_identity_association" "karpenter" {
      ...
      + namespace = "kube-system"
      + region = "us-east-1"
      + role_arn = "arn:aws:iam::492***148:role/KarpenterIRSA-atlas-eks-test-1-33-cluster"
      + service_account = "karpenter"
...

eks_pod_identity_association will be created for the Kubernetes Namespace "kube-system".

If you have Karpenter running in a different namespace, you need to specify it explicitly when calling the module:

...
module "karpenter" {
  source = "terraform-aws-modules/eks/aws//modules/karpenter"
  version = "~> v21.0"

  cluster_name = module.eks.cluster_name
  namespace = "karpenter"
...

Otherwise, Karpenter will broke, and the WorkerNode Group upgrade will fail because a Node will wait for the Karpenter, which will be in the CrashLoopbackoff, and the Group upgrade will fail.

`eks_managed_node_groups`: attribute "taints": map of object required

Now there is an error with node group tags:

...
│ The given value is not suitable for module.atlas_eks.module.eks.var.eks_managed_node_groups declared at .terraform/modules/atlas_eks.eks/variables.tf:1205,1-35: element "test-1-33-default": attribute "taints": map of object required.
...

Why? Because:

Variable definitions now contain detailed object types in place of the previously used any type.

See diff 20 vs 21:

So now it should be map(object):

...
  type = map(object({
    key = string
    value = optional(string)
    effect = string
  }))
...

And I have taints currently been passed from a variable with an object set(map(string)):

...
variable "eks_managed_node_group_params" {
  description = "EKS Managed NodeGroups setting, one item in the map() per each dedicated NodeGroup"
  type = map(object({
    min_size = number
    max_size = number
    desired_size = number
    instance_types = list(string)
    capacity_type = string
    taints = set(map(string))
    max_unavailable_percentage = number
  }))
}
...

With the following values:

...
eks_managed_node_group_params = {
  default_group = {
    min_size = 1
    max_size = 1
    desired_size = 1
    instance_types = ["t3.medium"]
    capacity_type = "ON_DEMAND"
    taints = [
      {
        key = "CriticalAddonsOnly"
        value = "true"
        effect = "NO_SCHEDULE"
      },
      {
        key = "CriticalAddonsOnly"
        value = "true"
        effect = "NO_EXECUTE"
      }
    ]
    max_unavailable_percentage = 100
  }
}
...

So what needs to be done is to change the declaration of the variable in my code:

...
variable "eks_managed_node_group_params" {
  description = "EKS Managed NodeGroups setting, one item in the map() per each dedicated NodeGroup"
  type = map(object({
    min_size = number
    max_size = number
    desired_size = number
    instance_types = list(string)
    capacity_type = string
    #taints = set(map(string))
    taints = optional(map(object({
      key = string
      value = optional(string)
      effect = string
    })))
    max_unavailable_percentage = number
  }))
}
...

And update the values — add keys for map{}:

...
eks_managed_node_group_params = {
  default_group = {
    min_size = 1
    max_size = 1
    desired_size = 1
    instance_types = ["t3.medium"]
    capacity_type = "ON_DEMAND"
    # taints = [
    # {
    # key = "CriticalAddonsOnly"
    # value = "true"
    # effect = "NO_SCHEDULE"
    # },
    # {
    # key = "CriticalAddonsOnly"
    # value = "true"
    # effect = "NO_EXECUTE"
    # }
    # ]
      taints = {
        critical_no_sched = {
          key = "CriticalAddonsOnly"
          value = "true"
          effect = "NO_SCHEDULE"
        },
        critical_no_exec = {
          key = "CriticalAddonsOnly"
          value = "true"
          effect = "NO_EXECUTE"
        }
      }
    max_unavailable_percentage = 100
  }
}
...

Run terraform plan again, and now everything works without errors.

Let’s deploy the updates.

Deploying changes

Run terraform apply, and now we have a new resource with EKS Pod Identity Association for Karpenter - module.atlas_eks.module.karpenter.aws_eks_pod_identity_association.karpenter:

Which wasn’t here in the old cluster with v20.

ALB Controller error: “failed to fetch VPC ID from instance metadata”

There was also a problem with AWS Load Balancer Controller, because after the upgrade it could not connect to IMDS, probably due to switching to v2, see AWS: Instance Metadata Service v1 vs IMDS v2 and working with Kubernetes Pod and Docker containers:

...
{"level":"error","ts":"2025-08-06T07:25:40Z"," logger":"setup","msg":"unable to initialize AWS cloud","error":"failed to get VPC ID: failed to fetch VPC ID from instance metadata: error in fetching vpc id through ec2 metadata: get mac metadata: operation error ec2imds: GetMetadata, canceled, context deadline exceeded"}
...

Actually, we can just pass the parameters explicitly; see the documentation Using the Amazon EC2 instance metadata server version 2 (IMDSv2).

Note the --aws-vpc-tag-key:

optional flag — aws-vpc-tag-key if you have a different key for the tag other than “Name”

First, let’s try setting the parameters manually to check that it works:

Everything is working now.

Now the parameters for the Helm chart, see its values.yaml#L163 — my controllers are installed from aws-ia/eks-blueprints-addons/aws in Terraform when creating the cluster, so set here:

...
    values = [
      <<-EOT
        replicaCount: 1
        region: ${var.aws_region}
        vpcId: ${var.vpc_id}
        tolerations:
        - key: CriticalAddonsOnly
          operator: Exists
      EOT
    ]
...

Start the deployment:

Everything works.

Issue: Node Group Status CREATE_FAILED

Here I will describe a problem that arose only when creating a new EKS cluster with module v21 — upgrading an existing cluster proceeds without these issues.

Actually, here’s the problem: the cluster was created, everything seems OK, but it hangs for a long time on creating the Node Group, and then crashes with the error “ unexpected state ‘CREATE_FAILED’ ”:

...
╷
│ Error: waiting for EKS Node Group (atlas-eks-test-1-33-cluster:test-1-33-default-20250801112636765600000014) create: unexpected state 'CREATE_FAILED', wanted target 'ACTIVE'. last error: i-03f2c73c7211880f7: NodeCreationFailure: Unhealthy nodes in the kubernetes cluster
...

Although there is an EC2 Auto Scaling Group created, and it has an EC2 up and running.

Why?

So, the problem is that WorkerNode has been created but cannot connect to Kubernetes.

The first thing that comes to mind is to check the Security Group, but everything appears to be correct here — all the rules are correct. I compared it with the current EKS cluster, which was created with AWS EKS Terraform module v20.x — everything is the same.

Problem with IAM? EC2 doesn’t have permissions to access the cluster? Again, compare with the old cluster, and everything is OK.

“Check the logs, Billy!”

The funny thing is that SSH is configured on all my EC2 instances, but only for nodes created with Karpenter, as I wrote in AWS: Karpenter and SSH for Kubernetes WorkerNodes.

The current problem arose in the “default” NodeGroup, where various controllers are launched.

So, let’s connect via the AWS Console and select Connect:

Then, in EC2 Instance Connect, select “Connect using a Private IP” and select an existing EC2 Instance Connect Endpoint or quickly create a new one.

Set the username — for Amazon Linux, it is ec2-user:

And let’s look at the logs:

“Container runtime network not ready — cni plugin not initialized”

Actually:

Aug 01 13:26:04 ip-10-0-48-198.ec2.internal kubelet[1619]: E0801 13:26:04.989799 1619 kubelet.go:3126] "Container runtime network not ready" networkReady="NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni plugin not initialized"

Wow…

Okay, what’s the situation with VPC CNI?

Let’s go check out EKS Add-ons, and…

It’s completely empty.

Let’s look at the log terraform apply - and we see "Read complete", but there is no "Creating...":

...
module.atlas_eks.module.eks.data.aws_eks_addon_version.this["vpc-cni"]: Read complete after 0s [id=vpc-cni]
...

Let’s check if there are any containers on the node — maybe there are some errors?

Wow, once again…

Nothing at all.

Even then, I went back to GitHub Issues and searched for “addon” and found this issue: Managed EKS Node Groups boot without CNI, but addon is added after node group.

Actually, yes — the problem arose due to the absence of the before_compute parameter.

Although it’s a little strange, because it was added in version v19.9, the last time I deployed a cluster from scratch was with v20, and this problem did not occur.

Even more, when I created the Testing cluster from the master branch, where none of the updates described here have been applied, and module version v20 is still used — everything is working without any problems.

And in diff 20 vs 21 I don’t see any significant changes related to before_compute.

However, since this only applies to creating a new cluster, we do not need to add before_compute when simply upgrading. But if you do add it, the add-ons will be recreated.

The before_compute itself was added to allow specifying which addons to create before WorkerNodes and which after. See main.tf#L797 and comments to PR #2478.

Add as in the examples EKS Managed Node Group:

...
    vpc-cni = {
      addon_version = var.eks_addon_versions.vpc_cni
      before_compute = true
      configuration_values = jsonencode({
        env = {
          ENABLE_PREFIX_DELEGATION = "true"
          WARM_PREFIX_TARGET = "1"
          AWS_VPC_K8S_CNI_EXTERNALSNAT = "true"
        }
      })
    }
    aws-ebs-csi-driver = {
      addon_version = var.eks_addon_versions.aws_ebs_csi_driver
      service_account_role_arn = module.ebs_csi_irsa_role.iam_role_arn
    }
    eks-pod-identity-agent = {
      addon_version = var.eks_addon_versions.eks_pod_identity_agent
      before_compute = true
    }
...

Run terraform apply again, and here it is:

...
module.atlas_eks.module.eks.aws_eks_addon.before_compute["vpc-cni"]: Creating...
...
module.atlas_eks.module.eks.aws_eks_addon.before_compute["vpc-cni"]: Creation complete after 46s [id=atlas-eks-test-1-33-cluster:vpc-cni]
...

And in the AWS Console:

NodeGroup created without errors:

...
module.atlas_eks.module.eks.module.eks_managed_node_group["test-1-33-default"].aws_eks_node_group.this[0]: Still creating... [01m40s elapsed]
module.atlas_eks.module.eks.module.eks_managed_node_group["test-1-33-default"].aws_eks_node_group.this[0]: Creation complete after 1m49s [id=atlas-eks-test-1-33-cluster:test-1-33-default-20250801142042855800000003]
...

Done.

Originally published at RTFM: Linux, DevOps, and system administration.

Kubernetes: PVC in a StatefulSet, and the “Forbidden updates to statefulset spec” error

Arseny Zinchenko — Wed, 17 Sep 2025 12:36:49 +0000

Kubernetes: PVC in StatefulSet, and the “Forbidden updates to statefulset spec” error

We have a VictoriaLogs Helm chart with a PVC size of 30 GB, which is no longer enough for us, and we need to increase it.

But the problem is that .spec.volumeClaimTemplates[*].spec.resources.requests.storage in STS is immutable, that is, we can't just change the size through values.yaml file, because it will lead to the error "Forbidden: updates to statefulset spec for fields other than 'replicas', 'ordinals', 'template', 'updateStrategy', 'revisionHistoryLimit', 'persistentVolumeClaimRetentionPolicy' and 'minReadySeconds' are forbidden".

The chart values now look like this:

victoria-logs-single:
  server:
    persistentVolume:
      enabled: true
      storageClassName: gp2-retain
      size: 30Gi
    retentionPeriod: 7d

And with the default type of StatefulSet in the chart, the volumeClaimTemplates is used to create PVCs:

...  
volumeClaimTemplates:
    - apiVersion: v1
      kind: PersistentVolumeClaim
      metadata:
        name: server-volume
        ...
      spec:
        ...
        resources:
          requests:
            storage: {{ $app.persistentVolume.size }}
...

If instead of STS there was a Deployment type, then in the VictoriaLogs chart this would lead to the creation of a separate PVC — see the pvc.yaml.

You could simply create a separate PVC yourself and connect it through the existingClaim value, but you already have a PersistentVolume, and you don't want to create a new one and migrate data (although you can if you need to, see VictoriaMetrics: migrating VMSingle and VictoriaLogs data between Kubernetes clusters, but there will be a down time), so let's see how we can solve this differently - without deleting Pods and without stopping the service.

storageClassName and AllowVolumeExpansion

The storageClas used to create a Persistent Volume must support AllowVolumeExpansion - see Volume expansion:

$ kk describe storageclass gp2-retain
Name: gp2-retain
...
Provisioner: kubernetes.io/aws-ebs
Parameters: <none>
AllowVolumeExpansion: True
MountOptions: <none>
ReclaimPolicy: Retain
VolumeBindingMode: WaitForFirstConsumer
...

Create this storageClass when creating an EKS cluster from a simple manifest:

...
resource "kubectl_manifest" "storageclass_gp2_retain" {

  yaml_body = <<YAML
    apiVersion: storage.k8s.io/v1
    kind: StorageClass
    metadata:
      name: gp2-retain
    provisioner: kubernetes.io/aws-ebs
    reclaimPolicy: Retain
    allowVolumeExpansion: true
    volumeBindingMode: WaitForFirstConsumer
  YAML
}
...

Although there is a dedicated storage_class resource for Terraform, and would be better to use it instead for the kubectl_manifest.

And the kubernetes.io/aws-ebs driver is already deprecated (OMG, since Kubernetes 1.17!), it's time to update to ebs.csi.aws.com.

But we’ll fix this later, right now the goal is to simply increase the disk.

Reproducing the issue

For the test, let’s write our own STS with volumeClaimTemplates:

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: demo-sts
spec:
  serviceName: demo-sts-svc
  replicas: 1
  selector:
    matchLabels:
      app: demo
  template:
    metadata:
      labels:
        app: demo
    spec:
      containers:
        - name: app
          image: busybox
          command: ["sh", "-c", "sleep 3600"]
          volumeMounts:
            - name: data
              mountPath: /data
  volumeClaimTemplates:
    - metadata:
        name: data
      spec:
        accessModes: ["ReadWriteOnce"]
        storageClassName: gp2-retain
        resources:
          requests:
            storage: 1Gi

In volumeClaimTemplates, set the storageClassName and the size to 1 gigabyte.

Deploy:

$ kk apply -f test-sts-pvc.yaml 
statefulset.apps/demo-sts created

Check the PVC:

$ kk get pvc
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS VOLUMEATTRIBUTESCLASS AGE
data-demo-sts-0 Bound pvc-31a9a547-7547-4d34-bb2d-2c7015b9e0f3 1Gi RWO gp2-retain <unset> 15s

Now, if we want to increase the size via volumeClaimTemplates from 1Gi to 2Gi:

...
  volumeClaimTemplates:
    ...
        resources:
          requests:
            storage: 2Gi

Then we get an error:

$ kk apply -f test-sts-pvc.yaml 
The StatefulSet "demo-sts" is invalid: spec: Forbidden: updates to statefulset spec for fields other than 'replicas', 'ordinals', 'template', 'updateStrategy', 'revisionHistoryLimit', 'persistentVolumeClaimRetentionPolicy' and 'minReadySeconds' are forbidden

The Fix

But we can get around this very easily:

edit the PVC manually — set a new size
delete STS with the --cascade=orphan - see Delete owner objects and orphan dependents
create STS again
…
profit!

Let’s try it.

Note : before changing disks, don’t forget about backups!

Edit the PVC manually — change resources.requests.storage from 1Gi to 2Gi:

Check the Events of this PVC:

$ kk describe pvc data-demo-sts-0
...
  Normal ExternalExpanding 40s volume_expand CSI migration enabled for kubernetes.io/aws-ebs; waiting for external resizer to expand the pvc
  Normal Resizing 40s external-resizer ebs.csi.aws.com External resizer is resizing volume pvc-31a9a547-7547-4d34-bb2d-2c7015b9e0f3
  Normal FileSystemResizeRequired 35s external-resizer ebs.csi.aws.com Require file system resize of volume on node

And after a few more seconds, it’s done:

...
  Normal FileSystemResizeSuccessful 19s kubelet

Check CAPACITY:

$ kk get pvc
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS VOLUMEATTRIBUTESCLASS AGE
data-demo-sts-0 Bound pvc-31a9a547-7547-4d34-bb2d-2c7015b9e0f3 2Gi RWO gp2-retain <unset> 4m7s

2Gi, everything is OK.

And now we also have 2 gigabytes in the Pod itself:

$ kk exec -ti demo-sts-0 -- df -h /data
Filesystem Size Used Available Use% Mounted on
/dev/nvme7n1 1.9G 24.0K 1.9G 0% /data

But if we try to deploy the changes to volumeClaimTemplates.spec.resources.requests.storage again, we will still get an error:

$ kk apply -f test-sts-pvc.yaml 
The StatefulSet "demo-sts" is invalid: spec: Forbidden: updates to statefulset spec for fields other than 'replicas', 'ordinals', 'template', 'updateStrategy', 'revisionHistoryLimit', 'persistentVolumeClaimRetentionPolicy' and 'minReadySeconds' are forbidden

So, delete the STS itself, but leave all its dependent objects:

$ kubectl delete statefulset demo-sts --cascade=orphan 
statefulset.apps "demo-sts" deleted

Check if the Pod is alive:

$ kk get pod
NAME READY STATUS RESTARTS AGE
demo-sts-0 1/1 Running 0 3m13s

And now we just create STS again, with a new value in the volumeClaimTemplates.spec.resources.requests.storage:

$ kk apply -f test-sts-pvc.yaml 
statefulset.apps/demo-sts created

Done.

Originally published at RTFM: Linux, DevOps, and system administration.

Kubernetes: what are the Kubernetes Operator and CustomResourceDefinition

Arseny Zinchenko — Tue, 16 Sep 2025 07:27:07 +0000

Perhaps everyone has used operators in Kubernetes, for example, PostgreSQL operator, VictoriaMetrics Operator.

But what’s going on under the hood? How and to what are CustomResourceDefinition (CRD) applied, and what is an “operator”?

And finally, what is the difference between a Kubernetes Operator and a Kubernetes Controller?

In the previous part — Kubernetes:Kubernetes APIs, API Groups, CRDs, etcd — we dug a little deeper into how the Kubernetes API works and what a CRD is, and now we can try to write our own micro-operator, a simple MVP, and use it as an example to understand the details.

Kubernetes Controller vs Kubernetes Operator
What is: Kubernetes Controller
What is: Kubernetes Operator
Kubernetes Operator frameworks
Creating a CustomResourceDefinition
Creating a Kubernetes Operator with Kopf
Resource templates: Kopf and Kubebuilder
And what about in real operators?

Kubernetes Controller vs Kubernetes Operator

So, what is the main difference between Controllers and Operators?

What is: Kubernetes Controller

Simply put, a Controller is just some service that monitors resources in a cluster and brings their state in line with how this state is described in the database — etcd.

In Kubernetes, we have a set of default controllers — Core Controllers within the Kube Controller Manager, such as the ReplicaSet Controller, which checks the number of pods in the Deployment against the replicas value, or the Deployment Controller, which controls the creation and update of ReplicaSets, or the PersistentVolume Controller and PersistentVolumeClaim Binder for working with disks, etc.

In addition to these default controllers, you can create your own controller or use an existing one, such as ExternalDNS Controller. These are examples of custom controllers.

Controllers work in a control loop — a cyclic process in which they constantly check the resources assigned to them — either to change existing resources in the system or to respond to the addition of new ones.

During each check*(reconciliation loop*), the Controller compares the current state of the resource and compares it with the desired state — that is, the parameters specified in its manifest when the resource was created or updated.

If the desired state does not correspond to the current state, the controller performs the necessary actions to bring these states into alignment.

What is: Kubernetes Operator

Kubernetes Operator, in turn, is a kind of “controller on steroids”: in fact, Operator is a Custom Controller in the sense that it has its own service in the form of a Pod that communicates with the Kubernetes API to receive and update information about resources.

But if ordinary controllers work with “default” resource types (Pod, Endpoint Slice, Node, PVC), then for Operator we describe our own custom resources using a manifest with Custom Resource.

And how these resources will look like and what parameters they will have — we set through CustomResourceDefinition which are written to the Kubernetes database and added to the Kubernetes API, and thus the Kubernetes API allows our custom Controller to operate with these resources.

That is:

Controller is a component, a service, and Operator is a combination of one or more custom Controllers and corresponding CRDs
Controller — responds to changes in resources, and Operator — adds new types of resources + controller that controls these resources

Kubernetes Operator frameworks

There are several solutions that simplify the creation of operators.

The main ones are Kubebuilder, a framework for creating controllers in Go, and Kopf, a framework in Python.

There is also the Operator SDK, which allows you to work with controllers even with Helm, without code.

At first, I was thinking of doing it in bare Go, without any frameworks, to better understand how everything works under the hood — but this post started to turn into 95% Golang.

And since the main idea of the post was to show conceptually what a Kubernetes Operator is, what role CustomResourceDefinitions play, and how they interact with each other and allow you to manage resources, I decided to use Kopf because it’s very simple and quite suitable for these purposes.

Creating a CustomResourceDefinition

Let’s start with writing the CRD.

Actually, CustomResourceDefinition is just a description of what fields our custom resource will have so that the controller can use them through the Kubernetes API to create real resources — whether they are some resources in Kubernetes itself, or external ones like AWS Load Balancer or AWS Route 53.

What we will do: we will write a CRD that will describe the MyApp resource, and this resource will have fields for the Docker image and a custom field with some text that will then be written to the Kubernetes Pod logs.

Kubernetes documentation on CRD — Extend the Kubernetes API with CustomResourceDefinitions.

Create the file myapp-crd.yaml:

apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: myapps.demo.rtfm.co.ua
spec:
  group: demo.rtfm.co.ua
  versions:
    - name: v1
      served: true
      storage: true
      schema:
        openAPIV3Schema:
          type: object
          properties:
            spec:
              type: object
              properties:
                image:
                  type: string
                banner: 
                  type: string
                  description: "Optional banner text for the application"
  scope: Namespaced
  names:
    plural: myapps
    singular: myapp
    kind: MyApp
    shortNames:
      - ma

Here:

spec.group: demo.rtfm.co.ua: create a new API Group, all resources of this type will be available at /apis/demo.rtfm.co.ua/...
versions: list of versions of the new resource
name.v1: we will have an only one version
served: true: add a new resource to the Kube API - you can do kubectl get myapp (GET /apis/demo.rtfm.co.ua/v1/myapps)
storage: true: this version will be used for storage in etcd (if several versions are described, only one should be with storage: true)
schema:
openAPIV3Schema: describe the API scheme according to the OpenAPI v3
type: object: describe an object with nested fields (key: value)
properties: what fields the object will have
spec: what we can use in YAML manifests when creating
type: object - describe the following fields:
properties:
image.type: string: a Docker image
banner.type: string: our custom field through which we will add some entry to the resource logs
scope: Namespaced: all resources of this type will exist in a specific Kubernetes Namespace
names:
plural: myapps: the resources will be available through /apis/demo.rtfm.co.ua/v1/namespaces/<ns>/myapps/, and how we can "access" the resource (kubectl get myapp), used in RBAC where you need to specify resources:["myapps"]
singular: myapp: alias for convenience
shortNames:[ma]: short alias for convenience

Let’s start Minikube:

$ minikube start

Add the CRD:

$ kk apply -f myapp-crd.yaml 
customresourcedefinition.apiextensions.k8s.io/myapps.demo.rtfm.co.ua created

Let’s look at the Groups API:

$kubectl api-versions 
...
demo.rtfm.co.ua/v1 
...

And a new resource in this API Group:

$ kubectl api-resources --api-group=demo.rtfm.co.ua
NAME SHORTNAMES APIVERSION NAMESPACED KIND
myapps ma demo.rtfm.co.ua/v1 true MyApp

OK — we have created a CRD, and now we can even create a CustomResource (CR).

Create the file myapp-example-resource.yaml:

apiVersion: demo.rtfm.co.ua/v1 # matches the CRD's group and version
kind: MyApp # kind from the CRD's 'spec.names.kind'
metadata:
  name: example-app # name of this custom resource
  namespace: default # namespace (CRD has scope: Namespaced)
spec:
  image: nginx:latest # container image to use (from our schema)
  banner: "This pod was created by MyApp operator 🚀"

Deploy:

$ kk apply -f myapp-example-resource.yaml 
myapp.demo.rtfm.co.ua/example-app created

And check:

$ kk get myapp
NAME AGE
example-app 15s

But there are no resources of type Pod — because we do not have a controller that will work with this type of resources.

Creating a Kubernetes Operator with Kopf

So, we will use Kopf to create a Kubernetes Pod, but using our own CRD.

Create a Python virtual environment:

$ python -m venv venv 
$ . ./venv/bin/activate 
(venv)

Add dependencies — requirements.txt file :

kopf 
kubernetes
PyYAML

Install them — with pip or uv:

$ pip install -r requirements.txt

Let’s write the operator code:

import os
import kopf
import kubernetes
import yaml

# use kopf to register a handler for the creation of MyApp custom resources
@kopf.on.create('demo.rtfm.co.ua', 'v1', 'myapps')
# this function will be called when a new MyApp resource is created
def create_myapp(spec, name, namespace, logger, **kwargs):
    # get image value from the spec of the CustomResource manifest
    image = spec.get('image')
    if not image:
        raise kopf.PermanentError("Field 'spec.image' must be provided.")

    # get optional banner value from the CR manifest spec
    banner = spec.get('banner')

    # load pod template YAML from file
    path = os.path.join(os.path.dirname( __file__ ), 'pod.yaml')
    with open(path, 'rt') as f:
        pod_template = f.read()

    # render pod YAML with provided values
    pod_yaml = pod_template.format(
        name=f"{name}-pod",
        image=image,
        app_name=name,
    )
    # create Pod difinition from the rendered YAML
    # it uses PyYAML to parse the YAML string into a Python dictionary
    # which can be used by Kubernetes API client
    # it is used to create a Pod object in Kubernetes
    pod_spec = yaml.safe_load(pod_yaml)

    # inject banner as environment variable if provided
    if banner:
        # it is used to add a new environment variable into the container spec
        container = pod_spec['spec']['containers'][0]
        env = container.setdefault('env', [])
        env.append({
            'name': 'BANNER',
            'value': banner
        })

    # create Kubernetes CoreV1 API client
    # used to interact with the Kubernetes API
    api = kubernetes.client.CoreV1Api()

    try:
        # it sends a request to the Kubernetes API to create a new Pod
        # uses 'create_namespaced_pod' method to create the Pod in the specified namespace
        # 'namespace' is the namespace where the Pod will be created
        # 'body' is the Pod specification that was created from the YAML template
        api.create_namespaced_pod(namespace=namespace, body=pod_spec)
        logger.info(f"Pod {name}-pod created.")
    except kubernetes.client.exceptions.ApiException as e:
        logger.error(f"Failed to create pod {name}-pod: {e}")

Create a template that will be used by our Operator to create resources:

apiVersion: v1
kind: Pod
metadata:
  name: {name}
  labels:
    app: {app_name}
spec:
  containers:
    - name: {app_name}
      image: {image}
      ports:
        - containerPort: 80
      env:
        - name: BANNER
          value: "" # will be overridden in code if provided
      command: ["/bin/sh", "-c"]
      args:
        - |
          if [-n "$BANNER"]; then
            echo "$BANNER";
          fi
          exec sleep infinity

Run the operator with kopf run myoperator.py.

We already have a CustomResource created, and the Operator should see it and create a Kubernetes Pod:

$ kopf run myoperator.py --verbose
...
[2025-07-18 13:59:58,201] kopf._cogs.clients.w [DEBUG] Starting the watch-stream for customresourcedefinitions.v1.apiextensions.k8s.io cluster-wide.
[2025-07-18 13:59:58,201] kopf._cogs.clients.w [DEBUG] Starting the watch-stream for myapps.v1.demo.rtfm.co.ua cluster-wide.
[2025-07-18 13:59:58,305] kopf.objects [DEBUG] [default/example-app] Creation is in progress: {'apiVersion': 'demo.rtfm.co.ua/v1', 'kind': 'MyApp', 'metadata': {'annotations': {'kubectl.kubernetes.io/last-applied-configuration': '{"apiVersion":"demo.rtfm.co.ua/v1","kind":"MyApp","metadata":{"annotations":{},"name":"example-app","namespace":"default"},"spec":{"banner":"This pod was created by MyApp operator 🚀","image":"nginx:latest","replicas":3}}\n'}, 'creationTimestamp': '2025-07-18T09:55:42Z', 'generation': 2, 'managedFields': [{'apiVersion': 'demo.rtfm.co.ua/v1', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:metadata': {'f:annotations': {'.': {}, 'f:kubectl.kubernetes.io/last-applied-configuration': {}}}, 'f:spec': {'.': {}, 'f:banner': {}, 'f:image': {}, 'f:replicas': {}}}, 'manager': 'kubectl-client-side-apply', 'operation': 'Update', 'time': '2025-07-18T10:48:27Z'}], 'name': 'example-app', 'namespace': 'default', 'resourceVersion': '2955', 'uid': '8b674a99-05ab-4d4b-8205-725de450890a'}, 'spec': {'banner': 'This pod was created by MyApp operator 🚀', 'image': 'nginx:latest', 'replicas': 3}}
...
[2025-07-18 13:59:58,325] kopf.objects [INFO] [default/example-app] Pod example-app-pod created.
[2025-07-18 13:59:58,326] kopf.objects [INFO] [default/example-app] Handler 'create_myapp' succeeded.
...

Check the Pod:

$ kk get pod
NAME READY STATUS RESTARTS AGE
example-app-pod 1/1 Running 0 68s

And its logs:

$ kk logs -f example-app-pod 
This pod was created by MyApp operator 🚀

So, the Operator launched the Pod using our CustomResource in which he took the spec.banner field with the string "This pod was created by MyApp operator 🚀", and executed the command /bin/sh -c " $BANNER" command in the pod.

Resource templates: Kopf and Kubebuilder

Instead of having a separate pod-template.yaml file, we could describe everything directly in the operator code.

That is, you can describe something like this:

...
    # get optional banner value
    banner = spec.get('banner', '')

    # define Pod spec as a Python dict
    pod_spec = {
        "apiVersion": "v1",
        "kind": "Pod",
        "metadata": {
            "name": f"{name}-pod",
            "labels": {
                "app": name,
            },
        },
        "spec": {
            "containers": [
                {
                    "name": name,
                    "image": image,
                    "env": [
                        {
                            "name": "BANNER",
                            "value": banner
                        }
                    ],
                    "command": ["/bin/sh", "-c"],
                    "args": [f'echo "$BANNER"; exec sleep infinity'],
                    "ports": [
                        {
                            "containerPort": 80
                        }
                    ]
                }
            ]
        }
    }

    # create Kubernetes API client
    api = kubernetes.client.CoreV1Api()
...

And in the case of Kubebuilder, a function is usually created that uses the CustomResource manifest (cr *myappv1.MyApp) and forms an object of type *corev1.Pod using the Go structures corev1.PodSpec and corev1.Container:

...
// newPod is a helper function that builds a Kubernetes Pod object
// based on the custom MyApp resource. It returns a pointer to corev1.Pod,
// which is later passed to controller-runtime's client.Create(...) to create the Pod in the cluster.
func newPod(cr *myappv1.MyApp) *corev1.Pod {
    // `cr` is a pointer to your CustomResource of kind MyApp
    // type MyApp is generated by Kubebuilder and lives in your `api/v1/myapp_types.go`
    // it contains fields like cr.Spec.Image, cr.Spec.Banner, cr.Name, cr.Namespace, etc.
    return &corev1.Pod{
        // corev1.Pod is a Go struct representing the built-in Kubernetes Pod type
        // it's defined in "k8s.io/api/core/v1" package (aliased here as corev1)
        // we return a pointer to it (`*corev1.Pod`) because client-go methods like
        // `client.Create()` expect pointer types

        ObjectMeta: metav1.ObjectMeta{
            // metav1.ObjectMeta comes from "k8s.io/apimachinery/pkg/apis/meta/v1"
            // it defines metadata like name, namespace, labels, annotations, ownerRefs, etc.
            Name: cr.Name + "-pod", // generate Pod name based on the CR's name
            Namespace: cr.Namespace, // place the Pod in the same namespace as the CR
            Labels: map[string]string{ // set a label for identification or selection
                "app": cr.Name, // e.g., `app=example-app`
            },
        },

        Spec: corev1.PodSpec{
            // corev1.PodSpec defines everything about how the Pod runs
            // including containers, volumes, restart policy, etc.

            Containers: []corev1.Container{
                // define a single container inside the Pod

                {
                    Name: cr.Name, // use CR name as container name (must be DNS compliant)
                    Image: cr.Spec.Image, // container image (e.g., "nginx:1.25")

                    Env: []corev1.EnvVar{
                        // corev1.EnvVar is a struct that defines environment variables
                        {
                            Name: "BANNER", // name of the variable
                            Value: cr.Spec.Banner, // value from the CR spec
                        },
                    },

                    Command: []string{"/bin/sh", "-c"},
                    // override container ENTRYPOINT to run a shell command

                    Args: []string{
                        // run a command that prints the banner and sleeps forever
                        // fmt.Sprintf(...) injects the value at runtime into the string
                        fmt.Sprintf(`echo "%s"; exec sleep infinity`, cr.Spec.Banner),
                    },

                    // optional: could also add ports, readiness/liveness probes, etc.
                },
            },
        },
    }
}
...

And what about in real operators?

But we did this for “internal” Kubernetes resources.

What about external resources?

Here’s just an example — I haven’t tested it, but the general idea is this: just take an SDK (in the Python example, it’s boto3), and using the fields from the CustomResource (for example, subnets or schema), make the appropriate API requests to AWS through the SDK.

An example of such a CustomResource:

apiVersion: demo.rtfm.co.ua/v1
kind: MyIngress
metadata:
  name: myapp
spec:
  subnets:
    - subnet-abc
    - subnet-def
  scheme: internet-facing

And the code that could create an AWS ALB from it:

import kopf
import boto3
import botocore
import logging

# create a global boto3 client for AWS ELBv2 service
# this client will be reused for all requests from the operator
# NOTE: region must match where your subnets and VPC exist
elbv2 = boto3.client("elbv2", region_name="us-east-1")

# define a handler that is triggered when a new MyIngress resource is created
@kopf.on.create('demo.rtfm.co.ua', 'v1', 'myingresses')
def create_ingress(spec, name, namespace, status, patch, logger, **kwargs):
    # extract the list of subnet IDs from the CustomResource 'spec.subnets' field
    # these subnets must belong to the same VPC and be public if scheme=internet-facing
    subnets = spec.get('subnets')

    # extract optional scheme (default to 'internet-facing' if not provided)
    scheme = spec.get('scheme', 'internet-facing')

    # validate input: at least 2 subnets are required to create an ALB
    if not subnets:
        raise kopf.PermanentError("spec.subnets is required.")

    # attempt to create an ALB in AWS using the provided spec
    # using the boto3 ELBv2 client
    try:
        response = elbv2.create_load_balancer(
            Name=f"{name}-alb", # ALB name will be derived from CR name
            Subnets=subnets, # list of subnet IDs provided by user
            Scheme=scheme, # 'internet-facing' or 'internal'
            Type='application', # we are creating an ALB (not NLB)
            IpAddressType='ipv4', # only IPv4 supported here (could be 'dualstack')
            Tags=[ # add tags for ownership tracking
                {'Key': 'ManagedBy', 'Value': 'kopf'},
            ]
        )
    except botocore.exceptions.ClientError as e:
        # if AWS API fails (e.g. invalid subnet, quota exceeded), retry later
        raise kopf.TemporaryError(f"Failed to create ALB: {e}", delay=30)

    # parse ALB metadata from AWS response
    lb = response['LoadBalancers'][0] # ALB list should contain exactly one entry
    dns_name = lb['DNSName'] # external DNS of the ALB (e.g. abc.elb.amazonaws.com)
    arn = lb['LoadBalancerArn'] # unique ARN of the ALB (used for deletion or listeners)

    # log the creation for operator diagnostics
    logger.info(f"Created ALB: {dns_name}")

    # save ALB info into the CustomResource status field
    # this updates .status.alb.dns and .status.alb.arn in the CR object
    patch.status['alb'] = {
        'dns': dns_name,
        'arn': arn,
    }

    # return a dict, will be stored in the finalizer state
    # used later during deletion to clean up the ALB
    return {'alb-arn': arn}

In the case of Go and Kubebuilder, we would use the aws-sdk-go library:

import (
    "context"
    "fmt"

    elbv2 "github.com/aws/aws-sdk-go-v2/service/elasticloadbalancingv2"
    "github.com/aws/aws-sdk-go-v2/aws"
    metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
    networkingv1 "k8s.io/api/networking/v1"
)

func newALB(ctx context.Context, client *elbv2.Client, cr *networkingv1.Ingress) (string, error) {
    // build input for the ALB
    input := &elbv2.CreateLoadBalancerInput{
        Name: aws.String(fmt.Sprintf("%s-alb", cr.Name)),
        Subnets: []string{"subnet-abc123", "subnet-def456"}, // replace with real subnets
        Scheme: elbv2.LoadBalancerSchemeEnumInternetFacing,
        Type: elbv2.LoadBalancerTypeEnumApplication,
        IpAddressType: elbv2.IpAddressTypeIpv4,
        Tags: []types.Tag{
            {
                Key: aws.String("ManagedBy"),
                Value: aws.String("MyIngressOperator"),
            },
        },
    }

    // create ALB
    output, err := client.CreateLoadBalancer(ctx, input)
    if err != nil {
        return "", fmt.Errorf("failed to create ALB: %w", err)
    }

    if len(output.LoadBalancers) == 0 {
        return "", fmt.Errorf("ALB was not returned by AWS")
    }

    // return the DNS name of the ALB
    return aws.ToString(output.LoadBalancers[0].DNSName), nil
}

In the real AWS ALB Ingress Controller, the creation of an ALB is called in the elbv2.go file :

...
func (c *elbv2Client) CreateLoadBalancerWithContext(ctx context.Context, input *elasticloadbalancingv2.CreateLoadBalancerInput) (*elasticloadbalancingv2.CreateLoadBalancerOutput, error) {
  client, err := c.getClient(ctx, "CreateLoadBalancer")
  if err != nil {
    return nil, err
  }
  return client.CreateLoadBalancer(ctx, input)
}
...

Actually, that’s all there is to it.

Originally published at RTFM: Linux, DevOps, and system administration.

AWS: creating an OpenSearch Service cluster and configuring authentication and authorization

Arseny Zinchenko — Tue, 16 Sep 2025 06:55:43 +0000

In the previous part, AWS: Getting Started with OpenSearch Service as a Vector Store, we looked at AWS OpenSearch Service in general, figured out how data is organized in it, what shards and nodes are, and what types of instances we actually need for data nodes.

The next step is to create a cluster and look at authentication, which, in my opinion, is even more complicated than AWS EKS. Although, maybe it’s just a matter of habit.

What we’re going to do today is manually create an AWS OpenSearch Service cluster, look at the main options for creating a cluster, and then dive into the settings for accessing the cluster and OpenSearch Dashboards with AWS IAM and Fine-grained access control of OpenSearch itself and its Security plugin.

And in the next part, if I have time to write it, we’ll get to Terraform.

Manually creating a cluster in AWS Console
Storage
Nodes
Network
Access && permissions
Authentication and authorization
Configuring Domain Access policy
Resource-based policy
IP-based policies and access to the OpenSearch Dashboards
Identity-based policy
Fine-grained access control
Configuring the Fine-grained access control
Creating an OpenSearch Role

Manually creating a cluster in AWS Console

We will do a minimal PoC to play around, i.e., with t3 instances in one Availability Zone and without Master Nodes.

In Production, we also plan to have one small cluster with three dev/staging/prod indexes as a vector store for AWS Bedrock Knowledge Base.

Documentation from AWS — Creating OpenSearch Service domains.

Go to Amazon OpenSearch Service > Domains, click “Create domain”.

Set a name, select “Standard create” to have access to all options:

In “Templates”, select “Dev/test” — then you can choose a configuration without Master Nodes and deploy in a single Availability Zone.

In “Deployment option(s)”, select “Domain without standby” — then you will have access to t3 instances:

The case conveniently shows us the entire setup right away.

Storage

We discussed the number of shards per cluster in the previous post. Let’s assume that we plan to have a maximum of 20–30 GiB of data, so we will create 1 primary shard and 1 replica. But the shards will be configured later, when we create indexes with Terraform and opensearch_index_template.

And for these two shards, we will create two Data Nodes — one for the primary shard and one for the replica.

“Engine options” are described in Features by engine version in Amazon OpenSearch Service. Just leave the default value, the latest version.

For “Instance family” select “General purpose”, and for “Instance type,” select t3.small.search.

For the “EBS storage size per node” we will take 50 GiB — 20–30 gigabytes for data and a little extra for the operating system itself:

Nodes

Leave “Number of master nodes” and “Dedicated coordinator nodes” unchanged, i.e. without them:

Network

We are not changing anything in “Custom endpoint” yet, but later you can add your own domain from Route53 with a certificate from AWS Certificate Manager to access the cluster, see Creating a custom endpoint for Amazon OpenSearch Service.

In the “Network”, we are going with the simplest option for now, “Public access”, but for Production, we will do it inside the VPC:

However, you will need to test access to Dashboards, because if the cluster is created in VPC subnets, IP-based policies cannot be applied to it, see About access policies on VPC domains. We will discuss IP-based policies further here.

Access && permissions

Fine-grained access control (FGAC) — we’ll disable it for now and take a closer look at this mechanism later. Although I’m not sure it will be necessary, because you can easily divide access to different indexes in a single cluster using IAM.

SAML, JWT, and IAM Identity Center depend on FGAC, so we’ll skip them too, and I don’t plan to use them in the future, as they are not relevant to our case.

Cognito is also out of the question — we don’t use it (although later, I may look into integrating with Auth0 or Cognito for Dashboards):

“Access policy”” can be compared to S3 Access Policy, or to IAM Policy for EKS, which allows IAM users to access the cluster.

We will discuss this in more detail in the section on authentication. For now, let’s just leave the default - "Do not set domain level access policy” option selected:

The “Off-peak window” is the time of lowest load for installing updates and performing Auto-tune operations.

Our off-peak time will be at night in the US, so Production will be Central Time (CT) 05:00 UTC.

But since this is a test PoC, we’ll skip that too.

Auto-Tune is also well described and unavailable for our t3 instances.

Automatic software update is a useful feature for Production and will be performed at the time specified in the Off-peak window:

In еру “Advanced cluster settings” you can disable rest.action.multi.allow_explicit_index, but I don't know how our queries will be built, and I think I read somewhere that it can break the Dashboard, so let's leave the default enabled:

And that’s it, as a result we have the following setup:

Click “Create” and go have some tea, because creating a cluster takes a long time — longer than EKS, and creating OpenSearch took about 20 minutes.

Authentication and authorization

Now, perhaps, the most interesting part — users and access.

After creating a cluster, by default we have limited access rights to the OpenSearch API itself:

Because in the “Security Configuration” we have an explicit Deny:

Access to AWS OpenSearch Service has three “levels” — network, IAM, and OpenSearch’s own Security Plugin.

In IAM, we have two entities — Domain Access Policy , which we see in Security Configuration > Access Policy (attribute access_policies in Terraform), and Identity-based policies - which are regular AWS IAM Policies.

If we talk about these levels in more detail, they look something like this:

Network: Network > VPC access or Public access parameter: we set the access limit at the network level (see Launching your Amazon OpenSearch Service domains within a VPC)
or, if we take an analogy with EKS, these are Public and Private API endpoints, or with RDS, creating an instance in public or private subnets
AWS IAM:
Domain Access Policies:
Resource-based policies: policies that are described directly in the cluster settings
access is set for IAM Role, IAM User, AWS Accounts to a specific OpenSearch domain
IP-based policies: essentially the same as Resource-based policies, but with the ability to allow access without authentication for specific IPs (only if the access type is Public, see VPC versus public domains)
Identity-based policies: if Resource-based policies are part of the cluster’s security policy settings, then Identity-based policies are regular AWS IAM Policies that are added to a specific user or role
Fine-grained access control (FGAC): OpenSearch’s own Security Plugin — the advanced_security_options attribute in Terraform
if in Resource-based policies and Identity-based policies we set rules at the cluster (domain) and index levels, then in FGAC we can additionally describe restrictions on specific documents or fields
and even if Resource-based policies and Identity-based policies allow access to a resource in the cluster, it can be “trimmed” through Fine-grained access control

That is, the authentication and authorization flow will be as follows:

AWS API receives a request from the user, for example es:ESHttpGet
AWS IAM performs authentication — checks ACCESS:SECRET keys or Session token
AWS IAM performs authorization:

checks the user’s IAM Policy ( Identity-based policy ), if there is explicit permission here — we skip
checks the Domain Access Policy ( Resource-based policy ) of the cluster, if there is explicit permission here — we skip

The request comes to OpenSearch itself
If Fine-grained access control is not enabled, we allow it
If Fine-grained access control is configured, we check internal roles, and if the user is allowed, we execute the request

Let’s make some accesses and see how it all works.

Configuring Domain Access policy

The basic option is to add IAM User access to the cluster.

Resource-based policy

Edit the “Access policy” and specify your user, API operation types, and domain:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::492***148:user/arseny.zinchenko"
      },
      "Action": "es:*",
      "Resource": "arn:aws:es:us-east-1:492***148:domain/test/*"
    }
  ]
}

Wait a minute, and now we have access to the OpenSearch API (because Cluster health in the AWS Console is obtained from OpenSearch — see Cluster Health API):

And now we can use curl and --aws-sigv4 to access the cluster (see Authenticating Requests (AWS Signature Version 4)):

$ curl --aws-sigv4 "aws:amz:us-east-1:es" \
> --user "AKI ***B7A:pAu*** 2gW" \
> https://search-test-***.us-east-1.es.amazonaws.com/_cluster/health?pretty
{
  "cluster_name" : "492***148:test",
  "status" : "green",
  "timed_out" : false,
  "number_of_nodes" : 2,
  "number_of_data_nodes" : 2,
  "discovered_master" : true,
  "discovered_cluster_manager" : true,
  "active_primary_shards" : 5,
  "active_shards" : 10,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 0,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0,
  "task_max_waiting_in_queue_millis" : 0,
  "active_shards_percent_as_number" : 100.0
}

IP-based policies and access to the OpenSearch Dashboards

Similarly, through Domain Access Policy, we can open access to Dashboards — the simplest option, but it only works with Public domains. If the cluster is in VPC, additional authentication will be required, see Controlling access to Dashboards.

Edit the policy, add condition IpAddress.aws:SourceIp:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::492***148:user/arseny.zinchenko"
      },
      "Action": "es:*",
      "Resource": "arn:aws:es:us-east-1:492***148:domain/test/*"
    },
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": "*"
      },
      "Action": "es:ESHttp*",
      "Resource": "arn:aws:es:us-east-1:492***148:domain/test/*",
      "Condition": {
        "IpAddress": {
          "aws:SourceIp": "178. ***.***.184"
        }
      }
    }
  ]
}

And now we have access to the Dashboards:

Identity-based policy

Now, the second option is to create a separate IAM User and connect a separate IAM Policy to it.

Add a user in AWS IAM:

We can just take a ready-made AWS managed policies for Amazon OpenSearch Service:

Next, we simply create access keys for the Command Line Interface (CLI) and, without changing anything in the cluster’s Access policy, check access:

$ curl --aws-sigv4 "aws:amz:us-east-1:es" --user "AKI ***YUK:fXV*** 34I" https://search-test-***.us-east-1.es.amazonaws.com/_cluster/health?pretty
{
  "cluster_name" : "492***148:test",
  "status" : "green",
  "timed_out" : false,
  "number_of_nodes" : 2,
  "number_of_data_nodes" : 2,
  "discovered_master" : true,
  "discovered_cluster_manager" : true,
  "active_primary_shards" : 5,
  "active_shards" : 10,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 0,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0,
  "task_max_waiting_in_queue_millis" : 0,
  "active_shards_percent_as_number" : 100.0
}

So now we have a Domain Access Policy that grants access specifically to my user, and there is a separate IAM Policy — an identity-based policy — that grants access to the test user.

But there is one important point here: in the IAM Policy, we specify either the entire domain or only its subresources.

That is, if instead of the AmazonOpenSearchServiceFullAccess policy, we create our own policy in which we specify "Resource":***:domain/test/*":

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "es:*"
            ],
            "Resource": "arn:aws:es:us-east-1:492***148:domain/test/*"
        }
    ]
}

So we can execute es:ESHttpGet (GET _cluster/health) - but we cannot execute cluster-level operations, such as es:AddTags, even though we have permission for all calls in the Actions of the IAM policy - es:*:

$ aws --profile test-os opensearch add-tags --arn arn:aws:es:us-east-1:492***148:domain/test --tag-list Key=environment,Value=test

An error occurred (AccessDeniedException) when calling the AddTags operation: User: arn:aws:iam::492 ***148:user/test-opesearch-identity-based-policy is not authorized to perform: es:AddTags on resource: arn:aws:es:us-east-1:492*** 148:domain/test because no identity-based policy allows the es:AddTags action

If we want to allow all operations with the cluster, we need to set "Resource" to "arn:aws:es:us-east-1:492***148:domain/test", and then we can add tags.

See all API actions in Actions, resources, and condition keys for Amazon OpenSearch Service.

Fine-grained access control

Documentation — Fine-grained access control in Amazon OpenSearch Service.

The basic idea is very similar to Kubernetes RBAC.

In OpenSearch, there are three main concepts:

users — like Kubernetes Users and ServiceAccounts
roles — like Kubernetes RBAC Roles
mappings — like Kubernetes Role Bindings

Users can be from both AWS IAM and the internal OpenSearch database.

As in Kubernetes, OpenSearch has a set of default roles — see Predefined roles.

At the same time, roles, as in Kubernetes, can be cluster-wide or index-specific — analogous to ClusterRoleBinding and simply namespaced RoleBinding in Kubernetes, plus in OpenSearch FGAC you can additionally have document level or field level permissions.

Configuring the Fine-grained access control

Important note : once FGAC is enabled, you will not be able to revert to the old scheme. However, all accesses from IAM will remain, even if you switch to the internal database.

Edit “Security configuration” and enable “Fine-grained access control”:

First, we need to set up a Master user, which can be specified from IAM or created locally in OpenSearch.

If we create a user via the “Create master user” option, we specify a regular login:password, and in this case, OpenSearch will connect to the internal user database (internal_user_database_enabled in Terraform).

If we use the internal OpenSearch database, we can have regular users and perform HTTP basic authentication. See the AWS documentation — Tutorial: Configure a domain with the internal user database and HTTP basic authentication and Defining users and roles in the OpenSearch documentation itself, as these are its internal mechanisms.

This makes sense if you don’t want to use Cognito or SAML, and if each cluster will have its own user settings.

If you set an IAM user, the scheme will be similar to AIM authentication for RDS and IAM database authentication — access to the cluster is controlled by AWS IAM, but internal access to schemas and databases is controlled by PostgreSQL or MariaDB roles, see AWS: RDS with IAM database authentication, EKS Pod Identities, and Terraform.

In this case, AWS IAM will only perform user authentication, while authorization (access rights verification) will be handled by the Security plugin and OpenSearch roles.

Let’s try a local database, and I think we’ll use this scheme in Production as well:

We can leave “Access Policy” as it is:

Switching to the internal database will take some time because it will trigger a blue/green deployment of the new cluster, see Making configuration changes in Amazon OpenSearch Service.

And it took a long time, more than an hour , even though there is no data of ours in the cluster.

Once the changes are applied, Dashboards will now ask for a login and password. Use our Master user:

The master user receives two connected roles: all_access and security_manager.

It is security_manager that provides access to the Security and Users sections in the dashboard:

At the same time, we still have access to our AIM users, and we can continue to use curl: IAM users will be mapped to the default_role, which allows GET/PUT on all indexes - see About the default_role:

Let’s check our test user’s access now:

$ curl --aws-sigv4 "aws:amz:us-east-1:es" --user "AKI ***YUK:fXV*** 34I" https://search-test-***.us-east-1.es.amazonaws.com/_cluster/health?pretty
{
  "cluster_name" : "492***148:test",
  "status" : "green",
  "timed_out" : false,
  "number_of_nodes" : 2,
...

Now let’s cut off access to all IAM users.

Creating an OpenSearch Role

To see how it works, let’s add a test index and map our test user with access to this index.

Add the index:

Go to Security > Roles, add a role:

Set Index permissions — full access to the index (crud):

Next, in this role, we move on to Mapped users > Map users:

And add the ARN of our test user:

Delete the default role:

Now our user does not have access to GET _cluster/health - here we get an error 403, no permissions :

$ curl --aws-sigv4 "aws:amz:us-east-1:es" --user "AKI ***YUK:fXV*** 34I" https://search-test-***.us-east-1.es.amazonaws.com/_cluster/health?pretty
{
  "error" : {
    ...
    "type" : "security_exception",
    "reason" : "no permissions for [cluster:monitor/health] and User [name=arn:aws:iam::492***148:user/test-opesearch-identity-based-policy, backend_roles=[], requestedTenant=null]"
  },
  "status" : 403
}

But has access to the test index:

$ curl --aws-sigv4 "aws:amz:us-east-1:es" --user "AKI ***YUK:fXV*** 34I" https://search-test-***.us-east-1.es.amazonaws.com/test-allowed-index/_search?pretty -d '{
    "query": {
      "match_all": {}
    }
  }' -H 'Content-Type: application/json'
{
  "took" : 78,
  "timed_out" : false,
  "_shards" : {
    "total" : 1,
    "successful" : 1,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : {
      "value" : 0,
      "relation" : "eq"
    },
    "max_score" : null,
    "hits" : []
  }
}

Done.

Originally published at RTFM: Linux, DevOps, and system administration.

DEV Community: Arseny Zinchenko

AWS: ALB and Cloudflare - Configuring mTLS and AWS Security Rules

AWS: ALB and Cloudflare - Configuring mTLS and AWS Security Rules

Contents

AWS ALB cost and LCU

AWS Load Balancer mTLS use cases

Configuring ALB mutual TLS with Cloudflare

Configuring Cloudflare

Adding a new domain to Cloudflare

Configuring the Application Load Balancer

Getting the Cloudflare Certificate Authority

Creating a Trust Store

Configuring mTLS for the ALB Listener

Protection with AWS Security Group and Cloudflare IP ranges

Creating an IAM Role for the EC2 Instance profile

Shell script for updating the ALB Security Group

Replacing the Security Group on the ALB

Running from crontab

AWS: Monitoring AWS OpenSearch Service Cluster with CloudWatch

Contents

CloudWatch metrics

Memory monitoring

kNN Memory usage

JVM Memory usage

Collecting metrics to VictoriaMetrics

Creating a Grafana dashboard

VictoriaMetrics/Prometheus sum(), avg() and max()`

HELP aws_es_cpuutilization_average CloudWatch metric AWS/ES CPUUtilization Dimensions: [ClientId, DomainName, NodeId] Statistic: Average Unit: Percent

TYPE aws_es_cpuutilization_average gauge

Cluster status

Nodes status

CPUUtilization: Stats

CPUUtilization: Graph

JVMMemoryPressure: Graph

JVMGCYoungCollectionCount and JVMGCOldCollectionCount

KNNHitCount vs KNNMissCount

Final result

t3.small.search vs t3.medium.search on graphs

Creating Alerts

Terraform: creating an AWS OpenSearch Service cluster and users

Contents

Terraform files structure

PROD

destroy-prod:

terraform destroy -var-file=envs/prod/prod.tfvars

Project planning

Creating a cluster

ROOT

generate root password

waiting for write-only: https://github.com/hashicorp/terraform-provider-aws/pull/43621

then will update it with the ephemeral type

store the root password in AWS Parameter Store

Custom endpoint configuration

TLS for the Custom Domain

Terraform Outputs

Creating OpenSearch Users

Error: elastic: Error 403 (Forbidden)

Creating Internal Users

Internal database users

KRAKEN

store the root password in AWS Parameter Store

Create a user

And a full user, role and role mapping example:

Adding IAM Users

BACKEND TEAM

Creating AWS Bedrock IAM Roles and OpenSearch Role mappings

MAIN ROLE FOR KNOWLEDGE BASE

grants permissions for AWS Bedrock to interact with other AWS services

IAM policy for Knowledge Base to access OpenSearch Managed

BEDROCK

'aws_iam_role' is defined in iam.tf

Creating OpenSearch indexes

Dev Index

Terraform: using Ephemeral Resources and Write-Only Attributes

Contents

Example without ephemeral values and write-only arguments

Using Write-Only Attributes

Using Ephemeral Resources

The “This output value is not declared as returning an ephemeral value” error

The “Ephemeral outputs are not allowed in context of a root module” error

VictoriaMetrics/Prometheus `sum()`, `avg()` and max()`

`t3.small.search` vs `t3.medium.search` on graphs

The interface “magic”: the `iface` structure

Creating a `resty` client

`resty` JSON Unmarshall

sanitizing names - formatting values with `strings.Replace()`

Create metrics with `NewGauge()`

Recording metrics to VictoriaMetrics with `InitPush()` and `PushMetrics()`

Inside the `InitPush()` implementation

Inside the `PushMetrics()` function implementation

Creating `context` and calling `PushMetrics()`

Using `gocron` for tasks scheduling

Bonus: How execution control works with Go `context`

Closing the channel with `cancel()`

How the `context` channel is created

“Write-path”: `vminsert` and `vmstorage`

`vmstorage`, IndexDB, and TSID

`vmstorage` and data storage on disk

“Read-path”: search for data from `vmselect` and `vmstorage`

Upgrade `terraform-aws-modules/eks/aws`

Renamed variables в `terraform-aws-modules/eks/aws`

Removed variables в terraform-aws-`modules/eks/aws//modules/karpenter`

`eks_managed_node_groups`: attribute "taints": map of object required