DEV Community: Paweł Swiridow

TypeSafe AI Jev explained: examples, use cases, and LLM comparisons

Paweł Swiridow — Wed, 23 Sep 2026 19:00:00 +0000

TypeSafe AI Jev is a model for making bounded decisions inside software. You give it text or structured state plus questions whose valid answers you define. It returns typed choices, scores, probabilities, and confidence values instead of generating prose.

That makes Jev useful for classification, routing, scoring, and review gates. It does not replace GPT, Claude, or another generative model when you need writing, code, explanations, or open-ended reasoning. Think of it as a fuzzy decision function, not a chatbot wearing a smaller hat.

Quick answer

Use Jev when:

Every valid answer can be defined before the call.
The hard part is semantic judgment, not arithmetic or deterministic policy.
Software needs probabilities or confidence to decide whether to act, retry, or escalate.
The workload repeats often enough for latency and per-call cost to matter.

Use something else when:

The output must contain newly written text, code, or an explanation.
Correctness depends on exact math, dates, permissions, or database state.
The task requires long, multi-step reasoning.
The model would be the only security or authorization boundary.

What is TypeSafe AI Jev?

TypeSafe calls Jev its first System One model. That is the company's product category for models built to make fast, structured decisions that software can consume directly. It is not a new chat interface and it is not an industry-standard model category.

The contract is simple:

state + typed questions -> Jev -> typed answers + probabilities -> application code

The state can be a string, JSON object, or array containing text. Each question defines its answer space. Jev evaluates independent questions against the same state in parallel, then your code decides what happens next.

At the time of writing, jev-latest resolves to jev-1.13.0. TypeSafe's model documentation lists it at $0.042 per million input tokens with free output tokens. The documented context limit is 64,000 tokens per request, with a separate 32,000-token limit for the state plus the longest question. Input is text only, so images, audio, and video must be converted before they reach Jev.

Pin a version when tuning thresholds and rerun evaluation before upgrading. A moving alias is convenient until it changes a production route at 02:00.

TypeSafe's launch material describes Jev as unable to hallucinate because it cannot emit a value outside the schema. That claim needs a narrower reading. Jev cannot invent a fourth result when your Choice defines three, but it can choose the wrong one. Type safety protects the shape of an answer. It does not make the judgment correct. Independent analysis from Arize makes the same distinction.

How Choice, Score, and Noul work

Jev exposes three question types:

Primitive	Question it answers	Returned data	Good fit
Choice	Which predefined option fits best?	selected option, option probabilities, confidence	intent routing, taxonomy classification
Score	Where does this fall on an ordered rubric?	weighted score, level probabilities, legend, confidence	severity, quality, urgency
Noul	How likely is this yes/no proposition?	probability from 0 to 1	detection, verification, binary review signals

A Choice can contain up to 255 options. A Score accepts between two and ten ordered levels. Choice and Score return a confidence value derived from their probability distributions; Noul returns the probability of yes without a separate confidence field.

Questions in one request are independent. If question B depends on question A, use another call or express the dependency in code. Do not assume that two differently phrased questions will obey arithmetic identities either. TypeSafe documents cases where a Noul and an equivalent-looking Choice produce different numbers because they ask different statistical questions.

Keep questions atomic. Ask separately whether a refund was requested, score impact, classify intent, then let code apply policy. Models judge fuzzy meaning; code does not reinterpret accounting rules overnight.

Run a Jev decision in Python

Install the SDK

The Python SDK requires an API key in TYPESAFE_API_KEY:

python3 -m pip install typesafe-sdk
export TYPESAFE_API_KEY='replace-with-a-key-from-console.typesafe.ai'

Use your normal secret manager in CI and production. Committing the key beside the example would turn a quick start into an incident report.

Classify and route a support ticket

This example asks one Choice, one Score, and one Noul in a single call. It then applies routing thresholds in ordinary Python.

import os
import time

from typesafe_sdk import Choice, Noul, Score, TypeSafeClient

if "TYPESAFE_API_KEY" not in os.environ:
    raise SystemExit("Set TYPESAFE_API_KEY before running this example")

INPUT_USD_PER_MTOK = 0.042  # TypeSafe list price at the time of writing

ticket = {
    "subject": "Checkout fails after payment",
    "message": (
        "Our card was charged twice and checkout still shows an error. "
        "We need this fixed today because customers cannot place orders."
    ),
    "plan": "business",
}

started = time.perf_counter()

with TypeSafeClient(model="jev-1.13.0") as client:
    response = client.system_one(
        state=ticket,
        questions={
            "department": Choice(
                instructions="Which team should own this ticket first?",
                criteria={
                    "billing": "Charges, invoices, refunds, or subscriptions",
                    "technical": "Bugs, outages, failed integrations, or broken product behavior",
                    "account": "Login, identity, permissions, or account access",
                    "other": "The ticket does not fit the other categories",
                },
            ),
            "severity": Score(
                instructions="How severely does this issue block the customer's work?",
                criteria=[
                    "Minor inconvenience with a clear workaround",
                    "Important feature is degraded but work can continue",
                    "Critical workflow is blocked or customers are losing money",
                ],
            ),
            "requests_refund": Noul(
                instructions="Does the customer explicitly ask for a refund?",
                criteria={
                    "true": "The customer directly requests money back",
                    "false": "The customer reports a charge but does not request a refund",
                },
            ),
        },
    )

elapsed = time.perf_counter() - started

department = response.choices["department"]
severity = response.scores["severity"]
refund_probability = response.nouls["requests_refund"].noul

if department.confidence < 0.65 or severity.confidence < 0.55:
    destination = "human-review"
elif severity.score > 1.5:
    destination = "incident-response"
else:
    destination = department.choice

input_tokens = response.usage.input_tokens or 0
estimated_cost = input_tokens / 1_000_000 * INPUT_USD_PER_MTOK

print(f"model={response.model}")
print(f"destination={destination}")
print(f"department_confidence={department.confidence:.2f}")
print(f"severity={severity.score:.2f}")
print(f"refund_probability={refund_probability:.2f}")
print(f"input_tokens={input_tokens}")
print(f"output_tokens={response.usage.output_tokens}")
print(f"elapsed_seconds={elapsed:.2f}")
print(f"estimated_cost_usd={estimated_cost:.8f}")

The SDK exposes separate collections for Choice, Score, and Noul answers. The thresholds are examples, not TypeSafe defaults. Set them from errors and review costs measured on your own data.

Pinning jev-1.13.0 prevents alias drift. In production, log the criteria version, probability distributions, chosen route, and human outcome.

Sample output and cost

A real run of the example above against jev-1.13.0 printed this:

model=jev-1.13.0
destination=incident-response
department_confidence=0.95
severity=2.00
refund_probability=0.06
input_tokens=528
output_tokens=77
elapsed_seconds=0.64
estimated_cost_usd=0.00002218

The usage block in the response is what makes the cost measurable without opening a billing dashboard. That call moved 528 input tokens. TypeSafe bills $0.042 per million input tokens and does not bill output tokens, so the run cost 528 / 1,000,000 * 0.042, or roughly $0.000022 at current list prices. A thousand calls of this size cost about two cents, and one dollar buys roughly 45,000 of them.

elapsed_seconds covers client setup plus the round trip, not just server time. Values move with the input, criteria, and model version, so treat the transcript as a real example rather than a fixed contract.

Practical Jev use cases

Route support requests

Support routing is the natural first pilot. Put the ticket text and relevant account context in state. Use Choice for the owning queue, Score for impact, and Noul for independent conditions such as an explicit refund request:

{
  "state": {
    "channel": "email",
    "subject": "Wrong item delivered, replacement needed before Friday",
    "message": "Order 48213 arrived with two left boots. I need the replacement before the conference on Friday or I want to cancel.",
    "account_tier": "enterprise",
    "earlier_tickets": 2
  },
  "model": "jev-1.13.0",
  "questions": {
    "queue": {
      "type": "choice",
      "instructions": "Which queue should own this ticket first?",
      "criteria": {
        "returns": "Wrong, damaged, or missing items, replacements, and shipping problems",
        "billing": "Charges, invoices, refunds, and subscriptions",
        "technical": "Product failures, errors, and defects",
        "account": "Access, identity, and permissions",
        "other": "None of the other queues fit"
      }
    },
    "impact": {
      "type": "score",
      "instructions": "How much does the problem block the customer?",
      "criteria": [
        "Minor inconvenience with a workaround",
        "The item or order cannot be used as delivered",
        "Time-critical situation or total loss of the purchase"
      ]
    },
    "wants_cancel": {
      "type": "noul",
      "instructions": "Is the customer asking to cancel an order or close the account?"
    }
  }
}

Code can accept a confident route, send uncertain tickets to a person, and apply exact entitlement rules from the account database.

Review agent actions

An agent can propose a tool call while Jev supplies a bounded semantic risk signal:

{
  "state": {
    "user_goal": "Remove generated build artifacts",
    "tool": "shell",
    "command": "rm -rf dist"
  },
  "model": "jev-1.13.0",
  "questions": {
    "risk": {
      "type": "choice",
      "instructions": "Classify the operational risk of the proposed action",
      "criteria": {
        "read_only": "Does not modify state",
        "reversible_write": "Changes state but has a reliable rollback",
        "irreversible_write": "Deletes or overwrites state without a reliable rollback",
        "unclear": "The available state is insufficient"
      }
    },
    "matches_goal": {
      "type": "noul",
      "instructions": "Does the proposed action directly match the user's stated goal?"
    }
  }
}

This is a review signal, not permission to execute. Allowlist enforcement, path validation, sandboxing, and user confirmation still belong in code. Jev's own documentation warns that adversarial text in state can move an answer, so it must not become the sole security boundary.

Check RAG evidence before generation

Jev can classify whether a retrieved passage supports a claim before a generative model writes the answer:

{
  "state": {
    "question": "Does the service retain customer prompts for training?",
    "claim": "Customer requests are not used to train Jev.",
    "passage": "Jev is not trained on customer requests or responses."
  },
  "model": "jev-1.13.0",
  "questions": {
    "support": {
      "type": "choice",
      "instructions": "How well does the passage support the claim?",
      "criteria": {
        "supported": "The passage directly supports the claim",
        "contradicted": "The passage conflicts with the claim",
        "insufficient": "The passage does not settle the claim"
      }
    },
    "contains_instruction": {
      "type": "noul",
      "instructions": "Does the passage contain an instruction aimed at changing the evaluator's behavior?"
    }
  }
}

The application still needs to preserve the source URL and quote. Jev judges the relationship between claim and passage; it does not browse, establish provenance, or write the final cited answer.

Triage moderation and operations queues

Moderation reports, invoices, incidents, and agent traces all end up in queues. Jev can classify the queue, score severity, and flag missing evidence:

{
  "state": {
    "source": "monitoring alert",
    "service": "checkout-api",
    "summary": "p99 latency rose from 180 ms to 2.4 s over the last 20 minutes while the error rate stayed near 0.2%",
    "context": "Deploy checkout-api v2.31.0 finished 25 minutes ago"
  },
  "model": "jev-1.13.0",
  "questions": {
    "owner": {
      "type": "choice",
      "instructions": "Which team should take the first response for this alert?",
      "criteria": {
        "service_team": "The reporting service's own code or configuration",
        "platform": "Shared infrastructure, networking, or managed dependencies",
        "database": "Storage, replication, or query performance",
        "unclear": "The available evidence does not point to one owner"
      }
    },
    "severity": {
      "type": "score",
      "instructions": "How much customer impact does the summary describe?",
      "criteria": [
        "No confirmed customer impact, worth watching",
        "Degraded latency for some customers",
        "Customers blocked or data at risk"
      ]
    },
    "missing_evidence": {
      "type": "noul",
      "instructions": "Is the alert missing context a first responder would need before acting?"
    }
  }
}

Low-confidence or high-impact items, and anything flagged for missing evidence, go to a human reviewer. Exact totals, due dates, and account permissions stay in deterministic code because Jev 1.13 is explicitly weak at counting, arithmetic, and date comparison.

Route work to the right model or human

Use Jev in front of more expensive handlers. A Choice can select deterministic lookup, a small generative model, a reasoning model, or human review:

{
  "state": {
    "request": "Summarize the attached vendor contract and flag the clauses that change our data-retention obligations",
    "attachments": ["vendor-contract-v7.pdf"],
    "requester": "procurement"
  },
  "model": "jev-1.13.0",
  "questions": {
    "route": {
      "type": "choice",
      "instructions": "Which handler should receive this request?",
      "criteria": {
        "deterministic": "A lookup, calculation, or template can answer it exactly",
        "small_model": "Short generation or extraction a cheap model handles well",
        "reasoning_model": "Long-document analysis or multi-step reasoning",
        "human": "Judgment, negotiation, or accountability beyond the model stack"
      }
    },
    "in_scope": {
      "type": "noul",
      "instructions": "Does the request fall inside the documented scope of the available handlers?"
    }
  }
}

Confidence determines whether the route is safe to accept. Below the threshold, send the request to a human instead of accepting a cheap guess that the matching handler will only have to redo later. This architecture spends generation tokens only when the task needs generation, rather than asking a novelist to sort every envelope.

Jev vs LLMs, classifiers, and rules

Jev is not automatically better than the alternatives. It occupies a specific slot:

Dimension	Rules and code	Traditional classifier	LLM structured output	TypeSafe Jev
Best job	exact deterministic checks	stable prediction task with labeled data	generation, extraction, explanation, broad reasoning	bounded semantic decisions
Output	exact programmed value	fixed trained labels and scores	generated text constrained to a schema	predefined Choice, Score, or Noul answers
New taxonomy	edit code	usually retrain	change prompt or schema	change request criteria
Probability signal	not applicable	often available, calibration varies	usually not a reliable application probability	first-class probabilities; local validation still required
Explanation	code is inspectable	usually limited	can generate one	no generated explanation
Exact math and policy	best option	depends on trained features	possible but unreliable	documented poor fit; keep it in code
Deployment tradeoff	engineering maintenance	labels, training, serving	latency, cost, parsing, model variance	hosted early access, bounded interface, no generation

Plain JSON mode guarantees valid JSON, not adherence to a supplied schema. Strict structured-output APIs can enforce supported JSON schemas, including required keys and enum values.

Schema enforcement is therefore not unique to Jev. Its distinction is a decision-specific interface built around Choice, Score, and Noul, with probability distributions and confidence for bounded judgments.

A conventional classifier may be the better choice for a stable, high-volume task when you already have good labels, training infrastructure, and predictable categories. In one early spam experiment, Jev reached 98.6% accuracy on a 5,733-email ham, spam, and phishing test without task-specific fine-tuning or labeled examples in its requests, while a TF-IDF logistic-regression baseline trained on labeled messages reached 98.9% (full writeup). The criteria had been refined after reviewing labeled examples, so this was useful early evidence, not magic zero-data victory.

Rules win whenever the answer can be computed exactly. Do not ask any model whether an invoice is overdue if two parsed dates and a comparison operator can settle it.

The cost and latency story is still notable.

TypeSafe reports 70 to 500 ms end-to-end latency and $0.042 per million input tokens for suitable requests.

Its launch workflow evaluation produced headline claims of 193.6 times faster and 444.6 times cheaper than its selected LLM comparison, but TypeSafe says those figures are likely near the high end of real-world gains and notes possible benchmark bias.

When not to use TypeSafe AI Jev

Jev 1.13 is not trained to write text. It will not draft a customer reply, produce code, summarize a report, or explain why it chose an answer. Use a generative model when the output itself must be created. The missing explanation also matters for evaluation work: an LLM judge can describe a failure, while Jev returns probabilities that tell you where to inspect, not how to fix the problem.

Keep arithmetic, counting, numeric interpolation, and date comparisons in code. TypeSafe's limitations page says Jev can read dates as text but does not reliably treat them as ordered values. It also warns about literal phrasing, multi-hop indirection, irrelevant context, contradictory criteria, and adversarial content.

More context is not automatically better. Retrieve and filter first, then send the fields needed for the decision. A 64,000-token limit is capacity, not a challenge.

Test non-English workloads separately. TypeSafe says English is Jev's strongest training language and other languages are not handled equally well.

Most importantly, separate a valid answer from a correct answer. A typed output prevents parser failures and out-of-schema labels. It does not prevent a confident, valid, expensive mistake.

How to evaluate Jev on your workload

Start in shadow mode:

Pick one reversible decision with a closed answer set.
Freeze a representative labeled dataset and the exact criteria text.
Run Jev and the current baseline on identical records.
Measure per-class errors, false automatic actions, human-review rate, calibration, end-to-end latency, and cost per completed decision.
Tune thresholds by consequence. Sending a ticket to the wrong queue and approving a destructive action do not deserve the same confidence floor.
Pin the model version and rerun the suite before changing either model or criteria.
Keep permissions, arithmetic, and state changes in deterministic code.

Do not optimize only for overall accuracy. A model can look good while failing badly on a rare queue that matters. Track confusion by class and inspect probability bands: cases scored near 0.9 should be correct more often than cases scored near 0.6. If that relationship does not hold on your data, the threshold is decoration.

Include review capacity in the evaluation. A cautious threshold that sends 70 percent of cases to a human may be safe, but it has not automated much. The useful operating point balances wrong automatic actions against the cost of review.

Should you use TypeSafe AI Jev?

Pilot TypeSafe AI Jev when the answer space is bounded, the judgment is semantic, and software consumes the result often enough for decision latency and cost to matter. Support routing, relevance checks, moderation labels, model routing, and broad evaluation coverage are credible starting points.

Keep an LLM when the output must be written, explained, or reasoned through. Keep code when the answer can be computed exactly. Keep a human in the path when a wrong but valid decision has serious consequences and your own calibration data is thin.

Jev's useful idea is not that every LLM should be replaced. It is that generation and judgment do not have to be the same model call. Split them, measure them independently, and make code responsible for the final action.

How to get fixed IP addresses for an AWS ALB with Global Accelerator

Paweł Swiridow — Wed, 09 Sep 2026 11:00:00 +0000

An AWS Application Load Balancer gives you a DNS name, not client-facing addresses you can promise to another company forever. The practical workaround is to publish the two anycast IPv4 addresses from AWS Global Accelerator and route that traffic to the ALB behind the scenes.

Application Load Balancers are great until someone asks, "What IP addresses should we whitelist?"

That is where the fun starts. The addresses behind the ALB name can change as AWS scales and maintains the service. Telling a customer to whitelist whatever resolves today is not architecture. It is a calendar reminder for a future outage.

This keeps the firewall conversation simple without replacing your load balancer, target groups, certificates, listener rules, or the other ALB machinery you already use.

Quick answer

Use this pattern when the other side only accepts IP-based allowlists, but your service still needs ALB features such as HTTPS listeners, host routing, path routing, target groups, and health checks.

Implementation checklist:

Create a Global Accelerator accelerator.
Add a TCP listener, usually on port 443.
Create an endpoint group in the same Region as the ALB.
Register the ALB as the endpoint.
Send both accelerator IPs to the firewall owner.
Add a Route 53 alias to the accelerator if clients can connect by name.

The pattern looks like this:

Client network
  -> allowlist 2 Global Accelerator IPs
  -> AWS edge location
  -> Global Accelerator listener :443
  -> regional endpoint group
  -> Application Load Balancer
  -> targets

Whitelisting, allowlisting, firewall exception, "please make networking stop yelling" - same practical problem.

Why ALB alone does not work

An ALB is addressed by DNS:

my-api-1234567890.eu-west-1.elb.amazonaws.com

That DNS name resolves to AWS-managed addresses. They are not yours, they are not fixed, and they are not a contract.

You can put a friendly name in Route 53:

api.example.com -> my-api-1234567890.eu-west-1.elb.amazonaws.com

That helps humans. It does not help a firewall that only accepts destination IPs.

Do not solve this by resolving the ALB name and sending the current addresses to a customer. Those addresses can change. The customer will eventually block your production traffic and then everyone gets to pretend DNS was the surprising part.

How Global Accelerator gives an ALB fixed inbound IPs

AWS Global Accelerator creates a fixed entry point for your application.

For an IPv4 accelerator, AWS gives you two static anycast IP addresses:

203.0.113.10
203.0.113.11

Those are the addresses your customer allowlists as outbound destinations. You can point your own DNS name at them:

api.example.com -> Global Accelerator alias

or, when the other side insists on raw IPs, hand them the two accelerator addresses.

Global Accelerator can use an Application Load Balancer as a standard accelerator endpoint. It also monitors endpoint health and can route to another Region if you add one later.

The important detail: the two Global Accelerator IPs are client-facing destination IPs. They are not the source IPs your ALB sees. Do not create an ALB security group rule that says "allow inbound from these two Global Accelerator IPs" and expect it to work. That is the common foot-gun, and it wears steel-toed boots.

Configure AWS Global Accelerator for an ALB with Terraform

Assume you already have an ALB listening on HTTPS.

The Global Accelerator pieces are:

An accelerator
A listener on TCP 443
An endpoint group in the ALB Region
The ALB as an endpoint

"aws_globalaccelerator_accelerator" "api" {
  name            = "api-prod"
  ip_address_type = "IPV4"
  enabled         = true

  attributes {
    flow_logs_enabled = false
  }
}

"aws_globalaccelerator_listener" "https" {
  accelerator_arn = aws_globalaccelerator_accelerator.api.id
  protocol        = "TCP"
  client_affinity = "NONE"

  port_range {
    from_port = 443
    to_port   = 443
  }
}

"aws_globalaccelerator_endpoint_group" "eu_west_1" {
  listener_arn          = aws_globalaccelerator_listener.https.id
  endpoint_group_region = "eu-west-1"
  traffic_dial_percentage = 100

  endpoint_configuration {
    endpoint_id                    = aws_lb.api.arn
    weight                         = 128
    client_ip_preservation_enabled = true
  }
}

After terraform apply, output the addresses:

"global_accelerator_ips" {
  value = aws_globalaccelerator_accelerator.api.ip_sets[0].ip_addresses
}

Example output:

global_accelerator_ips = [
  "203.0.113.10",
  "203.0.113.11",
]

These are the addresses to give to the other side for inbound access to your service from their point of view. They allow outbound traffic to those destination IPs. You keep your ALB behind the accelerator.

Add DNS without losing the fixed IPs

Create an alias record for the accelerator DNS name.

"aws_route53_record" "api" {
  zone_id = aws_route53_zone.example.zone_id
  name    = "api.example.com"
  type    = "A"

  alias {
    name                   = aws_globalaccelerator_accelerator.api.dns_name
    zone_id                = aws_globalaccelerator_accelerator.api.hosted_zone_id
    evaluate_target_health = false
  }
}

Customers that can allowlist by FQDN can use:

api.example.com

Customers that require IPs use the two accelerator addresses. Yes, both. Sending only one is how you turn high availability into a coin toss with extra paperwork.

Protect the ALB from direct access

Global Accelerator solves the fixed destination IP problem. It does not automatically make an internet-facing ALB impossible to reach directly through its own DNS name.

If the ALB is internet-facing, someone can still try:

curl https://my-api-1234567890.eu-west-1.elb.amazonaws.com

That bypasses your pretty api.example.com name and goes straight to the ALB. Security groups and AWS WAF still apply, but the request did not enter through the Global Accelerator IPs.

The cleaner pattern is to use an internal ALB as the accelerator endpoint. Global Accelerator supports internal Application Load Balancer endpoints and creates elastic network interfaces in your VPC to deliver the traffic.

"aws_lb" "api" {
  name               = "api-prod"
  internal           = true
  load_balancer_type = "application"
  subnets            = module.vpc.private_subnets
  security_groups    = [aws_security_group.alb.id]
}

Then allow the application port on the ALB security group. With client IP preservation enabled for an ALB endpoint, your application can still read the original client address from X-Forwarded-For.

"aws_security_group_rule" "alb_https_in" {
  type              = "ingress"
  security_group_id = aws_security_group.alb.id
  protocol          = "tcp"
  from_port         = 443
  to_port           = 443
  cidr_blocks       = ["0.0.0.0/0"]
}

That rule may look too open, but the ALB is internal. The point is that the ALB does not have a public route of its own. Traffic arrives through Global Accelerator into the VPC path AWS creates for the endpoint.

If you must keep the ALB internet-facing, add compensating controls:

Use AWS WAF on the ALB.
Require valid host headers.
Require authentication at the application layer.
Restrict known customer source ranges in the ALB security group if the client list is small and stable.

Do not rely on obscuring the ALB DNS name. Obscurity is not a control. It is a shrug with a certificate.

Keep the real client IP

For ALB endpoints, Global Accelerator and ALB preserve the original client IP in X-Forwarded-For.

Your application will receive a header like this:

X-Forwarded-For: 198.51.100.25

If there are multiple proxies, the header can contain multiple addresses:

X-Forwarded-For: 198.51.100.25, 10.0.12.34

Make sure your app or ingress layer knows which proxy chain it trusts. Blindly trusting every X-Forwarded-For value from the internet is how logs become fan fiction.

For nginx behind the ALB, configure trusted proxy ranges for your VPC or ALB layer, not the whole internet:

set_real_ip_from 10.0.0.0/8;
real_ip_header X-Forwarded-For;
real_ip_recursive on;

Adjust the CIDR to your actual network.

Verify the fixed IP setup

Get the accelerator IPs:

aws globalaccelerator list-accelerators \
  --query 'Accelerators[?Name==`api-prod`].IpSets[].IpAddresses[]' \
  --output text

Test HTTPS against each IP while sending the expected host header:

curl -v --resolve api.example.com:443:203.0.113.10 https://api.example.com/health
curl -v --resolve api.example.com:443:203.0.113.11 https://api.example.com/health

Check the accelerator health:

aws globalaccelerator describe-endpoint-group \
  --endpoint-group-arn "$ENDPOINT_GROUP_ARN" \
  --query 'EndpointGroup.EndpointDescriptions[*].{EndpointId:EndpointId,HealthState:HealthState,Weight:Weight}' \
  --output table

Check your application logs for the client IP:

grep 'X-Forwarded-For' /var/log/app/access.log

Or query ALB access logs in Athena:

SELECT
  client_ip,
  request_url,
  target_status_code,
  time
FROM alb_logs
WHERE request_url LIKE '%/health%'
ORDER BY time DESC
LIMIT 20;

Troubleshooting common errors

If traffic does not reach the ALB, check these first:

Symptom	Likely cause	What to check
`curl --resolve` times out	The accelerator endpoint is unhealthy or the ALB security group blocks traffic	Check `describe-endpoint-group`, ALB listener rules, target group health, and security group ingress
TLS certificate mismatch	The request reached the ALB with the wrong host name	Use `--resolve api.example.com:443:<accelerator-ip>` and make sure the ALB certificate covers `api.example.com`
Customer allowlisted one IP but traffic still fails sometimes	Only one Global Accelerator IP was added to the firewall	Give them both accelerator IPs
ALB is still reachable directly	The ALB is internet-facing	Use an internal ALB endpoint or add WAF, host-header checks, authentication, and source restrictions
App logs show a proxy IP instead of the client IP	The app does not trust the right proxy chain	Configure trusted proxy ranges and read `X-Forwarded-For` correctly

The most common mistake is treating the two Global Accelerator IPs as source IPs for the ALB security group. They are destination IPs for clients. Your ALB will not see requests as coming from those two addresses.

Multi-Region is the obvious next step

You do not need multiple Regions to justify Global Accelerator. Fixed ingress IPs are already enough.

But once it is in place, adding another ALB in another Region is straightforward:

"aws_globalaccelerator_endpoint_group" "us_east_1" {
  listener_arn          = aws_globalaccelerator_listener.https.id
  endpoint_group_region = "us-east-1"
  traffic_dial_percentage = 0

  endpoint_configuration {
    endpoint_id                    = aws_lb.api_us_east_1.arn
    weight                         = 128
    client_ip_preservation_enabled = true
  }
}

Start with traffic_dial_percentage = 0, validate the stack, then dial traffic up when you are ready:

traffic_dial_percentage = 10

This is useful for migrations, blue-green deployments, and disaster recovery plans that contain fewer hopes and prayers.

Things to remember

Global Accelerator is a good fit when external parties need stable destination IPs for your AWS-hosted service.

Use this pattern when:

Clients must allowlist fixed IPs.
You want an ALB behind those fixed IPs.
You may want regional failover later.
You want to avoid managing proxy instances or static Elastic IP hacks.

Be clear about what it does not do:

It does not give the ALB fixed IPs.
It does not make an internet-facing ALB private.
It does not let you use the two accelerator IPs as ALB security group sources.
It does not replace application authentication or WAF.

The short version: if you need fixed IP addresses for an AWS ALB, use AWS Global Accelerator. ALB gives you flexible HTTP load balancing. Global Accelerator gives you two stable anycast IPs in front of it. Put them together when the network team on the other side says, "We need IPs, not DNS," and you would prefer not to build your own little networking sadness machine.

Argo CD GitLab Authentication: A Complete GitOps-Friendly Setup

Paweł Swiridow — Wed, 01 Jul 2026 10:00:00 +0000

Argo CD can use GitLab as an authentication provider, so users log in with their GitLab accounts instead of local Argo CD users. That gives you centralized identity, cleaner access control, and one less password database waiting to become a security incident with a logo.

The setup uses OIDC through Dex, stores the GitLab OAuth credentials as encrypted Kubernetes secrets with SOPS and age, and lets Argo CD decrypt them with KSOPS during reconciliation.

How authentication works

Argo CD does not talk to GitLab directly for user login. It uses Dex as an identity broker.

The flow looks like this:

User -> Argo CD -> Dex -> GitLab -> Dex -> Argo CD -> RBAC

The short version:

The user opens Argo CD.
Argo CD redirects the user to Dex.
Dex redirects the user to GitLab.
GitLab authenticates the user and returns OIDC claims.
Argo CD reads those claims and applies RBAC rules.

Authentication tells Argo CD who the user is. RBAC tells Argo CD what damage they are allowed to do. Ideally, not much.

Create a GitLab OAuth application

First, create an OAuth application in GitLab.

In GitLab, go to your user, group, or admin application settings and create a new application with these values:

Name: Argo CD
Redirect URI: https://argocd.example.com/api/dex/callback
Scopes: openid, read_user

GitLab gives you two important values:

Application ID
Secret

In the Argo CD Dex config, these become:

dex.gitlab.clientID: "<gitlab-application-id>"
dex.gitlab.clientSecret: "<gitlab-application-secret>"
dex.gitlab.redirectURI: "https://argocd.example.com/api/dex/callback"

Do not commit the secret in plain YAML. That is not GitOps. That is just leaving your keys under the doormat and calling it automation.

Encrypt the Argo CD secret with SOPS and age

Use SOPS with age to encrypt the Kubernetes secret before committing it.

Generate an age key if you do not have one yet:

age-keygen -o age.key

Store the public key in .sops.yaml:

creation_rules:
  - encrypted_regex: '^(data|stringData)$'
    age: age1examplepublickeyreplacewithyourrealpublickey

Create argocd-secret.yml:

apiVersion: v1
kind: Secret
metadata:
  name: argocd-secret
  namespace: argocd
type: Opaque
stringData:
  dex.gitlab.clientID: "<gitlab-application-id>"
  dex.gitlab.clientSecret: "<gitlab-application-secret>"
  dex.gitlab.redirectURI: "https://argocd.example.com/api/dex/callback"

Encrypt it:

sops --encrypt argocd-secret.yml > argocd-secret.sops.yml

Then remove the unencrypted file:

rm argocd-secret.yml

Yes, really remove it. Git has a long memory and an even longer blame history.

Add KSOPS to the Argo CD repo server

Argo CD does not decrypt SOPS files by default. To make this work with Kustomize, install the KSOPS plugin in the Argo CD repo server.

The plugin lives here:

https://github.com/viaduct-ai/kustomize-sops

If you deploy Argo CD with Helm, add this to your values:

repoServer:
  volumeMounts:
    - name: custom-tools
      mountPath: /usr/local/bin/kustomize
      subPath: kustomize
    - name: custom-tools
      mountPath: /usr/local/bin/ksops
      subPath: ksops
  volumes:
    - name: custom-tools
      emptyDir: {}
  initContainers:
    - name: install-ksops
      image: viaductoss/ksops:v4.5.1
      command:
        [
          "/usr/local/bin/ksops",
          "install",
          "--with-kustomize",
          "/custom-tools",
        ]
      volumeMounts:
        - mountPath: /custom-tools
          name: custom-tools

This installs ksops and a Kustomize binary with plugin support into the repo server container.

Provide the age private key to Argo CD

The repo server needs the age private key to decrypt the SOPS file.

Create a Kubernetes secret from your age key:

kubectl -n argocd create secret generic sops-age-key \
  --from-literal=key="$(cat age.key)"

Expose it to the repo server as SOPS_AGE_KEY:

repoServer:
  env:
    - name: SOPS_AGE_KEY
      valueFrom:
        secretKeyRef:
          name: sops-age-key
          key: key
  volumeMounts:
    - name: custom-tools
      mountPath: /usr/local/bin/kustomize
      subPath: kustomize
    - name: custom-tools
      mountPath: /usr/local/bin/ksops
      subPath: ksops
  volumes:
    - name: custom-tools
      emptyDir: {}

Protect this key properly. Anyone with the private key can decrypt the secrets. Kubernetes secrets are base64 encoded, not magically protected by tiny YAML elves.

Create the KSOPS generator

Create secret-generator.yml:

apiVersion: viaduct.ai/v1
kind: ksops
metadata:
  name: argocd-secret-generator
  annotations:
    config.kubernetes.io/function: |
      exec:
        path: ksops
files:
  - argocd-secret.sops.yml

This tells Kustomize to call ksops and decrypt argocd-secret.sops.yml during the build.

Reference the generator in Kustomize

In kustomization.yml, add the generator:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

generators:
  - secret-generator.yml

When Argo CD renders this application, Kustomize runs the KSOPS generator and produces the decrypted argocd-secret resource.

Configure Dex in Argo CD

Now configure Argo CD to use GitLab through Dex.

In Helm values, configure argocd-cm like this:

configs:
  cm:
    kustomize.buildOptions: "--enable-alpha-plugins --enable-exec"
    dex.config: |
      connectors:
        - type: gitlab
          id: gitlab
          name: GitLab
          config:
            baseURL: https://gitlab.example.com
            clientID: $dex.gitlab.clientID
            clientSecret: $dex.gitlab.clientSecret

The $dex.gitlab.* values are loaded from argocd-secret. That is why the secret names must match exactly.

The Kustomize build options matter:

kustomize.buildOptions: "--enable-alpha-plugins --enable-exec"

This allows Kustomize to execute the KSOPS plugin.

Do not enable exec plugins for repositories you do not trust. Otherwise, you have invented remote command execution with extra YAML. Security teams love that. Briefly.

Configure Argo CD RBAC

Authentication is not authorization. After GitLab login works, configure Argo CD RBAC.

For simple user mapping by email:

helm:
  valuesObject:
    configs:
      rbac:
        policy.csv: |
          g, alice@example.com, role:admin
          g, bob@example.com, role:readonly
        scopes: "[email,groups]"

This maps:

alice@example.com -> role:admin
bob@example.com -> role:readonly

For larger teams, prefer GitLab groups instead of individual users. User by user RBAC starts cute and ends as spreadsheet archaeology.

Apply the configuration

If Argo CD is managed by another Argo CD application, commit the files and let Argo CD sync them.

If you are applying locally for a test, run:

kustomize build --enable-alpha-plugins --enable-exec . | kubectl apply -f -

Then restart the Argo CD components if needed:

kubectl -n argocd rollout restart deployment argocd-server
kubectl -n argocd rollout restart deployment argocd-dex-server
kubectl -n argocd rollout restart deployment argocd-repo-server

Verify the setup

Check that the secret exists:

kubectl -n argocd get secret argocd-secret

Check the Dex logs:

kubectl -n argocd logs deploy/argocd-dex-server

Check the Argo CD server logs:

kubectl -n argocd logs deploy/argocd-server

Check the repo server if decryption fails:

kubectl -n argocd logs deploy/argocd-repo-server

Open Argo CD in the browser:

https://argocd.example.com

You should see a GitLab login option. After login, the user should land in Argo CD with the expected role.

Troubleshoot common failures

If GitLab returns a redirect error, check the redirect URI.

It must match exactly:

https://argocd.example.com/api/dex/callback

If login works but RBAC does not, check which claim Argo CD receives. The email in policy.csv must match the user claim.

If the secret does not render, check the repo server pod:

kubectl -n argocd describe pod -l app.kubernetes.io/name=argocd-repo-server

Common causes:

KSOPS is not installed in the repo server.
-enable-alpha-plugins --enable-exec is missing.
The age private key is not available in the repo server environment.
SOPS_AGE_KEY is missing or points to the wrong Kubernetes secret key.
The encrypted file name does not match secret-generator.yml.
The secret keys do not match $dex.gitlab.clientID, $dex.gitlab.clientSecret, or $dex.gitlab.redirectURI.

You can test decryption locally if you have the age key:

SOPS_AGE_KEY="$(cat age.key)" sops --decrypt argocd-secret.sops.yml

Security notes

Keep the boring security parts boring. Boring is good. Boring means nobody is explaining an outage in a meeting with too many directors.

Encrypt the GitLab OAuth secret with SOPS.
Protect the age private key.
Limit who can change Argo CD applications that use Kustomize exec plugins.
Use GitLab groups for larger teams.
Rotate the GitLab OAuth secret periodically.
Audit role:admin assignments.
Avoid giving admin access to users because they asked nicely in Slack.

Final result

With this setup, GitLab becomes the login provider for Argo CD, secrets stay encrypted in Git, and Argo CD RBAC controls what authenticated users can do.

The moving parts are not complicated:

GitLab OAuth app
SOPS encrypted argocd-secret
KSOPS in repo server
Dex GitLab connector
Argo CD RBAC policy

That is a reasonable amount of YAML for centralized authentication. Not small, obviously. This is Kubernetes. We measure configuration in geological layers.

Secure Access to Private EKS Clusters Without Bastion Hosts Using SSM

Paweł Swiridow — Wed, 13 May 2026 13:00:00 +0000

Accessing Private EKS Clusters Without Losing Your Mind

Locking down your Kubernetes control plane is a basic requirement for any production environment. Exposing the EKS API server to the public internet is just asking for automated scanners to ruin your weekend. However, securing the endpoint creates an operational headache: how do you actually run kubectl when the API is sealed inside a private subnet?

The traditional answer was a bastion host. But managing SSH keys, rotating credentials, and maintaining yet another publicly exposed EC2 instance is tedious. We all know that a "temporary" bastion host spun up on a Friday afternoon will inevitably become a load-bearing production pillar by Monday.

Instead, we can use AWS Systems Manager (SSM) Session Manager. By leveraging the SSM agent already running on your EKS worker nodes, we can securely tunnel our local traffic directly to the private API endpoint without opening inbound ports or managing SSH keys.

The Mechanics of the SSM Tunnel

The flow is straightforward:

Your local machine initiates an SSM port forwarding session targeting a specific EKS worker node.
The SSM session is instructed to forward traffic to a remote host (the private EKS API endpoint URL) on port 443.
You update your kubeconfig to point to localhost on your chosen forwarded port.

Because the worker node is already in the VPC and authorized to talk to the EKS control plane, it acts as a highly secure, identity-aware proxy. Access is governed entirely by IAM, meaning you can audit every connection via CloudTrail.

Prerequisite: IAM Configuration

For this to work, your EKS worker nodes must have the SSM agent installed (the official EKS optimized AMIs have this by default) and the correct IAM permissions.

Here is a Terraform snippet demonstrating how to attach the necessary SSM policy to your existing EKS node IAM role.

# Assumes you already have an aws_iam_role defined for your worker nodes
# named 'eks_node_role'

resource "aws_iam_role_policy_attachment" "ssm_managed_instance_core" {
  policy_arn = "arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore"
  role       = aws_iam_role.eks_node_role.name
}

# Optional but recommended: Restrict who can start sessions in IAM
resource "aws_iam_policy" "ssm_user_access" {
  name        = "EKS-SSM-Tunnel-Access"
  description = "Allows users to port forward to EKS nodes"
  policy      = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Effect = "Allow"
        Action = "ssm:StartSession"
        Resource = [
          "arn:aws:ec2:*:*:instance/*",
          "arn:aws:ssm:*:*:document/AWS-StartPortForwardingSessionToRemoteHost"
        ]
        # In a real environment, restrict the instance resource via tags
        Condition = {
          StringEquals = {
            "ssm:resourceTag/eks:cluster-name" = "my-production-cluster"
          }
        }
      }
    ]
  })
}

This ensures your nodes can communicate with the SSM service and restricts which IAM users can actually initiate the tunnel.

Establishing the Tunnel

Once the nodes are registered in SSM, you need a script to extract a valid instance ID, locate the cluster API endpoint, and start the tunnel.

Here is a Bash script you can execute locally to handle the heavy lifting. It requires the AWS CLI and the Session Manager plugin to be installed on your workstation.

#!/bin/bash
set -euo pipefail

CLUSTER_NAME="my-production-cluster"
REGION="us-east-1"
LOCAL_PORT="8443"

# Fetch the private endpoint of the EKS cluster
echo "Fetching EKS endpoint for ${CLUSTER_NAME}..."
EKS_ENDPOINT=$(aws eks describe-cluster \
  --name "${CLUSTER_NAME}" \
  --region "${REGION}" \
  --query "cluster.endpoint" \
  --output text | sed 's/https:\/\///')

# Find an active worker node instance ID using tags
echo "Finding an active worker node..."
INSTANCE_ID=$(aws ec2 describe-instances \
  --region "${REGION}" \
  --filters "Name=tag:eks:cluster-name,Values=${CLUSTER_NAME}" "Name=instance-state-name,Values=running" \
  --query "Reservations[0].Instances[0].InstanceId" \
  --output text)

if [ "$INSTANCE_ID" == "None" ]; then
  echo "Error: No running worker nodes found."
  exit 1
fi

echo "Establishing SSM tunnel through ${INSTANCE_ID} to ${EKS_ENDPOINT}..."
echo "Leave this terminal open. Access EKS via https://localhost:${LOCAL_PORT}"

# Start the port forwarding session
aws ssm start-session \
  --region "${REGION}" \
  --target "${INSTANCE_ID}" \
  --document-name AWS-StartPortForwardingSessionToRemoteHost \
  --parameters "{\"host\":[\"${EKS_ENDPOINT}\"],\"portNumber\":[\"443\"],\"localPortNumber\":[\"${LOCAL_PORT}\"]}"

Run this script, and it will bind localhost:8443 to the private API endpoint.

Updating Kubeconfig

The final step is modifying your local Kubernetes configuration. You cannot simply run aws eks update-kubeconfig and call it a day, because that will configure the private AWS endpoint, which your machine still cannot route to directly.

You need to manually alter the server field for your cluster to point to the local port.

When you port-forward the EKS API server to your local machine, connecting to https://localhost:8443 introduces a new problem. The API server presents a TLS certificate minted for its internal AWS endpoint (e.g., 1234567890ABCDEF.yl4.us-east-1.eks.amazonaws.com), not localhost.

The quick, dirty fix is to add insecure-skip-tls-verify: true to your kubeconfig. But nothing screams "I definitely passed my SOC2 audit" quite like explicitly disabling TLS validation in production. It is the infrastructure equivalent of putting black tape over a check engine light.

Instead of turning off validation, we can instruct kubectl to connect via our local port but validate the TLS certificate against the actual EKS endpoint hostname. We do this by utilizing the tls-server-name parameter.

apiVersion: v1
clusters:
- cluster:
    server: https://localhost:8443
    # Validate the certificate against the real AWS endpoint
    tls-server-name: 1234567890ABCDEF.yl4.us-east-1.eks.amazonaws.com
  name: arn:aws:eks:us-east-1:123456789012:cluster/my-production-cluster
# ... contexts and users remain unchanged

Once saved, kubectl get pods will route securely through the SSM tunnel, across the worker node, and hit the control plane.

Wrap-Up

Relying on SSM port forwarding eliminates the need for VPNs, bastion hosts, and complex routing rules just to run operational commands against an isolated EKS cluster. By utilizing the existing IAM-integrated agent on your worker nodes, you shrink your external attack surface while maintaining strict audit trails for developer access.

Decoupling Ingress with TargetGroupBinding in EKS

Paweł Swiridow — Wed, 28 Jan 2026 11:00:00 +0000

As we scale our EKS clusters, relying solely on Kubernetes Ingress objects to provision AWS Application Load Balancers (ALBs) can become restrictive. Sometimes we need to attach an EKS Service to a pre-existing ALB managed by Terraform, or we need complex routing rules that are easier to manage in HCL (Terraform) than in K8s annotations.

The AWS Load Balancer Controller supports a custom resource called TargetGroupBinding (TGB). This allows us to provision the Load Balancer and Target Group in Terraform (Infrastructure layer) and simply "bind" our Kubernetes Service to it at runtime (Application layer).

This guide walks through how to set up an AWS Target Group in Terraform and register a Prometheus instance to it using Helm values.

Prerequisites

Before proceeding, ensure your environment meets the following requirements. This architecture relies on specific AWS components to route traffic directly to Pods:

Amazon EKS Cluster: A running EKS cluster is required.
AWS VPC CNI Plugin: We will be using target_type = "ip". This mode requires the AWS VPC CNI (the default networking plugin for EKS), which assigns native AWS VPC IP addresses to Pods. This allows the ALB to route traffic directly to the Pod IP, bypassing the worker node's kube-proxy.
AWS Load Balancer Controller: You must have the AWS Load Balancer Controller (v2.0+) installed and running in your cluster. This controller is responsible for installing the TargetGroupBinding CRD and actively managing the registration of targets.
- Note: Ensure the controller has the necessary IAM permissions to elasticloadbalancing:RegisterTargets and elasticloadbalancing:DeregisterTargets.

Part 1: Infrastructure (Terraform)

First, we need to create the Target Group. The critical setting here is target_type = "ip".

When using the AWS Load Balancer Controller with the AWS VPC CNI, we want the ALB to send traffic directly to the Pod IP addresses, bypassing NodePorts.

`main.tf`

resource "aws_lb_target_group" "prometheus_tg" {
  name        = "eks-prometheus-tg"
  port        = 9090
  protocol    = "HTTP"
  vpc_id      = module.vpc.vpc_id

  # CRITICAL: Must be 'ip' for direct Pod routing via AWS LB Controller
  target_type = "ip"

  health_check {
    path                = "/-/healthy"
    protocol            = "HTTP"
    matcher             = "200"
    interval            = 30
    timeout             = 5
    healthy_threshold   = 3
    unhealthy_threshold = 3
  }

  tags = {
    Environment = "production"
    ManagedBy   = "terraform"
  }
}

# We need to export this ARN to pass it to our Helm chart later
output "prometheus_tg_arn" {
  value = aws_lb_target_group.prometheus_tg.arn
}

Note: Do not use aws_lb_target_group_attachment in Terraform. The AWS Load Balancer Controller running inside the cluster will manage the targets dynamically as Pods come and go.

Part 2: Security (IAM Least Privilege)

By default, the generic AWS Load Balancer Controller policy is permissive (often using Resource: *). In a production environment - especially one with multiple teams sharing an AWS account - we should adhere to Least Privilege.

We must restrict the Controller's ability so it can only register/deregister targets for this specific Target Group, preventing it from accidentally modifying other Load Balancers.

Add this policy to the IAM Role used by your Load Balancer Controller ServiceAccount:

`iam.tf`

data "aws_iam_policy_document" "lb_controller_tgb_policy" {
  statement {
    sid       = "AllowRegisterTargets"
    effect    = "Allow"
    actions   = [
      "elasticloadbalancing:RegisterTargets",
      "elasticloadbalancing:DeregisterTargets"
    ]
    # Scope permissions strictly to the specific Target Group ARN created above
    resources = [aws_lb_target_group.prometheus_tg.arn]
  }

  statement {
    sid       = "AllowDescribeHealth"
    effect    = "Allow"
    actions   = [
      "elasticloadbalancing:DescribeTargetHealth"
    ]
    resources = [aws_lb_target_group.prometheus_tg.arn]
  }
}

resource "aws_iam_policy" "tgb_strict_policy" {
  name        = "eks-alb-controller-tgb-restricted"
  description = "Restricted access for TargetGroupBinding to specific TGs only"
  policy      = data.aws_iam_policy_document.lb_controller_tgb_policy.json
}

Part 3: The Glue (TargetGroupBinding CRD)

The TargetGroupBinding CRD tells the controller: "Watch this Kubernetes Service, and whenever its endpoints change, update this specific AWS Target Group."

A raw TGB manifest looks like this:

apiVersion: elbv2.k8s.aws/v1beta1
kind: TargetGroupBinding
metadata:
  name: prometheus-tgb
  namespace: monitoring
spec:
  serviceRef:
    name: prometheus-k8s # The name of your K8s Service
    port: 9090           # The port defined in the Service
  targetGroupARN: <YOUR_TF_OUTPUT_ARN>

Part 4: Application Deployment (Prometheus Helm Values)

We don't want to apply that YAML manually. We want it version-controlled with our Prometheus deployment.

Most Prometheus Helm charts (like kube-prometheus-stack) support an extraManifests or additionalManifests property in their values.yaml. This allows us to inject arbitrary K8s objects-like our TGB-directly during the Helm install.

Here is how you configure your values.yaml to register Prometheus to the Terraform-managed Target Group.

`values.yaml`

# Configuration for kube-prometheus-stack
prometheus:
  service:
    port: 9090
    # Ensure the service selector matches what the TGB expects
    # usually standard, but good to verify.

# Injecting the Custom Resource Definition
extraManifests:
  - apiVersion: elbv2.k8s.aws/v1beta1
    kind: TargetGroupBinding
    metadata:
      name: prometheus-binding
      namespace: monitoring # Must match the release namespace
    spec:
      serviceRef:
        name: prometheus-kube-prometheus-prometheus # Default name in the stack
        port: 9090
      targetGroupARN: "arn:aws:elasticloadbalancing:us-east-1:1234567890:targetgroup/eks-prometheus-tg/6d0ecf831eec9f09"

Summary of Flow

Terraform creates the empty Target Group (Mode: IP).
Helm deploys Prometheus + the TargetGroupBinding CR.
AWS Load Balancer Controller sees the new Binding CR.
Controller looks up the Pod IPs backing the Prometheus Service.
Controller registers those Pod IPs into the AWS Target Group automatically.

This approach gives us the stability of Terraform-managed Infrastructure (the ALB and Listeners) with the flexibility of Kubernetes-managed endpoints.

Secure AWS Access in Kubernetes: Transitioning from Secrets to IRSA or Pod Identity

Paweł Swiridow — Mon, 22 Sep 2025 09:07:05 +0000

Traditional approaches to AWS authorization from Kubernetes involved creating IAM users with permanent access keys and storing these credentials within the cluster. Modern solutions eliminate stored credentials entirely, using temporary tokens that expire automatically and follow the principle of least privilege.
Think of it like upgrading from physical keys to smart cards for your Kubernetes workloads - instead of storing a master key in every pod, your containers get temporary access cards that only work for specific resources and expire automatically.

The old way: stored credentials (what to avoid)

Before diving into modern solutions, let's understand what we're moving away from. Traditional approaches to AWS access from Kubernetes typically involved:

Create IAM users with access keys (AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY)
Store these credentials in Kubernetes secrets, environment variables, or config files
Hardcode them in container images or application code

Here's what this looked like in a typical Kubernetes deployment:

apiVersion: v1
kind: Secret
metadata:
  name: aws-credentials
  namespace: default
type: Opaque
data:
  AWS_ACCESS_KEY_ID: ABCDEF
  AWS_SECRET_ACCESS_KEY: GHIJKL
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: legacy-app
  namespace: default
spec:
  replicas: 2
  selector:
    matchLabels:
      app: legacy-app
  template:
    metadata:
      labels:
        app: legacy-app
    spec:
      containers:
      - name: legacy-app
        image: my-app:latest
        env:
        - name: AWS_ACCESS_KEY_ID
          valueFrom:
            secretKeyRef:
              name: aws-credentials
              key: AWS_ACCESS_KEY_ID
        - name: AWS_SECRET_ACCESS_KEY
          valueFrom:
            secretKeyRef:
              name: aws-credentials
              key: AWS_SECRET_ACCESS_KEY
        - name: AWS_REGION
          value: "us-west-2"

This approach creates several security and operational challenges in Kubernetes environments:

Security risks:

Long-lived credentials don't expire automatically
Keys can be extracted from running containers or Kubernetes secrets
Credential leaks provide persistent access until manually revoked
Difficult to audit credential usage across pods and namespaces

Operational challenges:

Manual key rotation across all pods and clusters
Credential sprawl as keys get copied across namespaces and environments
Complex secret management across multiple Kubernetes clusters

Modern authorization methods for Kubernetes

Modern AWS authorization from Kubernetes eliminates stored credentials by using temporary tokens that are automatically refreshed. Two primary solutions exist for Kubernetes-to-AWS authentication:

IRSA (IAM roles for service accounts)

IRSA uses OpenID Connect (OIDC) to bridge Kubernetes service accounts with AWS IAM roles. When a pod needs AWS access, it presents a JWT token that AWS validates through the cluster's OIDC provider, receiving temporary credentials in return.

How it works:

Pod receives a JWT token from the Kubernetes API server automatically
AWS SDK includes this token in API requests
AWS STS validates the token through the EKS cluster's OIDC provider
STS returns temporary AWS credentials to the pod
Pod uses these credentials to access AWS services

EKS pod identity

EKS Pod Identity is AWS's newer approach, using a cluster add-on that handles credential exchange automatically. A Pod Identity agent runs as a DaemonSet on each Kubernetes node, intercepting AWS API calls and fetching credentials from AWS's Pod Identity service.

How it works:

Pod makes AWS API calls through the AWS SDK
Pod Identity agent (running as DaemonSet) intercepts these calls
Agent requests credentials from the EKS Pod Identity service
Service returns temporary credentials to the agent
Original API call proceeds with valid credentials

Implementation with Terraform

Let's implement both approaches using consistent IAM policies. We'll use a common S3 access policy for both examples that pods can use to access AWS services:


resource "aws_iam_policy" "s3_access" {
  name        = "kubernetes-app-s3-access"
  description = "S3 access policy for Kubernetes applications"

  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Effect = "Allow"
        Action = [
          "s3:GetObject",
          "s3:PutObject",
          "s3:DeleteObject"
        ]
        Resource = "arn:aws:s3:::my-k8s-app-bucket/*"
      },
      {
        Effect = "Allow"
        Action = [
          "s3:ListBucket"
        ]
        Resource = "arn:aws:s3:::my-k8s-app-bucket"
      }
    ]
  })
}

Setting up IRSA for EKS

Configure your EKS cluster with OIDC enabled to support IRSA:


module "eks" {
  source  = "terraform-aws-modules/eks/aws"
  version = "~> 19.0.0"

  cluster_name    = "my-k8s-cluster"
  cluster_version = "1.28"

# create an OpenID Connect Provider for EKS to enable IRSA (IAM Roles for Service Accounts)
  enable_irsa = true

  vpc_id     = var.vpc_id
  subnet_ids = var.private_subnets

  eks_managed_node_groups = {
    main = {
      desired_size = 2
      max_size     = 4
      min_size     = 1
      instance_types = ["t3.medium"]
    }
  }
}

Create the IAM role that your Kubernetes service account will assume:


module "irsa_role" {
  source = "terraform-aws-modules/iam/aws//modules/iam-role-for-service-accounts-eks"

  role_name = "k8s-app-irsa-role"

  role_policy_arns = {
    s3_access = aws_iam_policy.s3_access.arn
  }

  oidc_providers = {
    main = {
      provider_arn               = module.eks.oidc_provider_arn
      namespace_service_accounts = ["default:k8s-app-service-account"]
    }
  }
}

Setting up EKS pod identity

Enable the Pod Identity add-on in your EKS cluster:


module "eks" {
  source  = "terraform-aws-modules/eks/aws"
  version = "~> 19.0.0"

  cluster_name    = "my-k8s-cluster"
  cluster_version = "1.28"

  cluster_addons = {
    eks-pod-identity-agent = {
      addon_version = "v1.3.4-eksbuild.1"
    }
  }

  vpc_id     = var.vpc_id
  subnet_ids = var.private_subnets

  eks_managed_node_groups = {
    main = {
      desired_size = 2
      max_size     = 4
      min_size     = 1
      instance_types = ["t3.medium"]
    }
  }
}

Create the Pod Identity association to link your Kubernetes service account with AWS permissions:


module "pod_identity" {
  source = "terraform-aws-modules/eks-pod-identity/aws"

  name = "k8s-app-pod-identity"

  additional_policy_arns = {
    S3Access = aws_iam_policy.s3_access.arn
  }

  associations = {
    default = {
      cluster_name    = module.eks.cluster_name
      namespace       = "default"
      service_account = "k8s-app-service-account"
    }
  }
}

Kubernetes manifests

IRSA application deployment


apiVersion: v1
kind: ServiceAccount
metadata:
  name: k8s-app-service-account
  namespace: default
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::123456789012:role/k8s-app-irsa-role
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: k8s-app-irsa
  namespace: default
spec:
  replicas: 2
  selector:
    matchLabels:
      app: k8s-app-irsa
  template:
    metadata:
      labels:
        app: k8s-app-irsa
    spec:
      serviceAccountName: k8s-app-service-account
      containers:
      - name: app
        image: my-app:latest
        env:
        - name: AWS_REGION
          value: "us-west-2"
        ports:
        - containerPort: 8080

Pod identity application deployment


apiVersion: v1
kind: ServiceAccount
metadata:
  name: k8s-app-service-account
  namespace: default
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: k8s-app-pod-identity
  namespace: default
spec:
  replicas: 2
  selector:
    matchLabels:
      app: k8s-app-pod-identity
  template:
    metadata:
      labels:
        app: k8s-app-pod-identity
    spec:
      serviceAccountName: k8s-app-service-account
      containers:
      - name: app
        image: my-app:latest
        env:
        - name: AWS_REGION
          value: "us-west-2"
        ports:
        - containerPort: 8080

Notice how both modern approaches eliminate the need for stored credentials entirely in your Kubernetes cluster. The Pod Identity deployment is even simpler, requiring no special annotations on the service account.

Comparing modern approaches for Kubernetes

IRSA strengths and limitations

Strengths:

Works with any Kubernetes cluster, not just EKS
Mature technology with extensive community support
Fine-grained control over JWT token configuration
Compatible with other OIDC-based systems and service meshes

Limitations:

More complex setup and troubleshooting in Kubernetes
Requires OIDC provider configuration at the cluster level
JWT tokens can become large and cause issues with some applications
Manual annotation management required for each service account

Pod identity strengths and limitations

Strengths:

Simplified setup and ongoing management in EKS
Native AWS integration with better performance
Automatic credential refresh and rotation handled by the agent
Built-in audit logging capabilities through CloudTrail
No JWT token size limitations or complexity

Limitations:

EKS-only solution, not portable to other Kubernetes distributions
Newer technology with less community knowledge and troubleshooting guides
Requires EKS cluster version 1.24 or higher
Less customization flexibility compared to OIDC-based approaches

Security best practices for Kubernetes workloads

Regardless of which modern approach you choose for your Kubernetes cluster, follow these security principles:

Principle of least privilege:


# Good: Specific permissions for specific resources used by pods
resource "aws_iam_policy" "restricted_s3" {
  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Effect = "Allow"
        Action = ["s3:GetObject"]
        Resource = "arn:aws:s3:::k8s-specific-bucket/app-folder/*"
      }
    ]
  })
}

# Avoid: Overly broad permissions that pods don't need
resource "aws_iam_policy" "too_broad" {
  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Effect = "Allow"
        Action = ["s3:*"]
        Resource = "*"
      }
    ]
  })
}

Additional security measures for Kubernetes:

Use Kubernetes namespace isolation to limit service account scope
Implement regular auditing of role usage and permissions across clusters
Monitor credential usage through AWS CloudTrail and Kubernetes audit logs
Set up alerts for unusual access patterns from pods
Use Kubernetes Network Policies to restrict pod-to-pod communication
Regularly review and rotate any remaining long-lived credentials in your cluster

Choosing the right approach

Choose IRSA when:

You need multi-cloud or hybrid Kubernetes support
You have complex OIDC integration requirements
Your team has existing IRSA expertise
You need fine-grained token control

Choose Pod Identity when:

You're using EKS exclusively
You want simplified management overhead
You're starting new projects from scratch
You prefer AWS-native solutions

Migrate from stored credentials when:

You're currently using long-lived access keys
You want to improve security posture
You need better credential rotation
You want to reduce operational overhead

Conclusion

Modern AWS authorization methods eliminate the security risks and operational complexity of stored credentials in Kubernetes environments. Both IRSA and EKS Pod Identity provide secure, temporary credential access that follows cloud security best practices while integrating seamlessly with Kubernetes workflows.

For new EKS projects, Pod Identity offers the simplest path forward with native AWS integration and reduced management overhead. For existing Kubernetes environments or multi-platform requirements, IRSA provides proven reliability and flexibility across different cluster types.

The most important step is moving away from stored credentials entirely in your Kubernetes clusters. Choose the modern approach that best fits your team's expertise and infrastructure requirements - both represent significant improvements over traditional credential management and will enhance your cluster's security posture while simplifying operations.

Secure AWS Access in Kubernetes: Transitioning from Secrets to IRSA or Pod Identity

Paweł Swiridow — Mon, 22 Sep 2025 07:00:00 +0000

The old way: stored credentials (what to avoid)

Before diving into modern solutions, let's understand what we're moving away from. Traditional approaches to AWS access from Kubernetes typically involved:

Create IAM users with access keys (AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY)
Store these credentials in Kubernetes secrets, environment variables, or config files
Hardcode them in container images or application code

Here's what this looked like in a typical Kubernetes deployment:

apiVersion: v1
kind: Secret
metadata:
  name: aws-credentials
  namespace: default
type: Opaque
data:
  AWS_ACCESS_KEY_ID: ABCDEF
  AWS_SECRET_ACCESS_KEY: GHIJKL
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: legacy-app
  namespace: default
spec:
  replicas: 2
  selector:
    matchLabels:
      app: legacy-app
  template:
    metadata:
      labels:
        app: legacy-app
    spec:
      containers:
      - name: legacy-app
        image: my-app:latest
        env:
        - name: AWS_ACCESS_KEY_ID
          valueFrom:
            secretKeyRef:
              name: aws-credentials
              key: AWS_ACCESS_KEY_ID
        - name: AWS_SECRET_ACCESS_KEY
          valueFrom:
            secretKeyRef:
              name: aws-credentials
              key: AWS_SECRET_ACCESS_KEY
        - name: AWS_REGION
          value: "us-west-2"

This approach creates several security and operational challenges in Kubernetes environments:

Security risks:

Long-lived credentials don't expire automatically
Keys can be extracted from running containers or Kubernetes secrets
Credential leaks provide persistent access until manually revoked
Difficult to audit credential usage across pods and namespaces

Operational challenges:

Manual key rotation across all pods and clusters
Credential sprawl as keys get copied across namespaces and environments
Complex secret management across multiple Kubernetes clusters

Modern authorization methods for Kubernetes

Modern AWS authorization from Kubernetes eliminates stored credentials by using temporary tokens that are automatically refreshed. Two primary solutions exist for Kubernetes-to-AWS authentication:

IRSA (IAM roles for service accounts)

How it works:

Pod receives a JWT token from the Kubernetes API server automatically
AWS SDK includes this token in API requests
AWS STS validates the token through the EKS cluster's OIDC provider
STS returns temporary AWS credentials to the pod
Pod uses these credentials to access AWS services

EKS pod identity

How it works:

Pod makes AWS API calls through the AWS SDK
Pod Identity agent (running as DaemonSet) intercepts these calls
Agent requests credentials from the EKS Pod Identity service
Service returns temporary credentials to the agent
Original API call proceeds with valid credentials

Implementation with Terraform

Let's implement both approaches using consistent IAM policies. We'll use a common S3 access policy for both examples that pods can use to access AWS services:


resource "aws_iam_policy" "s3_access" {
  name        = "kubernetes-app-s3-access"
  description = "S3 access policy for Kubernetes applications"

  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Effect = "Allow"
        Action = [
          "s3:GetObject",
          "s3:PutObject",
          "s3:DeleteObject"
        ]
        Resource = "arn:aws:s3:::my-k8s-app-bucket/*"
      },
      {
        Effect = "Allow"
        Action = [
          "s3:ListBucket"
        ]
        Resource = "arn:aws:s3:::my-k8s-app-bucket"
      }
    ]
  })
}

Setting up IRSA for EKS

Configure your EKS cluster with OIDC enabled to support IRSA:


module "eks" {
  source  = "terraform-aws-modules/eks/aws"
  version = "~> 19.0.0"

  cluster_name    = "my-k8s-cluster"
  cluster_version = "1.28"

# create an OpenID Connect Provider for EKS to enable IRSA (IAM Roles for Service Accounts)
  enable_irsa = true

  vpc_id     = var.vpc_id
  subnet_ids = var.private_subnets

  eks_managed_node_groups = {
    main = {
      desired_size = 2
      max_size     = 4
      min_size     = 1
      instance_types = ["t3.medium"]
    }
  }
}

Create the IAM role that your Kubernetes service account will assume:


module "irsa_role" {
  source = "terraform-aws-modules/iam/aws//modules/iam-role-for-service-accounts-eks"

  role_name = "k8s-app-irsa-role"

  role_policy_arns = {
    s3_access = aws_iam_policy.s3_access.arn
  }

  oidc_providers = {
    main = {
      provider_arn               = module.eks.oidc_provider_arn
      namespace_service_accounts = ["default:k8s-app-service-account"]
    }
  }
}

Setting up EKS pod identity

Enable the Pod Identity add-on in your EKS cluster:


module "eks" {
  source  = "terraform-aws-modules/eks/aws"
  version = "~> 19.0.0"

  cluster_name    = "my-k8s-cluster"
  cluster_version = "1.28"

  cluster_addons = {
    eks-pod-identity-agent = {
      addon_version = "v1.3.4-eksbuild.1"
    }
  }

  vpc_id     = var.vpc_id
  subnet_ids = var.private_subnets

  eks_managed_node_groups = {
    main = {
      desired_size = 2
      max_size     = 4
      min_size     = 1
      instance_types = ["t3.medium"]
    }
  }
}

Kubernetes manifests

IRSA application deployment


apiVersion: v1
kind: ServiceAccount
metadata:
  name: k8s-app-service-account
  namespace: default
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::123456789012:role/k8s-app-irsa-role
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: k8s-app-irsa
  namespace: default
spec:
  replicas: 2
  selector:
    matchLabels:
      app: k8s-app-irsa
  template:
    metadata:
      labels:
        app: k8s-app-irsa
    spec:
      serviceAccountName: k8s-app-service-account
      containers:
      - name: app
        image: my-app:latest
        env:
        - name: AWS_REGION
          value: "us-west-2"
        ports:
        - containerPort: 8080

Pod identity application deployment


apiVersion: v1
kind: ServiceAccount
metadata:
  name: k8s-app-service-account
  namespace: default
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: k8s-app-pod-identity
  namespace: default
spec:
  replicas: 2
  selector:
    matchLabels:
      app: k8s-app-pod-identity
  template:
    metadata:
      labels:
        app: k8s-app-pod-identity
    spec:
      serviceAccountName: k8s-app-service-account
      containers:
      - name: app
        image: my-app:latest
        env:
        - name: AWS_REGION
          value: "us-west-2"
        ports:
        - containerPort: 8080

Comparing modern approaches for Kubernetes

IRSA strengths and limitations

Strengths:

Works with any Kubernetes cluster, not just EKS
Mature technology with extensive community support
Fine-grained control over JWT token configuration
Compatible with other OIDC-based systems and service meshes

Limitations:

More complex setup and troubleshooting in Kubernetes
Requires OIDC provider configuration at the cluster level
JWT tokens can become large and cause issues with some applications
Manual annotation management required for each service account

Pod identity strengths and limitations

Strengths:

Simplified setup and ongoing management in EKS
Native AWS integration with better performance
Automatic credential refresh and rotation handled by the agent
Built-in audit logging capabilities through CloudTrail
No JWT token size limitations or complexity

Limitations:

EKS-only solution, not portable to other Kubernetes distributions
Newer technology with less community knowledge and troubleshooting guides
Requires EKS cluster version 1.24 or higher
Less customization flexibility compared to OIDC-based approaches

Security best practices for Kubernetes workloads

Regardless of which modern approach you choose for your Kubernetes cluster, follow these security principles:

Principle of least privilege:


# Good: Specific permissions for specific resources used by pods
resource "aws_iam_policy" "restricted_s3" {
  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Effect = "Allow"
        Action = ["s3:GetObject"]
        Resource = "arn:aws:s3:::k8s-specific-bucket/app-folder/*"
      }
    ]
  })
}

# Avoid: Overly broad permissions that pods don't need
resource "aws_iam_policy" "too_broad" {
  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Effect = "Allow"
        Action = ["s3:*"]
        Resource = "*"
      }
    ]
  })
}

Additional security measures for Kubernetes:

Use Kubernetes namespace isolation to limit service account scope
Implement regular auditing of role usage and permissions across clusters
Monitor credential usage through AWS CloudTrail and Kubernetes audit logs
Set up alerts for unusual access patterns from pods
Use Kubernetes Network Policies to restrict pod-to-pod communication
Regularly review and rotate any remaining long-lived credentials in your cluster

Choosing the right approach

Choose IRSA when:

You need multi-cloud or hybrid Kubernetes support
You have complex OIDC integration requirements
Your team has existing IRSA expertise
You need fine-grained token control

Choose Pod Identity when:

You're using EKS exclusively
You want simplified management overhead
You're starting new projects from scratch
You prefer AWS-native solutions

Migrate from stored credentials when:

You're currently using long-lived access keys
You want to improve security posture
You need better credential rotation
You want to reduce operational overhead

Conclusion

Configuring EKS Managed Node Groups to Use a Proxy with Terraform

Paweł Swiridow — Thu, 18 Sep 2025 08:33:05 +0000

In enterprise environments, security and network policies are paramount. It's common practice for servers, including Kubernetes worker nodes, to reside in private subnets with no direct outbound internet access. Instead, all egress traffic must be routed through a centrally managed and monitored HTTP/HTTPS proxy. While this enhances security, it introduces a configuration challenge for services like Amazon EKS, which need to pull container images, communicate with AWS APIs, and perform other bootstrap tasks.

When using the popular terraform-aws-modules/eks/aws module to provision your cluster, you need a reliable, automated way to inject this proxy configuration into your managed node groups. The goal is to ensure that the operating system, the container runtime (containerd), and the Kubernetes components themselves are all proxy-aware from the moment they boot.

This article details a robust solution using the cloudinit_pre_nodeadm hook provided by the module to configure your EKS nodes for a proxy environment.

The core challenge: comprehensive proxy awareness

Simply setting HTTP_PROXY and HTTPS_PROXY environment variables is not enough. A modern EKS node, built on the Amazon Linux 2 EKS Optimized AMI, has several key components that must be configured independently:

The Base Operating System: For shell sessions and general system utilities.
The Container Runtime (containerd): This is critical. Without proxy settings, containerd will fail to pull any images from public registries like Docker Hub or even private registries outside the VPC, such as Amazon ECR.
The EKS Bootstrap Process (nodeadm): The new bootstrapping agent for EKS AMIs needs to communicate with the EKS control plane. This communication should bypass the proxy.
Package Managers (yum): If you need to install any additional packages as part of your user data, yum must also be configured to use the proxy.

The solution is to use a cloud-init script that runs before the main EKS bootstrapping logic, ensuring the entire environment is correctly configured before any Kubernetes processes are started.

Understanding the bootstrap hook: `cloudinit_pre_nodeadm`

Before diving into the code, it's essential to understand the mechanism we're using. The name cloudinit_pre_nodeadm breaks down into two key concepts:

cloud-init: This is the de facto industry standard for early-stage initialization of cloud instances. When a new EC2 instance boots for the first time, it runs cloud-init, which executes a series of user-provided instructions (user data) to configure the operating system, install packages, create files, and set up users before the machine is considered fully "ready".
nodeadm: Starting with the EKS Optimized Amazon Linux 2023 AMI, nodeadm is the official bootstrap agent responsible for configuring a node and joining it to an EKS cluster. It handles tasks like retrieving cluster certificates, configuring the kubelet, and applying node labels and taints. It has replaced the legacy bootstrap.sh script.

The cloudinit_pre_nodeadm parameter, provided by the terraform-aws-modules/eks/aws module, is a powerful hook that lets you run a custom cloud-init script at a precise moment: after basic OS initialization but before the nodeadm service is started.

This timing is critical. By executing our script before nodeadm, we ensure that the foundational network environment—including our proxy settings—is already in place. When nodeadm, containerd, and the kubelet eventually start, they inherit the correct configuration from the environment, allowing them to function properly within the restricted network.

The solution: using `cloudinit_pre_nodeadm` in Terraform

Our solution will pass the Kubernetes service CIDR from our Terraform configuration directly into the user data script. This removes hardcoded values and makes the solution reusable across different clusters.

Here is the Terraform code block that implements the complete proxy configuration.

module "eks" {
  source  = "terraform-aws-modules/eks/aws"
  version = "~> 21.0"

  # Define the service CIDR for the cluster
  cluster_service_ipv4_cidr = "10.100.0.0/16"

  # ... other EKS cluster configuration ...

  eks_managed_node_groups = {
    main_nodes = {
      # ... other node group configuration like instance_types, min_size, etc. ...

      cloudinit_pre_nodeadm = [
        {
          content_type = "text/x-shellscript"
          content      = <<-EOT
            #!/bin/bash
            set -ex

            # Define your proxy endpoint
            PROXY="http://your-proxy-url:3128" # standard proxy port

            # Use IMDSv2 to securely fetch instance metadata
            TOKEN=`curl -s -X PUT "http://169.254.169.254/latest/api/token" -H "X-aws-ec2-metadata-token-ttl-seconds: 21600"`
            MAC=$(curl -s -H "X-aws-ec2-metadata-token: $TOKEN" http://169.254.169.254/latest/meta-data/mac)
            VPC_CIDR=$(curl -s -H "X-aws-ec2-metadata-token: $TOKEN" http://169.254.169.254/latest/meta-data/network/interfaces/macs/$MAC/vpc-ipv4-cidr-blocks | xargs | tr ' ' ',')
                        K8S_SERVICE_CIDR="${module.eks.cluster_service_ipv4_cidr}"

            # Define your NO_PROXY list. This is critical for internal and AWS service communication.
            # You might need to add your Pod CIDR as well.
            NO_PROXY="$VPC_CIDR,$K8S_SERVICE_CIDR,localhost,127.0.0.1,169.254.169.254,.amazonaws.com,.svc.cluster.local,.svc,.cluster.local"

            # 1. Configure system-wide proxy settings
            echo "Setting up /etc/environment"
            cat << EOF > /etc/environment
            HTTP_PROXY=$PROXY
            HTTPS_PROXY=$PROXY
            NO_PROXY=$NO_PROXY
            http_proxy=$PROXY
            https_proxy=$PROXY
            no_proxy=$NO_PROXY
            EOF

            # 2. Configure containerd proxy via systemd override
            echo "Configuring containerd service"
            mkdir -p /etc/systemd/system/containerd.service.d
            cat << EOF > /etc/systemd/system/containerd.service.d/http-proxy.conf
            [Service]
            EnvironmentFile=/etc/environment
            EOF

            # 3. Configure nodeadm proxy via systemd override
            echo "Configuring nodeadm service"
            mkdir -p /etc/systemd/system/nodeadm.service.d
            cat << EOF > /etc/systemd/system/nodeadm.service.d/http-proxy.conf
            [Service]
            EnvironmentFile=/etc/environment
            EOF

            # 4. Configure yum proxy
            echo "Configuring yum"
            echo "proxy=$PROXY" >> /etc/yum.conf

            # 5. Reload systemd daemon and restart containerd to apply changes
            echo "Reloading systemd and restarting containerd"
            systemctl daemon-reload
            systemctl restart containerd
          EOT
        }
      ]
    }
  }

  # ... other module configuration ...
}

Dissecting the script

Let's break down the content of the cloud-init script to understand each step.

set -ex: This is a standard best practice for shell scripting. set -e ensures the script will exit immediately if a command fails, and set -x prints each command before it is executed, providing clear debug output in the system logs.
Fetching metadata with IMDSv2: The script dynamically fetches the VPC CIDR block directly from the EC2 metadata service. This is crucial for the NO_PROXY variable, ensuring that any traffic destined for other resources within the VPC bypasses the proxy. It uses the more secure IMDSv2 method, which requires a session token.
Defining NO_PROXY: The NO_PROXY variable is just as important as HTTP_PROXY. It specifies a comma-separated list of domains and IP ranges that should not use the proxy. Our list includes:
- The dynamically fetched $VPC_CIDR for all internal VPC traffic.
- The Kubernetes service CIDR passed from module via $K8S_SERVICE_CIDR (e.g., 10.100.0.0/16).
- localhost and the metadata service address (169.254.169.254).
- Key AWS and Kubernetes service endpoints to ensure direct communication with the control plane and other AWS APIs. Note the addition of .amazonaws.com to ensure services like ECR, S3, and EC2 are accessed directly via VPC Endpoints if they exist.
/etc/environment: This file provides the system-wide environment variables for all users and processes. It's the foundation of our configuration.
Systemd overrides: The modern, correct way to modify a systemd service like containerd or nodeadm is to create an override file. We create http-proxy.conf for both services. The EnvironmentFile=/etc/environment directive instructs systemd to load all variables from our global configuration file before starting the service. This is cleaner and more maintainable than modifying the main service files directly.
yum.conf: A simple line added to /etc/yum.conf makes the package manager proxy-aware.
Reload and restart: Finally, systemctl daemon-reload forces systemd to re-read its configuration files, and systemctl restart containerd applies the new environment variables to the container runtime immediately. nodeadm will pick up its configuration when it runs shortly after this script completes.

Successfully deploying EKS worker nodes in a network firewalled behind an HTTP proxy is a common enterprise challenge, but it doesn't have to be a complex one. By leveraging the cloudinit_pre_nodeadm hook within the terraform-aws-modules/eks/aws module, you gain precise control over the node's bootstrap sequence.

This method provides a declarative, automated, and robust solution. By injecting a dynamic script that configures the operating system, containerd, and nodeadm before these critical services start, you ensure nodes join the cluster and pull images reliably. This Infrastructure as Code approach not only solves the immediate technical problem but also creates a repeatable, version-controlled pattern for building secure and compliant Kubernetes platforms on AWS.

Centralized EKS Logging Across AWS Accounts Using Fluent Bit, Pod Identity and OpenSearch

Paweł Swiridow — Wed, 13 Aug 2025 08:47:52 +0000

Centralized log collection is essential in distributed Kubernetes environments. Amazon EKS combined with Fluent Bit offers a lightweight and flexible log forwarding solution. In this guide, we configure Fluent Bit to run on an EKS cluster in one AWS account (Account A) and forward logs to an OpenSearch domain hosted in another account (Account B).

We’ll secure the communication using EKS Pod Identity and a cross-account IAM role protected with an external_id. This ensures that no long-lived credentials are exposed and the trust relationship cannot be abused.

What is Fluent Bit?

Fluent Bit is a lightweight, high-performance log processor and forwarder. Designed with efficiency in mind, it’s ideal for environments like Kubernetes, where resource constraints matter. It collects logs from various sources (e.g., files, containers, syslog), applies filtering and transformation, and then routes them to different destinations such as OpenSearch, Elasticsearch, or cloud services.

What is OpenSearch?

OpenSearch is an open-source search and analytics engine derived from Elasticsearch 7.10. It’s developed and maintained by AWS and the community. OpenSearch supports full-text search, distributed indexing, and real-time analytics — making it an excellent backend for log aggregation, monitoring, and observability platforms.

Architecture overview

account A: hosts the EKS cluster and Fluent Bit
account B: hosts the OpenSearch domain
Fluent Bit pods use a pod identity to assume a cross-account role that grants access to write logs into OpenSearch

To prevent the confused deputy problem, the trust policy in account B includes an external_id that must be known and passed by the caller (Fluent Bit).

Prerequisites

EKS cluster deployed in account A with Pod Identity Agent installed
OpenSearch domain deployed in account B
Terraform v1.6+
kubectl access to the cluster
helm

Creating the pod identity role (account A)

resource "aws_iam_policy" "log_writer_assuming_policy" {
  name = "LogWriterAssumingPolicy"

  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Effect = "Allow"
        Action = [
          "sts:AssumeRole",
          "sts:TagSession"
        ]
        Resource = "arn:aws:iam::ACCOUNT_B_ID:role/fluentbit-opensearch-writer"
      }
    ]
  })
}

module "fluentbit_pod_identity" {
  source  = "terraform-aws-modules/eks-pod-identity/aws"
  version = "1.11.0"

  name = "fluentbit-pod-identity"

  additional_policy_arns = {
    LogWriterAssumingPolicy = aws_iam_policy.log_writer_assuming_policy.arn
  }

  associations = {
    main = {
      namespace       = "middlewares"
      service_account = "fluentbit"
      cluster_name    = var.cluster_name
    }
  }
}

The log_writer_assuming_policy is a policy that allows sts:AssumeRole on the cross-account writer role in account B.

EKS Pod Identity uses session tags (therefore sts:TagSession action) to pass metadata during the AssumeRole operation — specifically:

Kubernetes namespace
Service account name
Cluster name

These are passed as session tags when the EKS control plane assumes the IAM role on behalf of the pod. AWS uses those tags internally to enforce the correct role-to-service-account association, ensuring that only the intended pods can assume that role.

Creating the OpenSearch writer role (account B)

This role trusts only account A to assume it and requires an external ID for additional protection:

data "aws_iam_policy_document" "trust_policy" {
  statement {
    effect = "Allow"

    principals {
      type        = "AWS"
      identifiers = ["arn:aws:iam::ACCOUNT_A_ID:role/fluentbit-pod-identity"]
    }

    actions = ["sts:AssumeRole"]

    condition {
      test     = "StringEquals"
      variable = "sts:ExternalId"
      values   = ["shared-external-id-between-a-and-b-accounts"]
    }
  }
}

resource "aws_iam_role" "fluentbit_writer" {
  name               = "fluentbit-opensearch-writer"
  assume_role_policy = data.aws_iam_policy_document.trust_policy.json
}

Attaching OpenSearch permissions

This policy grants Fluent Bit permission to write to the OpenSearch domain (cluster):

resource "aws_iam_policy" "write_to_opensearch" {
  name = "opensearch-write-access"

  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Effect = "Allow"
        Action = [
          "es:ESHttpPost",
          "es:ESHttpPut"
        ]
        Resource = "arn:aws:es:AWS_REGION:ACCOUNT_B_ID:domain/OPENSEARCH_DOMAIN_NAME/*"
      }
    ]
  })
}

resource "aws_iam_role_policy_attachment" "attach_writer_policy" {
  role       = aws_iam_role.fluentbit_writer.name
  policy_arn = aws_iam_policy.write_to_opensearch.arn
}

Deploying Fluent Bit on EKS with helm

We use the official Helm chart from the eks-charts repo. It supports pod identity and AWS-signed requests to OpenSearch.

Make sure the IAM role from account B is correctly assumed by Fluent Bit using the pod identity role from account A. Also, pass the external ID as a Helm value. Remember that AWS_Role_ARN has to be an ARN of role in account B. Service account name should be the same as we set up in fluentbit_pod_identity module.

helm repo add eks-charts https://aws.github.io/eks-charts
helm repo update

helm upgrade --install fluentbit eks-charts/aws-for-fluent-bit \
  --namespace middlewares \
  --create-namespace \
  --set serviceAccount.name="fluentbit-service-account-name" \
  --set output.opensearch.enabled=true \
  --set output.opensearch.host="vpc-my-domain.region.es.amazonaws.com" \
  --set output.opensearch.port=443 \
  --set output.opensearch.awsAuth=true \
  --set output.opensearch.awsExternalId="shared-external-id-between-a-and-b-accounts" \
  --set output.opensearch.AWS_Role_ARN="arn:aws:iam::ACCOUNT_B_ID:role/fluentbit-opensearch-writer" \
  --set output.opensearch.index="index-name" \
  --set output.opensearch.type="_doc"

Verifying the setup

Check the logs of the Fluent Bit pods:

kubectl logs -n middlewares -l app.kubernetes.io/name=fluentbit

You should see lines indicating successful writes:

[ info] [output:es:es.0] HTTP status=200

You can also search in the OpenSearch dashboard to confirm logs are being indexed in eks-logs.

Conclusion

This setup allows Fluent Bit to securely ship logs from EKS in account A to OpenSearch in account B. By using EKS Pod Identity and a cross-account IAM role with an external ID, we avoid exposing secrets while still maintaining a strong trust boundary.

This architecture is well-suited for multi-account environments and aligns with AWS best practices for identity and access control.