DEV Community: Jakub

Secure Private EKS Access and SSO-Protected Frontends with Cloudflare Tunnel on EC2

Jakub — Mon, 18 May 2026 17:47:29 +0000

What I Built

The system uses a Cloudflare Tunnel running on a single EC2 instance to replace traditional VPN infrastructure. It provides zero-trust VPC access via WARP for engineers and identity-aware frontend application delivery through a private ALB, exposing services on public subdomains without opening inbound firewall ports.

System Architecture

The runtime infrastructure groups components strictly around a single egress-only tunnel instance that services two access models:

EC2 instance — A t4g.micro instance running RHEL 9 in a private subnet with no public IP, no SSH access, and no inbound ports.

Cloudflare Tunnel — A daemon running as a systemd service on the EC2 instance that opens outbound QUIC connections to Cloudflare's edge network.

Cloudflare Access Applications — Edge configurations mapping public subdomains to the internal load balancer while enforcing identity provider SSO.

Private ALB — An AWS Application Load Balancer with no internet-facing listener, fronting frontend services inside EKS.

AWS Secrets Manager — Secure persistent storage holding the tunnel token retrieved by the instance at boot.

Security group — An AWS network firewall configured with restrictive egress-only rules for the tunnel instance.

IAM role — An execution role scoped strictly to read permissions for AWS Secrets Manager and AWS Systems Manager.

EKS cluster security group rule — A network policy rule allowing internal ingress traffic from the tunnel instance.

All system management occurs via AWS Systems Manager Session Manager.

Core Technical Behavior

At runtime, the EC2 instance retrieves the authentication token from AWS Secrets Manager and starts the cloudflared daemon. The process initiates outbound QUIC connections to Cloudflare's edge network over ports 7844 and 7845.

For engineer network routing, users connect via the local Cloudflare WARP client. Traffic destined for the VPC CIDR routes through the tunnel, giving direct network access to the EKS API server, internal RDS databases, and private cluster services.

For web traffic, Cloudflare Access serves as an identity-aware reverse proxy. External web requests to public domains hit the Cloudflare edge, which evaluates user sessions against an identity provider. Authenticated requests pass through the QUIC tunnel to the private ALB, which forwards traffic directly to frontend pods running inside EKS.

Frontend Request Flow

User → app.jakops.cloud (Cloudflare Edge, TLS + SSO)
     → Cloudflare Tunnel (encrypted QUIC)
     → EC2 cloudflared instance (private subnet)
     → Private ALB (VPC)
     → EKS frontend pods

Instance Egress Security Group Rule

egress {
  from_port   = 7844
  to_port     = 7845
  protocol    = "udp"
  cidr_blocks = ["0.0.0.0/0"]
  description = "Allow outbound QUIC for Cloudflare Tunnel"
}

VPC Internal Ingress Security Group Rule

ingress {
  from_port   = 0
  to_port     = 0
  protocol    = "-1"
  cidr_blocks = [var.vpc_cidr_block]
  description = "Allow all traffic from VPC for WARP routing"
}

Key Engineering Decisions

Dual-purpose tunnel consolidates L3/L4 network proxying via WARP and L7 application routing via Cloudflare Access onto a single EC2 footprint.

Public domains with private infrastructure keeps public DNS records pointed to Cloudflare's edge while leaving the AWS footprint completely invisible without public endpoints.

SSO at the edge forces authentication before traffic ever enters the AWS network, removing the requirement for application-level authentication gates on internal frontends.

Private ALB configuration removes internet-facing listeners, eliminating public DDoS surface area, public security group tracking, and AWS-side certificate rotation.

ARM architecture selection leverages t4g.micro instances to save approximately 20% on compute cost compared to t3 variants while running the lightweight cloudflared Go binary.

Hardcoded AMI pinning prevents unintended infrastructure tearing and instance recreation during terraform apply actions when upstream OS images change.

Single instance deployment without an Auto Scaling Group trades automated sub-minute failover for simplified configuration on staging and developer environments.

IMDSv2 requirement mitigates SSRF-based IAM credential theft from the EC2 instance metadata service endpoint.

Dedicated system user constraints execute the cloudflared binary as a non-login user with no system shell to restrict localized blast radius.

KMS-encrypted EBS enforces protection of the root volume data at rest via a customer-managed key.

Trade-offs

Optimized for: cost, simplicity, zero-trust posture, unified access control, operational minimalism, elimination of public attack surface.

Sacrificed: high availability (single instance), self-healing (no ASG), automated AMI rotation, independent scaling of frontend proxy vs. WARP routing, centralized logging (logs stay on-instance via journald).

Results / Cost Impact

The implementation reduced ongoing infrastructure spend to an explicit total of approximately $7.33 per month.

t4g.micro (on-demand) — ~$6.13
8GB GP3 EBS — ~$0.80
Secrets Manager secret — ~$0.40

This architecture replaced a managed VPN product and a public ALB setup including WAF, certificate validation, and public DNS operations that had cost over $75 per month. Operational overhead for certificate rotation, WAF rule maintenance, and network auditing was removed.

Conclusion

A single Cloudflare Tunnel instance provides concurrent infrastructure routing for developers and SSO-gated public domain ingress for external stakeholders. By keeping the target AWS load balancer private, the entire internal network remains closed to inbound public traffic while supporting edge-authenticated web delivery.

Combining WARP routing with Cloudflare Access applications on a single tunnel gives you both L3 infrastructure access and L7 application delivery with SSO on real domains with zero public infrastructure for under $8/month.

Need Help?

If you want to deploy a zero-trust setup including Cloudflare Tunnel on EC2, WARP routing, private ALB ingress for EKS, and Terraform modules with IMDSv2 and KMS encryption, you can find assistance directly at https://jakops.cloud.

Migrating a Terraform Monolith to Terragrunt: State Slicing Without Downtime

Jakub — Fri, 08 May 2026 07:39:10 +0000

What I Built

I decomposed a monolithic Terraform state containing 19 logical AWS infrastructure components into a Terragrunt monorepo. This migration established isolated state files for each component—including VPC, EKS, and RDS—to enable independent locking, reduced blast radius, and faster plan performance without triggering any infrastructure changes or downtime.

System Architecture

Monolith State — A single S3-backed state file containing all 19 infrastructure components under a nested module hierarchy.

Terragrunt Modules — 13 independent module directories, each inheriting root configuration and managing a unique S3 state key.

Dependency Graph — Explicit inter-module wiring using Terragrunt dependency blocks to pass versioned outputs between isolated states.

Core Technical Behavior

The system runtime behavior changed from a single global lock to a per-component locking model. In the monolith, any change to a load balancer rule required a full re-evaluation of the entire stack, including RDS and EKS clusters. By slicing the state, I isolated the execution flow so that Terraform only reconciles the resources relevant to a specific logical component.

The migration process relied on address rewriting to drop the top-level parent prefixes used in the monolith. For example, a resource originally located at module.client_stage.module.database.module.rds.aws_db_instance.this[0] was moved to module.rds.aws_db_instance.this[0] within the new isolated rds module state.

Pulling the monolith state to a local file for immutable processing

terraform state pull > monolith.tfstate

Dynamically discovering direct child modules from the state list

DIRECT_MODULES=$(echo "$STATE_LIST" | grep "^${MODULE_PREFIX}\.module\." | \
  sed "s|^${MODULE_PREFIX}\.module\.||" | \
  sed 's/^\([^.[]*\).*/\1/' | sort -u)

Executing the state move from the local monolith source to individual module states

terraform state mv \
  -state="$MONOLITH_STATE" \
  -state-out="$TARGET_STATE" \
  "$resource" "$new_address"

Final runtime verification involved a run-all plan across the dependency graph. This confirmed that downstream modules could successfully read VPC IDs and RDS endpoints from upstream modules via typed outputs stored in the new isolated state files.

Key Engineering Decisions

Script-driven slicing over manual commands was implemented to ensure the move of hundreds of resources across 13 modules remained reproducible and free of manual typos.

Immutable source state management used separate -state and -state-out files to ensure the local monolith snapshot was never modified during the slicing process, allowing for clean retries.

Dynamic module discovery derived module names directly from the state list rather than a hardcoded inventory, preventing the silent omission of existing infrastructure from the migration.

Python-based regex processing was utilized for address rewriting to correctly handle dot-separated and bracket-indexed resource patterns that are not safely handled by standard shell tools.

Local backend validation was performed before migrating to S3 to verify each module against a zero-diff plan, ensuring the state perfectly matched live infrastructure before pushing to remote storage.

Trade-offs

Optimized for: blast radius reduction, per-module state locking, and faster iteration via targeted plan/apply cycles.

Sacrificed: operational simplicity during the migration window, requiring a change freeze to prevent drift while state existed in both monolithic and sliced forms.

Results / Cost Impact

The platform now operates 13 independent state files in S3, each protected by its own DynamoDB lock.

Parallel workstreams no longer block each other, as a Kubernetes deployment change no longer locks the VPC or database state.

The system enforces explicit ownership boundaries, where changes are restricted to specific infrastructure concerns without the risk of affecting adjacent resources in the same state file.

Conclusion

This migration turned a monolithic bottleneck into a scalable management boundary by performing state surgery instead of infrastructure re-creation. The resulting system maintains zero-drift compared to the original monolith while enabling the team to execute parallel changes with isolated failure modes.

The correctness of a state migration is guaranteed when every isolated module produces a clean plan with zero diff.

Need Help?

If you're working on a similar state decomposition or evaluating Terragrunt adoption for a growing SaaS platform, feel free to reach out at hello@jakops.cloud.

https://jakops.cloud

Athena Cost Kill Switch: Automated IAM Credential Revocation with CloudWatch, EventBridge, and Lambda

Jakub — Wed, 06 May 2026 12:16:38 +0000

How to design an automated kill switch for an Athena data platform that disables service credentials within seconds of a scan threshold breach.

What I Built

This system provides an automated response to excessive AWS Athena scan costs generated by external services. It monitors Athena workgroup metrics and immediately revokes IAM access keys when pre-defined data processing thresholds are exceeded, preventing unmonitored cost spikes without requiring human intervention.

System Architecture

The architecture is composed of four distinct layers operating in sequence to monitor, route, and execute the revocation.
Athena Workgroups - Dedicated workgroups for PowerBI and OpenMetadata that enforce a 1 GB per-query scan cutoff and publish CloudWatch metrics.

CloudWatch Alarms - Three independent alarms monitoring the OpenMetadata workgroup for sustained high usage, high failure rates, and rapid consumption spikes.
EventBridge Rule - A routing layer that pattern-matches CloudWatch Alarm State Change events to trigger the execution logic.
Lambda Kill Switch - A Python-based function that retrieves service credentials from Secrets Manager and executes the IAM revocation call.
Secrets Manager - A KMS-encrypted store for the OpenMetadata IAM username and access key ID, keeping the execution logic stateless.

Core Technical Behaviour

The system remains passive until a threshold is breached. CloudWatch tracks ProcessedBytes and query failure counts at the workgroup level. When a metric crosses a threshold, the alarm transitions to ALARM state.
EventBridge detects this state change and triggers the Lambda function. The Lambda performs two primary operations: it fetches the target IAM metadata from Secrets Manager and calls the IAM API to set the specific access key status to Inactive.
Python

# One-line caption: Disabling the IAM access key via Boto3
iam_client.update_access_key(
    UserName=username,
    AccessKeyId=access_key_id,
    Status="Inactive"
)

The execution flow is asynchronous. While the Lambda disables the credential, SNS simultaneously sends email notifications to the engineering team. Once the key is inactive, all subsequent Athena queries from the external service fail with authentication errors until manual rotation or reactivation occurs.

Key Engineering Decisions

IAM user with static credentials was used because OpenMetadata does not support IAM role assumption. Disabling the access key provides the fastest possible revocation without modifying IAM policies or workgroup configurations.
Storing the access key ID and IAM username in Secrets Manager keeps the Lambda stateless. This ensures that credential rotation can occur within the security layer without requiring code changes or redeployments of the Lambda infrastructure.
Three independent alarms were chosen over a composite alarm to ensure any single failure mode - sustained volume, high failure rates, or sudden spikes - triggers the switch immediately. A composite alarm would have required multiple conditions to be met simultaneously.
Direct EventBridge-to-Lambda integration was selected over Step Functions for this path. While the platform's S3-triggered Glue pipeline uses Step Functions for stateful orchestration, the kill switch is a single, stateless API call where added orchestration would only increase latency.
The use of configurable Terraform variables for thresholds allows for environment-specific tuning. This enables tighter cost controls in staging and more relaxed limits in production without modifying the underlying logic.

Trade-offs

Optimized for: speed of response and operational simplicity. The system executes in seconds with a minimal codebase and no external dependencies beyond native AWS APIs.

Sacrificed: self-healing. The system requires a deliberate manual action by a platform engineer to investigate the root cause and re-enable or rotate credentials.

The system lacks a dead-letter queue on the Lambda invocation. If the IAM API call fails or Secrets Manager is throttled, the system relies on standard Lambda async retries without a secondary alerting path for the kill switch's own failure.

The spike alarm uses a fixed 60-second period. This fixed window cannot be adjusted via Terraform variables, meaning a legitimate high-volume schema discovery scan could trigger a false positive that requires a code change to tune.

The high-usage alarm does not have a direct Lambda action assigned in its configuration. It relies entirely on the EventBridge rule pattern match for routing, which differs from how the other alarms utilize direct actions.

Results / Cost Impact

The system introduces negligible ongoing costs as it is entirely event-driven. It eliminates the response window between a threshold breach and human intervention, which is critical for Athena where billing is processed per byte scanned. The platform team receives immediate SNS notifications while the automated response ensures that financial exposure is capped within seconds of a breach.

Conclusion

This architecture uses CloudWatch, EventBridge, and Lambda to create a production-grade cost control mechanism for managed query services. By targeting the IAM credential layer, the system provides a reversible but immediate response to misbehaving external connectors.
Automated cost control is most effective when it targets well-defined service boundaries with fast event routing and manual recovery.

Need Help?

If you're working on similar infrastructure challenges around AWS cost control, data platform access governance, or IAM-level automation, feel free to reach out at hello@jakops.cloud.

DEV Community: Jakub

Secure Private EKS Access and SSO-Protected Frontends with Cloudflare Tunnel on EC2

What I Built

System Architecture

Core Technical Behavior

Key Engineering Decisions

Trade-offs

Results / Cost Impact

Conclusion

Further Reading

Need Help?

Migrating a Terraform Monolith to Terragrunt: State Slicing Without Downtime

What I Built

System Architecture

Core Technical Behavior

Key Engineering Decisions

Trade-offs

Results / Cost Impact

Conclusion

Further Reading

Need Help?

Athena Cost Kill Switch: Automated IAM Credential Revocation with CloudWatch, EventBridge, and Lambda

What I Built

System Architecture

Core Technical Behaviour

Key Engineering Decisions

Trade-offs

Results / Cost Impact

Conclusion

Further Reading

Need Help?