DEV Community: saifeddine Rajhi

Amazon EKS enhanced network policies: Admin and DNS-based controls explained

saifeddine Rajhi — Mon, 05 Jan 2026 05:30:44 +0000

A guide to new Amazon EKS Admin and DNS (FQDN) network policies released in December: what they are, how they work, and how to use them with practical YAML.

Amazon EKS enhanced network policies: Admin and DNS-based controls explained

In December, Amazon EKS introduced enhanced network policy capabilities:

Admin network policies for cluster-wide control with tiers (Admin and Baseline).
Application (DNS/FQDN) network policies for domain-based egress in EKS Auto Mode.

This post explains what changed, why it matters, how policies are evaluated, and how to start using them today with simple examples.

TL;DR

Admin policies give platform/security teams cluster-wide, non-overridable controls (Admin tier) and default, overridable controls (Baseline tier).
Application Network Policies let you allow egress to domains (FQDNs) like api.example.com or *.s3.amazonaws.com — no more IP lists.
Evaluation order: Admin tier (Deny/Allow/Pass) → namespace NetworkPolicy/ApplicationNetworkPolicy → Baseline tier → default deny.
Requirements: Kubernetes 1.29+, VPC CNI v1.21.1+ for Admin policies on EC2-based clusters; DNS/FQDN policies are available in EKS Auto Mode.
Start with deny-by-default at the cluster level, then allow only what you need.

What’s new in EKS:

1) Admin Network Policies (ClusterNetworkPolicy CRD)

Cluster-scoped policies applied across namespaces.
Two tiers:
- Admin: mandatory controls, cannot be overridden.
- Baseline: defaults that can be overridden by namespace policies.
Use for org-wide guardrails (e.g., block IMDS, force isolation, allow monitoring).

2) Application Network Policies (ApplicationNetworkPolicy CRD)

Namespace-scoped, DNS/FQDN-aware egress rules.
Filter traffic by domain names at L7 (e.g., *.salesforce.com, api.internal.company.com).
Great for SaaS access and hybrid/on-prem connectivity without tracking changing IPs.
Available in EKS Auto Mode.

Why this matters

Stable rules: domains don’t change as often as IPs.
Central control: security teams set guardrails once, developers work safely within them.
Less toil: fewer IP updates, simpler operations.

How policy evaluation works (in order):

1) Admin tier ClusterNetworkPolicy

Deny: highest precedence; blocks immediately.
Allow: accepts and stops evaluation.
Pass: skips remaining Admin tier, defers to namespace policies. 2) Namespace policies
ApplicationNetworkPolicy (DNS/FQDN) and traditional NetworkPolicy are evaluated.
Can further restrict, but cannot override an Admin Deny. 3) Baseline tier ClusterNetworkPolicy
Defaults that can be overridden by namespace policies. 4) Default deny if nothing matches.

Think of it as organization guardrails first, then team-level details, then safe defaults.

Requirements and availability

Kubernetes: 1.29+.
EKS Auto Mode: supports Admin + DNS/FQDN ApplicationNetworkPolicy.
EKS on EC2: supports Admin policies via Amazon VPC CNI.
- Amazon VPC CNI: v1.21.1+.
DNS-based policies are exclusive to EKS Auto Mode-launched EC2 instances.

Quick start

Enable network policy support in the VPC CNI (EC2-based clusters):

Update the VPC CNI add-on (v1.21.1+ recommended).
Set configuration values to enable network policies.

Example (Console → EKS → Add-ons → VPC CNI → Edit → Configuration values):

{ "enableNetworkPolicy": "true" }

Verify CNI pods:

kubectl get pods -n kube-system | grep 'aws-node\|amazon'

Note:

ApplicationNetworkPolicy (DNS/FQDN) requires EKS Auto Mode.
ClusterNetworkPolicy works on both EKS Auto Mode and EC2-based EKS with the VPC CNI requirements above.

Practical examples

1) Admin: block EC2 Instance Metadata Service (IMDS) for all pods

Mandatory, cluster-wide protection that can't be overridden.

apiVersion: networking.k8s.aws/v1alpha1
kind: ClusterNetworkPolicy
metadata:
  name: block-instance-metadata-service
spec:
  tier: Admin
  priority: 10
  subject:
    namespaces: {}  # all namespaces
  egress:
    - name: deny-metadata-service
      action: Deny
      to:
        - networks:
            - cidr: "169.254.169.254/32"

2) Admin: isolate a sensitive namespace from the rest

Blocks all ingress from other namespaces into a protected namespace.

apiVersion: networking.k8s.aws/v1alpha1
kind: ClusterNetworkPolicy
metadata:
  name: protect-sensitive-workload
spec:
  tier: Admin
  priority: 20
  subject:
    namespaces:
      matchLabels:
        kubernetes.io/metadata.name: earth
  ingress:
    - action: Deny
      name: select-all-deny-all
      from:
        - namespaces:
            matchLabels: {}  # match all namespaces

3) Admin: allow monitoring and DNS egress everywhere

Enforce visibility and reliable DNS resolution cluster-wide.

apiVersion: networking.k8s.aws/v1alpha1
kind: ClusterNetworkPolicy
metadata:
  name: cluster-wide-allow-monitoring-and-dns
spec:
  tier: Admin
  priority: 30
  subject:
    namespaces: {}
  ingress:
    - action: Accept
      name: allow-monitoring-ns-ingress
      from:
        - namespaces:
            matchLabels:
              kubernetes.io/metadata.name: monitoring
  egress:
    - action: Accept
      name: allow-kube-dns-egress
      to:
        - pods:
            namespaceSelector:
              matchLabels:
                kubernetes.io/metadata.name: kube-system
            podSelector:
              matchLabels:
                k8s-app: kube-dns

4) Application (DNS/FQDN): allow a backend to call an on-prem domain

Namespace-scoped, for EKS Auto Mode.

apiVersion: networking.k8s.aws/v1alpha1
kind: ApplicationNetworkPolicy
metadata:
  name: moon-backend-egress
  namespace: moon
spec:
  podSelector:
    matchLabels:
      role: backend
  policyTypes:
    - Egress
  egress:
    - name: allow-onprem-api
      to:
        - domainNames:
            - "myapp.mydomain.com"
      ports:
        - protocol: TCP
          port: 8080

5) Application (DNS/FQDN): allow access to AWS services by domain

Great for multi-tenant setups without managing IPs.

Allow pods labeled security-tier=low to use S3 from the default namespace:

apiVersion: networking.k8s.aws/v1alpha1
kind: ApplicationNetworkPolicy
metadata:
  name: access-to-s3
  namespace: default
spec:
  podSelector:
    matchLabels:
      security-tier: low
  policyTypes:
    - Egress
  egress:
    - name: allow-access-to-s3
      action: Accept
      to:
        - domainNames:
            - "*.s3.us-east-1.amazonaws.com"

Allow one app to reach DynamoDB in another namespace:

apiVersion: networking.k8s.aws/v1alpha1
kind: ApplicationNetworkPolicy
metadata:
  name: access-to-dynamodb
  namespace: cart-services
spec:
  podSelector:
    matchLabels:
      app: checkout-service
  policyTypes:
    - Egress
  egress:
    - name: allow-dynamodb-access
      action: Accept
      to:
        - domainNames:
            - "*.dynamodb.us-east-1.amazonaws.com"

Best practices:

Start deny-by-default: Use Admin tier Deny for high-risk destinations; then allow only what’s required.
Use labels for segmentation: e.g., security-tier=high vs security-tier=low across namespaces.
Prefer specific domains over wildcards: *.amazonaws.com is convenient but broad; use exact endpoints when possible.
Layer policies: Combine Admin policies with namespace-level ApplicationNetworkPolicy and traditional NetworkPolicy for defense-in-depth.
Keep policies small and readable: Short, focused rules are easier to review and audit.

Monitoring, audit, and Route 53 DNS Firewall:

Enable policy decision logging and forward to CloudWatch or your SIEM.
Audit “denied” flows regularly to catch misconfigurations or suspicious behavior.
Remember: If EKS policy allows a domain but Route 53 DNS Firewall blocks it, DNS resolution fails and the connection won’t establish. These layers complement each other.

Common gotchas:

DNS/FQDN policies require EKS Auto Mode; they won’t work on EC2-based clusters without Auto Mode.
Make sure your VPC and routing allow egress where needed (Transit Gateway, NAT, firewalls).
Order matters: Admin Deny beats everything; Pass hands control to namespace level; Baseline applies last.
Validate in staging first: replicate production network/DNS behavior to avoid surprises.

Quick checklist:

Kubernetes 1.29+.
For EC2-based EKS clusters: VPC CNI v1.21.1+ and enable network policy support.
For DNS/FQDN egress: use EKS Auto Mode.
Implement Admin Deny guardrails (e.g., IMDS).
Add specific ApplicationNetworkPolicy rules for allowed external domains.
Monitor, audit, and iterate.

Wrap-up:

Amazon EKS now makes it much easier to enforce strong, centralized network security while keeping developer workflows simple:

Admin policies create clear, cluster-wide guardrails.
Application (DNS/FQDN) policies let teams express intent using domain names instead of IPs.

Adopt a deny-by-default stance, allow only what you need, and keep auditing. Your clusters will be safer — with less operational overhead.

Happy shipping!

References ©️

AWS blog: https://aws.amazon.com/blogs/containers/enhance-amazon-eks-network-security-posture-with-dns-and-admin-network-policies/

Thank You 🖤

Until next time, keep innovating and securing your cloud journey!

💡 Thank you for reading!

Until next time, つづく 🎉

🙌🏻😁📃 see you in the next blog.🤘 Until next time 🎉

🚀 Thank you for sticking up till the end. If you have any questions/feedback regarding this blog feel free to connect with me:

♻️ LinkedIn: https://www.linkedin.com/in/rajhi-saif/

♻️ X/Twitter: https://x.com/rajhisaifeddine

The end ✌🏻

🔰 Keep Learning !! Keep Sharing !! 🔰

📅 Stay updated

Subscribe to our newsletter for more insights on AWS cloud computing and containers.

Native Amazon EKS Backups with AWS Backup

saifeddine Rajhi — Mon, 29 Dec 2025 06:30:20 +0000

TL;DR: AWS Backup now protects Amazon EKS. It backs up your cluster state and persistent data (EBS, EFS, S3) using one policy-driven workflow. You can restore a full cluster, a namespace, or specific volumes even into a brand-new EKS cluster.

It’s been available since Nov 2025, and it’s now a dependable option for production teams.

Why this matters:

Before this, many teams used scripts or tools like Velero to back up Kubernetes. It worked, but it was hard to run at scale.

Now you get:

One place to manage backups for EKS and other AWS services.
Policy-based schedules, retention, encryption, and immutability.
A single “composite” recovery point that keeps cluster state and data in sync.
Stress-free restores, including creating a new EKS cluster during restore.

What AWS Backup protects in EKS:

AWS Backup creates a composite recovery point(cluster state + persistent storage «EBS, EFS, S3» as a single, consistent recovery point) with child recovery points:

Cluster state (full backup)
- Examples: cluster name and settings, IAM role, VPC and network settings, logging, encryption, add-ons, access entries, managed node groups, Fargate profiles, pod identity associations, Kubernetes manifests.
Persistent storage (incremental where supported)
- Amazon EBS, Amazon EFS, Amazon S3 buckets (bucket-level snapshot backups)

What is not included:

Container images in external registries (ECR, Docker, etc.)
Infrastructure like VPCs and subnets
Auto-generated runtime objects like nodes, auto-generated pods, events, leases, jobs
Some CSI/ACK plugin scenarios (see Limits below)

How to enable and create a backup (Console):

Open AWS Backup → Settings → Configure resources → opt in to Amazon EKS.
Go to Protected resources → Create on-demand backup.
Choose Resource type: Amazon EKS → select your cluster.
Choose an IAM role with:
- AWSBackupServiceRolePolicyForBackup
- AWSBackupServiceRolePolicyForRestores
- If your cluster uses S3: also add AWSBackupServiceRolePolicyForS3Backup
Configure window, retention, and (optionally) lifecycle to cold storage for supported child recovery points.
Create the on-demand backup.

Tip: Set EKS Cluster authorization mode to API or API_AND_CONFIG_MAP so AWS Backup can create Access Entries.

Understanding recovery points and status:

Composite recovery point (parent): the EKS backup as a whole.
Child recovery points: cluster state and each persistent store.

Statuses:

Completed: everything finished; the cluster is protected.
Partial: some parts completed, others failed. You can re-run failed parts and still restore the successful ones.
Failed: the job did not complete; fix the issue and try again.

Restore options (Console):

In Protected resources, pick your EKS composite recovery point → Restore.
Scope:
- Full cluster
- Namespace (only to an existing cluster)
- Individual persistent volumes
Destination:
- Existing cluster (non-destructive; only the delta is applied)
- Original cluster
- New cluster (AWS Backup can provision it during restore)
Choose IAM role for restore. Review settings → Restore.

You can monitor child recovery points during restore. If some parts fail, you can restore the successful ones (for example, EBS volumes) separately.

Copying, vaults, and immutability:

You can copy composite recovery points across Regions/accounts (where supported).
Use backup vaults for organization, access control, and immutability.
Child recovery points for persistent storage support lifecycle transitions to cold storage.

Limits to know:

Persistent volumes using CSI migration, in-tree storage plugins, or ACK controllers are not supported.
S3 backups: only bucket-level snapshots are supported (no prefix-level backup via CSI mount points).
FSx via CSI driver: not supported.
EKS on AWS Outposts: not supported.
Subject to backup/restore quotas. Check the AWS Backup feature matrix for details.

Best practices:

Set EKS authorization mode to API or API_AND_CONFIG_MAP.
Use managed policies:
- AWSBackupServiceRolePolicyForBackup
- AWSBackupServiceRolePolicyForRestores
- If S3 is involved: AWSBackupServiceRolePolicyForS3Backup
Choose the right KMS key per backup vault and confirm encryption behavior for each storage type.
Prefer backup plans (scheduled, policy-driven) over ad-hoc jobs.
Test restores often (including restoring into a new cluster).
Continue GitOps for config management—backups are your safety net, not a replacement.

Quick-start checklist:

Opt in to EKS in AWS Backup Settings.
Attach the correct IAM policies to your backup/restore role.
Confirm EKS auth mode is API or API_AND_CONFIG_MAP.
Create an on-demand backup to validate end-to-end.
Set a backup plan (e.g., every 6 hours, retain 30 days).
[✔️] Test a full-cluster restore and a namespace-only restore.
[✔️] Review costs and lifecycle/cold storage options.

Costs

Expect charges for snapshots, storage, cross-Region/account copies, and retention. Persistent data size (EBS/EFS/S3) and frequency drive most costs.

Final thoughts

This feature turns EKS backup from homegrown scripts into managed reliability. With composite recovery points, clear policies, and flexible restore targets, teams get predictable protection with less operational effort. For production EKS, it’s a practical way to reduce risk during upgrades, incidents, and day‑to‑day changes.

References ©️

AWS Backup: https://docs.aws.amazon.com/aws-backup/latest/devguide/whatisbackup.html
Amazon EKS: https://docs.aws.amazon.com/eks/latest/userguide/what-is-eks.html
EKS + AWS Backup: https://docs.aws.amazon.com/eks/latest/userguide/integration-backup.html
AWS Blog: https://aws.amazon.com/blogs/aws/secure-eks-clusters-with-the-new-support-for-amazon-eks-in-aws-backup/
Shittu Sulaimon (Barry)'s blog: https://dev.to/sadebare/eks-disaster-recovery-simplified-native-backups-with-aws-backup-15g4

Thank You 🖤

Until next time, keep innovating and securing your cloud journey!

💡 Thank you for reading!

Until next time, つづく 🎉

🙌🏻😁📃 see you in the next blog.🤘 Until next time 🎉

🚀 Thank you for sticking up till the end. If you have any questions/feedback regarding this blog feel free to connect with me:

♻️ LinkedIn: https://www.linkedin.com/in/rajhi-saif/

♻️ X/Twitter: https://x.com/rajhisaifeddine

The end ✌🏻

🔰 Keep Learning !! Keep Sharing !! 🔰

📅 Stay updated

Subscribe to our newsletter for more insights on AWS cloud computing and containers.

Introducing the AWS EKS best practices Mindmap

saifeddine Rajhi — Sun, 05 Oct 2025 19:12:02 +0000

A Visual guide to secure, scalable, and cost-optimized Kubernetes on AWS

Managing Kubernetes on AWS can be complex, but with the right practices, it becomes an excellent platform to run and deploy modern applications. Inspired by William Quiles’ renowned security mindmaps, I’ve created an AWS EKS Best Practices Mindmap that brings together the most useful strategies for security, scalability, networking, and cost optimization—all in one visual guide.

This mindmap is designed to help engineers, architects, and platform teams simplify EKS operations while maintaining security, reliability, and efficiency.

AWS EKS best practices mindmap overview

The mindmap covers the following key areas:

1. Security & compliance

Protecting your EKS clusters is a must. This section focuses on:

Risk assessment & mitigation: Identify and reduce risks without compromising business value.
IAM & RBAC best practices: Control access at every level.
Data protection: Encrypt data in transit and at rest.
Continuous monitoring: Use AWS-native tools like GuardDuty and Security Hub.

2. Reliability

Reliability ensures your applications run smoothly. Best practices include:

Applications: Design for resilience and fault tolerance.
Control plane: Leverage AWS-managed control plane for stability.
Data plane: Keep worker nodes updated and healthy.

3. Cluster autoscaling

Scaling is key to cost and performance optimization. The mindmap highlights three approaches:

Karpenter:

An open-source tool that automates node provisioning and deprovisioning based on pod requirements. It supports flexible instance types and advanced scheduling constraints.
Cluster Autoscaler:

A Kubernetes-native solution that adjusts node counts by monitoring unschedulable pods and underutilized nodes. Integrated with AWS Auto Scaling Groups.
EKS Auto mode:

A fully managed scaling solution by AWS. It provisions nodes automatically using Bottlerocket AMIs and integrates load balancer controllers, pod identity, and security features—reducing operational overhead.

4. Networking

Understanding Kubernetes networking is essential for cluster efficiency:

Pod networking: EKS uses the VPC CNI plugin for native AWS VPC integration.
Underlay mode: Pods and nodes share the same network layer for consistent IP addressing.
Configurable options: VPC CNI supports multiple operating modes and advanced configurations for scalability and security.

5. Scalability

Maximize the work a single EKS cluster can handle:

Use one large cluster for reduced operational overhead (with trade-offs for multi-region and isolation).
Optimize resource allocation and scheduling for high performance.

6. Cluster upgrades

Keep your clusters up-to-date:

Control Plane: AWS manages upgrades, but you initiate them.
Data Plane: Upgrade worker nodes (self-managed, managed node groups, Karpenter, or Fargate) to match the control plane version.

7. Cost optimization

Achieve business goals at the lowest cost:

Right-sizing: Monitor CPU and memory usage with CloudWatch Container Insights.
Flexible purchasing: Use On-Demand, Spot, and Savings Plans.
Continuous optimization: Regularly review workloads for efficiency.

8. Specialized workloads

Windows containers: Secure and optimize Windows-based workloads.
Hybrid deployments: Extend EKS to on-premises or edge environments.
AI/ML workloads: Ensure performance and cost-efficiency for ML pipelines.

Practical benefits

Enhanced security: Strong IAM, encryption, and monitoring.
Operational efficiency: Automation reduces manual effort.
Cost savings: Intelligent scaling and resource optimization.

Conclusion

The AWS EKS Best Practices Mindmap is your go-to visual guide for building secure, scalable, and cost-effective Kubernetes environments on AWS.

Thank You 🖤

Until next time, keep innovating and securing your cloud journey!

💡 Thank you for reading!

Until next time, つづく 🎉

🙌🏻😁📃 see you in the next blog.🤘 Until next time 🎉

🚀 Thank you for sticking up till the end. If you have any questions/feedback regarding this blog feel free to connect with me:

♻️ LinkedIn: https://www.linkedin.com/in/rajhi-saif/

♻️ X/Twitter: https://x.com/rajhisaifeddine

The end ✌🏻

🔰 Keep Learning !! Keep Sharing !! 🔰

📅 Stay updated

Subscribe to our newsletter for more insights on AWS cloud computing and containers.

EKS'pert Automation: Amazon EKS Auto Mode and Karpenter in action

saifeddine Rajhi — Mon, 20 Jan 2025 08:34:48 +0000

Automate your Amazon EKS journey with Auto Mode and Karpenter in Action

🚀 Introduction

In the quest for seamless Kubernetes management, automation is key.

This blog explores how Karpenter and EKS Auto Mode transform Amazon Kubernetes cluster management.

By overcoming the limitations of traditional scaling tools, optimizing resource usage, and automating infrastructure provisioning and updates, these tools ensure your clusters are efficient and cost-effective.

EKS Auto Mode further enhances deployment by managing compute, networking, and storage, dynamically scaling resources based on application needs, and maintaining security through automatic updates and health monitoring.

Introduction to Amazon EKS and Karpenter

Amazon Elastic Kubernetes Service EKS simplifies the deployment and management of Kubernetes (k8s) clusters.

Kubernetes is a well-known container orchestration platform, but managing its control plane can be complex and resource-intensive.

EKS provides a managed control plane, ensuring scalability, high availability, and security.

Scaling Applications with Traditional Tools

Traditional tools like Cluster Autoscaler have several challenges:

Node Groups Management: Requires creating and managing multiple node groups.
Instance Family Specification: Users must specify instance families and their priorities.
Autoscaling Groups: Each node group needs a separate autoscaling group, complicating management.
AMI Selection and Refresh: Managing AMI (Amazon Machine Image) selection and updates can be cumbersome.

Karpenter: A Modern Solution for Scaling

Karpenter is a Kubernetes-native autoscaler designed to overcome these limitations. It interacts directly with EC2 Fleet, bypassing the need for node groups and autoscaling groups, resulting in faster and more efficient instance provisioning. Karpenter uses YAML files (NodePool and EC2NodeClass) to manage its behavior, offering flexibility and control.

Key Insights

🔍 What is Karpenter?
Karpenter automatically adds or removes servers (nodes) in a Kubernetes cluster, adjusting resources dynamically to match workloads.
⚖️ Karpenter vs. Cluster Autoscaler
Karpenter communicates directly with cloud providers (like AWS EC2) and can add new resources in seconds instead of minutes.
📊 Choosing the Right Resources Automatically
Karpenter selects the best type of server based on application needs (CPU, memory, etc.), simplifying setup and saving time.
🔄 Save Money with Resource Consolidation
Karpenter combines workloads on fewer servers, moving workloads around and shutting down idle servers to reduce costs.
⚙️ Works with Kubernetes Scheduling
Karpenter integrates seamlessly with Kubernetes, respecting all scheduling rules and supporting specialized applications.
💡 Supports All Types of Workloads
Karpenter handles high-performance servers, GPUs, specific configurations, and works across different environments, including multi-cloud setups.
📚 Easy to Learn and Use
This guide covers everything from understanding scaling to implementing Karpenter in real-world scenarios.

Key Features of Karpenter

NodePool YAML: Specify instance families, availability zones, architecture types (x86 or ARM), and capacity types (spot and on-demand).
Scheduling Constraints: Supports node selectors, node affinity, taints and tolerations, and topology spread.
User-Defined Labels: Enables custom labels, taints, and annotations.
Cost Optimization: Automatically consolidates underutilized nodes.
Disruption Budget: Controls when nodes can be disrupted, with configurable policies.
Drift Management: Ensures the desired state matches the current state by reconciling node configurations.
EC2NodeClass: Manages AMI selection, subnets, security groups, and other configurations.

Advanced AMI Management

Karpenter supports using Amazon EKS optimized AMIs, including BottleRocket and Amazon Linux 2. Users can pin worker nodes to specific AMI versions for testing before rolling out updates. Custom AMIs are also supported, allowing selection by tag, name, owner, or ID. If multiple AMIs match the criteria, Karpenter uses the latest one.

New Features in Karpenter v1

Enhanced Disruption Controls by Reason
Specify disruption budgets based on reasons like Underutilized, Empty, or Drifted.
Forceful Disruption Mode
Balance application availability against security requirements with assertive node lifecycle management.
Expanded consolidateAfter Functionality
Define how long Karpenter should wait before consolidating underutilized nodes.

Challenges and Solutions Beyond Karpenter

While Karpenter addresses many scaling challenges, managing core cluster capabilities like networking, service discovery, and load balancing remains complex. Upgrading and securing Kubernetes clusters also require dedicated expertise and continuous time investment.

Introduction to Amazon EKS Auto Mode

Amazon EKS Auto Mode offloads more operational responsibilities to AWS, extending AWS's responsibility from the control plane to the data plane, including compute, networking, and storage.

EKS Auto Mode dynamically scales compute resources based on application needs, optimizes costs through dynamic scaling and capacity planning, and automatically updates worker nodes and applies security fixes.

Benefits of EKS Auto Mode

Simplified Deployment: Reduces the steps needed to deploy applications by automating infrastructure provisioning and management.
Managed Core Capabilities: Provides managed compute, network, and storage capabilities out of the box.
Health Monitoring and Auto Repair: Includes health monitoring and auto-repair features.
Networking Enhancements: Simplifies networking with managed VPC CNI, network policies, and in-cluster service load balancing.
Shared Responsibility Model: Shifts more responsibility to AWS, allowing customers to focus on application innovation.

Getting Started with EKS Auto Mode

EKS Auto Mode simplifies cluster creation with a single-click setup and built-in best practices.

It provides two default node pools (general purpose and system) supporting a mix of instance types and architectures (Graviton and x86). Users can also define custom node pools for specific use cases like GPU instances, spot instances, and tenant isolation.

Key Features of EKS Auto Mode:

General Purpose Node Pool: Supports a mix of on-demand instance types, consolidation enabled by default, and a default node expiry of 14 days.
System Node Pool: Dedicated for critical add-ons, supports both AMD and ARM architectures, and includes a special taint to prevent scheduling regular workloads.
User-Defined Node Pools: Allows customization for specific use cases, such as GPU instances, spot instances, and tenant isolation.
EKSNodeClass: Manages security groups and subnets without requiring AMI selectors, as AWS handles the lifecycle of EC2 instances.

Day Two Operations and Updates

EKS Auto Mode handles updates for core capabilities and data plane nodes, respecting disruption budgets and ensuring minimal impact on running applications. Security updates are automatically applied, and worker nodes are updated in a rolling deployment fashion.

Automatic Updates:

Core Capabilities: Automatically updates core capabilities like Karpenter, storage controllers, and network controllers.
Data Plane Nodes: Worker nodes are updated to the latest AMI versions, respecting disruption budgets.
Security Updates: Security patches are applied automatically, ensuring a secure environment.

Conclusion

Amazon EKS, Karpenter, and EKS Auto Mode collectively enhance Kubernetes cluster management by simplifying scaling, optimizing costs, and automating infrastructure provisioning and updates. These tools provide robust solutions for modern cloud-based applications, allowing users to focus on innovation while AWS handles the operational complexities.

This comprehensive approach ensures efficient, secure, and cost-effective Kubernetes cluster management, making it easier for users to deploy and manage their applications at scale.

Thank You 🖤

Until next time, つづく 🎉

💡 Thank you for Reading !! 🙌🏻😁📃, see you in the next blog.🤘 Until next time 🎉

🚀 Thank you for sticking up till the end. If you have any questions/feedback regarding this blog feel free to connect with me:

♻️ LinkedIn: https://www.linkedin.com/in/rajhi-saif/

♻️ X/Twitter: https://x.com/rajhisaifeddine

The end ✌🏻

🔰 Keep Learning !! Keep Sharing !! 🔰

📅 Stay updated

Subscribe to our newsletter for more insights on AWS cloud computing and containers.

Node Health Monitoring and Auto-Repair for Amazon EKS

saifeddine Rajhi — Mon, 23 Dec 2024 07:21:25 +0000

⚡ Introduction

Amazon Elastic Kubernetes Service (Amazon EKS) continues to innovate beyond re:Invent with the introduction of Node Health Monitoring and Auto-Repair.

This groundbreaking feature is designed to automatically monitor and repair EC2 instances (nodes) in your Kubernetes clusters, ensuring your infrastructure remains at peak performance.

This is particularly beneficial for workloads using accelerated instances for machine learning.

Overview

Managing the health of nodes in a Kubernetes cluster can be a daunting task, often requiring significant operational effort to ensure resilience and performance. Amazon EKS now simplifies this process with Node Health Monitoring and Auto-Repair, which continuously monitors the health of nodes and takes automatic corrective actions when issues are detected.

Key Features

Node Monitoring Agent

The Node Monitoring Agent is a powerful tool that continuously reads node logs to detect a variety of health issues specific to Kubernetes environments. This agent identifies problems such as storage and networking issues and applies dedicated NodeConditions to worker nodes. These conditions are then surfaced in the observability dashboard, providing detailed insights into the health of your nodes.

Automatic Detection: The agent parses node logs to detect failures and surface status information.
NodeConditions: Specific conditions are applied for detected issues, such as KernelReady, NetworkingReady, and StorageReady.
Observability Dashboard: Detailed descriptions of detected health issues are available for monitoring.

Node Auto Repair

Node Auto Repair works in tandem with the Node Monitoring Agent to ensure that any detected issues are promptly addressed, thereby maintaining the overall health and availability of your cluster.

Continuous Monitoring: The feature continuously monitors node health and reacts to detected problems.
Automatic Replacement: Nodes with issues are automatically cordoned and replaced to maintain cluster health.
Enhanced Reliability: Addresses intermittent node issues, such as unresponsive kubelets and increased device errors.

How It Works

When the Node Monitoring Agent detects a health issue, it applies a specific NodeCondition to the affected node. If Node Auto Repair is enabled, it will automatically take corrective actions based on the type of issue detected. For example, if a node is experiencing networking issues, it may be cordoned and replaced to prevent disruption to your workloads.

Enabling Node Health Monitoring and Auto-Repair

To enable these features, you need to install the EKS node monitoring agent add-on and enable node auto-repair in the EKS managed node group APIs or AWS Console. EKS Auto Mode comes with both features enabled by default.

AWS Console: Activate the "Enable node auto repair" checkbox for the managed node group.
AWS CLI: Use the --node-repair-config enabled=true option with the eks create nodegroup or eks update-nodegroup-config command.

Availability

Node Health Monitoring and Auto-Repair is available at no additional cost in all AWS Regions, except AWS GovCloud (US) and China Regions. This ensures that you can leverage these powerful features to maintain the health of your Kubernetes clusters without incurring extra costs.

Benefits

Improved Cluster Availability

By automatically detecting and addressing node health issues, these features help ensure that your Kubernetes clusters remain highly available. This is particularly important for applications that require high availability and minimal downtime.

Reduced Operational Overhead

Managing node health manually can be time-consuming and complex. With Node Health Monitoring and Auto-Repair, much of this work is automated, freeing up your team to focus on other critical tasks.

Enhanced Performance for Machine Learning Workloads

For workloads that use accelerated instances, such as those for machine learning, maintaining node health is crucial. These features help ensure that your nodes are always in optimal condition, which is essential for performance-intensive applications.

Detailed Node Health Issues

The Node Monitoring Agent can detect a wide range of health issues, categorized into conditions and events. Conditions are terminal issues that warrant remediation actions like instance replacement or reboot, while events are temporary issues or sub-optimal configurations that do not trigger auto-repair actions.

Kernel Node Health Issues

ForkFailedOutOfPID: A fork or exec call has failed due to the system being out of process IDs or memory.
AppBlocked: A task has been blocked for a long period, usually due to input/output blocking.
AppCrash: An application on the node has crashed.
ApproachingKernelPidMax: The number of processes is nearing the maximum allowed by the kernel.
ConntrackExceededKernel: Connection tracking exceeded the maximum for the kernel, leading to packet loss.

Networking Node Health Issues

InterfaceNotRunning: The network interface is not running or there are network issues.
IPAMDNotReady: IPAMD fails to connect to the API server.
BandwidthInExceeded: Inbound bandwidth exceeded the maximum for the instance, causing packet loss.
ConntrackExceeded: Connection tracking exceeded the maximum for the instance, leading to packet loss.

NVIDIA Node Health Issues

NvidiaDoubleBitError: A double bit error was produced by the GPU driver, requiring node replacement.
NvidiaXID13Error: A graphics engine exception, requiring node reboot.
NvidiaXID79Error: The GPU driver found the GPU inaccessible over PCI Express, requiring node replacement.

Conclusion

The introduction of Node Health Monitoring and Auto-Repair for Amazon EKS is a significant advancement in maintaining the health and performance of your Kubernetes clusters. By automating the detection and remediation of node health issues, these features help ensure that your clusters remain resilient and operational with minimal manual intervention.

For more information, visit the Amazon EKS documentation.

💡 Thank you for Reading !! 🙌🏻😁📃, see you in the next blog.🤘 Until next time 🎉

🚀 Thank you for sticking up till the end. If you have any questions/feedback regarding this blog feel free to connect with me:

♻️ LinkedIn: https://www.linkedin.com/in/rajhi-saif/

♻️ X/Twitter: https://x.com/rajhisaifeddine

The end ✌🏻

🔰 Keep Learning !! Keep Sharing !! 🔰

📻🧡 Resources

📅 Stay updated

Subscribe to our newsletter for more insights on AWS cloud computing and containers.

AWS Skill Builder: Build Cloud Skills with Amazon Web Services

saifeddine Rajhi — Mon, 16 Dec 2024 07:39:37 +0000

Overview of AWS Skill Builder

AWS Skill Builder is an online learning center designed to help individuals and organizations build in-demand cloud skills. It offers a wide range of training resources, including over 600 free courses, certification exam preparation, hands-on labs, and more.

Key Features

Free Learning Content: AWS Skill Builder provides access to over 600 free courses and learning plans. These cover various topics, from basic cloud concepts to advanced AWS services.
AWS Exam Prep: The platform offers comprehensive exam preparation resources, including official question sets, practice exams, and pretests. This helps learners gain confidence and be well-prepared for AWS certification exams.
AWS Builder Labs: These labs provide guided instructions to help users develop practical skills in a sandbox environment. This allows learners to practice building without the risk of incurring unexpected costs.
AWS Digital Classroom: Available with the annual subscription, this feature includes expert-led videos, hands-on labs, knowledge checks, and course assessments. It offers a structured learning experience similar to traditional classroom training.
Game-Based Learning: AWS Skill Builder includes interactive learning experiences like AWS Cloud Quest and AWS Jam. These game-based simulations and challenges make learning engaging and fun.
Community and Networking: Users can connect with a large community of AWS professionals to share knowledge, ask questions, and network. This community aspect enhances the learning experience by providing support and collaboration opportunities.

Subscription Options

AWS Skill Builder offers several subscription plans to cater to different needs:

Free Account: Access to 600+ on-demand courses, standard exam prep courses, and digital badges.
Individual Monthly Subscription: Priced at $29 per month, it includes enhanced exam prep courses, hands-on labs, and more.
Individual Annual Subscription: Priced at $299 per year, it offers everything in the monthly subscription plus access to the AWS Digital Classroom.
Team Subscription: Designed for organizations, this plan costs $449 per learner per year and includes additional features like administrator dashboards, comprehensive reporting, and SSO support.

Specialized Learning Paths

AWS Skill Builder offers tailored learning paths for different roles and industries. These paths help learners focus on the skills most relevant to their career goals.

For example, there are specific tracks for cloud practitioners, solutions architects, developers, DevOps, serverless, containers and more.

Practical Experience

The platform emphasizes hands-on learning through its extensive lab offerings. With over 1,000 lab experiences, learners can apply their knowledge in real-world scenarios. This practical approach ensures that users not only understand theoretical concepts but also know how to implement them.

Continuous Updates

AWS Skill Builder is regularly updated to reflect the latest AWS services and best practices. This ensures that learners are always getting the most current and relevant information.

Additional Features

Login to skills Builder: Users can log in using their Builder ID, AWS Partner Network credentials, Organization SSO, or Amazon employee single sign-on, making access seamless and secure.

Generative AI-Powered Simulations: The platform includes AI-powered simulations that provide a dynamic and interactive learning experience, helping users understand complex concepts through practical application.
Instructor-Driven Digital Courses: These courses offer a blend of video instruction, hands-on labs, and assessments, providing a comprehensive learning experience.
Jam Journey Challenges: These challenges present real-world, open-ended problems in an AWS Console environment, enhancing problem-solving skills and preparing users for professional scenarios.
Ramp-Up Guides: Downloadable guides are available for further study, allowing users to delve deeper into specific roles or solutions within the AWS ecosystem.
Activities Board: This feature helps users track their course progress, completions, and test scores, providing insights into their learning journey and keeping them motivated.

Conclusion

AWS Skill Builder is an excellent training platform that caters to a wide range of learning needs. Whether you're new to the cloud or an experienced professional, it offers the resources and support needed to build and validate your AWS skills. The combination of free and paid content, practical labs, and community support makes it a valuable tool for anyone looking to advance their career in cloud computing.

If you're looking for alternatives, DataScientest offers AWS training courses that teach you how to use cloud services to make your infrastructure scalable, identify migration procedures, and design solutions integrating security, authentication, and access functions with AWS. Their training is delivered entirely remotely, and they are recognized by the French government, making their courses eligible for funding under the Compte Personnel de Formation scheme.

💡 Thank you for Reading !! 🙌🏻😁📃, see you in the next blog.🤘 Until next time 🎉

🚀 Thank you for sticking up till the end. If you have any questions/feedback regarding this blog feel free to connect with me:

♻️ LinkedIn: https://www.linkedin.com/in/rajhi-saif/

♻️ X/Twitter: https://x.com/rajhisaifeddine

The end ✌🏻

🔰 Keep Learning !! Keep Sharing !! 🔰

📻🧡 Resources

📅 Stay updated

Subscribe to our newsletter for more insights on AWS cloud computing and containers.

AWS re:Invent 2024: Key Announcements on Containers and Serverless

saifeddine Rajhi — Mon, 09 Dec 2024 07:05:43 +0000

AWS re:Invent 2024: Key Announcements on Containers and Serverless

🔖 Introduction

During re:Invent 2024, AWS made several significant announcements regarding their container and serverless offerings. Here's what you need to know.

1. Amazon EKS Auto Mode

AWS introduced Amazon Elastic Kubernetes Service (Amazon EKS) Auto Mode, a new capability designed to streamline Kubernetes cluster management. With EKS Auto Mode, you can manage compute, storage, and networking from provisioning to ongoing maintenance with a single click. This feature enhances agility, performance, and cost-efficiency by automating cluster management tasks, allowing you to focus on building innovative applications instead of managing infrastructure.

2. Amazon EKS Hybrid Nodes

Another major announcement was the general availability of Amazon EKS Hybrid Nodes. This feature allows you to attach your on-premises and edge infrastructure as nodes to EKS clusters in the cloud. By unifying Kubernetes management across cloud and on-premises environments, EKS Hybrid Nodes enable you to leverage the scalability and availability of Amazon EKS while using your existing hardware. This integration supports consistent operational practices and tooling across your environments.

3. Amazon Aurora Serverless v2

AWS also announced that Amazon Aurora Serverless v2 now supports scaling to zero capacity. This feature allows databases to scale down to zero, with resume times typically around 15 seconds. It's particularly useful for low-usage databases, such as those in development or user acceptance testing (UAT) environments, offering significant cost savings for small and medium-sized businesses (SMBs).

🔚 Conclusion

These announcements underscore AWS's commitment to providing flexible, efficient, and cost-effective solutions for managing container and serverless environments.

Until next time, つづく 🎉

💡 Thank you for Reading !! 🙌🏻😁📃, see you in the next blog.🤘 Until next time 🎉

🚀 Thank you for sticking up till the end. If you have any questions/feedback regarding this blog feel free to connect with me:

♻️ LinkedIn: https://www.linkedin.com/in/rajhi-saif/

♻️ X/Twitter: https://x.com/rajhisaifeddine

The end ✌🏻

🔰 Keep Learning !! Keep Sharing !! 🔰

📅 Stay updated

Subscribe to our newsletter for more insights on AWS cloud computing and containers.

Enhanced Observability for Amazon EKS with CloudWatch Container Insights

saifeddine Rajhi — Mon, 02 Dec 2024 11:19:58 +0000

CloudWatch Boosts EKS Observability

Introduction

AWS recently announced that Amazon CloudWatch Container Insights now offers enhanced observability for Amazon EKS on EC2, providing deeper insights into your Kubernetes clusters. This update includes additional telemetry from Kubernetes control plane components and detailed health and performance metrics at the container level.

Amazon CloudWatch Container Insights is a fully managed monitoring and observability service that provides engineers with out-of-the-box visibility into their containerized applications and microservice environments.

With Amazon CloudWatch Container Insights, you can monitor, isolate, and diagnose issues in your Kubernetes clusters with minimal effort. It delivers infrastructure telemetry like CPU, memory, network, and disk usage for your clusters, services, and pods in the form of metrics and logs that can be easily visualized in the CloudWatch console.

🔍 Key Features

Comprehensive Metrics and Logs

With enhanced observability, you can monitor CPU, memory, network, and disk usage for your clusters, services, and pods. The new features also include detailed metrics from the Kubernetes API server and etcd, helping you isolate and diagnose issues more effectively. Customers also have the capability to add CloudWatch alarms to get notified of anomalies for proactive actions.

Kube-State Metrics

With Kube-State metrics, you get a complete view of the core components and overall health of your Kubernetes clusters. You can monitor the real-time state and quickly identify any issues or bottlenecks. Detailed container-level metrics allow you to visually navigate through different layers of your cluster, making it easier to spot problems like memory leaks in individual containers. This helps you resolve issues faster and more efficiently.

Proactive Risk Management

Identify risks and take proactive actions even without pre-set alarms. Set alarms on unmonitored components or allocate more resources to mitigate risks preemptively, ensuring a smooth end-user experience. The enhanced observability feature facilitates early risk identification and proactive mitigation without relying on customer actions, helping prevent issues that could negatively impact the end-user experience.

🛠️ How to Enable Enhanced Observability

To enable enhanced observability for your Amazon EKS cluster, follow these steps:

Set Up IAM Permissions: Ensure your worker nodes have the necessary IAM permissions by attaching the CloudWatchAgentServerPolicy to your IAM role:

aws iam attach-role-policy --role-name my-worker-node-role --policy-arn arn:aws:iam::aws:policy/CloudWatchAgentServerPolicy

Install the EKS Add-on: The Amazon EKS add-on simplifies the process of enabling enhanced observability. It installs the CloudWatch agent and Fluent Bit, providing infrastructure and container log insights. Run the following command, replacing my-cluster-name with the name of your cluster:

aws eks create-addon --cluster-name my-cluster-name --addon-name amazon-cloudwatch-observability

View Enhanced Metrics and Logs: Access the CloudWatch console to start seeing enhanced metrics and logs for your EKS clusters.

Source

📊 Enhanced Dashboards

The enhanced Container Insights dashboards provide various views to analyze performance, including:

Cluster-wide Performance Dashboard: Overview of resource utilization across the entire cluster.
Node Performance View: Metrics at the individual node level.
Pod Performance View: Focus on pod-level metrics for CPU, memory, network, etc.
Container Performance View: Drill down into utilization metrics for individual containers.

These dashboards allow you to quickly identify and address performance issues at different levels of your Kubernetes environment.

🌟 Conclusion

With the new enhanced observability features in Amazon CloudWatch Container Insights, you can monitor and troubleshoot your Amazon EKS clusters more effectively. Enable these features today to gain deeper insights and ensure optimal performance for your Kubernetes workloads.

References

https://aws.amazon.com/blogs/mt/new-container-insights-with-enhanced-observability-for-amazon-eks/

Until next time, つづく 🎉

💡 Thank you for Reading !! 🙌🏻😁📃, see you in the next blog.🤘 Until next time 🎉

🚀 Thank you for sticking up till the end. If you have any questions/feedback regarding this blog feel free to connect with me:

♻️ LinkedIn: https://www.linkedin.com/in/rajhi-saif/

♻️ X/Twitter: https://x.com/rajhisaifeddine

The end ✌🏻

🔰 Keep Learning !! Keep Sharing !! 🔰

📅 Stay updated

Subscribe to our newsletter for more insights on AWS cloud computing and containers.

Simplify Authorization Management with Cedar by AWS

saifeddine Rajhi — Mon, 25 Nov 2024 11:59:41 +0000

Simplify Authorization Management with Cedar by AWS

⚡ Introduction

AWS recently introduced Cedar that simplifies managing authorization policies. Cedar allows you to define and manage complex authorization rules as reusable components, making your life easier and your applications more secure.

What is Cedar?

Cedar is an open-source project by AWS that helps you create and manage fine-grained authorization policies. By defining these policies separately from your application code, you can update, analyze, and audit them independently.

How does Cedar work?

Cedar uses a simple yet expressive language to define authorization policies. When you apply a policy, Cedar:

Validates the policy against a schema.
Evaluates access requests in real-time.
Provides detailed analysis and optimization tools.

Cedar Policies

A Cedar policy specifies who can do what with which resources. It supports common authorization models like Role-Based Access Control (RBAC) and Attribute-Based Access Control (ABAC).

Example Cedar Policy

Here's an example of a Cedar policy that allows a user named "alice" to view photos in the "jane_vacation" album:

permit (
  principal == User::"alice",
  action == Action::"view",
  resource in Album::"jane_vacation"
);

Example Entities

Cedar represents principals, resources, and actions as entities. Here's an example of entities in JSON format:

[
    {
        "uid": { "type": "User", "id": "alice"} ,
        "attrs": {"age": 18},
        "parents": []
    },
    {
        "uid": { "type": "Photo", "id": "VacationPhoto94.jpg"},
        "attrs": {},
        "parents": [{ "type": "Album", "id": "jane_vacation" }]
    }
]

Getting Started with Cedar

Installation

To use Cedar in your application, add the cedar-policy crate as a dependency:

cargo add cedar-policy

Testing Your Policy

You can test your policy using the Cedar CLI:

cargo run authorize \
    --policies policy.cedar \
    --entities entities.json \
    --principal 'User::"alice"' \
    --action 'Action::"view"' \
    --resource 'Photo::"VacationPhoto94.jpg"'

CLI output:

ALLOW

This request is allowed because "VacationPhoto94.jpg" belongs to "jane_vacation", and "alice" can view photos in that album.

Cedar for Kubernetes

Cedar for Kubernetes allows users to enforce access control on Kubernetes API requests using Cedar policies. Users can dynamically create authorization policies for Kubernetes that support features like request or user attribute-based rules, label-based access controls, conditions, and enforce denial policies. Users can also create admission policies in the same file as authorization policy, giving policy authors a single language to write and reason about.

Cedar Access Controls for Kubernetes

AWS has announced a new open-source project, Cedar access controls for Kubernetes. This project brings the power of Cedar to Kubernetes authorization and admission validation, enabling cluster administrators to use a unified access control language for principals making API calls. With Cedar access controls for Kubernetes, administrators can dynamically create authorization policies that support features like request or user attribute-based rules, label-based access controls, conditions, and denial policies.

Background: Kubernetes Authorization and Admission

Kubernetes is an open-source system for automating deployment, scaling, and management of containerized applications. Central to Kubernetes is an API server that supports multiple extension points, and Cedar integrates into two of these steps: the authorization and validating admission phase. After a request is authenticated, Kubernetes performs authorization for nodes, a pluggable authorization to an external webhook (powered by Cedar in this case), and the built-in Role-Based Access Control (RBAC).

Authorization in Kubernetes is intended to be fast, so the body of the requested action is not serialized or evaluated: just the authentication information, the verb, and the contents of the URL are evaluated. In a Kubernetes request, the URL contains data like the API group, what resource type is being acted on ("pods", "secrets", "nodes", etc.), a possible namespace, or the specific resource's name. If a request passes authorization, it proceeds through several phases and then reaches the validating admission step. Admission validation allows administrators to allow or deny mutating requests (create/update/delete) with the requested object included, and Cedar policies can also be evaluated in this step.

Demonstration

Let's walk through an example of capabilities that Cedar enables. As a Kubernetes administrator, you want to allow some users to create and manage Kubernetes ConfigMaps to store configuration data, but you only want those users to view and modify ConfigMaps they own. Enforcing this kind of control is possible today in Kubernetes, but requires you to use RBAC and a validating admission webhook with a separate policy language.

First, we'll write an authorization policy permitting users to view and manage ConfigMaps.

permit (
    principal in k8s::Group::"requires-labels",
    action in [
        k8s::Action::"create",
        k8s::Action::"update",
        k8s::Action::"patch",
        k8s::Action::"delete",
        k8s::Action::"list",
        k8s::Action::"watch"],
    resource is k8s::Resource
) when {
    resource.apiGroup == "" && // "" is the core Kubernetes API group
    resource.resource == "configmaps"
};

This policy grants users in the group "requires-labels" the ability to create, update, patch, delete, list, and watch ConfigMap objects. So far, this is comparable to an RBAC policy in Kubernetes. Now, let's add a policy to prevent users from listing or watching for ConfigMaps they do not own.

forbid (
    principal is k8s::User in k8s::Group::"requires-labels",
    action in [k8s::Action::"list", k8s::Action::"watch"],
    resource is k8s::Resource
) unless {
    resource has labelSelector &&
    resource.labelSelector.contains({
        "key": "owner",
        "operator": "=",
        "values": [principal.name]
    })
};

Unlike RBAC, we can define explicit denials in authorization. This policy forbids users in the group "requires-labels" from making list or watch requests against any resource unless they include a label selector in their request. This policy leverages a new alpha-level feature gate AuthorizeWithSelectors in Kubernetes v1.31 that allows the Cedar authorizer to make decisions on field and label selectors included in a request. Now let's restrict what the users can create or update.

forbid (
    principal is k8s::User in k8s::Group::"requires-labels",
    action in [
        k8s::admission::Action::"create",
        k8s::admission::Action::"update",
        k8s::admission::Action::"delete"],
    resource
) unless {
    resource has metadata &&
    resource.metadata has labels &&
    resource.metadata.labels.contains({"key": "owner", "value": principal.name})
};

You may notice that the action in this policy has a different prefix than the authorization policies. Because Kubernetes does not include the content of the resource in authorization but does in admission validations, we use actions in separate Cedar namespaces for authorization and admission. Authorization actions apply to k8s::Resource types and admission actions apply to Kubernetes types. Finally, we need one more admission policy:

forbid (
    principal is k8s::User in k8s::Group::"requires-labels",
    action == k8s::admission::Action::"update",
    resource
) unless {
    resource has oldObject &&
    resource.oldObject has metadata &&
    resource.oldObject.metadata has labels &&
    resource.oldObject.metadata.labels.contains(
        {"key": "owner", "value": principal.name})
};

Similar to the first admission policy, we write a policy that applies to update operations. When a user makes an update request, the validating admission request not only includes the updated object, but also includes the object before it was modified. This enables us to prevent principals from overwriting the owner label on a resource that doesn't belong to the principal.

Now we can try this policy out! We have created a local Kubernetes cluster in a VM using Kind, and applied the above policy. We have created a kubeconfig for a user named "sample-user", and included them in the group "requires-labels" (the GitHub project has full setup instructions for you to try this out).

# set our KUBECONFIG file to a sample user in our Kind cluster
$ export KUBECONFIG=./mount/sample-user-kubeconfig.yaml
$ kubectl auth whoami
ATTRIBUTE   VALUE
Username    sample-user
Groups      [sample-group requires-labels system:authenticated]

Now, let's see what that user can do. First, we'll try to see what ConfigMaps exist, and create a new one.

$ kubectl get configmap
Error from server (Forbidden): configmaps is forbidden: \
  User "sample-user" cannot list resource "configmaps" in API group "" \
  in the namespace "default": \
  {"reasons":[{"policy":"label-enforcement-policy1","position":{
      "filename":"label-enforcement-policy","offset":671,"line":21,"column":1}}]}

$ kubectl create configmap sample-config --from-literal=k1=v1
error: failed to create configmap: \
    admission webhook "vpolicy.cedar.k8s.aws" denied the request: \
    [{"policy":"label-enforcement-policy2","position":{
         "filename":"label-enforcement-policy","offset":1226,"line":36,"column":1}}]

In the first request, our kubectl get request resulted in a list operation. This was permitted in the first policy, but denied in the second policy (the zero-indexed "label-enforcement-policy1") that forbids list/watch resource requests without a label selector. The second request was similarly permitted in the first authorization policy, but denied in admission policy "label-enforcement-policy2".

Next, let's try to make requests that use ownership labels.

$ kubectl get configmap --selector owner=sample-user --show-labels
No resources found in default namespace.

# Construct a configmap with the proper owner label
$ cat << EOF > sample-config.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: sample-config
  labels:
    owner: sample-user
data:
  stage: test
EOF

$ kubectl create -f ./sample-config.yaml
configmap/sample-config created

$ kubectl get configmap --selector owner=sample-user --show-labels
NAME            DATA   AGE   LABELS
sample-config   1      19s   owner=sample-user

And they all succeeded! When we made list or create requests using the owner labels, they were permitted because each forbid statement's unless clause was satisfied.

Additional Features

Beyond the ability to enforce Cedar-based permissions in Kubernetes, the project provides a few other features:

A Custom Resource Definition (CRD) for storing Cedar policies in a Kubernetes cluster.
A converter that works with any RBAC bindings and policies to rewrite them in Cedar. RBAC policies continue to work in clusters with Cedar enabled, they are only evaluated when no Cedar rules explicitly allow or forbid a request.
Support for authorizing Kubernetes impersonation on UIDs, usernames, and groups.
A Cedar schema generator for Kubernetes built-in types and CRDs, so policies can be validated before creation.

Limitations

This project is a public experiment for how Cedar policies can be used to enforce authorization in Kubernetes. We think this project holds a lot of possibilities for enabling more secure policies in Kubernetes, but there are limitations and tradeoffs to be aware of. One worth mentioning here is that Cedar doesn't work well with optional nested fields on sets of structures. While Cedar has operators that can test set membership or set overlap, it does not have a way to map a general operation on a set of objects. This is by design: It ensures that Cedar policies are efficiently and precisely analyzable, in the sense that they can be represented in formal logic for purposes of automating policy reasoning. As a result, Cedar is not able to enforce common Kubernetes policies like ensuring all container images on a pod are from a specific container registry, or that all containers have CPU and memory limits set. If this is an issue for you, Cedar is still a great upgrade for the authorization component, but other tooling like Open Policy Agent Gatekeeper or Kyverno may be a better fit for those policies. For a full list of features, limitations, and a walkthrough to try this out for yourself in a Kind cluster, see the GitHub repository. We can't wait to see what kind of policies you write!

Can I use Cedar for Kubernetes policy enforcement?

While Cedar offers powerful authorization guarantees, there are policy enforcement requirements common to Kubernetes that are not formally analyzable. For example, enforcing that all containers in all pods have a maximum memory limit set. Cedar is powered by automated reasoning, including an SMT solver, which does not implement loops or map functions. Rather than viewing Cedar as a replacement for tools like Open Policy Agent/Gatekeeper or Kyverno, it is best seen as an additional tool for access control enforcement.

Conclusion

Cedar by AWS offers a powerful and flexible way to manage authorization policies. By separating policy management from application code, you can ensure your security model is robust and adaptable. For more details, check out the Cedar documentation.

📻🧡 For more information, check out the Cedar documentation and explore the Cedar examples repository.

💡 Thank you for Reading !! 🙌🏻😁📃, see you in the next blog.🤘 Until next time 🎉

🚀 Thank you for sticking up till the end. If you have any questions/feedback regarding this blog feel free to connect with me:

♻️ LinkedIn: https://www.linkedin.com/in/rajhi-saif/

♻️ X/Twitter: https://x.com/rajhisaifeddine

The end ✌🏻

🔰 Keep Learning !! Keep Sharing !! 🔰

📅 Stay updated

Subscribe to our newsletter for more insights on AWS cloud computing and containers.

Simplify Kubernetes Resource Management with KRO by AWS

saifeddine Rajhi — Mon, 18 Nov 2024 12:27:30 +0000

⚡ Introduction

AWS recently introduced Kube Resource Orchestrator (kro) that simplifies managing Kubernetes resources. kro allows you to define and manage complex Kubernetes resources as reusable components, making your life easier and your deployments more efficient.

What is kro?

Kube Resource Orchestrator (kro) is an open-source project by AWS that helps you create and manage custom groups of Kubernetes resources. By defining these groups as ResourceGroups, you can deploy multiple resources together in a consistent and controlled way.

How does kro work?

kro uses core Kubernetes primitives to simplify resource grouping and dependency management. When you apply a ResourceGroup to your cluster, kro:

Verifies the ResourceGroup specification.
Creates a new Custom Resource Definition (CRD).
Deploys a dedicated controller to manage the lifecycle of the resources.

ResourceGroups

A ResourceGroup is the fundamental building block in kro. It allows you to define, organize, and manage sets of related Kubernetes resources as a single, reusable unit. A ResourceGroup acts as a blueprint, defining:

Schema: What users can configure.
Resources: What resources to create.
Dependencies: How resources reference each other.
Conditions: When resources should be included.
Status: What status to expose.

Anatomy of a ResourceGroup

A ResourceGroup consists of three main parts:

Metadata: Includes name, namespace, labels, etc.
Spec: Defines the structure and properties of the ResourceGroup.
Status: Reflects the current state of the ResourceGroup.

The spec section contains:

Schema: Defines the API structure, including fields users can configure and status information.
Resources: Specifies the Kubernetes resources to create, their templates, dependencies, conditions for inclusion, and readiness criteria.

Example ResourceGroup

Here's an example of a ResourceGroup that includes a Deployment, Service, and Ingress:

apiVersion: kro.run/v1alpha1
kind: ResourceGroup
metadata:
  name: my-application
spec:
  schema:
    apiVersion: v1alpha1
    kind: Application
    spec:
      name: string
      image: string | default="nginx"
      ingress:
        enabled: boolean | default=false
    status:
      deploymentConditions: ${deployment.status.conditions}
      availableReplicas: ${deployment.status.availableReplicas}
  resources:
    - id: deployment
      template:
        apiVersion: apps/v1
        kind: Deployment
        metadata:
          name: ${schema.spec.name}
        spec:
          replicas: 3
          selector:
            matchLabels:
              app: ${schema.spec.name}
          template:
            metadata:
              labels:
                app: ${schema.spec.name}
            spec:
              containers:
                - name: ${schema.spec.name}
                  image: ${schema.spec.image}
                  ports:
                    - containerPort: 80
    - id: service
      template:
        apiVersion: v1
        kind: Service
        metadata:
          name: ${schema.spec.name}-service
        spec:
          selector: ${deployment.spec.selector.matchLabels}
          ports:
            - protocol: TCP
              port: 80
              targetPort: 80
    - id: ingress
      includeWhen:
        - ${schema.spec.ingress.enabled}
      template:
        apiVersion: networking.k8s.io/v1
        kind: Ingress
        metadata:
          name: ${schema.spec.name}-ingress
          annotations:
            kubernetes.io/ingress.class: alb
            alb.ingress.kubernetes.io/scheme: internet-facing
            alb.ingress.kubernetes.io/target-type: ip
            alb.ingress.kubernetes.io/healthcheck-path: /health
            alb.ingress.kubernetes.io/listen-ports: '[{"HTTP": 80}]'
            alb.ingress.kubernetes.io/target-group-attributes: stickiness.enabled=true,stickiness.lb_cookie.duration_seconds=60
        spec:
          rules:
            - http:
                paths:
                  - path: "/"
                    pathType: Prefix
                    backend:
                      service:
                        name: ${service.metadata.name}
                        port:
                          number: 80

ResourceGroup Instances

Once kro processes your ResourceGroup, it creates a new API in your cluster. Users can then create instances of this API to deploy resources in a consistent, controlled way. An instance represents your deployed application and contains your configuration values, serving as the single source of truth for your application's desired state.

Example ResourceGroup Instance

Here's an example of how an instance for the SimpleWebApp might look:

apiVersion: kro.run/v1alpha1
kind: SimpleWebApp
metadata:
  name: my-web-app
spec:
  appName: awesome-app
  image: nginx:latest
  replicas: 3

Getting Started with kro

Installation

To install kro, you need Helm and kubectl configured for your Kubernetes cluster. Follow these steps:

Fetch the latest release version:

export KRO_VERSION=$(curl -sL https://api.github.com/repos/awslabs/kro/releases/latest | jq -r '.tag_name | ltrimstr("v")')

Install kro using Helm:

helm install kro oci://public.ecr.aws/kro/kro --namespace kro --create-namespace --version=${KRO_VERSION}

Verify the installation:

helm -n kro list
kubectl get pods -n kro

Creating Your First ResourceGroup

Define a ResourceGroup:
Save the following as resourcegroup.yaml:

apiVersion: kro.run/v1alpha1
kind: ResourceGroup
metadata:
  name: my-application
spec:
  schema:
    apiVersion: v1alpha1
    kind: Application
    spec:
      name: string
      image: string | default="nginx"
      ingress:
        enabled: boolean | default=false
    status:
      deploymentConditions: ${deployment.status.conditions}
      availableReplicas: ${deployment.status.availableReplicas}
  resources:
    - id: deployment
      template:
        apiVersion: apps/v1
        kind: Deployment
        metadata:
          name: ${schema.spec.name}
        spec:
          replicas: 3
          selector:
            matchLabels:
              app: ${schema.spec.name}
          template:
            metadata:
              labels:
                app: ${schema.spec.name}
            spec:
              containers:
                - name: ${schema.spec.name}
                  image: ${schema.spec.image}
                  ports:
                    - containerPort: 80
    - id: service
      template:
        apiVersion: v1
        kind: Service
        metadata:
          name: ${schema.spec.name}-service
        spec:
          selector: ${deployment.spec.selector.matchLabels}
          ports:
            - protocol: TCP
              port: 80
              targetPort: 80
    - id: ingress
      includeWhen:
        - ${schema.spec.ingress.enabled}
      template:
        apiVersion: networking.k8s.io/v1
        kind: Ingress
        metadata:
          name: ${schema.spec.name}-ingress
          annotations:
            kubernetes.io/ingress.class: alb
            alb.ingress.kubernetes.io/scheme: internet-facing
            alb.ingress.kubernetes.io/target-type: ip
            alb.ingress.kubernetes.io/healthcheck-path: /health
            alb.ingress.kubernetes.io/listen-ports: '[{"HTTP": 80}]'
            alb.ingress.kubernetes.io/target-group-attributes: stickiness.enabled=true,stickiness.lb_cookie.duration_seconds=60
        spec:
          rules:
            - http:
                paths:
                  - path: "/"
                    pathType: Prefix
                    backend:
                      service:
                        name: ${service.metadata.name}
                        port:
                          number: 80

Apply the ResourceGroup:
```
kubectl apply -f resourcegroup.yaml
```
Inspect the ResourceGroup:
```
kubectl get rg my-application -o wide
```

Creating an Application Instance

Define an Application instance:
Save the following as instance.yaml:

apiVersion: kro.run/v1alpha1
kind: Application
metadata:
  name: my-application-instance
spec:
  name: my-awesome-app
  ingress:
    enabled: true

Apply the Application instance:
```
kubectl apply -f instance.yaml
```
Inspect the Application instance:
```
kubectl get applications
```

Inspect the resources:

kubectl get deployments,services,ingresses

Deleting the Application Instance

To clean up resources, delete the Application instance:

kubectl delete application my-application-instance

🌟 Conclusion

kro by AWS simplifies Kubernetes resource management by allowing you to define and manage complex resource configurations as reusable components. Whether you're deploying simple applications or complex multi-service architectures, kro can help streamline your Kubernetes operations.

🌐 Community Participation

Development and discussion are coordinated in the Kubernetes Slack (invite link) channel #kro channel.

We welcome questions, suggestions, and contributions from the community! To get involved, check out our contributing guide. For bugs or feature requests, feel free to submit an issue. You’re also invited to join our community.

📚 Documentation

For more detailed information, check out the following resources:

Until next time, つづく 🎉

💡 Thank you for Reading !! 🙌🏻😁📃, see you in the next blog.🤘 Until next time 🎉

🚀 Thank you for sticking up till the end. If you have any questions/feedback regarding this blog feel free to connect with me:

♻️ LinkedIn: https://www.linkedin.com/in/rajhi-saif/

♻️ X/Twitter: https://x.com/rajhisaifeddine

The end ✌🏻

🔰 Keep Learning !! Keep Sharing !! 🔰

📅 Stay updated

Subscribe to our newsletter for more insights on AWS cloud computing and containers.

HardenEKS: The Easy Way to keep your AWS EKS clusters secure and compliant

saifeddine Rajhi — Mon, 11 Nov 2024 07:08:55 +0000

Guide to automating Amazon EKS best practices compliance

🚀 Introduction

Amazon Elastic Kubernetes Service (EKS) is a managed Kubernetes service that makes it easy to run Kubernetes on AWS. However, ensuring that your EKS clusters are secure and compliant with best practices can be a daunting task. That's where HardenEKS comes in.

HardenEKS is an open source Python CLI that helps you programmatically validate if your EKS clusters follow best practices defined in the AWS EKS Best Practices Guide EBPG. The EBPG covers security, reliability, autoscaling, networking, and scalability. HardenEKS has incorporated and codified the pillars of the EBPG into a set of rules.

HardenEKS is easy to use. Simply install it, run it against your EKS cluster, and it will generate a report of any violations. You can then use this report to fix the violations and ensure that your clusters are secure and compliant.

HardenEKS is a valuable tool for any EKS administrator who wants to keep their clusters secure and compliant. It's easy to use, it's open source, and it's backed by the AWS community.

This blog post is for anyone who wants to learn how to secure their Kubernetes clusters. 🕵‍♂️

Enjoy! 💪

📣 Introducing HardenEKS 📣

HardenEKS is a potent Python-based Command Line Interface (CLI), capable of systematically assessing whether Amazon Elastic Kubernetes Service (Amazon EKS) clusters comply with the strict guidelines laid out in the AWS EKS Best Practices Guide (EBPG). This comprehensive guide covers six fundamental pillars of best practices for Amazon EKS clusters:

Security
Reliability
Autoscaling
Networking
Scalability
Windows Containers

HardenEKS, a technical marvel in its own right, primarily focuses on these rules derived from the EBPG that can be executed automatically. With a robust set of over 40 carefully designed automated rules already available and more on the way, HardenEKS ensures that your Amazon EKS cluster's adherence to best practices is nothing short of outstanding.

What's impressive is that you don't need to install HardenEKS within the cluster you're validating. Instead, it conducts its validation of all rules from an external standpoint, ensuring a non-invasive and secure assessment of best practices. This accessibility, regardless of your level of experience, makes HardenEKS an indispensable tool for all.

Here's a sneak peek at what you can expect to learn from the article:

Getting and installing HardenEKS.
Running the validation against your operational Amazon EKS cluster.
Creating a comprehensive report with the results.
Exploring the structure and details of this generated report.
Diving into an illustrative best practice scenario, explaining how fixing the best practice issues brings significant benefits.

🕸️ Prerequisites

As a prerequisite for running the tool, you must already have access to a Kubernetes cluster configured through the kubeconfig file.

To run the tool on an EKS cluster, some minimum permissions must be met, both within AWS (IAM Policy) and within the EKS cluster (RBAC).

Below are the minimum required permissions for the IAM Policy:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": "eks:ListClusters",
            "Resource": "*"
        },
        {
            "Effect": "Allow",
            "Action": "eks:DescribeCluster",
            "Resource": "*"
        },
        {
            "Effect": "Allow",
            "Action": "ecr:DescribeRepositories",
            "Resource": "*"
        },
        {
            "Effect": "Allow",
            "Action": "inspector2:BatchGetAccountStatus",
            "Resource": "*"
        },
        {
            "Effect": "Allow",
            "Action": "ec2:DescribeFlowLogs",
            "Resource": "*"
        },
        {
            "Effect": "Allow",
            "Action": "ec2:DescribeInstances",
            "Resource": "*"
        }
    ]
}

For more details on configuring IAM policies, refer to the AWS IAM Documentation.

Below ClusterRole that must be created in the cluster:

kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: hardeneks-runner
rules:
- apiGroups: [""]
  resources: ["namespaces", "resourcequotas", "persistentvolumes", "pods", "services"]
  verbs: ["list"]
- apiGroups: ["rbac.authorization.k8s.io"]
  resources: ["clusterroles", "clusterrolebindings", "roles", "rolebindings"]
  verbs: ["list"]
- apiGroups: ["networking.k8s.io"]
  resources: ["networkpolicies"]
  verbs: ["list"]
- apiGroups: ["storage.k8s.io"]
  resources: ["storageclasses"]
  verbs: ["list"]
- apiGroups: ["apps"]
  resources: ["deployments", "daemonsets", "statefulsets"]
  verbs: ["list", "get"]
- apiGroups: ["autoscaling"]
  resources: ["horizontalpodautoscalers"]
  verbs: ["list"]

Hands-on

HardenEKS, I suppose it is a great Shift-Left Testing tool that can add value to your hardening practices.

Moreover, it can easily be implemented in your GitOps approach by adding it to the pipeline.

You can run HardenEKS with just a few commands:

python3 -m venv /tmp/.venv
source /tmp/.venv/bin/activate
pip install hardeneks
hardeneks

The previous command generates an HTML file that highlights the rules that have been violated. Below is a snapshot of the report that HardenEKS provides.

HardenEKS systematically checks each section of the EBPG against your cluster. If a rule is found to be violated, HardenEKS not only identifies the specific configuration that caused the violation, but it also provides relevant information about the corresponding EKS Best Practice.

Each result in the report is accompanied by a link that takes you to detailed information about the best practices related to that particular violation. These results can be used to help you assess and address any issues in your cluster that need attention.

For example, if the report shows that the "Pod Security Policy" rule has been violated, the link will take you to the documentation for the Pod Security Policy best practice. This documentation will provide you with information about how to configure your cluster to comply with this best practice.

The results in the HardenEKS report can be a valuable resource for helping you keep your cluster secure and compliant with best practices. By understanding the results of the report, you can identify and address any issues in your cluster that need to be fixed.

To use the tool, just run the command below:

hardeneks

Below are some prints of the tool's outputs:

At the end of each check that the tool performs, it provides a link to the best practices guide.

🛠️ Using HardenEKS for Cluster Configuration Validation and Drift Detection

HardenEKS isn't just a one-time validation tool; it's a dynamic solution that can continually monitor your cluster's configuration. Here's how it works:

✅ Baseline Configuration

Before and after making significant changes to your cluster, HardenEKS captures a snapshot of your cluster's configuration status. This baseline serves as a reference point.

✅ Continuous Monitoring

Once changes are implemented, you can create a new baseline to reflect the updated configuration. This process can be automated and scheduled at regular intervals. It ensures that your cluster remains aligned with best practices automatically.

✅ Detecting Configuration Drift

Comparing these baselines allows you to detect drift—any unintended variations between configurations. Identifying drift is crucial for maintaining the integrity of your Amazon EKS clusters and ensuring that unexpected changes do not compromise their stability.

HardenEKS facilitates this ongoing validation process and supports JSON output, which is highly beneficial. Here's an example of how to export data as JSON and use it for automated validation:

hardeneks --export-json hardeneks_report.json

HardenEKS facilitates this ongoing validation process and supports JSON output, which is highly beneficial. Here's an example of how to export data as JSON and use it for automated validation:

# write StorageClass.yaml with encryption parameter false
cat > StorageClass.yaml <<EOF
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: ebs-sc
provisioner: ebs.csi.aws.com
parameters:
  csi.storage.k8s.io/fstype: xfs
  type: io1
  iopsPerGB: "50"
  encrypted: "false"
EOF

kubectl apply -f StorageClass.yaml
hardeneks --export-json report.json
kubectl delete -f StorageClass.yaml

# write StorageClass.yaml with encryption parameter true
cat > StorageClass.yaml <<EOF
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: ebs-sc
provisioner: ebs.csi.aws.com
parameters:
  csi.storage.k8s.io/fstype: xfs
  type: io1
  iopsPerGB: "50"
  encrypted: "true"
EOF

kubectl apply -f StorageClass.yaml
hardeneks --export-json report2.json

# Shows the difference in the reports before and after the StorageClass change was made
cat report.json | jq --raw-output  '.cluster_wide.security.encryption_secrets."EBS Storage Classes should have encryption parameter.".status'
false
cat report2.json | jq --raw-output  '.cluster_wide.security.encryption_secrets."EBS Storage Classes should have encryption parameter.".status'
true

❇️ Advanced configuration customizablerRules

There is the possibility to inform the hardeneks configuration file with this it is possible to disable some checks of the tool.
Below is a configuration file where some checks can be disabled:

---
ignore-namespaces:
  - kube-node-lease
  - kube-public
  - kube-system
  - kube-apiserver
  - karpenter
  - kubecost
  - external-dns
  - argocd
  - aws-for-fluent-bit
  - amazon-cloudwatch
  - vpa
rules: 
  cluster_wide:
    security:
      iam:
        - disable_anonymous_access_for_cluster_roles
        - check_endpoint_public_access
        - check_aws_node_daemonset_service_account
        - check_access_to_instance_profile
        - restrict_wildcard_for_cluster_roles
      multi_tenancy:
        - ensure_namespace_quotas_exist
      detective_controls:
        - check_logs_are_enabled
      network_security:
        - check_vpc_flow_logs
        - check_awspca_exists
        - check_default_deny_policy_exists
      encryption_secrets:
        - use_encryption_with_ebs
        - use_encryption_with_efs
        - use_efs_access_points
      infrastructure_security:
        - deploy_workers_onto_private_subnets
        - make_sure_inspector_is_enabled
      pod_security:
        - ensure_namespace_psa_exist
      image_security:
        - use_immutable_tags_with_ecr
    reliability:
      applications:
        - check_metrics_server_is_running
        - check_vertical_pod_autoscaler_exists
  namespace_based:
    security: 
      iam:
        - disable_anonymous_access_for_roles
        - restrict_wildcard_for_roles
        - disable_service_account_token_mounts
        - disable_run_as_root_user
        - use_dedicated_service_accounts_for_each_deployment
        - use_dedicated_service_accounts_for_each_stateful_set
        - use_dedicated_service_accounts_for_each_daemon_set
      pod_security:
        - disallow_container_socket_mount
        - disallow_host_path_or_make_it_read_only
        - set_requests_limits_for_containers
        - disallow_privilege_escalation
        - check_read_only_root_file_system
      network_security:
        - use_encryption_with_aws_load_balancers
      encryption_secrets:
        - disallow_secrets_from_env_vars    
      runtime_security:
        - disallow_linux_capabilities
    reliability:
      applications:
        - check_horizontal_pod_autoscaling_exists
        - schedule_replicas_across_nodes
        - run_multiple_replicas
        - avoid_running_singleton_pods

🔧 Additional options

HardenEKS offers several additional options to tailor the checks to your specific needs:

Namespace-Specific Checks: Perform the check in a given namespace.
Region-Specific Checks: Run the check in a given region.
Context or Cluster Name Checks: Perform the check in a given context or by cluster name.
Report Export: Export the report to HTML or TXT format.

The commitment is to achieve comprehensive coverage of the AWS EKS Best Practices Guide (EBPG). We aim to incorporate as many EBPG rules as possible into HardenEKS, ensuring that it remains a robust tool for best practices validation.

🤝 Contributions welcome

If you're interested in contributing to HardenEKS, we invite you to review our contribution guidelines. We value the collaboration of both contributors and collaborators who play a pivotal role in shaping the future versions of HardenEKS.

🛤️ Roadmap ahead

HardenEKS maintains a publicly accessible roadmap outlining the planned features for upcoming versions. We encourage users to actively participate by creating GitHub issues, thus influencing the direction and priorities of HardenEKS's development journey.

🔚 Conclusion

The idea of this post was to demonstrate the use of the hardeneks tool to carry out a check of good practices in the use of kubernetes.
Also, we've demonstrated how Amazon EKS operators can leverage programmable validation to assess the compliance of their Amazon EKS clusters with the EBPG.

It's always a rewarding endeavor to explore innovative methods for assisting customers in optimizing their Day 2 Operations with Amazon EKS.

We hope that you have found this blog post helpful. If you have any other tips or tricks that you would like to share, please leave a comment below.

https://giphy.com/gifs/Everdale-supercell-everdale-bigs-the-builder-BDGZ5LdDUkHCS8kS8R

Until next time, つづく 🎉

💡 Thank you for Reading !! 🙌🏻😁📃, see you in the next blog.🤘 Until next time 🎉

🚀 Thank you for sticking up till the end. If you have any questions/feedback regarding this blog feel free to connect with me:

♻️ LinkedIn: https://www.linkedin.com/in/rajhi-saif/

♻️ X/Twitter: https://x.com/rajhisaifeddine

The end ✌🏻

🔰 Keep Learning !! Keep Sharing !! 🔰

📅 Stay updated

Subscribe to the newsletter for more insights on AWS cloud computing and containers.

Speed Up Microservices Development with Dapr on AWS EKS

saifeddine Rajhi — Mon, 04 Nov 2024 08:42:32 +0000

Use Dapr to Build Distributed Applications Easily on Kubernetes 🎩

🐳 Introduction

Building distributed applications means creating software that runs on multiple computers within a network, working together to achieve a common goal. This involves coordinating various components or modules that might be spread across different infrastructures.

Organizations often opt for distributed applications for two main reasons. First, they allow multiple development teams to work independently while contributing to a larger system. Second, they enable the integration of components built with different programming languages, enhancing interoperability.

This flexibility is important in today's diverse tech landscape, where different teams might prefer different tools and languages.

However, developing distributed applications comes with several challenges. Ensuring that numerous components work together fluently requires careful attention to resiliency (the ability to recover from failures), observability (monitoring the system's health), security, and scalability across various services and runtimes.

Additionally, these applications often interact with message brokers (like Kafka or RabbitMQ), data stores (like databases), and external services (like third-party APIs), necessitating a thorough understanding of specific APIs and SDKs, which adds to the complexity. The need for robust error handling, efficient load balancing, and seamless service discovery further complicates the development process.

In this blog, we will explore how the open-source Dapr (Distributed Application Runtime) can assist us in building reliable and secure distributed applications. Dapr provides a set of building blocks for common microservice patterns, such as service invocation (calling services), state management (handling data), and pub/sub messaging (publish/subscribe communication), which can significantly reduce the development effort.

By using Dapr’s built-in best practices and patterns, we will also highlight common use cases for Dapr on AWS EKS (Elastic Kubernetes Service), demonstrating how it can simplify and enhance your microservices architecture. This approach not only speeds up development but also ensures that your applications are scalable and maintainable.

For more on Dapr, check out Part 1 of our series. You can also learn more about Kubernetes, AWS EKS, and microservices architecture to deepen your understanding of the concepts discussed in this post.

🚀 What is Dapr and Why Do We Need It?

Dapr (Distributed Application Runtime) is an open-source project designed to simplify the development of microservices. It provides a set of building blocks that address common challenges in building distributed applications, such as service-to-service communication, state management, and pub/sub messaging.

🌟 Main Features of Dapr

Service Invocation:
- Dapr makes it easy for microservices to communicate with each other. Instead of writing custom code to handle HTTP or gRPC calls, you can use Dapr's Service Invocation API. This API abstracts the complexity and provides built-in retries, timeouts, and error handling. The middleware components are used with the service invocation building block.
State Management:
- Managing state in a distributed system can be complex. Dapr provides a State Management API that allows you to store and retrieve state across different services. This API supports various state stores like Redis, DynamoDB, and Cosmos DB, making it flexible and easy to use.
Publish/Subscribe Messaging:
- For event-driven architectures, Dapr offers a Pub/Sub API. This allows services to publish events and other services to subscribe to them, enabling asynchronous communication. You can integrate with message brokers like AWS SNS, SQS, and Kafka without changing your application code.
Bindings:
- Dapr supports input and output bindings to interact with external systems. This means you can easily connect to databases, message queues, and other services using Dapr's Binding API.
Workflow:
- Dapr provides a built-in workflow feature to orchestrate logic across various microservices.
Actors:
- Dapr supports the actor model, allowing you to encapsulate code and data in reusable actor objects, which is a common microservices design pattern.
Secrets Management:
- Security is a critical aspect of any distributed system. Dapr includes features like mutual TLS (mTLS) for secure service-to-service communication and secret management to handle sensitive information like API keys and database credentials.
Configuration:
- Dapr provides a Configuration API to manage and be notified of application configuration changes.
Distributed Lock:
- Dapr offers a Distributed Lock API to provide mutually exclusive access to shared resources from an application.
Cryptography:
- Dapr includes a Cryptography API to perform cryptographic operations without exposing keys to your application.
Jobs:
- Dapr provides a Jobs API to manage the scheduling and orchestration of jobs.

By leveraging these features, Dapr simplifies the development of distributed applications, allowing developers to focus on writing business logic rather than dealing with the complexities of distributed systems. This leads to faster development cycles, more reliable applications, and easier maintenance.

❓ Why Do We Need Dapr?

Developing distributed applications involves several challenges:

Complex Communication: Ensuring reliable communication between services can be difficult, especially when dealing with different protocols and error handling.
State Management: Keeping track of state across multiple services requires a consistent and reliable approach.
Event-Driven Architecture: Implementing pub/sub messaging patterns can be complex and requires integration with various message brokers.
External Integrations: Connecting to external systems like databases and third-party APIs often involves writing boilerplate code.
Observability: Monitoring and diagnosing issues in a distributed system requires comprehensive logging, metrics, and tracing.
Security: Ensuring secure communication and managing secrets are essential for protecting your application.
Isolation: Dapr namespacing provides isolation and multi-tenancy across many capabilities, giving greater security. Typically applications and components are deployed to namespaces to provide isolation in a given environment, such as Kubernetes.

Dapr addresses these challenges by providing a set of standardized APIs and components that simplify the development process. By using Dapr, developers can focus on writing business logic instead of dealing with the complexities of distributed systems. This leads to faster development cycles, more reliable applications, and easier maintenance.

For more details, check the official documentation.

🛠️ How-To: Invoke Services Using Middleware Component

In this section, we'll demonstrate how to deploy services with unique application IDs, allowing other services to discover and call endpoints using Dapr's service invocation over HTTP.

🆔 Step 1: Choose an ID for Your Service

Dapr allows you to assign a global, unique ID for your app. This ID encapsulates the state for your application, regardless of the number of instances it may have.

🛠️ Step 2: Set an App-ID When Deploying to Kubernetes

In Kubernetes, set the dapr.io/app-id annotation on your pod:

apiVersion: apps/v1
kind: Deployment
metadata:
    name: my-app
    namespace: default
    labels:
        app: my-app
spec:
    replicas: 1
    selector:
        matchLabels:
            app: my-app
    template:
        metadata:
            labels:
                app: my-app
            annotations:
                dapr.io/enabled: "true"
                dapr.io/app-id: "order-processor"
                dapr.io/app-port: "6001"
                dapr.io/app-protocol: "http"  # Use "https" if your app uses TLS

📡 Step 3: Invoke the Service

To invoke an application using Dapr, you can use the invoke API on any Dapr instance. The sidecar programming model encourages each application to interact with its own instance of Dapr. The Dapr sidecars discover and communicate with one another.

Below is an example in Python that leverages Dapr SDKs for service invocation:

import random
from time import sleep
import logging
import requests
import json

base_url = "http://localhost:3500/v1.0/invoke/order-processor/method"
headers = {'Content-Type': 'application/json'}

logging.basicConfig(level=logging.INFO)

while True:
        sleep(random.randrange(50, 5000) / 1000)
        order_id = random.randint(1, 1000)
        order = {'orderId': order_id, 'item': 'laptop', 'quantity': 1}

        # Invoke a service
        result = requests.post(
                url=f'{base_url}/orders',
                data=json.dumps(order),
                headers=headers
        )

        logging.info(f'Order requested: {order_id}')
        logging.info(f'Result: {result.text}')

🌐 Additional URL Formats

To invoke a GET endpoint:

curl http://localhost:3500/v1.0/invoke/order-processor/method/orders/100

Dapr provides multiple ways to call the service invocation API:

Change the address in the URL to localhost:<dapr-http-port>.
Add a dapr-app-id header to specify the ID of the target service, or alternatively pass the ID via HTTP Basic Auth: http://dapr-app-id:<service-id>@localhost:3500/path.

For example, the following command:

curl http://localhost:3500/v1.0/invoke/order-processor/method/orders/100

is equivalent to:

curl -H 'dapr-app-id: order-processor' 'http://localhost:3500/orders/100' -X GET

or:

curl 'http://dapr-app-id:order-processor@localhost:3500/orders/100' -X GET

Using CLI:

dapr invoke --app-id order-processor --method orders/100

🔍 Including a Query String in the URL

You can append a query string or a fragment to the end of the URL, and Dapr will pass it through unchanged. For example:

curl 'http://dapr-app-id:order-processor@localhost:3500/orders/100?basket=1234&key=abc' -X GET

🏷️ Using Namespaces

When running on namespace-supported platforms, include the namespace of the target app in the app ID. For example, use order-processor.production.

Invoking the service with a namespace would look like:

curl http://localhost:3500/v1.0/invoke/order-processor.production/method/orders/100 -X GET

This example demonstrates how to use Dapr's service invocation to call services securely and efficiently within a Kubernetes environment. For more details, check the official Dapr documentation. Let me know if you need any further details or adjustments!

Demo: Set Up an AWS EKS Cluster and App test

In this section, we'll walk you through setting up an Elastic Kubernetes Service (EKS) cluster and deploying a sample application to test Dapr integration.

📋 Prerequisites

Before you begin, ensure you have the following installed:

kubectl
AWS CLI
eksctl
An existing VPC and subnets

🔐 Authenticating to AWS

To interact with AWS services like DynamoDB from your EKS pods, you need to ensure that your pods have the necessary permissions. This can be achieved using IAM Roles for Service Accounts (IRSA) or a pod identity agent.

Using IAM Roles for Service Accounts (IRSA)

Create an IAM Role: Create an IAM role with the necessary permissions for accessing AWS services. Attach the required policies to this role.

aws iam create-role --role-name EKS-DynamoDB-Role --assume-role-policy-document file://trust-policy.json
aws iam attach-role-policy --role-name EKS-DynamoDB-Role --policy-arn arn:aws:iam::aws:policy/AmazonDynamoDBFullAccess

The trust-policy.json should contain the trust relationship allowing EKS to assume this role:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Service": "eks.amazonaws.com"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}

Associate the IAM Role with a Kubernetes Service Account:

eksctl create iamserviceaccount \
  --name my-service-account \
  --namespace default \
  --cluster my-cluster \
  --attach-role-arn arn:aws:iam::<AWS_ACCOUNT_ID>:role/EKS-DynamoDB-Role \
  --approve

Update Your Pod Specification: Update your pod specification to use the service account:

apiVersion: v1
kind: Pod
metadata:
  name: my-pod
spec:
  serviceAccountName: my-service-account
  containers:
  - name: my-container
    image: my-image

Using a Pod Identity Agent

Deploy the Pod Identity Agent: Follow the installation instructions to deploy the agent to your EKS cluster.

Annotate Your Pods: Annotate your pods with the IAM role to be assumed:

apiVersion: v1
kind: Pod
metadata:
  name: my-pod
  annotations:
    iam.amazonaws.com/role: EKS-DynamoDB-Role
spec:
  containers:
  - name: my-container
    image: my-image

By following these steps, you can ensure that your EKS pods have the necessary permissions to interact with AWS services securely.

🛠️ Step 1: Deploy an EKS Cluster

Log into AWS:
```
aws configure
```
Create an EKS Cluster:

To create an EKS cluster, use the following command. You can specify a specific version of Kubernetes using the --version flag (1.13.x or newer version required).
```
eksctl create cluster --name dapre-eks-demo --region eu-west-1 --version 1.30 --vpc-private-subnets subnet-12345,subnet-67890 --without-nodegroup
```
Adjust the --vpc-private-subnets values to meet your requirements. You can also specify public subnets by changing --vpc-private-subnets to --vpc-public-subnets.
Verify kubectl Context:
Ensure your kubectl context is set to the new EKS cluster.
```
kubectl config current-context
```

Update Security Group Rules:
Allow the EKS cluster to communicate with the Dapr sidecar by creating an inbound rule for port 4000.

aws ec2 authorize-security-group-ingress --region eu-west-1 \
--group-id sg-0123456789abcdef0 \
--protocol tcp \
--port 4000 \
--source-group sg-0123456789abcdef0

Step 2: Deploy Sample Applications

First, we need to clone the repo that has the code snippet:

git clone https://github.com/dapr/quickstarts.git
cd quickstarts/tutorials/hello-kubernetes

Then, we need to set up Dapr dev mode on our Kubernetes cluster.

Follow the steps below to deploy Dapr to Kubernetes using the --dev flag.

Note: Any previous Dapr installations in the Kubernetes cluster need to be uninstalled first. You can use dapr uninstall -k to remove Dapr.

With the dapr init -k --dev command, the CLI will also install the Redis and Zipkin containers dapr-dev-redis and dapr-dev-zipkin in the default namespace apart from the Dapr control plane in the dapr-system namespace.

The statestore, pubsub, and appconfig default components and configuration are applied in the default Kubernetes namespace if they do not exist.

You can use dapr components -k and dapr configurations -k to see these.

For more details, check the official Dapr documentation.

dapr init -k --dev
Expected output in a fresh Kubernetes cluster without Dapr installed:

⌛  Making the jump to hyperspace...
ℹ️  Note: To install Dapr using Helm, see here: https://docs.dapr.io/getting-started/install-dapr-kubernetes/#install-with-helm-advanced

ℹ️  Container images will be pulled from Docker Hub
✅  Deploying the Dapr control plane with latest version to your cluster...
✅  Deploying the Dapr dashboard with latest version to your cluster...
✅  Deploying the Dapr Redis with 17.14.5 version to your cluster...
✅  Deploying the Dapr Zipkin with latest version to your cluster...
ℹ️  Applying "statestore" component to Kubernetes "default" namespace.
ℹ️  Applying "pubsub" component to Kubernetes "default" namespace.
ℹ️  Applying "appconfig" zipkin configuration to Kubernetes "default" namespace.
✅  Success! Dapr has been installed to namespace dapr-system. To verify, run `dapr status -k' in your terminal. To get started, go here: https://aka.ms/dapr-getting-started

🐍 Deploy the Python App

Next, we deploy the Python app. This is a basic Python app that posts JSON messages to localhost:3500, the default listening port for Dapr. You can invoke the Node.js application's neworder endpoint by posting to v1.0/invoke/nodeapp/method/neworder. The message contains some data with an orderId that increments once per second:

import time
import requests

n = 0
dapr_url = "http://localhost:3500/v1.0/invoke/nodeapp/method/neworder"

while True:
    n += 1
    message = {"data": {"orderId": n}}

    try:
        response = requests.post(dapr_url, json=message)
        print(response.json())
    except Exception as e:
        print(e)

    time.sleep(1)

To deploy the Python app to the Kubernetes cluster:

kubectl apply -f ./deploy/python.yaml

🟢 Deploy the Node.js App

To deploy the Node.js app to Kubernetes, use the following command:

kubectl apply -f ./deploy/node.yaml

This will deploy the Node.js app to Kubernetes. The Dapr control plane will automatically inject the Dapr sidecar to the Pod. If you take a look at the node.yaml file, you will see how Dapr is enabled for that deployment:

dapr.io/enabled: true - This tells the Dapr control plane to inject a sidecar to this deployment.
dapr.io/app-id: nodeapp - This assigns a unique ID or name to the Dapr application, so it can be sent messages to and communicated with by other Dapr apps.
dapr.io/enable-api-logging: "true" - This is added to node.yaml file by default to see the API logs.

You'll also see the container image that you're deploying. If you want to update the code and deploy a new image, see the Next Steps section.

🔄 Accessing the Kubernetes Service

There are several different ways to access a Kubernetes service depending on which platform you are using. Port forwarding is one consistent way to access a service, whether it is hosted locally or on a cloud Kubernetes provider like AKS.

kubectl port-forward service/nodeapp 8080:80

This will make your service available on http://localhost:8080.

Configure the Redis Statestore Component

Apply the redis.yaml file and observe that your state store was successfully configured!

kubectl apply -f ./deploy/redis.yaml

📜 Viewing Logs

Now that the Node.js and Python applications are deployed, watch messages come through:

Node.js App Logs

Get the logs of the Node.js app:

kubectl logs --selector=app=node -c node --tail=-1

If all went well, you should see logs like this:

Got a new order! Order ID: 1
Successfully persisted state for Order ID: 1
Got a new order! Order ID: 2
Successfully persisted state for Order ID: 2
Got a new order! Order ID: 3
Successfully persisted state for Order ID: 3

API Call Logs

Observe API call logs:

Node.js App API Logs

Get the API call logs of the Node.js app:

kubectl logs --selector=app=node -c daprd --tail=-1

When save state API calls are made, you should see logs similar to this:

time="2024-11-02T22:46:09.82121774Z" level=info method="POST /v1.0/state/statestore" app_id=nodeapp instance=nodeapp-7dd6648dd4-7hpmh scope=dapr.runtime.http-info type=log ver=1.7.2
time="2024-11-02T22:46:10.828764787Z" level=info method="POST /v1.0/state/statestore" app_id=nodeapp instance=nodeapp-7dd6648dd4-7hpmh scope=dapr.runtime.http-info type=log ver=1.7.2

Python App API Logs

Get the API call logs of the Python app:

kubectl logs --selector=app=python -c daprd --tail=-1

time="2024-11-02T02:47:49.972688145Z" level=info method="POST /neworder" app_id=pythonapp instance=pythonapp-545df48d55-jvj52 scope=dapr.runtime.http-info type=log ver=1.7.2
time="2024-11-02T02:47:50.984994545Z" level=info method="POST /neworder" app_id=pythonapp instance=pythonapp-545df48d55-jvj52 scope=dapr.runtime.http-info type=log ver=1.7.2

✅ Confirm Successful Persistence

Call the Node.js app's order endpoint to get the latest order. Grab the external IP address that you saved before, append /order, and perform a GET request against it (enter it into your browser, use Postman, or curl it!):

curl $NODE_APP/order
{"orderID":"42"}

You should see the latest JSON in response!

This will spin down each resource defined by the .yaml files in the deploy directory, including the state component.

Note: This will also delete the state store component. If the --dev flag was used for Dapr init, and you want to use the dapr-dev-redis deployment as state store, replace the redisHost value inside ./deploy/redis.yaml with dapr-dev-redis-master:6379 and also the secretKeyRef, name with dapr-dev-redis. Then run the command kubectl apply -f ./deploy/redis.yaml, to apply the file again. This will create a statestore Dapr component pointing to dapr-dev-redis deployment.

For more details, check the official Dapr documentation.

🏁 Conclusion

In this blog post, we explored how Dapr can significantly simplify the development and management of distributed applications on AWS EKS. By leveraging Dapr's building blocks, such as service invocation, state management, and pub/sub messaging, developers can focus more on business logic and less on the complexities of distributed systems.

We demonstrated how to set up an AWS EKS cluster, deploy sample applications, and integrate Dapr to enhance microservices architecture. This approach not only accelerates development but also ensures that your applications are scalable, secure, and maintainable.

Dapr's open-source nature and extensive documentation make it an excellent choice for organizations looking to streamline their microservices development on Kubernetes. Whether you are just starting with microservices or looking to optimize your existing architecture, Dapr provides the tools and best practices to help you succeed.

For more information and detailed guides, be sure to check out the official Dapr documentation.

Happy coding! 🚀

💡 Thank you for Reading !! 🙌🏻😁📃, see you in the next blog.🤘 Until next time 🎉

🚀 Thank you for sticking up till the end. If you have any questions/feedback regarding this blog feel free to connect with me:

♻️ LinkedIn: https://www.linkedin.com/in/rajhi-saif/

♻️ X/Twitter: https://x.com/rajhisaifeddine

The end ✌🏻

🔰 Keep Learning !! Keep Sharing !! 🔰

📅 Stay updated

Subscribe to our newsletter for more insights on AWS cloud computing and containers.