DEV Community: Camille Chang

OpenLens Cannot Connect to AWS EKS Cluster: executable aws not found

Camille Chang — Wed, 07 Jan 2026 23:24:31 +0000

Overview

Attempting to use OpenLens locally to access a remote AWS EKS environment.
AWS SSO has been configured successfully, and access via the local terminal works as expected. However, OpenLens fails to connect to the cluster.

Environment Setup

AWS SSO login and verification:

aws sso login --profile profileA aws sts get-caller-identity --profile profileA

EKS kubeconfig update:

aws eks update-kubeconfig --name clusterA --profile profileA

Actual Result:

Access via local terminal works:

kubectl get nodes

OpenLens fails to connect to the cluster and returns the following error: Error while proxying request: getting credentials: exec: executable aws not found. It looks like you are trying to use a client-go credential plugin that is not installed. To learn more about this feature, consult the documentation available at: https://kubernetes.io/docs/reference/access-authn-authz/authentication/#client-go-credential-plugins Failed to get /version for clusterId=: Internal Server Error

The Fix

1. Verify AWS CLI installation Run this in a terminal:

aws --version

2. Add AWS CLI to system PATH Check:

which aws

Typical locations:
macOS (Intel): /usr/local/bin/aws macOS (Apple Silicon): /opt/homebrew/bin/aws

Then make it globally available.
macOS (Apple Silicon – most common)
sudo ln -s /opt/homebrew/bin/aws /usr/local/bin/aws

Restart OpenLens completely.

3. Or Hardcode full path to AWS CLI in kubeconfig. Edit your kubeconfig: ~/.kube/config Change: command: aws To: command: /opt/homebrew/bin/aws This guarantees OpenLens can execute it.

aws dop-c02 - notes

Camille Chang — Sun, 04 Jan 2026 11:12:59 +0000

ECR

ECR Cache Rule https://docs.aws.amazon.com/AmazonECR/latest/userguide/pull-through-cache-creating-rule.html ### Fault Injection Service (AWS FIS) experiment， These experiments stress an application by creating disruptive events so that you can observe how your application responds. https://docs.aws.amazon.com/fis/latest/userguide/what-is.html

lightsail vs beanstalk

lightsail, Simple applications, small-scale deployments https://docs.aws.amazon.com/decision-guides/latest/lightsail-elastic-beanstalk-ec2/lightsail-elastic-beanstalk-ec2.html

AWS Config

conformance pack, a collection of AWS Config rules and remediation actions that can be easily deployed as a single entity in an account and a Region or across an organization in AWS Organizations. https://docs.aws.amazon.com/config/latest/developerguide/conformance-packs.html

CodeDeploy

List of lifecycle event hooks for an Amazon ECS deployment
- *BeforeInstall AfterInstall AfterAllowTestTraffic BeforeAllowTraffic AfterAllowTraffic *

https://docs.aws.amazon.com/codedeploy/latest/userguide/reference-appspec-file-structure-hooks.html

SQS

SQS Dead-Letter Queue, useful for debugging your application because you can isolate unconsumed messages to determine why processing did not succeed. https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-dead-letter-queues.html

EC2 Image Builder

fully managed AWS service that helps you to automate the creation, management, and deployment of customized, secure, and up-to-date server images.

ALB vs NLB

Both support websocket.
ALB stickiness,Uses HTTP/HTTPS cookies to bind user sessions to specific targets.

AWS Certified Developer - Associate (DVA-C02) Exam notes

Camille Chang — Sun, 14 Dec 2025 02:01:59 +0000

Notes preparation before the exam.

AWS X-Ray

X-Ray daemonis a software application that listens for traffic on UDP port 2000, gathers raw segment data, and relays it to the AWS X-Ray API.

*Note: * End-of-support notice – On February 25th, 2027, AWS X-Ray will discontinue support for AWS X-Ray SDKs and daemon. https://docs.aws.amazon.com/xray/latest/devguide/xray-daemon.html

S3

S3 Object Lambda, provide different views of data to multiple applications, allows you to add custom code to process and transform data retrieved from S3 buckets during standard GET, HEAD, and LIST API requests. This enables applications to access a tailored view of the data without needing to create or maintain separate, derivative copies or manage a proxy layer. https://aws.amazon.com/blogs/aws/introducing-amazon-s3-object-lambda-use-your-code-to-process-data-as-it-is-being-retrieved-from-s3/

DynamoDB

supports two different kinds of primary keys:
- Partition key – A simple primary key, composed of one attribute known as the partition key.
- Partition key and sort key – Referred to as a composite primary key, this type of key is composed of two attributes.
Secondary index
- Global secondary index – An index with a partition key and sort key that can be different from those on the table. The primary key values in global secondary indexes don't need to be unique.
- Local secondary index – An index that has the same partition key as the table, but a different sort key.

https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/HowItWorks.CoreComponents.html

Cloudfront

Only support ACM certificate in the US East (N. Virginia) Region (us-east-1). ### Amplify
Git-based workflow for hosting full-stack serverless web applications with continuous deployment.

LB

ALB, layer 7, HTTP/HTTPS
- X-Forwarded-For, helps you identify the IP address of a client https://docs.aws.amazon.com/elasticloadbalancing/latest/application/x-forwarded-headers.html
NLB, layer 4, TCP/UDP

Beanstalk

Elastic Beanstalk provisions Amazon EC2 instances, configures load balancing, sets up health monitoring, and dynamically scales your environment.

Amazon DevOps Guru

is a machine learning service designed to detect abnormal operating patterns

AWS CloudOps Engineer - Associate (SOA-C03)

Camille Chang — Sun, 07 Dec 2025 19:18:31 +0000

Although I've been using AWS for a while now, I still need to review a lot of the knowledge frequently.

Preparation:
1 Tried Official Practice Question Set https://awscertificationpractice.benchprep.com/app/official-practice-question-set-aws-certified-cloudops-engineer-associate-soa-c03#exams/details/315463, passed with 80%.

2 Knowledge points I forgot：

gateway VPC endpoint for Amazon S3 and DynamoDB
Gateway endpoints allow EC2 instances in private subnets to access Amazon S3 without using the internet or incurring data processing charges.
Reserved concurrency specifies the maximum number of concurrent instances of a Lambda function that can run at the same time.
Provisioned concurrency specifies the number of pre-initialized execution environments that a Lambda function has. If the initialization of the code is pre-provisioned, then the Lambda function will spend less time running.
IAM OIDC IdPs when you want to connect an external OIDC-compatible IdP to AWS resources.

-IAM Identity Center to provide user access from external IdPs to -AWS applications by using the SAML protocol. You would typically use IAM Identity Center to provide **single sign-on access **for external users to AWS services.

**CloudWatch Logs data protection **policies help identify and protect sensitive data within CloudWatch Logs. Sensitive data includes personally identifiable information (PII) or PHI. CloudWatch Logs data protection policies automatically detect sensitive data patterns. CloudWatch Logs data protection policies can invoke alerts or initiate actions when sensitive data is logged.
The SPF record specifies which IP addresses are allowed to send email for the domain. To prevent spoofing and to improve email deliverability, you should add an SPF record as a TXT record.
Rolling update deployment gradually replaces tasks with new versions. Rolling deployments maintain application availability by gradually updating tasks in small batches.
A canary deployment shifts a small percentage of traffic for validation. A canary deployment with weighted routing requires additional infrastructure to split traffic. A weighted routing configuration leads to higher costs for this scenario.

Route53

AAAA record-> IPv6
A record-> IPv4
Alias -> zone apex or ALB DNS -> CNAME, point a subdomain like www.example.com to the root domain example.com, or to map a domain to a different service provider.

RDS

RDS proxy, pool and share database connections

Rerfer: https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/rds-proxy.html

EC2

Cluster Placement Group, Packs instances close together inside an Availability Zone. low-latency network performance necessary for tightly-coupled node-to-node communication that is typical of high-performance computing (HPC) applications.
Partition Placement Group, Spreads your instances across logical partitions such that groups of instances in one partition do not share the underlying hardware with groups of instances in different partitions. This strategy is typically used by large distributed and replicated workloads, such as Hadoop, Cassandra, and Kafka.
Spread placement group, trictly places a small group of instances across distinct underlying hardware to reduce correlated failures.

https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/placement-groups.html

Storage

EFS, Can create only one mount target per Availability Zone https://docs.aws.amazon.com/efs/latest/ug/accessing-fs.html -EBS snapshot is an incremental backup, which means that we save only the blocks on the volume that have changed since the most recent snapshot. This minimizes the time required to create the snapshot and saves on storage costs by not duplicating data.
EBS fast snapshot restore (FSR) enables you to create a volume from a snapshot that is fully initialized at creation. This eliminates the latency of I/O operations on a block when it is accessed for the first time. Volumes that are created using fast snapshot restore instantly deliver all of their provisioned performance.
RDS Performance Insights. can visualize the database load on your Amazon RDS DB instance load and filter the load by waits, SQL statements, hosts, or users. Fhttps://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_PerfInsights.html
Aurora,
- PITR is the robust, long-term backup and restore solution that involves creating a new database cluster.
- Backtracking is a quick, in-place "undo" feature for Aurora MySQL to recover from recent, minor errors with low Recovery Time Objective (RTO).
Storage Gateway
- Gateway-stored volumes: Store a full copy of the data locally while asynchronously backing it up to AWS. For the backup application, it behaves like operating a local block storage device.
- Gateway-cached volumes: Store most of the data in AWS, with only recently accessed data cached locally.
ElastiCache
- Memcached is lightweight and simple, good for read-heavy, non-persistent caching.
- Redis is feature-rich, supports high availability, persistence, and advanced data types, making it suitable for mission-critical caching and real-time applications.

CloudFormation

A stack set lets you create stacks in AWS accounts across AWS Regions by using a single CloudFormation template
Custom resources, provisioning requirements involve complex logic or workflows that can't be expressed with CloudFormation's built-in resource types. https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/template-custom-resources.html

EC2

stop an instance, it shuts down.
start an instance, it is typically migrated to a new underlying host computer and assigned a new public IPv4 address. https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/Stop_Start.html
reboot, An instance reboot is equivalent to an operating system reboot. In most cases, it takes only a few minutes to reboot your instance. When you reboot an instance, it keeps the following:
- Public DNS name (IPv4)
- Private IPv4 address
- Public IPv4 address
- IPv6 address (if applicable)
- Any data on its instance store volumes
Terminate, After you terminate an instance, you can no longer connect to it, and it can't be recovered. All attached Amazon EBS volumes that are configured to be deleted on termination are also permanently deleted and can't be recovered.

Network

Customer gateway,
Route 53 Resolver,
- Inbound Resolver, endpoints allow DNS queries on-premises network or another VPC-> your VPC.
- Outbound Resolver, DNS queries your VPC -> your on-premises network or another VPC.
Route 53 routing policy
- Geolocation, route traffic based on the location of your users
- Geoproximity, route traffic based on the location of your resources
- Latency-based
- Multivalue answer,

CloudWatch

Collect process metrics with the procstat plugin
CloudWatch Synthetics canary, create canaries, configurable scripts that run on a schedule, to monitor your endpoints and APIs. https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/CloudWatch_Synthetics_Canaries.html

Tag Editor

user-defined cost allocation tags, you must activate them. https://docs.aws.amazon.com/awsaccountbilling/latest/aboutv2/activating-tags.html

Security

Trusted Advisor inspects your AWS environment, and then makes recommendations when opportunities exist to save money, improve system availability and performance, or help close security gaps.
- The support plan will affect the quantity of available Trusted Advisor checks

Others

AWS Service Catalog sharing, When you share a portfolio using account-to-account sharing or Organizations, you are sharing a reference of that portfolio. The products and constraints in the imported portfolio stay in sync with changes that you make to the shared portfolio, the original portfolio that you shared.

The recipient cannot change the products or constraints, but can add IAM access for end users. https://docs.aws.amazon.com/servicecatalog/latest/adminguide/catalogs_portfolios_sharing_how-to-share.html

AWS Personal Health Dashboard, a personalized view of AWS service events that may affect your AWS resources
OpsWorks supports Chef and Puppet
AWS Control Tower,
- Automated landing zone setup: Quickly sets up a well-architected, multi-account environment with features like dedicated log archive and audit accounts.
- Pre-configured controls: Provides a library of pre-packaged governance rules (guardrails) to enforce security, compliance, and operational policies. These can be preventive, detective, or proactive.
- Account Factory: Enables the provisioning of new AWS accounts that automatically comply with the established governance policies.

Attended and passed the exam on 7 Dec. There was one AI-related question.

AWS Security certificate SCS-C03 updates

Camille Chang — Mon, 20 Oct 2025 23:23:22 +0000

I attended the AWS Security exam on 18 October 2025 and received my result a little later than usual. Then I saw the news that the AWS Security exam has been updated to SCS-C03, and registration for the updated exam (SCS-C03) opens on November 18.

Cannot find the AWS Certified Security - Specialty (SCS-C03) Exam Guide yet, but according to AWS, the exam now covers emerging technologies, with a dedicated emphasis on generative AI and machine learning security. To better support security professionals, the exam domains have been restructured, introducing separate sections for Detection and Incident Response capabilities.

More details please check
https://aws.amazon.com/blogs/training-and-certification/big-news-aws-expands-ai-certification-portfolio-and-updates-security-certification/

Complete Beginner's Guide: Upload Files to AWS S3 with GitHub Actions

Camille Chang — Wed, 08 Oct 2025 03:57:07 +0000

If you're new to GitHub Actions and AWS, this guide will walk you through automating file uploads to an S3 bucket step by step. I'll share the common mistake I made and how to fix it, so you can avoid the same pitfalls!

🎯 What We're Building

By the end of this tutorial, you'll have a GitHub Action that automatically:

Connects securely to your AWS account
Uploads files to your S3 bucket whenever you push code

📋 Prerequisites

Before we start, make sure you have:

A GitHub repository
An AWS account
Basic familiarity with GitHub (knowing how to create files and commit changes)

🔧 Step 1: Set Up Your S3 Bucket

First, let's create an S3 bucket where your files will be stored:

Log into the AWS Console
Navigate to S3
Click "Create bucket"
Give it a unique name (like my-project-files-bucket)
Keep the default settings and create the bucket

🔐 Step 2: Create an IAM Role (The Tricky Part!)

This is where I initially got stuck, so let's break it down:

What's an IAM Role?

Think of an IAM role as a set of permissions that GitHub Actions can "borrow" to access your AWS resources. It's like giving GitHub a temporary key to your AWS account.

Creating the Role

Go to IAM in AWS Console
Click "Roles" → "Create role"
Choose "Web identity" as the trusted entity type
For Identity provider, select "OpenID Connect"
Add this provider URL: token.actions.githubusercontent.com
For Audience, enter: sts.amazonaws.com

Adding Permissions

Your role needs permission to upload files to S3. Attach this policy:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:PutObject",
                "s3:PutObjectAcl",
                "s3:GetObject"
            ],
            "Resource": "arn:aws:s3:::your-bucket-name/*"
        }
    ]
}

Important: Replace your-bucket-name with your actual S3 bucket name!

❌ The Problem I Ran Into

When I first tried this, I got this error:

Run aws-actions/configure-aws-credentials@v4
Configuring proxy handler for STS client
Error: Credentials could not be loaded, please check your action inputs: Could not load credentials from any providers

The issue? I forgot the most important part - the trust policy!

✅ Step 3: Fix the Trust Policy (The Missing Piece!)

Here's what I was missing. The IAM role needs to "trust" GitHub Actions. Here's the trust policy you need:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "Federated": "arn:aws:iam::YOUR-ACCOUNT-ID:oidc-provider/token.actions.githubusercontent.com"
            },
            "Action": "sts:AssumeRoleWithWebIdentity",
            "Condition": {
                "StringEquals": {
                    "token.actions.githubusercontent.com:sub": "repo:YOUR-GITHUB-USERNAME/YOUR-REPO-NAME:ref:refs/heads/main",
                    "token.actions.githubusercontent.com:aud": "sts.amazonaws.com"
                }
            }
        }
    ]
}

How to Apply This Trust Policy:

In your IAM role, click the "Trust relationships" tab
Click "Edit trust policy"
Replace the existing policy with the one above
Don't forget to replace:
- YOUR-ACCOUNT-ID with your 12-digit AWS account ID
- YOUR-GITHUB-USERNAME with your GitHub username
- YOUR-REPO-NAME with your repository name

🔍 How to Find Your AWS Account ID

Not sure what your AWS account ID is? Here's how to find it:

Click on your username in the top-right corner of AWS Console
Your account ID is shown in the dropdown menu

🚀 Step 4: Create Your GitHub Action

Now create a file in your repository at .github/workflows/upload-to-s3.yml:

name: Upload to S3

on:
  push:
    branches: [ main ]

jobs:
  upload:
    runs-on: ubuntu-latest

    # This is crucial - it allows the action to get temporary credentials
    permissions:
      id-token: write
      contents: read

    steps:
    - name: Checkout code
      uses: actions/checkout@v4

    - name: Configure AWS credentials
      uses: aws-actions/configure-aws-credentials@v4
      with:
        role-to-assume: arn:aws:iam::YOUR-ACCOUNT-ID:role/YOUR-ROLE-NAME
        aws-region: us-east-1

    - name: Upload files to S3
      run: |
        aws s3 cp ./your-file.txt s3://your-bucket-name/

Remember to replace:

YOUR-ACCOUNT-ID with your AWS account ID
YOUR-ROLE-NAME with the name of your IAM role
your-bucket-name with your S3 bucket name
your-file.txt with the file you want to upload

🎉 Testing Your Setup

Commit and push your workflow file to GitHub
Go to the "Actions" tab in your GitHub repository
You should see your workflow running
Check your S3 bucket - your files should appear there!

📚 Want to Learn More?

Some reflections after passing the AWS Certified Machine Learning Engineer - Associate (MLA-C01) exam:

Camille Chang — Sun, 05 Oct 2025 03:14:33 +0000

The exam itself is a bit easier than the MLS one — some questions can be answered at a glance.
Around 60%–80% of the questions focus on SageMaker.
Below are my summarised notes from this attempt — I hope they can help anyone preparing for the exam in the future.

🧠 SageMaker Overview

Data Wrangler

Provides a user-friendly interface to clean, preprocess, and transform data without needing to write custom code.
Includes built-in transformations to balance data, such as Random Oversampler/Undersampler and SMOTE (Synthetic Minority Over-sampling Technique).

Autopilot

Automates the process of building and deploying machine learning models.

Clarify

Identifies potential bias during data preparation and explains predictions without needing custom code.

Debugger

Provides tools to register hooks and callbacks to extract model output tensors.
Offers built-in rules to detect model convergence issues such as overfitting, underutilized GPU, and vanishing/exploding gradients.

Feature Attribution Drift

Use the ModelExplainabilityMonitor class to generate a feature attribution baseline and deploy a monitoring mechanism that evaluates whether feature attribution drift has occurred.
Then deploy the baseline to SageMaker Model Monitor. Learn more →

Shadow Testing

Enables testing of new ML models against production models using live data without impacting live inference traffic.
Helps identify potential configuration errors, performance issues, and other problems before full deployment.

Neo

Enables machine learning models to train once and run anywhere — both in the cloud and at the edge.

JumpStart

A machine learning hub with prebuilt models and solutions.

Ground Truth

Provides labeling workflows for creating high-quality training datasets.

FSx for Lustre

Designed for large-scale ML training and HPC workloads.
Can be linked directly to an S3 bucket, caching data as needed.
Requires minimal setup.

ML Lineage Tracking

Creates and stores metadata about ML workflow steps from data preparation to model deployment.
Enables reproducibility, model governance, and audit tracking.

Canvas

Allows users to import, prepare, transform, visualize, and analyze data using a visual interface.

📊 Model Monitoring in SageMaker

SageMaker Model Monitor provides the following types of monitoring:

Data Quality – Monitor drift in data quality.
Model Quality – Monitor drift in model metrics such as accuracy.
Bias Drift – Monitor bias in model predictions.
Feature Attribution Drift – Monitor changes in feature attribution.
SageMaker Endpoints can enable data capture and reuse that data for retraining.

Data capture docs →
Bring Your Own Containers (BYOC) — e.g., deploy ML models built with R.

Docs →

R guide →
Network Isolation — blocks internet and external network access.

Docs →
Asynchronous Inference — suitable for large payloads (up to 1 GB) and long processing times (up to 1 hour).

Auto-scales to zero when idle, reducing costs.

Docs →
Batch Transform — perform inference without persistent endpoints.

Docs →
Real-Time Inference — supports payloads up to 5 MB for synchronous requests.

⚖️ Model Explainability & Bias Detection

Difference in Proportions of Labels (DPL) — detects pre-training bias to prevent discriminatory models.

Docs →
Partial Dependence Plots (PDPs) — illustrate how predictions change with one input feature.

Docs →
Shapley Values — determine the contribution of each feature to model predictions.

Docs →

🧩 Other SageMaker Features

TensorBoard Integration — visualize the training process and debug model performance.

Docs →
Feature Store — create feature groups, ingest records, and build datasets for training.

Docs →
Managed Warm Pools — retain and reuse infrastructure after training jobs to reduce latency for iterative workloads.
Inference Recommender — automates load testing and helps select the best instance configuration for ML workloads.

🔍 Other AWS Services & Concepts

OpenSearch

Can be used as a Vector Database. Docs →

Data Augmentation

Generates synthetic data to improve model training and reduce overfitting. Docs →
Benefits:
- Enhanced model performance
- Reduced data dependency
- Mitigates overfitting

AppFlow

Fully managed integration service for secure data transfer between SaaS apps (e.g., Salesforce, SAP, Google Analytics) and AWS (e.g., S3, Redshift).

Forecast

Handles missing values in time-series forecasting. Docs →

Glue

ETL service for preparing and transforming data.

DataBrew

Visual data preparation tool with data quality rules, cleaning, and feature engineering.

🗣️ AI/ML Application Services

Service	Description
Lex	Chatbot and call center solutions
Polly	Text-to-speech service
Transcribe	Speech-to-text
Forecast	Time-series forecasting
Rekognition	Image and video analysis (object detection, facial recognition)
Comprehend	NLP for sentiment analysis, topic modeling, and PII redaction
Kendra	Intelligent enterprise search with GenAI Index for RAG and digital assistants
Bedrock	Managed API access to LLMs like Jurassic-2
Managed Service for Apache Flink	Fully managed real-time stream processing service (supports anomaly detection with `RANDOM_CUT_FOREST`)

🧩 General ML Concepts

Embeddings — high-dimensional vectors capturing semantic meaning.
RAG (Retrieval-Augmented Generation) — enriches responses with external knowledge sources.
Temperature — controls randomness of generative model output (low = focused, high = creative).
Top_k — limits token choices to top k probabilities; higher values increase diversity.
Recall — focuses on minimizing false negatives.
Precision — focuses on minimizing false positives.
Concept Drift — when data patterns change over time, degrading model accuracy.
MAE (Mean Absolute Error) — measures the average magnitude of prediction errors.
Learning Rate — controls training step size; too high overshoots, too low slows convergence.
Trainium Chips — AWS-built AI chips for efficient model training and inference.

📈 Performance Metrics

Common evaluation metrics for ML models:

Precision
Recall
Accuracy
F1 Score
ROC
AUC
RMSE
MAPE

How to Pass the AWS Certified Data Engineer – Associate (DEA-C01) Exam2025

Camille Chang — Wed, 24 Sep 2025 10:08:55 +0000

This year, I finally have the time to settle down, focus on studying and working, and set a few small goals for myself. One of them is to obtain all 12 AWS certifications before the end of 2026. Last week, I passed this exam, so while it’s still fresh in my mind, I want to record my thoughts and experiences.

How to Pass the AWS Certified Data Engineer – Associate Exam

If you’re preparing for the AWS Certified Data Engineer – Associate exam, here’s a simple and effective 4-step study guide to help you stay focused and increase your chances of passing.

Step 1: Check the Official Exam Guide

Start with the official exam guide. It clearly lists the AWS services and topics covered in the exam:

Official Exam Guide — AWS Certified Data Engineer – Associate

By reviewing the guide carefully, you’ll understand the exam scope and know which services to prioritise in your study plan.

Step 2: Watch Training Videos (Optional)

There are many training videos available, but some are too long or overly detailed. Instead of relying only on them, I recommend:

Creating an AWS account to get hands-on practice.
Watching shorter tutorials to see what the service UIs look like.
Paying attention to common service combinations, since these often appear in exam questions.

Recommended finish the free courses provided for AWS:

Step 3: Practice Questions

Practice is the key to success.
Recommended finish the free questions set provided by AWS:

https://skillbuilder.aws/learn/2JS5H1Z9KP/official-practice-question-set-aws-certified-data-engineer--associate-deac01--english/VX268Y5VBA

My suggestions:

Use practice questions from at least two different providers.
Don’t just memorize answers — read the explanations carefully to understand the reasoning behind them.

Step 4: Identify Your Weak Points

As you practice, you’ll quickly discover which areas are your weakest. Focus your study time on these services until you’re comfortable and confident with them.

Core Analytics Services You Should Know

Based on my preparation, these are the major services you should focus on. Creating a personal cheat sheet for them is highly recommended:

Airflow
- Orchestrates complex data pipelines with support for dependency management and scheduling.
- Cost is relatively high (an Airflow cluster runs continuously and incurs fixed expenses), so it may not be the most cost-effective choice for simple ETL scenarios.
- Open-source, cross-platform orchestration framework that can be deployed both on-premises and in the cloud.
- With MWAA (Managed Workflows for Apache Airflow), you can host Airflow in the cloud while maintaining compatibility with on-premises Airflow DAGs.
Amazon Athena
- Partition Projection: Normally, Athena needs to enumerate all partitions from the AWS Glue Data Catalog before running a query. If you have thousands or millions of partitions, query planning time becomes a bottleneck. Partition Projection solves this by not storing partitions in Glue. Instead, Athena calculates partitions dynamically at query time based on rules you define.
- MSCK REPAIR TABLE: This command scans the underlying paths in Amazon S3, detects new folders that represent partitions, and automatically adds them to the AWS Glue Data Catalog. This ensures Athena can query the most up-to-date data without manually adding partitions.
- Athena vs S3 Select
Amazon EMR
- For terabyte-scale archived data, high-throughput, distributed processing (such as Spark or Hadoop) is required.
- Amazon EMR is a mature and scalable choice that can process large volumes of data in parallel at once.
AWS Glue
- AWS Glue workflows → Orchestrates a series of crawlers, jobs, and triggers into a complete ETL pipeline。
- AWS Step Functions is a fully managed service for building and orchestrating workflows across multiple AWS services. It integrates directly with AWS Glue and Amazon EMR, supports retries, error handling, wait states, and parallel execution.
- Glue DynamicFrames,provides a higher-level abstraction for semi-structured data (JSON, Parquet) with built-in schema inference. DynamicFrames include file-grouping options:
- Automatically merges many small files into larger partitions
- Reduces job startup and I/O overhead, significantly improving throughput when converting to Parquet and loading into Redshift
- Glue DataBrew, a no-code data preparation tool aimed at analysts. 250+ built-in data cleansing and transformation operations (deduplication, missing value filling, format conversion). Provides live visual previews of sample data. -Glue Transforms, built-in transformations for common data cleansing and ML preprocessing (e.g., FindMatches for fuzzy deduplication). -Job Bookmarking, automatically tracks the last processed offset to avoid reprocessing data (incremental load scenarios). -Glue Studio: a visual interface for designing ETL jobs. Drag-and-drop to create data flows, which automatically generates PySpark/SparkSQL code.
AWS Lake Formation
- Centralizes and secures S3-based data lakes, simplifies permissions, cataloging, and governance
Amazon Kinesis Data Firehose
- Fully managed service for loading streaming data into S3, Redshift, or OpenSearch
- Can convert data formats, e.g., JSON → Parquet
Amazon Kinesis Data Streams
- Real-time streaming data ingestion and processing
Amazon Managed Service for Apache Flink
- Managed service for running Apache Flink applications for real-time stream processing
Amazon MSK (Managed Streaming for Apache Kafka)
- Fully managed Apache Kafka service for building streaming applications
Amazon OpenSearch Service
- Managed search and analytics service (formerly Amazon Elasticsearch Service)
Amazon QuickSight,
- SPICE (Super-fast, Parallel, In-memory Calculation Engine) is the in-memory data engine used by Amazon QuickSight.
- High performance: Stores data in-memory to provide fast query and visualization responses.
- Parallel processing: Can handle large datasets efficiently by distributing computations across multiple nodes.
- Scalable: Automatically scales to support increasing data volume and concurrent users.
- Use case: Ideal for interactive dashboards and analytics without waiting for live queries to complete.
Redshift
- Federated Query, Allows direct querying of external Aurora PostgreSQL (or RDS/Postgres). This enables combining real-time transactional data with warehouse data during analysis without replicating all real-time data from the transactional database into Redshift. This approach retains real-time insights and reduces replication costs.
- Query Editor v2, A web-based interface that supports query scheduling.
- materialized view is a precomputed, stored version of a query result.
  - It improves query performance because the database can read from the stored result instead of recomputing it every time.
  - Useful for complex or frequently used aggregations.
  - Can be refreshed periodically to keep the data up-to-date.
- Redshift Spectrum allows Amazon Redshift to query data directly in S3 without loading it into Redshift tables. - Works with data stored in formats like Parquet, ORC, JSON, or CSV.
- Federated Query enables Redshift to query external databases (like RDS, Aurora, or even MySQL/PostgreSQL) directly. - No need to move or copy data; queries run across Redshift and external sources.
- COPY Command
- Efficiently loads data from S3, DynamoDB, EMR, and Kinesis into Redshift
- Supports parallel loading of multiple files (using manifest files)
- UNLOAD Command
- Exports Redshift tables to S3
- Can be used as part of a data lake or as an intermediate step in ETL
- AWS Data Integration
- Glue ETL jobs can write directly into Redshift
- DMS (Database Migration Service) can synchronize data from RDS or on-premises databases into Redshift
- TRUNCATE in Redshift
- A DDL operation that is more efficient than DELETE
- Immediately reclaims storage space
- Recommended for clearing tables or materialized views while recovering storage
Amazon DataZone provides a fully managed data catalog and governance service with built-in support for business glossaries, custom metadata forms, and a user-friendly data portal. You can onboard your existing data assets, define glossaries, and capture business metadata without building or maintaining custom tables, APIs, or applications, minimizing operational overhead.

Final Thoughts

Passing this exam isn’t about memorizing every single detail — it’s about understanding how AWS data services work together to build real-world solutions.

Good luck 🚀

How I passed AWS Certified Machine Learning — Specialty 2025

Camille Chang — Sun, 14 Sep 2025 10:11:57 +0000

I passed the exam in April this year, and I’d like to share my personal experience and some tips for beginners. My background is not in data science, and I had only used some basic AWS services before starting my preparation. If you are new to AWS or don’t have a technical background, don’t worry — with the right approach, you can still pass the exam.

1. Give Yourself Enough Preparation Time
I recommend setting aside at least one month to prepare. Start by visiting the official AWS exam guide https://aws.amazon.com/certification/certified-machine-learning-specialty to understand which services and topics are covered. This helps you focus your study on the areas that are most important for the exam.

2. Take an Online Course
Choose one structured online course from any learning platform (Udemy, Coursera, A Cloud Guru, etc.). Most of the courses are quite dry, so don’t expect them to be very exciting, but they will give you the necessary foundation.

While following the course, take notes. If possible, create one your own AWS account and explore the services. Familiarising yourself with the AWS Management Console UI helps you remember what each service looks like.

For the confused part, watch YouTube, there are a lot of hands-on demos for you.

3. Practice with Questions
After finishing the course, start doing practice exams. These will show you your weak points. Whenever you found that a service or concept you don’t understand, search for a short, clear video on YouTube. Visual explanations can often make things much easier to grasp than just reading documentation.

4. Study Routine
My personal schedule was about one hour or two hours of study per day. This consistent pace worked well without overwhelming me. If you stay disciplined, one month of focused study is enough for many beginners.

5. Additional Tips
Don’t try to memorize everything — focus on understanding the use cases of each AWS service.
Learn how different services integrate (e.g., S3 with Lambda, Glue with Athena, etc.).
Use free-tier resources in your AWS account to test things hands-on. Doing is always better than just reading.
Right before the exam, review the services that come up most often in practice questions.
Final Thoughts
You don’t need to be a data scientist or an AWS expert to pass this exam. With a clear study plan, the right resources, and steady practice, you can achieve it too.
My biggest advice is: stay consistent, use practice questions, and don’t hesitate to look for simpler explanations when something feels confusing.

From Certification to Real-World AWS: Troubleshooting

Camille Chang — Thu, 11 Sep 2025 01:45:06 +0000

When I started working in IT, I thought AWS certifications would give me the confidence to handle real-world challenges. I passed several exams, but in practice, I realised certifications only provide a broad understanding of AWS services. The real lessons come when you’re stuck debugging production issues — that’s when the learning truly sticks.

Recently, I encountered a particularly challenging problem at work. In repo A, I wrote a feature in AWS SAM (YAML) using Lambda, SNS, EventBridge, and S3. Unexpectedly, my colleague said that our new feature needed to be in repo B instead. Repo B was written in Terraform, but it didn’t yet have reusable modules for Lambda — and I had zero Terraform experience. What I thought would take a single day to deploy and test ended up taking me several days, even with the help of AI. Finally, my PR was approved, the code was merged and deployed… and then IAM role issues appeared: insufficient permissions.

The role’s definition was in another repo, C, built with CloudFormation and CodeBuild. Many AWS policies couldn’t be reused because they used overly broad Resource * permissions. We needed fine-grained policies specifying the exact actions and resources. At first, I tried to add many actions at once into the role, but CodeBuild kept failing. I switched to adding them incrementally — compile, deploy repo B, test permissions, repeat. This wasted an entire day.

In the evening, a colleague joined me and pointed out the real problem:

Cannot exceed quota for policy size: 6144. ServiceLimiteExceeded.

At that moment, everything clicked. I created a new IAM policy, attached it to the role, and the issue was resolved.

Looking back, I realized I was only seeing part of the problem. Each CodeBuild run would generate logs in S3, and I only looked at those logs, which simply said “role failed to update.” I hadn’t gone deeper into the corresponding CloudFormation stack to investigate the exact reason. If I had checked that right away, I could have avoided wasting so much time adding permissions piece by piece.

Lessons Learned

Check root causes early – Logs in S3 only said “role failed to update.” The detailed error was in CloudFormation. I should have traced it sooner.
Don’t overload IAM policies – AWS IAM has a strict 6,144-character policy size limit. Split large policies into smaller ones.
Hands-on beats theory – Certifications gave me a foundation, but real troubleshooting taught me far more.
Ask for help – Sometimes a colleague’s perspective saves hours (or days).