DEV Community: Sachin Varghese

AWS CP CLF-02 Cheat Sheet

Sachin Varghese — Thu, 18 Jun 2026 06:01:26 +0000

AWS Certified Cloud Practitioner (CLF-C02) 2026 Cheat Sheet

An ultra-concise, tabular reference guide for the AWS Certified Cloud Practitioner exam (CLF-C02).

1. Exam Overview & Domains

Domain	Weight	Core Focus
Domain 1: Cloud Concepts	24%	Benefits of cloud, economics (CapEx/OpEx), architecture, and CAF.
Domain 2: Security and Compliance	30%	Shared Responsibility, IAM, infrastructure security, and compliance.
Domain 3: Cloud Technology and Services	34%	Core services (Compute, Storage, Database, Network, Developer, ML, Integration).
Domain 4: Billing, Pricing, and Support	12%	Pricing models, cost management tools, and Support Plans.

Official reference checked: AWS Certified Cloud Practitioner (CLF-C02) Exam Guide and in-scope service list on AWS Certification documentation (checked 2026-06-21).

2. Cloud Concepts & Economics

Concept	Key Keywords / Definition	Exam Focus / Use Case
High Availability	No single point of failure; running in multiple AZs.	System remains operational even if hardware fails.
Fault Tolerance	System survives component failures without degradation.	Critical apps needing zero downtime.
Scalability	Grow/shrink system capacity based on workload.	Handling traffic spikes (vertical/horizontal scaling).
Elasticity	Automated scaling; match resource supply to demand.	Auto Scaling scale-out/scale-in based on CPU usage.
Agility	Reduce time to spin up resources from weeks to minutes.	Rapid experimentation and faster time-to-market.
Economy of Scale	Lower pay-as-you-go prices as AWS grows and buys bulk.	Massive cost savings compared to private data centers.
CapEx vs. OpEx	CapEx: Upfront physical assets. OpEx: Pay-as-you-go costs.	Cloud changes CapEx (buying servers) into OpEx (utility bills).
Total Cost of Ownership (TCO)	Compare on-premises vs. AWS costs. Includes both direct (hardware, labor) and indirect (power, cooling, space) costs.	Used to build a financial business case for migrating to the cloud.
Cloud Adoption Framework (CAF)	Structure to migrate workloads. 6 Perspectives:	Business, People, Governance (Business); Platform, Security, Operations (Technical).
CAF Transformation Phases	Iterative and incremental cloud transformation. 4 Phases:	Envision → Align → Launch → Scale (repeat).
Envision	Define business outcomes and how cloud delivers measurable value.	First phase: identify transformation opportunities and stakeholder alignment.
Align	Identify capability gaps across people, process, and technology.	Build a concrete roadmap, prioritize initiatives with stakeholders.
Launch	Execute pilot projects to prove value with real workloads.	Learn from early results, build organizational confidence.
Scale	Expand successful pilots to full production across the enterprise.	Operationalize best practices, drive broad adoption.
Deployment Models	Public: Fully AWS. Private: On-premises. Hybrid: Combined.	Use Direct Connect / VPN to connect Hybrid clouds.
Rightsizing	Match resource size/type to actual workload demand.	Cost optimization: reduce over-provisioned EC2/RDS/EBS capacity.
BYOL vs. Included License	Bring existing software licenses or use license-included AWS pricing.	Licensing strategy questions, especially Windows/SQL Server/Oracle workloads.
Automation Benefits	Repeatable, faster, lower-error operations using IaC/scripts.	Reduces manual work and improves operational consistency.

Migration Strategies (7 Rs)

Strategy	Meaning	Exam Keyword
Retire	Remove apps no longer needed.	Reduce scope and cost.
Retain	Keep on-premises for now.	Not ready, compliance, dependency constraints.
Rehost	Lift and shift to AWS.	Fast migration to EC2, Application Migration Service.
Replatform	Move with small optimizations.	Managed DB, Elastic Beanstalk, minimal code change.
Repurchase	Move to SaaS/product replacement.	Replace self-managed app with marketplace/SaaS option.
Refactor / Re-architect	Redesign for cloud-native benefits.	Serverless, microservices, highest effort/highest benefit.
Relocate	Move infrastructure without purchasing new hardware or rewriting.	VMware Cloud on AWS migration.

3. Shared Responsibility Model

AWS Responsibility (Security OF the Cloud)	Customer Responsibility (Security IN the Cloud)
Physical infrastructure, data centers, host virtualization OS.	Customer data, application code, identity management (IAM).
Global Infrastructure (Edge locations, AZs, Regions).	Guest Operating Systems (patching EC2 virtual machines).
Managed databases (RDS OS patching, hardware failures).	Firewall configurations (Security Groups, Network ACLs).
Physical security, security audits, server destruction.	Encryption settings (At-rest using KMS, In-transit using SSL/TLS).

Responsibility shifts by service: EC2 = customer patches guest OS and apps; RDS = AWS patches database host/OS, customer manages data/access/config; Lambda = AWS manages servers/runtime infrastructure, customer manages code, data, IAM, and configuration.

4. AWS Well-Architected Framework (6 Pillars)

Pillar	Key Design Principle	Exam Focus / Keyword
Operational Excellence	Perform operations as code, make frequent, small, reversible changes.	Continuous improvement, post-mortems, automating deployment.
Security	Implement a strong identity foundation, protect data at rest/transit.	Principle of Least Privilege, traceability (logging), encrypt everything.
Reliability	Automatically recover from failure, scale horizontally.	Test recovery procedures, Multi-AZ design, fault tolerance.
Performance Efficiency	Use serverless architectures, go global in minutes.	Democratizing advanced technologies, mechanical sympathy.
Cost Optimization	Measure overall efficiency, stop spending money on undifferentiated work.	Analyze spend, use managed services, pay-as-you-go matching.
Sustainability	Maximize utilization, minimize resources required.	Shared responsibility for environmental impact, reduction of waste.

5. Core Technology Services

Global Infrastructure

Component	Definition	Exam Focus
Region	Physical geographic area with multiple Availability Zones.	Choose for latency, cost, service availability, and data residency.
Availability Zone (AZ)	One or more discrete data centers in a Region.	Multi-AZ improves high availability; AZs avoid shared single points of failure.
Edge Location	CDN/security edge site used by services like CloudFront and Route 53.	Cache and serve content closer to users for lower latency.
Multi-Region	Workloads/data deployed across Regions.	Disaster recovery, business continuity, global latency, sovereignty.

Compute Services

Service	Type	Key Keywords / Characteristics	Primary Exam Use Case / Scenario
Amazon EC2	Virtual Servers	IaaS, resizable capacity, full OS access.	Legacy apps, custom software needing specific OS config.
EC2 Instance Types	General, compute, memory, storage, accelerated computing.	Pick based on workload profile: CPU-heavy, RAM-heavy, local storage, GPU/ML.
Elastic Load Balancing (ELB)	ALB/NLB/GWLB, distributes traffic.	Improves availability and scales traffic across multiple targets/AZs.
Auto Scaling	Automatic scale out/in.	Provides elasticity by matching capacity to demand.
AWS Lambda	Serverless	FaaS, event-driven, runs code max 15 mins.	Run code without managing servers; pay only for execution time.
Amazon ECS	Container Orchestration	AWS-native, runs Docker containers.	Running microservices in Docker at scale.
Amazon EKS	Container Orchestration	Managed Kubernetes standard.	Migrating existing Kubernetes workloads to AWS.
Amazon ECR	Container Registry	Secure storage and sharing of container images.	Private Docker registry to store container images for ECS or EKS.
AWS Fargate	Serverless Compute	Container-only compute; no EC2 to manage.	Serverless Docker containers for ECS or EKS.
Elastic Beanstalk	PaaS	Quick deploy, upload code, AWS handles infrastructure.	Developers who want to deploy web apps without configuring infrastructure.
Amazon Lightsail	Virtual Servers	VPS, simple, low cost, predictable monthly pricing.	Simple websites, blogs, test environments, small business apps.
AWS Batch	Compute	Runs batch jobs at any scale.	High-throughput, automated large-scale batch processing.
AWS Outposts	Hybrid Compute	Run native AWS services on-premises.	Extremely low latency or local data residency requirements.
AWS Wavelength	Edge Compute	Connects to 5G networks, ultra-low latency.	Mobile edge applications (video streaming, gaming, IoT).
AWS Local Zones	Edge Compute	Places compute/storage near large cities.	Running low-latency applications close to end-users.
VMware Cloud on AWS	Hybrid Compute	Runs VMware workloads natively on AWS.	Migrating on-premises VMware vSphere environments without modifying workloads.

Storage Services

Service	Type	Key Keywords / Characteristics	Primary Exam Use Case / Scenario
Amazon S3	Object Storage	Key-value store, 99.999999999% durability, static hosting.	Unstructured files, backups, static websites, data lake storage.
S3 Storage Classes	Standard, Intelligent-Tiering, Standard-IA, One Zone-IA, Glacier Instant/Flexible/Deep Archive.	Match retrieval frequency, availability needs, and archive cost.
S3 Lifecycle Policies	Automatically transition/expire objects.	Move older logs/backups to cheaper storage or delete after retention period.
S3 Glacier	Archive	Glacier Instant/Flexible/Deep Archive (up to 12h retrieval).	Long-term backup/compliance archiving at ultra-low cost.
Amazon EBS	Block Storage	Persistent volume, tied to single AZ, attached to EC2.	Database storage or boot volumes for individual EC2 instances.
EC2 Instance Store	Ephemeral block storage physically attached to host.	Temporary cache/scratch data; data lost if instance stops/terminates/fails.
Amazon EFS	File Storage	Shared network file system, Linux, scalable, multi-AZ.	Shared storage for multiple EC2 instances simultaneously.
Amazon FSx	File Storage	Native Windows (FSx for Windows) or Lustre (high-perf).	High-performance computing or Windows server migration.
Storage Gateway	Hybrid	File Gateway, Volume Gateway (Cached/Stored), Tape Gateway.	Connects on-premises environments to cloud storage.
AWS Backup	Backup	Managed, centralized, automated backup across services.	Automating backup policies for EBS, RDS, S3, etc.

Database Services

Service	Type	Key Keywords / Characteristics	Primary Exam Use Case / Scenario
Amazon RDS	Relational	Managed SQL (MySQL, PostgreSQL, SQL Server, Oracle).	OLTP applications, complex queries, traditional databases.
Amazon Aurora	Relational	Proprietary RDS, MySQL/PostgreSQL compatible, 3-5x performance.	High-throughput, self-healing relational database requirements.
Amazon DynamoDB	NoSQL	Key-value, serverless, single-digit millisecond latency.	Shopping carts, user profiles, high-speed read/write web apps.
Amazon ElastiCache	In-Memory	Redis or Memcached compatible.	Caching frequently read database queries to reduce load.
Amazon Redshift	Relational	Columnar data warehouse, OLAP.	Large-scale data analytics, business intelligence (BI) reports.
Amazon DocumentDB	NoSQL	Managed MongoDB compatible.	Storing JSON data structures and content management.
Amazon Neptune	Graph DB	Managed graph database.	Social networks, fraud detection, recommendation engines.
AWS DMS / SCT	Migration	DMS moves data; Schema Conversion Tool converts schemas.	Migrating homogeneous/heterogeneous databases to AWS managed databases.

Networking & Content Delivery

Service	Key Keywords / Characteristics	Primary Exam Use Case / Scenario
Amazon VPC	Isolated virtual network, Subnets, Internet Gateway, NAT Gateway.	Logically isolating your AWS resources in a private network.
Security Group	Stateful, instance-level firewall.	Controlling inbound and outbound traffic for individual EC2 instances.
Network ACL (NACL)	Stateless, subnet-level firewall.	Securing entire VPC subnets with explicit allow/deny rules.
Route 53	Global DNS, health checks, domain registration, latency routing.	Mapping domain names to IP addresses; routing users to closest resources.
CloudFront	Global Content Delivery Network (CDN), Edge Locations, caching.	Fast content delivery (images, videos, APIs) to users worldwide.
Direct Connect	Dedicated physical cable, bypasses the internet, secure, consistent.	Establishing a high-speed, private connection from on-prem to AWS.
AWS VPN	Site-to-Site (IPsec) VPN, Client VPN (OpenVPN endpoint).	Securely connecting on-premises data centers or remote employees to VPC.
Transit Gateway	Hub-and-spoke network router.	Connecting thousands of VPCs and on-premises networks together.
Global Accelerator	Optimizes IP routing using the AWS global network.	Improving global user latency by up to 60% via Static IPs.
API Gateway	Managed API creation, hosting, and protection.	Exposing serverless backends (Lambda) as REST/WebSocket APIs.
AWS PrivateLink	Private service connectivity through VPC endpoints.	Access supported AWS/SaaS services privately without public internet exposure.

Route 53 Routing Policies

Policy	Core Behavior / Meaning	Primary Exam Use Case / Keyword
Simple	Single resource, basic DNS query resolution.	Redirecting domain traffic to a single resource; no health check failover.
Weighted	Routes traffic to multiple resources in proportions you specify.	Blue/Green deployments, testing new versions on a fraction of users (e.g., 90%/10%).
Latency	Routes traffic to the AWS Region with the lowest network latency.	Providing the fastest response times for globally distributed users.
Failover	Active-Passive disaster recovery.	Automatically routing users to a secondary resource (backup) if the primary is unhealthy.
Geolocation	Routes traffic based on the geographic location of the user.	Localizing website language, managing regional compliance, or content restrictions.
Geoproximity	Routes traffic based on user/resource distance; can adjust boundaries using Bias.	Shifting the geographic region boundaries served by specific resources.
Multivalue Answer	Returns up to 8 healthy IP addresses in response to DNS queries.	Simple, decentralized, DNS-based load balancing with health checks.

Analytics Services

Service	Type	Key Keywords / Characteristics	Primary Exam Use Case / Scenario
Amazon Athena	Serverless Query	Query S3 files directly using standard SQL.	Querying logs/data stored in S3 without loading them into a database.
Amazon EMR	Big Data / Hadoop	Elastic MapReduce, Spark, Hadoop, HBase.	Running and scaling petabyte-scale distributed data processing frameworks.
Amazon MSK	Streaming / Kafka	Managed Apache Kafka cluster.	Building and running real-time streaming data applications.
Amazon Kinesis	Streaming	Real-time data ingestion, processing, and analysis.	Ingesting real-time application logs or IoT device sensor data.
AWS Glue	ETL Service	Extract, Transform, Load; serverless data catalog.	Discovering schemas and preparing data for database/analytics platforms.
Amazon QuickSight	Business Intelligence	Serverless BI dashboards, ML-powered visualizations.	Creating interactive business reports and dashboards for stakeholders.
Amazon OpenSearch Service	Search/analytics	Log analytics, search, observability.	Searching and analyzing application logs/events.

End User Computing, Business Applications, & IoT

Service	Type	Key Keywords / Characteristics	Primary Exam Use Case / Scenario
Amazon WorkSpaces	End User Computing	DaaS, persistent virtual desktops (Windows/Linux).	Providing employees with remote access to virtual office desktops.
Amazon AppStream 2.0	End User Computing	Non-persistent desktop application streaming.	Streaming high-performance desktop apps to a web browser on any device.
WorkSpaces Secure Browser	End User Computing	Secure browser access to internal websites/SaaS.	Browser-only workforce access without full virtual desktops.
Amazon Connect	Business App	Omnichannel cloud contact center, customer service helpdesk.	Setting up a scalable customer support phone system and chat center.
Amazon SES	Business App	Simple Email Service, marketing and transaction emails.	Automatically sending order confirmation or newsletter emails to customers.
AWS Amplify	Frontend & Mobile	Full-stack web/mobile app build tools and hosting.	Rapidly building and hosting mobile and web frontends on AWS.
AWS AppSync	Frontend & Mobile	Managed GraphQL APIs.	Connecting web/mobile apps to data sources with real-time sync.
AWS IoT Core	IoT	Secure device-to-cloud connection, message broker.	Connecting and routing messages from millions of IoT sensors to AWS.

6. Security, Identity, & Compliance

Core Security & Identity

Service	Key Keywords / Characteristics	Primary Exam Use Case / Scenario
AWS IAM	Users, Groups, Roles, Policies, MFA, Access Analyzer.	Control who can access what in your AWS account (Least Privilege).
IAM Policies	JSON permissions; AWS managed, customer managed, inline.	Define allowed/denied actions on resources.
IAM Roles	Temporary credentials; assumed by users/services/accounts.	Preferred access for EC2/Lambda and cross-account access.
IAM Identity Center	Single Sign-On (SSO).	Centrally manage SSO access to multiple AWS accounts.
AWS STS	Security Token Service, temporary credentials.	Granting temporary access to resources (e.g., federation, IAM role assumption).
Amazon Cognito	Sign-up, Sign-in, Guest Access.	Identity provider for web/mobile apps (Google/Facebook login).
AWS KMS	Envelope encryption, customer managed keys (CMKs), shared hardware.	Creating, deleting, and rotating cryptographic encryption keys.
AWS Secrets Manager	Database credentials, automatic rotation.	Securely storing and rotating sensitive API/DB keys.
AWS Directory Service	Managed Active Directory.	Integrates AWS resources with existing on-premises AD.
AWS Certificate Manager (ACM)	SSL/TLS certificates, free public certificates.	Provisioning, managing, and deploying SSL/TLS encryption certificates.
AWS License Manager	BYOL license management, Dedicated Hosts, usage tracking.	Governing and tracking software license usage (Windows/SQL Server) to avoid overage penalties.

Root user exam tip: Protect with MFA, avoid daily use, do not create root access keys, and use IAM/admin roles for routine tasks. Some account-level tasks require root, such as changing root credentials, closing the account, and certain tax/support plan/account settings.

Security Protection & Auditing

Service	Key Keywords / Characteristics	Primary Exam Use Case / Scenario
AWS WAF	Layer 7 Web Application Firewall, SQL injection, XSS protection.	Blocking malicious web attacks targeting HTTP/HTTPS apps.
AWS Shield	Layer 3/4 DDoS protection, Standard (free) and Advanced.	Protecting applications from massive Distributed Denial of Service attacks.
AWS Firewall Manager	Centralized security rules across accounts.	Configuring and deploying firewall rules (WAF, Shield, Security Groups) for AWS Organizations.
Amazon GuardDuty	Threat detection, Machine Learning, continuously monitors logs.	Finding malicious activity (e.g., bitcoin mining, compromised instances).
Amazon Inspector	Vulnerability scanner, EC2, ECR container images, Lambda.	Scanning application software packages for known security exposures.
Amazon Macie	PII discovery, S3 buckets, Machine Learning.	Identifying and alerting on sensitive data (e.g., credit cards, SSNs).
AWS Artifact	Compliance portal, ISO/PCI/SOC reports.	Downloading official AWS compliance documents for audits.
AWS Security Hub	Security posture management, single dashboard.	Consolidated view of security alerts across GuardDuty, Inspector, Macie.
Amazon Detective	Security investigation.	Investigating and finding the root cause of security anomalies.
AWS CloudHSM	Dedicated hardware security module (FIPS 140-2 Level 3).	Managing encryption keys using dedicated cryptographic hardware in AWS.

7. Management, Governance, & Billing

Management & Monitoring

Service	Key Keywords / Characteristics	Primary Exam Use Case / Scenario
Amazon CloudWatch	Performance metrics, logs, alarms, dashboards.	Monitoring resource CPU utilization, setting alarms for high usage.
AWS CloudTrail	API auditing, user history, "Who did what, when, and where."	Reviewing which user deleted an S3 bucket or changed a route table.
AWS Config	Compliance auditing, configuration history.	Tracking changes to security group rules over time for compliance.
AWS Systems Manager	SSM, Run Command, Patch Manager, Session Manager.	Executing shell scripts or applying OS patches to hundreds of EC2s.
AWS Organizations	Multi-account management, OUs, Service Control Policies (SCPs).	Centrally applying security guardrails and consolidating bills.
AWS Control Tower	Automated landing zone setup, multi-account governance.	Setting up a secure, compliant multi-account environment.
AWS Trusted Advisor	Best practices advisor: Cost, Security, Reliability, Performance, Limits.	Finding idle EC2 instances or public S3 buckets.
Well-Architected Tool	Architecture review against 6 pillars.	Evaluating workload architectures to ensure they align with best practices.
Compute Optimizer	Machine learning analysis of usage.	Recommending optimal EC2/Lambda sizes to save money/boost performance.
AWS Health Dashboard	Two views: Service Health Dashboard (global status of all AWS services) and Personal Health Dashboard (personalized alerts for your account/resources).	Checking general AWS outages vs. getting specific alerts about services that affect your workloads.
AWS Service Catalog	Managed catalog of approved IT services.	Governing resource creation by allowing users to launch only pre-approved, compliant configurations.
AWS CloudFormation	Infrastructure as Code (IaC), declarative templates (JSON/YAML), drift detection, StackSets.	Provisioning and managing AWS resources consistently using repeatable, version-controlled templates.
AWS Resource Groups & Tag Editor	Organize and manage resources by tags; bulk tag editing across services/Regions.	Grouping EC2 instances by `Environment: Production` tag for centralized management and cost tracking.
AWS Service Quotas	View and manage service limits (formerly Service Limits); request quota increases.	Checking current EC2 instance count limits and requesting an increase before launching large workloads.

Billing & Cost Management

Service / Tool	Primary Purpose	Key Exam Scenario
AWS Billing Dashboard	Visual monthly invoice, payments.	High-level tracking of current month costs.
AWS Cost Explorer	Historic cost visualization, forecasting.	Identifying spend trends and predicting future cloud bills.
AWS Budgets	Custom cost/usage alerts.	Triggering email notifications when costs exceed 80% of budget.
Cost & Usage Report (CUR)	Most granular raw data (S3 export).	Deep dive cost analysis with Athena/QuickSight.
AWS Pricing Calculator	Estimate infrastructure costs.	Planning costs before deploying an application to AWS.
Cost Allocation Tags	Key-value tagging (`Environment: Production`).	Organizing and categorizing resource costs by department/project.
Consolidated Billing	Single bill across AWS Organizations accounts.	Volume discounts and centralized payment while keeping account isolation.
Savings Plans	1- or 3-year spend commitment.	Flexible compute savings across EC2, Fargate, and Lambda usage.
Reserved Instances	1- or 3-year instance commitment.	Steady-state EC2/RDS usage; can share benefits across Organizations accounts.
Spot Instances	Unused EC2 capacity with interruption risk.	Fault-tolerant, flexible workloads such as batch jobs.
Dedicated Hosts / Instances	Dedicated physical server or dedicated instance placement.	Compliance, server-bound licenses, physical isolation requirements.
Capacity Reservations	Reserve EC2 capacity in a specific AZ.	Ensure capacity availability without necessarily getting a billing discount.
Data Transfer Pricing	Inbound usually free; outbound and cross-Region usually charged.	Watch egress and inter-Region traffic costs.
AWS Free Tier	Free plan: Up to $200 credits over 6 months (new accounts). Always Free: 30+ services within monthly limits (both plans). Short-term trials: Time-limited trials for select paid services.	Starting hands-on experience with AWS at no cost; practicing core services without incurring charges.
AWS Marketplace	Digital catalog of third-party software.	Finding, buying, and deploying software that runs on AWS with unified billing.
AWS Cost Anomaly Detection	Machine Learning cost monitors.	Automatically detecting and alerting on anomalous or unexpected billing activity.
AWS Billing Conductor	Custom pro forma billing.	Customizing billing parameters and sharing billing views with business partners/clients.

8. Integration, Developer, & Machine Learning

Application Integration

Service	Architecture	Communication Model	Primary Exam Keyword / Scenario
Amazon SQS (Simple Queue Service)	Message Queue	Pull-based (Consumers pull messages)	Decoupling components; processing asynchronous transactions.
Amazon SNS (Simple Notification Service)	Pub/Sub Topic	Push-based (Fan-out pattern)	Broadcasting single notifications (Email, SMS) to multiple targets.
EventBridge	Serverless Event Bus	Push-based (Event routing & scheduling)	Routes real-time events from AWS services, SaaS apps, and custom apps using pattern matching rules; supports scheduled events via EventBridge Scheduler; sends to multiple targets (Lambda, SNS, SQS, Kinesis, Firehose, etc.); event transformation and filtering capabilities.
Step Functions	State Machine Workflow	Visual orchestration	Coordinating sequential multi-step serverless tasks (Lambda).
AWS SWF (Simple Work Flows)	Coordinated Workflow	Task-based (Deciders & Workers)	Orchestrating complex, long-running, programmatic workflows; human approval steps and custom retry/compensation logic.

Developer Tools

Service	Primary Function	Primary Exam Use Case
AWS CLI	Command Line Interface	Control AWS services using text commands in a terminal.
AWS CloudShell	Browser-based shell	Executing CLI scripts directly from the AWS Console without installs.
AWS Cloud9	Browser-based IDE	Writing and debugging code collaboratively in the cloud.
AWS CodeCommit	Git Repository	Hosting private Git repositories natively in AWS.
AWS CodeBuild	Build & Test	Compiling source code and running automated testing scripts.
AWS CodeDeploy	Code Deployment	Automating application updates onto EC2, ECS, or Lambda.
AWS CodePipeline	CI/CD Orchestration	Designing and managing the workflow from commit to deploy.
AWS X-Ray	Distributed tracing & debugging	Analyzing and debugging production, distributed serverless applications (visualizing service maps).
AWS CDK	Infrastructure as Code using programming languages	Defining repeatable AWS infrastructure with code.

Machine Learning & AI (No ML expertise required)

Service	Primary Function / Keyword	Primary Exam Scenario
Amazon SageMaker	Build, Train, Deploy custom ML.	Fully custom machine learning modeling workbench.
Amazon Bedrock	Generative AI, Foundation Models.	Building generative AI apps using API-based foundation models.
Amazon Lex	Conversational chatbots (Alexa tech).	Creating customer service chatbots for websites/apps.
Amazon Rekognition	Image & Video analysis.	Facial recognition, locating unsafe content, labeling objects in photos.
Amazon Transcribe	Speech-to-Text.	Generating text transcripts from audio recordings.
Amazon Polly	Text-to-Speech.	Converting written text into lifelike spoken voice.
Amazon Translate	Language translation.	Localizing application text content into multiple languages.
Amazon Comprehend	Natural Language Processing (NLP).	Analyzing customer feedback text for sentiment (Positive/Negative).
Amazon Textract	Document OCR + data extraction.	Extracting table structures and form data from scanned PDF invoices.
Amazon Kendra	Intelligent Document Search.	Finding answers across thousands of PDF and Word files.

9. Migration & Support

Migration & Transfer

Service	Key Keywords / Characteristics	Primary Exam Use Case / Scenario
Database Migration Service (DMS)	Minimal downtime, homogeneous/heterogeneous.	Migrating database to AWS while source remains operational.
Migration Hub	Single tracking dashboard.	Monitoring progress of application migrations across multiple tools.
Application Discovery Service	Discover inventory, profiling resources.	Cataloging on-premises server configurations to plan migrations.
Application Migration Service (MGN)	Lift-and-shift server replication.	Rehosting virtual/physical servers onto EC2 instances.
AWS Snow Family	Physical data transport. Snowcone < Snowball < Snowmobile.	Migrating massive datasets (TB/PB-scale) where internet is too slow.
AWS DataSync	Online automation, sync over WAN.	Synchronizing local NAS storage data to S3 or EFS on a schedule.
AWS Transfer Family	SFTP, FTPS, FTP wrapper.	Exposing S3 or EFS storage directly to users via SFTP protocol.

AWS Support Plans

Tier	Technical Support Response Times	Trusted Advisor Checks	Key Feature
Basic (Free)	No technical cases; account/billing support only	Core checks	24/7 customer service, docs, whitepapers, AWS re:Post, AWS Health Dashboard.
Developer	< 24h (general), < 12h (system impaired)	7 Core checks	Single contact, Email support (biz hours).
Business	< 4h (system impaired), < 1h (production down)	Full checks	Unlimited contacts, 24/7 Phone/Email/Chat.
Enterprise On-Ramp	Production/business-critical response faster than Business	Full checks	Pool of Technical Account Managers, concierge-style guidance for growing production workloads.
Enterprise	< 15m (business critical down)	Full checks	Technical Account Manager (TAM), Concierge Support.

Technical Resources & Partners

Resource	What It Is	Exam Use Case
AWS Documentation / Whitepapers / Blogs	Official technical guidance and best practices.	Find authoritative service details and architecture/security guidance.
AWS Knowledge Center	Troubleshooting articles and answers.	Resolve common AWS account/service issues.
AWS re:Post	Community Q&A and expert answers.	Ask/search technical questions outside a support case.
AWS Prescriptive Guidance	Migration, modernization, and architecture patterns.	Follow recommended implementation patterns.
AWS Professional Services	Paid consulting from AWS experts.	Hands-on help for large migrations or complex transformations.
AWS Solutions Architects	AWS technical advisors.	Architecture guidance during planning and adoption.
AWS Partner Network (APN)	AWS partner ecosystem: ISVs, system integrators, consultants.	Find third-party software/services and partner expertise.
Trust and Safety Team	Abuse reporting and investigation.	Report abuse of AWS resources such as spam, malware, or attacks.

10. Service Level Agreements (SLAs)

Service	SLA Commitment	Key Exam Detail
Amazon Route 53	100% (Hosted Zones)	Only AWS service with a 100% SLA; < 100% → ≥ 99.99% yields 10% credit.
Amazon EC2 (Multi-AZ)	99.99%	Region-Level SLA: instances deployed across ≥ 2 AZs.
Amazon EC2 (Single Instance)	99.5%	Instance-Level SLA: individual EC2 instance uptime.
Amazon S3 (Standard)	99.9%	S3 Standard, Express One Zone, Glacier Flexible, Glacier Deep Archive.
Amazon S3 (IA / Intelligent-Tiering)	99.0%	S3 Standard-IA, One Zone-IA, Intelligent-Tiering, Glacier Instant Retrieval.
Amazon RDS (Multi-AZ)	99.95%	Multi-AZ DB Instance or DB Cluster deployment.
Amazon RDS (Single DB Instance)	99.5%	Single-DB Instance deployment.

SLA Credit Process: Submit a support case within the second billing cycle after the incident. Credits apply as future service credits (not refunds), minimum $1 USD.

SLA Exclusions (common): Force majeure, customer actions/inactions, customer equipment/software, free-tier services, and services in beta/preview.

Python compare script 7

Sachin Varghese — Thu, 19 Jun 2025 12:42:03 +0000

import os
import difflib
import re

def extract_method_names(file_path):
    """Extract method names from a Java file."""
    method_names = []
    with open(file_path, 'r', encoding='utf-8', errors='ignore') as file:
        content = file.read()
        # Java method pattern: return_type methodName(parameters)
        # This regex looks for method declarations while avoiding comments and strings
        pattern = r'(?:public|protected|private|static|\s)+[\w\<\>\[\]]+\s+([\w]+)\s*\([\w\s,\[\]<>\.\]]*\)\s*(?:\{|[^;])'
        methods = re.findall(pattern, content)
        method_names.extend(methods)
    return method_names

def get_package_methods(repo_path):
    """Get all method names from Java files in a repository."""
    package_methods = {}
    for root, _, files in os.walk(repo_path):
        for file in files:
            if file.endswith('.java'):
                file_path = os.path.join(root, file)
                methods = extract_method_names(file_path)
                package_name = os.path.relpath(root, repo_path).replace(os.sep, '.')
                if package_name in package_methods:
                    package_methods[package_name].extend(methods)
                else:
                    package_methods[package_name] = methods
    return package_methods

def compare_methods(repo1_methods, repo2_methods):
    """Compare method names from two repositories and find methods unique to each."""
    comparison_results = {}
    all_packages = set(repo1_methods.keys()) | set(repo2_methods.keys())

    for package in all_packages:
        repo1_methods_set = set(repo1_methods.get(package, []))
        repo2_methods_set = set(repo2_methods.get(package, []))

        only_in_repo1 = repo1_methods_set - repo2_methods_set
        only_in_repo2 = repo2_methods_set - repo1_methods_set

        if only_in_repo1 or only_in_repo2:
            comparison_results[package] = {
                'only_in_repo1': sorted(list(only_in_repo1)),
                'only_in_repo2': sorted(list(only_in_repo2))
            }

    return comparison_results

def main(repo1_path, repo2_path, output_file='method_comparison_results.txt'):
    if not repo1_path or not repo2_path:
        print("Error: Repository paths must be specified")
        return

    repo1_methods = get_package_methods(repo1_path)
    repo2_methods = get_package_methods(repo2_path)
    comparison_results = compare_methods(repo1_methods, repo2_methods)

    # Write results to a text file
    with open(output_file, 'w', encoding='utf-8') as f:
        f.write("Method Comparison Results\n")
        f.write(f"Repo 1: {repo1_path}\n")
        f.write(f"Repo 2: {repo2_path}\n")
        f.write("=" * 80 + "\n\n")

        for package, result in comparison_results.items():
            f.write(f"InComparing methods in package: {package}\n")
            f.write("-" * 50 + "\n")
            if result['only_in_repo1']:
                f.write("Methods only in Repo1:\n")
                for method in result['only_in_repo1']:
                    f.write(f"  {method}\n")
                f.write("\n")
            if result['only_in_repo2']:
                f.write("Methods only in Repo2:\n")
                for method in result['only_in_repo2']:
                    f.write(f"  {method}\n")
                f.write("\n")

    print(f"Comparison results saved to {output_file}")

if __name__ == "__main__":
    # Replace these with the actual paths to your Java repositories
    repo1_path = '/path/to/your/first/java/repo'
    repo2_path = '/path/to/your/second/java/repo'

    # Optional: Specify the output file path
    output_file = 'java_method_comparison.txt'

    main(repo1_path, repo2_path, output_file)

Python compare script 6

Sachin Varghese — Thu, 19 Jun 2025 12:26:48 +0000

import os
import difflib
import re

def extract_method_names(file_path):
    """Extract method names from a Java file."""
    with open(file_path, 'r', encoding='utf-8', errors='ignore') as file:
        content = file.read()
    # Java method pattern: return_type methodName(parameters)
    # This regex looks for method declarations while avoiding comments and strings
    pattern = r'(?:(?:public|protected|private|static|s)[\w\s]*\s+[\w<>\[\]]+\s+([\w]+)\s*\()'
    method_names = re.findall(pattern, content)
    return method_names

def get_package_methods(repo_path):
    """Get all method names from Java files in a repository."""
    package_methods = {}
    for root, _, files in os.walk(repo_path):
        for file in files:
            if file.endswith('.java'):
                file_path = os.path.join(root, file)
                methods = extract_method_names(file_path)
                package_name = os.path.relpath(root, repo_path).replace(os.sep, '.')
                if package_name in package_methods:
                    package_methods[package_name].extend(methods)
                else:
                    package_methods[package_name] = methods
    return package_methods

def compare_methods(repo1_methods, repo2_methods):
    """Compare method names from two repositories and find methods unique to each."""
    comparison_results = {}
    all_packages = set(repo1_methods.keys()) | set(repo2_methods.keys())
    for package in all_packages:
        repo1_methods_set = set(repo1_methods.get(package, []))
        repo2_methods_set = set(repo2_methods.get(package, []))
        only_in_repo1 = repo1_methods_set - repo2_methods_set
        only_in_repo2 = repo2_methods_set - repo1_methods_set
        if only_in_repo1 or only_in_repo2:
            comparison_results[package] = {
                'only_in_repo1': sorted(list(only_in_repo1)),
                'only_in_repo2': sorted(list(only_in_repo2))
            }
    return comparison_results

def main(repo1_path, repo2_path):
    if not repo1_path or not repo2_path:
        print("Error: Repository paths must be specified")
        return
    repo1_methods = get_package_methods(repo1_path)
    repo2_methods = get_package_methods(repo2_path)
    comparison_results = compare_methods(repo1_methods, repo2_methods)
    for package, result in comparison_results.items():
        print(f"\nComparing methods in package: {package}")
        if result['only_in_repo1']:
            print("Methods only in Repo1:")
            for method in result['only_in_repo1']:
                print(f"+ {method}")
        if result['only_in_repo2']:
            print("Methods only in Repo2:")
            for method in result['only_in_repo2']:
                print(f"- {method}")

if __name__ == "__main__":
    # Replace these paths with the actual paths to your repositories
    repo1_path = ''
    repo2_path = ''
    main(repo1_path, repo2_path)

Py compare script 5

Sachin Varghese — Thu, 19 Jun 2025 08:51:32 +0000

import os
import re
import difflib

def extract_java_method_names(file_path):
    """Extract method names from a Java file."""
    method_names = []
    method_pattern = re.compile(r'^(\bpublic\b|\bprotected\b|\bprivate\b|\bstatic\b\s*)*(?=\w+\s+\w+\s*\([^)]*\))\s*\w+\s*\([^)]*\)\s*{?')
    with open(file_path, 'r', encoding='utf-8', errors='ignore') as file:
        for line in file:
            line = line.strip()
            match = method_pattern.match(line)
            if match:
                method_name = match.group(2)
                method_names.append(method_name)
    return method_names

def get_all_method_names(repo_path):
    """Get all method names from Java files in a repository."""
    package_methods = {}
    for root, _, files in os.walk(repo_path):
        for file in files:
            if file.endswith('.java'):
                file_path = os.path.join(root, file)
                methods = extract_java_method_names(file_path)
                package_name = os.path.relpath(root, repo_path).replace(os.sep, '.')
                if package_name not in package_methods:
                    package_methods[package_name] = []
                package_methods[package_name].extend(methods)
    return package_methods

def compare_methods(repo1_methods, repo2_methods):
    """Compare method names from two repositories."""
    comparison_results = {}
    all_packages = set(repo1_methods.keys()).union(repo2_methods.keys())
    for package in all_packages:
        methods1 = set(repo1_methods.get(package, []))
        methods2 = set(repo2_methods.get(package, []))
        only_in_repo1 = methods1 - methods2
        only_in_repo2 = methods2 - methods1
        if only_in_repo1 or only_in_repo2:
            comparison_results[package] = {
                'only_in_repo1': sorted(only_in_repo1),
                'only_in_repo2': sorted(only_in_repo2)
            }
    return comparison_results

def main(repo1_path, repo2_path):
    repo1_methods = get_all_method_names(repo1_path)
    repo2_methods = get_all_method_names(repo2_path)
    comparison_results = compare_methods(repo1_methods, repo2_methods)
    for package, diffs in comparison_results.items():
        print(f"\nComparing methods in package: {package}")
        if diffs['only_in_repo1']:
            print("Methods only in Repo1:")
            for method in diffs['only_in_repo1']:
                print(f"  - {method}")
        if diffs['only_in_repo2']:
            print("Methods only in Repo2:")
            for method in diffs['only_in_repo2']:
                print(f"  - {method}")

if __name__ == "__main__":
    # Replace these paths with the actual paths to your Java repositories
    repo1_path = ""
    repo2_path = ""
    main(repo1_path, repo2_path)

Py compare script 4

Sachin Varghese — Wed, 18 Jun 2025 15:15:08 +0000

import os
import difflib


def extract_method_names(file_path):
    """Extract method names from a Python file."""
    method_names = []
    with open(file_path, 'r') as file:
        for line in file:
            line = line.strip()
            if line.startswith('def '):
                # Extract method name
                method_name = line.split('(')[0][4:]  # Get the name after 'def'
                method_names.append(method_name)
    return method_names


def get_package_methods(repo_path):
    """Get all method names from Python files in a repository."""
    package_methods = {}
    for root, _, files in os.walk(repo_path):
        for file in files:
            if file.endswith('.py'):
                file_path = os.path.join(root, file)
                methods = extract_method_names(file_path)
                package_name = os.path.relpath(root, repo_path).replace(os.sep, '.')
                package_methods[package_name] = methods
    return package_methods


def compare_methods(repo1_methods, repo2_methods):
    """Compare method names from two repositories."""
    comparison_results = {}
    for package, methods in repo1_methods.items():
        if package in repo2_methods:
            repo2_method_names = repo2_methods[package]
            # Compare method names
            diff = difflib.unified_diff(
                methods,
                repo2_method_names,
                lineterm='',
                fromfile='Repo1',
                tofile='Repo2'
            )
            comparison_results[package] = list(diff)
    return comparison_results


def main(repo1_path, repo2_path):
    repo1_methods = get_package_methods(repo1_path)
    repo2_methods = get_package_methods(repo2_path)
    comparison_results = compare_methods(repo1_methods, repo2_methods)

    for package, diffs in comparison_results.items():
        print(f"Comparing methods in package: {package}")
        for line in diffs:
            print(line)


if __name__ == "__main__":
    # Replace these paths with the actual paths to your repositories
    repo1_path = '/path/to/repo1'
    repo2_path = '/path/to/repo2'
    main(repo1_path, repo2_path)

Py compare script 3

Sachin Varghese — Wed, 18 Jun 2025 15:14:20 +0000

`import os
import difflib

def extract_method_names(file_path):
"""Extract method names from a Python file."""
method_names = []
with open(file_path, 'r') as file:
for line in file:
line = line.strip()
if line.startswith('def '):
# Extract method name
method_name = line.split('(')[0][4:] # Get the name after 'def'
method_names.append(method_name)
return method_names

def get_package_methods(repo_path):
"""Get all method names from Python files in a repository."""
package_methods = {}
for root, _, files in os.walk(repo_path):
for file in files:
if file.endswith('.py'):
file_path = os.path.join(root, file)
methods = extract_method_names(file_path)
package_name = os.path.relpath(root, repo_path).replace(os.sep, '.')
package_methods[package_name] = methods
return package_methods

def compare_methods(repo1_methods, repo2_methods):
"""Compare method names from two repositories."""
comparison_results = {}
for package, methods in repo1_methods.items():
if package in repo2_methods:
repo2_method_names = repo2_methods[package]
# Compare method names
diff = difflib.unified_diff(
methods,
repo2_method_names,
lineterm='',
fromfile='Repo1',
tofile='Repo2'
)
comparison_results[package] = list(diff)
return comparison_results

def main(repo1_path, repo2_path):
repo1_methods = get_package_methods(repo1_path)
repo2_methods = get_package_methods(repo2_path)
comparison_results = compare_methods(repo1_methods, repo2_methods)

for package, diffs in comparison_results.items():
    print(f"Comparing methods in package: {package}")
    for line in diffs:
        print(line)

if name == "main":
# Replace these paths with the actual paths to your repositories
repo1_path = '/path/to/repo1'
repo2_path = '/path/to/repo2'
main(repo1_path, repo2_path)`

Py compare script 2

Sachin Varghese — Wed, 18 Jun 2025 15:13:14 +0000

import os
import difflib

def extract_method_names(file_path):
"""Extract method names from a Python file."""
method_names = []
with open(file_path, 'r') as file:
for line in file:
line = line.strip()
if line.startswith('def '): pc
# Extract method name
method_name = line.split('(')[0][4:] # Get the name after 'def'
method_names.append(method_name)
return method_names

def compare_methods(repo1_methods, repo2_methods):
"""Compare method names from two repositories."""
comparison_results = {}
for package, methods in repo1_methods.items():
if package in repo2_methods:
repo2_method_names = repo2_methods[package]
# Compare method names
diff = difflib.unified_diff(methods, repo2_method_names, lineterm='', fromfile='Repo1', tofile='Repo2')
comparison_results[package] = list(diff)
return comparison_results

if name == "main":
# Replace these paths with the actual paths to your repositories
repo1_path = '/path/to/repo1'
repo2_path = '/path/to/repo2'
main(repo1_path, repo2_path)

Py script to compare repo

Sachin Varghese — Wed, 18 Jun 2025 14:41:36 +0000

import os
import difflib

def compare_methods(repo1_methods, repo2_methods):
"""Compare method names from two repositories."""
comparison_results = {}
for package, methods in repo1_methods.items():
if package in repo2_methods:
repo2_method_names = repo2_methods[package]
# Compare method names
diff = difflib.unified_diff(methods, repo2_method_names, lineterm='', fromfile='Repo1', tofile='Repo2')
comparison_results[package] = list(diff)
return comparison_results

def main(repo1_path, repo2_path):
repo1_methods = get_package_methods(repo1_path)
repo2_methods = get_package_methods(repo2_path)
comparison_results = compare_methods(repo1_methods, repo2_methods)
for package, diffs in comparison_results.items():
print(f"Comparing methods in package: {package}")
for line in diffs:
print(line)
if name == "main":
# Replace these paths with the actual paths to your repositories
repo1_path = ''
repo2_path = ''
main(repo1_path, repo2_path)

Keep awake - ps script

Sachin Varghese — Sat, 14 Jun 2025 09:35:34 +0000

Thanks for the image — the issue is clear!

❌ Error Explained:

Method invocation failed because [System.Object[]] does not contain a method named 'op_Addition'

This means you're trying to add to a value that’s not a point object, probably because .Position is returning an array or something not cast correctly.

✅ Fix — Cast the mouse position correctly before doing math

Update your script like this:

Add-Type -AssemblyName System.Windows.Forms
Add-Type -AssemblyName System.Drawing

Clear-Host
Write-Host "...Lock screen avoider with mouse movement..."

$sleep = 30 # seconds

while ($true) {
# Get and cast current mouse position properly
$pos = [System.Windows.Forms.Cursor]::Position
$x = $pos.X
$y = $pos.Y

# Move slightly and back
[System.Windows.Forms.Cursor]::Position = New-Object System.Drawing.Point($x + 1, $y)
Start-Sleep -Milliseconds 100
[System.Windows.Forms.Cursor]::Position = New-Object System.Drawing.Point($x, $y)

Write-Host "Mouse moved to prevent lock. Waiting $sleep seconds..."
Start-Sleep -Seconds $sleep

}

✅ Why this works:

$pos.X and $pos.Y safely extract coordinates as numbers.

New-Object System.Drawing.Point(...) constructs a valid point object for cursor movement.

Avoids the invalid attempt to do math on an object array.

Let me know if you'd like:

To run it in the background silently

A scheduled task version

A .ps1 file ready to download and use

Letter01

Sachin Varghese — Wed, 23 Apr 2025 14:11:02 +0000

Here's a professional and concise email you can send to your Business Analyst to be forwarded to the client:

Subject: Clarification Required Regarding RAALL002 Script and OB Episode Error

Hi [Business Analyst's Name],

Could you please check with the client on the following points regarding the RAALL002 script and the OB Episode record error?

We are unable to replicate the issue in our environment (JEDI 2.0); it is working as expected.

RAALL002 is a pre-save rule, so it will be triggered first during the save and close of the OB Episode record. Please confirm if there are any other pre-save rules defined that may be executing before this one. (Specifically, any custom execution sequence implemented for OB Episode save/close in CareManage.)

Typically, this error occurs when attempting to insert/update the OB Episode record in the database. However, the RAALL002 script does not involve any direct database interaction, which could otherwise cause such an error.

Kindly let us know once the client confirms these details so we can proceed accordingly.

Best regards,
[Your Full Name]
[Your Job Title]
[Your Company Name]

Let me know if you'd like to add anything specific like a ticket number or reference screenshots/logs.

JSON to CSV - 01

Sachin Varghese — Wed, 23 Apr 2025 07:01:41 +0000

I'll write a Python script to convert your large JSON file to CSV format. Based on the snippet you've shared, I can see it's a complex JSON with nested structures that needs careful handling for efficient processing.

Here's a Python script optimized for handling large JSON files:

import json
import csv
import os
import gc
from itertools import islice
import ijson  # You'll need to install this: pip install ijson

def flatten_json(y):
    """Flatten a nested json structure into a single level dictionary"""
    out = {}

    def flatten(x, name=''):
        if type(x) is dict:
            for a in x:
                flatten(x[a], name + a + '_')
        elif type(x) is list:
            for i, a in enumerate(x):
                flatten(a, name + str(i) + '_')
        else:
            out[name[:-1]] = x

    flatten(y)
    return out

def process_json_to_csv(json_file, csv_file, batch_size=1000):
    """Process large JSON file to CSV with batching for memory efficiency"""
    # Use ijson to parse the JSON file as a stream
    with open(json_file, 'rb') as f:
        # For an array of objects
        objects = ijson.items(f, 'item')

        # Process the first object to get the headers
        first_batch = list(islice(objects, batch_size))
        if not first_batch:
            print("No data found in JSON file")
            return

        # Flatten the first batch to get all possible headers
        flattened_batch = [flatten_json(obj) for obj in first_batch]
        all_headers = set()
        for item in flattened_batch:
            all_headers.update(item.keys())

        headers = sorted(list(all_headers))

        # Write to CSV
        with open(csv_file, 'w', newline='', encoding='utf-8') as csvfile:
            writer = csv.DictWriter(csvfile, fieldnames=headers)
            writer.writeheader()

            # Write the first batch
            for item in flattened_batch:
                writer.writerow(item)

            # Process the rest of the file in batches
            batch = []
            for obj in objects:
                batch.append(flatten_json(obj))

                if len(batch) >= batch_size:
                    writer.writerows(batch)
                    batch = []
                    gc.collect()  # Force garbage collection

            # Write any remaining items
            if batch:
                writer.writerows(batch)

def main():
    json_file = 'your_large_file.json'  # Replace with your JSON file path
    csv_file = 'output.csv'

    print(f"Converting {json_file} to {csv_file}...")
    process_json_to_csv(json_file, csv_file)
    print(f"Conversion complete! Output saved to {csv_file}")
    print(f"CSV file size: {os.path.getsize(csv_file) / (1024 * 1024):.2f} MB")

if __name__ == "__main__":
    main()

Key optimization techniques used in this script:

Stream parsing with ijson: Instead of loading the entire JSON into memory, ijson parses the file as a stream, which is crucial for large files.
Batch processing: Processing the data in batches (1000 records at a time) to control memory usage.
Flattening nested structures: The flatten_json function converts nested JSON objects into a flat structure suitable for CSV.
Garbage collection: Explicitly calling gc.collect() after processing each batch to free up memory.
Optimized CSV writing: Using DictWriter for efficient CSV creation.

You may need to adjust the script based on the exact structure of your JSON. If your JSON is not an array at the root level but has a different structure, you'll need to modify the ijson path in ijson.items(f, 'item') to match your structure.

Also, if your JSON has deeply nested structures, the flattened column names could get very long. You might want to customize the flatten_json function to handle the naming scheme differently if needed.

Would you like me to explain any specific part of this script in more detail?

Py txt to CSV - 02

Sachin Varghese — Tue, 22 Apr 2025 17:03:07 +0000

The provided code is a well-structured script for converting a large text file containing concatenated JSON objects into a CSV file using Python, pandas, and a streaming approach to handle memory efficiently. Below is a detailed review of the code, including its strengths, potential issues, and suggestions for improvement.

Strengths

Memory Efficiency with Streaming:
- The stream_json_objects function reads the file in chunks (64KB) and processes JSON objects incrementally, avoiding loading the entire file into memory. This is critical for handling large files.
- The use of a buffer and regex (separator_re) to split concatenated JSON objects is robust for handling objects split across chunk boundaries.
Batch Processing:
- The json_to_csv_optimized function processes JSON objects in batches (batch_size=10000), normalizing them into a pandas DataFrame and writing to CSV incrementally. This balances memory usage and performance.
Error Handling:
- The code includes comprehensive error handling for:
  - File not found (FileNotFoundError).
  - JSON parsing errors (json.JSONDecodeError).
  - General exceptions during file processing or normalization.
- Warnings are printed with useful context (e.g., buffer snippets) to aid debugging.
Column Consistency:
- The script ensures consistent column headers across batches by determining columns from the first valid batch and reindexing subsequent batches to match. This prevents misaligned CSV output.
Clean File Management:
- Removes the default Excel file (result.xlsx) if it exists, avoiding confusion from previous runs.
- Uses 'a' (append) mode for CSV writing after the header is written, ensuring efficient file operations.
Modularity:
- The code is split into two clear functions: stream_json_objects for parsing and json_to_csv_optimized for conversion, making it reusable and maintainable.

Potential Issues and Suggestions

While the code is robust, there are a few areas where it could be improved or where edge cases might cause issues.

1. Edge Case: Malformed JSON Objects

Issue: If the input file contains malformed JSON objects or unexpected separators (e.g., }{ inside a string value), the stream_json_objects function skips the invalid segment but might lose data. The warning message helps, but it doesn’t allow for recovery of partial valid objects.

Suggestion: Add an option to log skipped segments to a separate file for post-processing or manual inspection. For example:

 def stream_json_objects(filepath, log_file=None):
     if log_file:
         log = open(log_file, 'w', encoding='utf-8')
     try:
         # ... existing code ...
         except json.JSONDecodeError as e:
             print(f"Warning: JSONDecodeError at position {match.start()+1}: {e}")
             if log_file:
                 log.write(f"Error at position {match.start()+1}: {potential_obj_str}\n")
             last_processed_index = match.end()
     finally:
         if log_file:
             log.close()

Usage: stream_json_objects(json_file_path, log_file='skipped_segments.txt').

2. Performance with Large JSON Objects

Issue: If individual JSON objects are very large (e.g., megabytes each), the 64KB chunk size may result in frequent buffer resizing and incomplete object parsing, slowing down processing.
Suggestion: Make the chunk size configurable to allow tuning based on the expected JSON object size:
```
 def stream_json_objects(filepath, chunk_size=65536):
     # ... use chunk_size in f.read(chunk_size) ...
```
Then call: stream_json_objects(json_file_path, chunk_size=1048576) for larger objects (e.g., 1MB).

3. Separator Regex Limitations

Issue: The regex }[ \t\r\n]*{ assumes JSON objects are separated by optional whitespace. If the file uses a different separator (e.g., commas, newlines only, or no separator), parsing will fail or produce incorrect splits.
Suggestion: Add flexibility to handle different separator patterns or detect them dynamically. For example:
```
 def stream_json_objects(filepath, separator_pattern=r"}[ \t\r\n]*{"):
     separator_re = re.compile(separator_pattern)
     # ... rest of the function ...
```
Alternatively, add a preprocessing step to detect the separator by scanning the first few KB of the file.

4. Empty or Invalid File Handling

Issue: If the input file is empty or contains no valid JSON objects, the script correctly reports “No valid JSON objects were parsed” but still creates an empty result.csv if any batch was attempted. This could be confusing.

Suggestion: Check if any valid objects were processed before creating the CSV file, or clean up the empty CSV file:

 if not header_written and all_columns is None:
     print("No valid JSON objects were parsed. No output CSV was created.")
     if os.path.exists(csv_file):
         os.remove(csv_file)
         print(f"Removed empty CSV file: {csv_file}")

5. CSV Encoding and Delimiter Flexibility

Issue: The CSV output is hardcoded to use UTF-8 encoding and the default pandas delimiter (,). Some systems or datasets may require different encodings (e.g., UTF-16) or delimiters (e.g., ;, \t).

Suggestion: Add parameters for encoding and delimiter:

 def json_to_csv_optimized(json_file, csv_file='result.csv', batch_size=5000, encoding='utf-8', delimiter=','):
     # ... in df.to_csv calls ...
     df.to_csv(csv_file, index=False, mode='w', header=True, encoding=encoding, sep=delimiter)
     # ... and in append mode ...
     df.to_csv(csv_file, index=False, mode='a', header=False, encoding=encoding, sep=delimiter)

Usage: json_to_csv_optimized(json_file_path, csv_output_path, encoding='utf-16', delimiter=';').

6. Progress Feedback

Issue: For very large files, the user only gets feedback per batch (e.g., every 10,000 objects). Long-running processes could benefit from more granular progress updates.

Suggestion: Add a counter for total objects processed and print progress every N objects:

 total_objects = 0
 for i, obj in enumerate(stream_json_objects(json_file)):
     total_objects += 1
     batch_data.append(obj)
     if total_objects % 1000 == 0:
         print(f"Processed {total_objects} objects...")
 # ... rest of the function ...
 print(f"Total objects processed: {total_objects}")

7. Commented-Out Excel Code

Issue: The commented-out json_to_excel function and openpyxl import are remnants of the original script. While harmless, they add clutter and could confuse maintainers.
Suggestion: Remove the commented-out code and the openpyxl import unless there’s a plan to reintroduce Excel support. If Excel output is needed, consider implementing it as an optional output format in json_to_csv_optimized with a parameter (e.g., output_format='csv' or 'excel').

8. Batch Size Tuning

Issue: The default batch_size=10000 may be too large for systems with limited memory or too small for very simple JSON objects, affecting performance.
Suggestion: Provide guidance in the docstring or add a dynamic batch size adjustment based on memory usage or object complexity. Alternatively, make it easier to tune via a command-line argument or config:
```
 json_to_csv_optimized(json_file_path, csv_output_path, batch_size=5000)  # Smaller batch for low-memory systems
```

9. Unused `io` Import

Issue: The io module is imported but not used in the provided code. This is minor but could indicate an oversight or leftover from earlier versions.
Suggestion: Remove the import io line unless there’s a specific plan to use it (e.g., for in-memory buffering).

10. Documentation and Type Hints

Issue: The docstrings are clear, but they could be enhanced with return types and parameter types for better IDE support and maintainability.

Suggestion: Add type hints and improve docstrings:

 from typing import Generator, Dict, List, Union
 import pandas as pd

 def stream_json_objects(filepath: str, chunk_size: int = 65536) -> Generator[Union[Dict, List], None, None]:
     """
     Streams JSON objects from a text file containing concatenated objects.
     Args:
         filepath: Path to the input text file.
         chunk_size: Size of chunks to read from file (in bytes).
     Yields:
         Parsed Python dictionary or list per JSON object.
     """
     # ... function body ...

 def json_to_csv_optimized(json_file: str, csv_file: str = 'result.csv', batch_size: int = 5000, encoding: str = 'utf-8') -> None:
     """
     Converts a text file with concatenated JSON objects to a CSV file.
     Args:
         json_file: Path to the input text file.
         csv_file: Path for the output CSV file.
         batch_size: Number of JSON objects to process per batch.
         encoding: Encoding for the output CSV file.
     """
     # ... function body ...

Specific Code Fixes

Here’s a consolidated version of the suggested changes applied to the code, keeping it concise:

import pandas as pd
import json
import re
import os
from typing import Generator, Dict, List, Union

def stream_json_objects(filepath: str, chunk_size: int = 65536) -> Generator[Union[Dict, List], None, None]:
    """
    Streams JSON objects from a text file containing concatenated objects.
    Args:
        filepath: Path to the input text file.
        chunk_size: Size of chunks to read from file (in bytes).
    Yields:
        Parsed Python dictionary or list per JSON object.
    """
    buffer = ""
    separator_re = re.compile(r"}[ \t\r\n]*{")
    last_processed_index = 0

    try:
        with open(filepath, 'r', encoding='utf-8') as f:
            while True:
                chunk = f.read(chunk_size)
                if not chunk:
                    break
                buffer += chunk
                for match in separator_re.finditer(buffer, last_processed_index):
                    potential_obj_str = buffer[last_processed_index : match.start() + 1]
                    try:
                        obj = json.loads(potential_obj_str)
                        yield obj
                        last_processed_index = match.end()
                    except json.JSONDecodeError as e:
                        print(f"Warning: JSONDecodeError at position {match.start()+1}: {e}")
                        print(f"Segment snippet (first 200 chars): {potential_obj_str[:200]}...")
                        last_processed_index = match.end()
                buffer = buffer[last_processed_index:]
                last_processed_index = 0
            buffer = buffer.strip()
            if buffer:
                try:
                    obj = json.loads(buffer)
                    yield obj
                except json.JSONDecodeError as e:
                    print(f"Warning: Could not parse remaining buffer: {e}")
                    print(f"Remaining buffer snippet (first 200 chars): {buffer[:200]}...")
    except FileNotFoundError:
        print(f"Error: File not found at {filepath}")
    except Exception as e:
        print(f"An error occurred: {e}")

def json_to_csv_optimized(json_file: str, csv_file: str = 'result.csv', batch_size: int = 5000, encoding: str = 'utf-8') -> None:
    """
    Converts a text file with concatenated JSON objects to a CSV file.
    Args:
        json_file: Path to the input text file.
        csv_file: Path for the output CSV file.
        batch_size: Number of JSON objects to process per batch.
        encoding: Encoding for the output CSV file.
    """
    if os.path.exists(csv_file):
        try:
            os.remove(csv_file)
            print(f"Removed existing CSV file: {csv_file}")
        except OSError as e:
            print(f"Error removing {csv_file}: {e}")

    all_columns = None
    header_written = False
    batch_data = []
    total_objects = 0

    print(f"Starting conversion of '{json_file}' to '{csv_file}'...")
    try:
        for i, obj in enumerate(stream_json_objects(json_file)):
            total_objects += 1
            batch_data.append(obj)
            if total_objects % 1000 == 0:
                print(f"Processed {total_objects} objects...")

            if len(batch_data) >= batch_size:
                try:
                    df = pd.json_normalize(batch_data)
                    if all_columns is None:
                        all_columns = list(df.columns)
                        df.to_csv(csv_file, index=False, mode='w', header=True, encoding=encoding)
                        header_written = True
                        print(f"Processed batch {i // batch_size + 1}. Header written.")
                    else:
                        df = df.reindex(columns=all_columns, fill_value=None)
                        df.to_csv(csv_file, index=False, mode='a', header=False, encoding=encoding)
                        print(f"Processed batch {i // batch_size + 1}.")
                    batch_data = []
                except Exception as e:
                    print(f"Warning: Error normalizing batch starting at object {i - len(batch_data) + 1}: {e}")
                    batch_data = []

        if batch_data:
            try:
                df = pd.json_normalize(batch_data)
                if all_columns is None:
                    all_columns = list(df.columns)
                    df.to_csv(csv_file, index=False, mode='w', header=True, encoding=encoding)
                    header_written = True
                    print(f"Processed remaining {len(batch_data)} objects. Header written.")
                else:
                    df = df.reindex(columns=all_columns, fill_value=None)
                    df.to_csv(csv_file, index=False, mode='a', header=False, encoding=encoding)
                    print(f"Processed remaining {len(batch_data)} objects.")
            except Exception as e:
                print(f"Warning: Error normalizing final batch: {e}")

        if not header_written and all_columns is None:
            print("No valid JSON objects were parsed. No output CSV created.")
            if os.path.exists(csv_file):
                os.remove(csv_file)
                print(f"Removed empty CSV file: {csv_file}")
        elif header_written:
            print(f"Data written successfully to '{csv_file}'")
        print(f"Total objects processed: {total_objects}")

    except Exception as e:
        print(f"An unexpected error occurred: {e}")

# Example usage
json_file_path = "sample2.txt"
csv_output_path = "result.csv"
json_to_csv_optimized(json_file_path, csv_output_path, batch_size=10000, encoding='utf-8')

Key Changes in the Fixed Version

Removed unused io and commented-out openpyxl code.
Added type hints for better code clarity.
Made chunk size configurable in stream_json_objects.
Added progress feedback every 1000 objects.
Ensured empty CSV files are removed if no valid objects are parsed.
Added encoding parameter to json_to_csv_optimized.
Improved docstrings with type information.
Removed the Excel file cleanup logic since it’s no longer relevant (replaced with CSV cleanup).

Testing Recommendations

To ensure the code works as expected, test it with the following scenarios:

Valid Input: A large file with correctly formatted JSON objects separated by }{ or whitespace.
Malformed JSON: A file with some invalid JSON objects to verify error handling and skipping.
Empty File: An empty file to check that no CSV is created.
Single Large Object: A file with one very large JSON object to test chunk size handling.
Different Separators: A file with JSON objects separated by commas or newlines to test regex robustness (may require modifying the separator pattern).
Low Memory: Run on a system with limited memory to tune batch_size and chunk_size.

Conclusion

The code is well-designed for its purpose, with strong memory efficiency and error handling. The suggested improvements enhance its flexibility, robustness, and usability without significantly altering its core functionality. The fixed version incorporates these changes and is ready for use with large JSON files. If you have specific requirements (e.g., support for Excel output, different separators), let me know, and I can tailor the code further!