DEV Community: Hoang Le

12 Hidden AWS Costs That Silently Drain Your Budget

Hoang Le — Wed, 14 Jan 2026 10:01:03 +0000

Originally published on INNOMIZE.

TL;DR: Most AWS waste comes from costs nobody thinks to check: snapshot accumulation ($2,500/month for 50TB), dormant resources ($100-500/month each), NAT Gateways ($100+/month), CloudWatch logs with infinite retention, and unattached Elastic IPs. The pattern is "set and forget." Quick win: delete snapshots older than 90 days. Enable Cost Anomaly Detection (free). Read on for all 12 costs and two real case studies.

Your AWS bill tells you what you spent. It doesn't tell you what you wasted.

If you've ever looked at your monthly invoice and thought "why is this so high?", you're not alone. We've audited dozens of AWS accounts over the years, and we keep finding the same pattern: teams optimize the obvious costs (EC2 instances, RDS databases) while overlooking charges that accumulate quietly in the background.

The problem isn't that these costs are hidden in the console. They're right there in Cost Explorer. The problem is that nobody thinks to look.

This is part 1 of our AWS Cost Optimization series. We'll walk through 12 costs that most teams miss, explain why they're easy to overlook, and show you what to check. In Part 2, we cover the optimization strategies that actually move the needle. In Part 3, we give you a step-by-step audit playbook.

Note: Part 2 and Part 3 will be published soon. Check back or subscribe for updates.

If you're looking for a quick overview of cost optimization best practices, we also published a comprehensive guide to AWS cost optimization that covers the fundamentals.

Data transfer: the invisible cost

Data transfer is one of the most commonly overlooked AWS charges. Most teams focus on compute and storage costs, assuming network charges are negligible. Then the bill arrives.

AWS charges $0.02 - $0.09 per GB (see Pricing) for inter-region transfers and internet egress. That doesn't sound like much until you do the math. A service sending 1TB of data between regions costs roughly $20/month. Multiply that by multiple services across regions, and you're looking at hundreds of dollars.

We've seen this catch teams off guard, especially in microservices architectures. Each service calls other services. Those calls cross availability zones. Sometimes they cross regions. The data transfer charges compound.

Data transfer costs are invisible until the bill arrives. Budget for them explicitly.

What to check: Look at cross-region API calls between microservices. Check backups replicated to other regions. Review large data exports to external systems. CDN origin fetches from S3 can add up too.

Snapshot accumulation: the backup nobody deletes

EBS snapshots cost $0.05 per GB-month (see Pricing). That's the straightforward part. The problem is how they accumulate.

Automated backup policies create snapshots on schedule. Daily backups. Weekly backups. Sometimes multiple backup tools running on the same volumes. The snapshots pile up. Nobody deletes them.

We've seen accounts with 50+ TB of snapshots. That's $2,500/month for data nobody will ever restore. When we ask teams about their retention policies, the answer is usually "we don't have one."

The fix is simple: set retention policies. Delete snapshots older than 90 days unless compliance requires otherwise. Review snapshots from deleted volumes, they don't get cleaned up automatically. Check for duplicate snapshots created by overlapping backup tools.

Dormant resources: the zombies in your account

Every AWS account has zombies. Load balancers with no targets. RDS instances nobody connects to. ElastiCache clusters for applications that were rewritten years ago. EC2 instances running at 5% CPU because nobody remembers what they do.

These resources get created for testing, demos, or projects that ended. Nobody tracks them after the project ends. The charges continue.

The cost varies widely, often $100-500/month per forgotten resource, but the pattern is consistent. We've never audited an account that didn't have at least a few.

What to check: EC2 instances with sustained low CPU utilization. Load balancers with zero healthy targets. RDS instances with zero connections over the past week. Elastic IPs not attached to running instances.

Case study: when "copy-paste infrastructure" costs $8,000/month extra

This one still stands out as one of our most challenging and rewarding optimization projects.

A client came to us managing multiple products across their organization. Their DevOps engineer had a straightforward approach: replicate the exact same environment setup for every project. Each product got its own EKS cluster (5 nodes), VPC, EC2 instances, bastion hosts, and VPN server.

The problem? Different projects had vastly different needs.

Some products didn't need Kubernetes at all. Others only required a single EC2 instance. A few genuinely needed the full EKS setup, but most didn't.

The result: roughly $10,000/month in AWS costs when it should have been $1,000-2,000/month. That's $8,000/month in waste, nearly $100,000/year, because of a "one-size-fits-all" infrastructure approach.

How we fixed it:

Audited each project's actual resource usage
Removed unused resources (idle EKS clusters, unnecessary bastion hosts)
Redefined their Infrastructure as Code to match real requirements
Created a clear playbook for how DevOps should provision and manage resources going forward

The fix wasn't just a one-time cleanup. We built a system that prevents the same pattern from recurring.

Standardization is good. Blind replication is expensive. Every environment should be right-sized to its actual workload, not copy-pasted from a template.

Case study: when instance sprawl blocks your Savings Plans

Another pattern we see repeatedly: teams that want to use Reserved Instances or Savings Plans but can't get meaningful coverage because their instance types are all over the place.

One client had a mix of RDS instances across their environments. Some were db.t3.medium, others db.r5.large, a few db.m5.xlarge. Different teams had provisioned different instance types based on whatever seemed right at the time. No standardization. No governance.

The problem? Reserved Instances are specific to instance type and size. Savings Plans are more flexible, but you still need predictable usage patterns to commit confidently. When every database is a different instance type, you can't commit to anything without risking underutilization.

The solution: We helped them standardize on a smaller set of instance types. Not everything needs to be identical, but having three standard sizes instead of twelve makes capacity planning possible. Once they consolidated, they could purchase Savings Plans covering 70% of their baseline, saving roughly 30% on RDS costs.

The broader lesson: instance sprawl isn't just an operational problem. It's a cost optimization blocker. You can't benefit from commitment discounts if you don't know what you're committing to.

We cover the Savings Plans vs Reserved Instances decision in detail in Part 2 of this series, including how to decide what commitment level makes sense for your organization.

NAT Gateway charges: the set-and-forget tax

NAT Gateways are essential infrastructure for private subnets that need outbound internet access. They're also expensive infrastructure that teams deploy without calculating the cost.

The pricing: $0.045/hour plus $0.045/GB processed (see Pricing). The hourly charge alone is $32/month per gateway. Teams often deploy NAT Gateways in every availability zone "for redundancy" without asking whether they need that redundancy. Three NAT Gateways means roughly $100/month before any data flows through them.

We're not saying don't use NAT Gateways. We're saying understand what you're paying for. Review the number of gateways versus actual HA requirements. Check the data volume flowing through each gateway. Consider whether VPC endpoints could replace NAT traffic for AWS service calls. They're often cheaper.

CloudWatch logs: infinite retention, infinite costs

Logs are essential. Teams enable them everywhere. Nobody sets retention policies.

CloudWatch pricing: $0.03/GB for ingestion plus $0.03/GB-month for storage (see Pricing). An application logging at 10GB/day costs $9/day for ingestion alone. After a year, you're storing 3.6TB at $108/month, for logs nobody queries.

The default retention for CloudWatch log groups is infinite. Every log you ever wrote stays there forever, charging storage fees, unless you explicitly set a retention period.

The EKS Control Plane logging trap

This catches many teams running Kubernetes on AWS. When you enable EKS Control Plane logging, including API server logs, audit logs, authenticator logs, controller manager logs, and scheduler logs, the volume can be massive.

A busy EKS cluster can generate gigabytes of control plane logs per day. Enable all five log types across multiple clusters, and you're looking at significant CloudWatch costs before you've logged a single application event.

The question to ask: do you actually need these logs in CloudWatch? For troubleshooting Kubernetes issues, maybe. For compliance, possibly. But many teams enable everything "just in case" without calculating the cost.

Consider self-hosted observability

Teams running EKS often find that self-hosted monitoring and observability stacks are more cost-effective than CloudWatch at scale. The Prometheus/Grafana ecosystem is mature, well-documented, and runs efficiently on Kubernetes.

The tradeoff is operational overhead. You're managing another system. But if you're already running Kubernetes, you have the infrastructure expertise. And the cost savings at scale can be substantial, especially when you factor in log aggregation with tools like Loki or the ELK stack.

Before enabling every log type in CloudWatch, calculate the cost. Self-hosted observability often makes more sense for EKS-heavy organizations.

What to check: Log groups with no retention policy. Log groups storing more than 100GB. Debug-level logging enabled in production (it probably shouldn't be). Duplicate logs, such as application logs, container logs, and infrastructure logs capturing the same events.

The small charges that add up

Some costs are individually small but signal poor hygiene. They matter because they indicate systemic issues with how resources are managed.

Unattached Elastic IPs cost $0.005/hour (see Pricing), about $3.60/month per IP. Not catastrophic. But teams accumulate them during infrastructure changes. Twenty unused IPs means $72/month. More importantly, it means nobody is cleaning up after themselves.

Idle load balancers cost at least $16/month each (see ALB Pricing) for the base charge. Dev and staging environments often sit idle. Five idle ALBs across dev, staging, and QA environments means $80/month for infrastructure serving no traffic.

Classic Load Balancers are often forgotten entirely. If you're still running them, you should probably migrate to ALB or NLB. The pricing is similar, but the features and performance are better.

Cross-region replication: enabled once, runs forever

Replication features are powerful for disaster recovery. They're also easy to forget about once enabled.

S3 cross-region replication for a 5TB bucket costs roughly $100/month in data transfer plus $115/month in destination storage. That's $215/month for one bucket. Is your DR strategy worth that? Maybe. But you should be making that decision consciously, not discovering it on your bill.

The same applies to RDS read replicas in other regions. DynamoDB global tables. Any replication that crosses regions incurs ongoing data transfer and storage charges.

Action: Review what you have enabled. Ask whether you still need it. If you do need it, budget for it explicitly.

Third-party marketplace licensing: the hidden multiplier

Marketplace charges are genuinely hidden in a way other costs aren't. They appear as "EC2-Other" in Cost Explorer. Teams see high EC2 costs without realizing it's software licensing, not compute.

A marketplace AMI for a database or security tool can add $500-2000/month on top of instance costs. We've seen teams pay 2-3x what they thought they were paying for compute because of marketplace licensing they didn't know they were using.

What to check: EC2 instances launched from Marketplace AMIs. "EC2-Other" line items in Cost Explorer. Whether the software could be replaced with open-source or AWS-native alternatives.

Request-based pricing at scale

Some AWS services charge per request. The per-request cost looks negligible until you scale.

API Gateway charges $1.00-3.50 per million requests depending on region and tier. At 100 million requests per month, that's $100-350. Many teams don't realize API Gateway has its own charges separate from Lambda.

S3 request pricing is similar. Everyone focuses on storage. Requests are an afterthought. But an application making 1 billion GET requests per month pays $400 in request charges (see S3 Pricing), potentially more than storage. LIST operations are 10x more expensive than GET.

What to check: Request volumes. Whether caching could reduce origin requests. For high-volume APIs, evaluate whether ALB plus Lambda would be cheaper than API Gateway.

Reserved capacity underutilization: paying for commitments you don't use

Reserved Instances and Savings Plans are purchased centrally. Usage is decentralized. This creates a disconnect.

We've seen accounts paying for Reserved Instances in us-east-1 while running workloads in eu-west-1. The reservation goes unused. The workload pays on-demand rates. Everyone loses.

What to check: Reserved Instance utilization reports. Savings Plans utilization. Instance types that don't match your reservations. If you reserved m5.large but you're running m5.xlarge, the reservation doesn't apply.

The pattern behind hidden costs

These 12 costs share common characteristics:

Set and forget. They're enabled once and never revisited. Backup policies. Replication rules. NAT Gateways. Log groups. Once configured, they run indefinitely.

Small individually. Each charge is easy to dismiss. $3.60/month for an Elastic IP? Not worth thinking about. But $50/month here and $100/month there adds up to "why is our bill so high?"

Compound over time. Snapshots accumulate. Logs grow. Zombie resources multiply. The costs grow even when nothing changes.

The fix isn't a one-time cleanup. It's building cost visibility into your operational routine.

What to do next

Three steps to take this week:

Quick win. Check your EBS snapshots. Delete anything older than 90 days that isn't required for compliance. This is often the single highest-impact change you can make in 30 minutes.

Enable monitoring. Turn on AWS Cost Anomaly Detection. It's free and catches unexpected charges before they accumulate.

Plan the audit. In Part 3 of this series, we provide a complete audit playbook. Schedule time to run through it.

In Part 2, we cover the six optimization strategies that actually move the needle: right-sizing, Savings Plans, auto-scaling, Spot instances, storage tiering, and serverless economics. Read that next if you want to understand how to systematically reduce costs, not just eliminate waste.

The Builds That Last Manifesto

Hoang Le — Sun, 11 Jan 2026 08:47:41 +0000

Originally published on Builds That Last.

I've been building and leading engineering teams for 15 years. Over 50 projects. Startups, enterprises, legacy systems, greenfield builds.

Same pattern every time.

Teams ship fast. Then they slow down. Not because engineers got lazy. Because the foundation was never there.

At my company, we maintain systems 20+ years old. No documentation. No standards. When we ask stakeholders about business logic, they say "read the code".

My teams sometimes decompile binary files just to understand what's inside. Debug at runtime because that's the only way to see how things work. Previous teams deployed code to AWS without committing to source control. Gone.

This is what engineering looks like for most of us. Not the AI demos. Not the apps shipped in a weekend.

The gap between posts and reality

You'll see posts about vibe coding, AI-augmented development, shipping apps in hours.

I'm not saying AI isn't real. I use these tools every day. The productivity gains are real.

But there's a gap between what people post and what I see in actual projects.

The posts show demos that work. Reality is production systems that break.

The posts celebrate shipping fast. Reality is teams spending months paying back technical debt.

The demo works. The demo always works. It's what comes after that separates software that lasts from software that collapses.

The iceberg problem

What you see is 20% above water. The shiny demos. The fast shipping. Vibe coding, agentic AI, apps built in hours.

What you don't see is the 80% below:

→ Maintenance
→ Technical debt
→ Engineers connecting systems never meant to work together
→ Data inconsistencies accumulated over years
→ Teams spending days understanding what the previous developer was thinking

That 80% is where my teams spend most of our time. And it's where the real lessons are.

The speed trap

I see this pattern repeat.

A team starts fast. AI tools, modern stack, motivated engineers. First version ships in weeks. Everyone celebrates.

Then users show up. Edge cases appear. The payment flow breaks. Data sync fails silently. Features that worked in demo crash under real load.

Suddenly the team that was "moving fast" spends months fixing things. Not building new features. Just paying back debt from shipping without foundation.

Leadership gets frustrated. "Why is the team slow now?"

The team isn't slow. They're doing work that should have been done upfront.

Speed without foundation creates the illusion of progress. Then reality catches up.

Maintenance costs more than building

Here's something most people don't think about until it's too late.

Maintaining software costs more than building it from scratch.

Think about repairing a house. You don't just fix the broken part. You investigate the structure. Remove old materials. Work around things that can't be changed. Then build the new thing.

Software is the same.

When you inherit a system without documentation, without standards, every change becomes an archaeology project. You spend more time understanding than building.

The time you "save" by skipping foundation gets paid back with interest during maintenance.

The AI paradox

AI makes code generation 10x faster. That's real.

AI also increases cognitive load by 30-40%.

When AI generates code, someone still needs to verify it's secure. Check for edge cases. Understand the logic before shipping. Maintain it when something breaks.

AI doesn't eliminate this work. It changes who's responsible for catching problems.

Anthropic's CEO said 90% of their internal code is now AI-generated. The follow-up: "We're not replacing engineers".

The 10% humans handle? That's the leverage zone. Architecture decisions. Debugging complex problems. Understanding why something should work, not just what it should do.

AI is a turbo, not a robot.

Good foundation? AI makes you faster at building good software.

Bad foundation? AI makes you faster at building bad software.

What this means for engineers

The fundamentals matter more than ever.

Everyone has access to AI now. The differentiator isn't who prompts better. It's who understands what they're building deeply enough to know when AI helps and when it hurts.

Own your code.

"The AI wrote it" isn't an excuse when something breaks at 2am. You shipped it. You're responsible. Read the code. Understand the logic. Ship with confidence.

The 80% below water is where you build real skills.

Legacy systems, maintenance, debugging. Not glamorous. But it's where you learn how software actually behaves. Don't avoid it. Embrace it.

What this means for leaders

You're only seeing 20% of what your team deals with.

The demos work. Sprint reports look fine. But your team might be drowning in the 80% you don't see. The legacy code. Missing documentation. Tribal knowledge that walks out when someone leaves.

Buying tools is easy. Training is hard.

Teams that succeed with AI invested in foundation first. Standards, process, documentation. Then added AI.

Teams that struggle added tools to existing chaos. Now they have faster chaos.

Remove friction before adding speed.

When you want to go faster, the instinct is to add more. More tools, more people, more pressure.

Usually, the answer is to remove things. Remove blockers. Remove unnecessary process. Remove friction slowing your team down.

The bottom line

Real speed comes from clarity, not from typing faster.

In the AI era, shipping is easy. Building to last is what matters.

Those legacy systems with no documentation? They taught me more than any greenfield project. Not because they were well-built. Because they showed what happens when foundation is missing.

Every time I build something new, I think about the engineer maintaining it in 10 years. Will they understand our decisions? Can they change things confidently? Or will they be stuck doing archaeology?

That's what foundation means. Building for the people who come after.

What's your experience with the 80% below water?

I write about foundation-first engineering at Builds That Last.

AWS re:Invent 2025: 3 Announcements That Matter for Your Architecture

Hoang Le — Wed, 07 Jan 2026 14:40:51 +0000

AWS re:Invent 2025 dropped a lot of announcements. Most won't change how you build. Three will.

Quick note: My last post here was November 2023. Life got busy. Running INNOMIZE, shipping products, building teams. I'm back to writing in public. If you've been following along, thanks for sticking around.

I spent December 2025 digging through the announcements that affect real architecture decisions. Not the flashy keynote demos. The ones that change cost models, operational overhead, and what's possible.

Here's the summary and where to go deeper.

1. Lambda gets Durable Functions and Managed Instances

Lambda now has two capabilities that fix its biggest limitations.

Durable Functions let you write multi-step workflows directly in Lambda code. No Step Functions. No ASL syntax. Just code with checkpoints that survive crashes. Can wait up to a year for external events.

// Checkpoint, wait, resume
const payment = await context.step('process_payment', async () => {
  return await paymentService.charge(event.amount);
});

await context.waitForCallback('manager_approval', { timeout: 86400000 });

await context.step('complete_order', async () => {
  return await orderService.complete(payment.id);
});

Managed Instances keep Lambda warm. No cold starts. Works with Savings Plans (up to 72% off with 3-year commitment). Multiple concurrent requests per instance.

Why this matters:

The "Lambda is expensive at scale" argument got weaker. For steady-state workloads processing millions of requests, Lambda Managed Instances with Savings Plans now competes with container pricing.

The decision tree changed. The question is now "which compute model fits this workload?"

Deep dive: AWS Lambda's Biggest Update in Years covers pricing comparisons, Durable Functions vs Step Functions, and Managed Instances trade-offs.

Bonus: Serverless vs Containers in 2025 with updated decision tree.

2. EKS Capabilities: managed ArgoCD, ACK, and KRO

EKS Capabilities is a managed layer for three Kubernetes-native tools:

→ Managed Argo CD: AWS handles installation, scaling, patching, HA
→ AWS Controllers for Kubernetes (ACK): Manage AWS resources via kubectl. S3, RDS, IAM through CRDs.
→ Kube Resource Orchestrator (KRO): Reusable resource bundles. Backed by AWS, Google, and Azure jointly.

Key architectural difference: these run in AWS-owned infrastructure, not on your worker nodes. Zero cluster overhead.

Why this matters:

Running self-managed ArgoCD? You're spending engineer time on upgrades, patches, and HA configuration. AWS estimates 70-80% reduction in platform team overhead.

ACK's resource adoption feature is interesting. You can migrate existing Terraform/CloudFormation resources without recreating them. Gradual migration becomes possible.

Trade-off: Less flexibility. No multi-namespace deployments. No Image Updater. No custom SSO providers. If you need those, stay self-managed.

Deep dive: AWS EKS Capabilities: What We Learned After Testing It covers feature comparisons, pricing, and when to use managed vs self-managed.

3. Database Savings Plans and SQL Server Developer Edition

Database Savings Plans: Commit to $/hour usage over 1-year term. No upfront payment required.

Type	Discount
Serverless (Aurora, ElastiCache, etc.)	Up to 35%
Provisioned instances	Up to 20%

The gap is intentional. AWS wants you on serverless.

SQL Server Developer Edition: Now available on RDS. All Enterprise features, no licensing costs. Dev/test environments got cheaper.

M7i/R7i with Optimize CPU: Disable SMT, reduce vCPU count by 50%, pay for half the SQL Server licenses. Same physical cores, near-equivalent performance.

Why this matters:

We audited one project. Found 12 RDS instances on db.t3 (2018 instance family). The migration decision isn't straightforward:

→ db.t4g: 10% cheaper on-demand, but no discount options at all
→ db.t3: Older, but Reserved Instance eligible (37% off)
→ db.r7g: Database Savings Plan eligible, better performance

For always-on dev environments, t3 with Reserved Instance can be cheaper than t4g on-demand. Counterintuitive but true.

For SQL Server projects, Developer Edition eliminates licensing costs for non-production. That's ~$240/month savings per db.r5.large instance.

Deep dive: AWS Database Cost Optimization covers the t3/t4g decision matrix, SQL Server migration path, and when Reserved Instances beat Savings Plans.

The pattern across all three

AWS is sending clear pricing signals:

→ Serverless gets better discounts: 35% for serverless databases vs 20% for provisioned
→ Managed services reduce overhead: EKS Capabilities, Lambda Managed Instances
→ Latest-gen instances get the deals: r7g/r8g qualify for Savings Plans, t4g doesn't

The message: adopt managed services, use latest-gen infrastructure, commit to spend.

What's next

At INNOMIZE, we're tracking these changes across client projects. The t3 → r7g migration is in progress. We're evaluating EKS Capabilities for new deployments. Lambda Managed Instances is on the roadmap for high-traffic APIs.

I'm launching a newsletter called Builds that Last. Covering platform engineering, cloud architecture, and engineering leadership. Subscribe here or connect on LinkedIn.

Which announcement impacts your architecture most?

Hoang Le is the founder of INNOMIZE, building cloud-native systems for startups and enterprises. Connect on LinkedIn for more on engineering leadership and platform engineering.

Exploring Serverless Billing Management: Architecture and Cost

Hoang Le — Tue, 07 Nov 2023 10:17:13 +0000

This article was originally published on our website.

Over the past five years, as we've developed and managed a serverless system, we've gained valuable insights into design considerations and cost optimization. In this article, we aim to share our expertise, offering businesses a blueprint for addressing similar challenges efficiently. Our primary focus is on cost reduction while ensuring the key aspects of high availability, scalability, security, extensibility, and ease of development. It's important to note that while this architecture can offer significant benefits, it may not be a universal solution. It's vital to thoroughly evaluate and adapt it to your specific requirements.

If you're new to the concept of Serverless Architecture, we recommend familiarizing yourself with it before embarking on this architectural journey.

Overview of the System

We were approached by a US-based client whose Billing Department grappled with the task of generating, managing, and collaborating with their legal team to send invoices to clients. The existing process involved producing paper invoices, manually sending them to attorneys for review, and often taking more than a month to complete a monthly billing cycle. With approximately 2,000 invoices to review and process each month, the manual nature of this process was not only time-consuming but also resource-intensive.

Additionally, the billing team had the arduous task of uploading LEDES files to over 20 carrier sites. Each upload required repetitive manual steps, such as logging in to the carrier site and verifying the results either on the site or via email.

Imagine how boring it is for users that they have to upload around 2000 files to various sites every month, how much effort is consumed?

Calculating this is straightforward. Let's assume that it takes a user roughly 3 minutes to log in, upload a file, check the results, and update the internal system for each file. So, with 2000 files, the total time required amounts to approximately 6,000 minutes (equivalent to 100 hours). In continuous operation, this task could span an entire month for a single person to complete. However, in reality, it may necessitate the effort of at least two full-time individuals over the course of a month to accomplish this job. For US workers, how much does it cost for labor expenses?

To address these challenges, we analyzed the client's requirements and developed a solution that streamlined the invoicing process. This solution allowed users to effortlessly view, track, and review assigned invoices while automating various steps in the process. This digital transformation not only reduced manual efforts but also significantly improved productivity.

System Architecture

After discussing the client's specific needs, which included a relatively small user base (around 150 users including Billing users and Attorneys) and the absence of in-house IT resources for system management, we opted for a serverless architecture. This choice allowed the client to focus on the features they needed while minimizing operational overhead.

Serverless Architecture for Billing Management System

The Serverless Billing Management System comprises the following key components:

Serverless Compute Services with Lambda, API Gateway

The backend system uses AWS Lambda integrated with API Gateway REST APIs to handle HTTP requests from clients, including both a web-based application and a mobile application.

We leveraged the Serverless Framework for building, testing, and deploying Lambda functions to AWS. The stack included Node.js, TypeScript, serverless-webpack, and serverless-offline plugins, which allowed us to simulate HTTP APIs locally and streamline the deployment of TypeScript code.

Datastore with Amazon RDS MySQL and S3

While we initially considered Amazon DynamoDB for our database needs, we ultimately selected Amazon RDS MySQL due to our specific schema and query requirements. We managed database changes through Liquibase, which tracked database schema and data changes via changesets. Our data access layer was implemented using sequelize-typescript, and for more complex queries and reporting, we used serverless-mysql and knex as a query builder.

S3 played a dual role, serving as the platform for our Angular-based billing portal and facilitating data replication from on-premises to AWS. The integration involved Node.js, S3, and Lambda to automate the process of capturing changes from a legacy system and synchronizing them with our database.

Authentication with Amazon Cognito

Security being a paramount concern, we selected Amazon Cognito as the user directory. It integrated seamlessly with other AWS services and provided a secure mechanism for protecting the backend services running on Lambda. We initiated user migration from ADFS using Cognito triggers and then disabled ADFS once all users had transitioned.

Event Sources with S3 and HTTPS Requests from API Gateway

Our primary event sources were triggered by Lambda from API Gateway and data replication. APIs were secured using TLS/SSL and Cognito Authorizer.

Billing Portal with Angular, S3, and CloudFront

We used Angular to develop a Single Page Application (SPA) for user interfaces, enabling smooth collaboration and review of invoices. Static assets and the website were deployed to an S3 bucket with CloudFront distribution.

Mobile Application with React Native

A mobile application was developed to enable users to view and book timesheets against Matter records. This app was created using React Native and seamlessly replicated all time entries back to the Case Management System running on-premise servers.

Headless Automation for LEDES Uploads with CDK, AWS Fargate, Selenium, and Protractor

To automate the time-consuming process of manually uploading invoices to carrier sites, we employed headless automation with technologies such as Selenium and Protractor. Given the potentially lengthy execution time, AWS Fargate (ECS Fargate) was used to handle the automation. We used AWS CDK to build and deploy ECS Fargate tasks and services, which were triggered by Lambda functions when invoices were ready for upload.

Monitoring and Logging with AWS CloudWatch, Sentry

We initially relied on third-party monitoring but later transitioned to CloudWatch for log and alert management. Sentry is used for error reporting and integrated with Lambda functions. Additionally, Slack was connected to Sentry to receive alerts on new errors.

SNS and SES were used for alerting and email notifications.

IaC with Serverless Framework and AWS CDK

The development process was facilitated by the Serverless Framework for building, testing, and deploying Lambda functions. In the absence of native support for certain functionalities, we utilized the following plugins:

serverless-secrets-plugin: For encrypting and decrypting secret files, allowing secure storage in source control.
serverless-split-stacks: To mitigate CloudFormation stack resource limits by splitting resources per Lambda function.
serverless-domain-manager: For creating custom domains for API Gateway.
serverless-plugin-warmup: A custom plugin to warm up Lambda functions with CloudWatch schedules.
serverless-s3-sync: Used to deploy the Angular application to an S3 bucket.

Amazon Web Services Cloud Development Kit (AWS CDK) was instrumental in building, testing, and deploying the automated processes with ECS Fargate. Its abstraction design allowed easy creation of a headless stack and integration with CI/CD pipelines. Jenkins was used for CI/CD, with the ability to manage multiple environments and deploy changes to AWS environments in minutes. Checkout the Long-running Serverless Web Scraping – An AWS Well-architected Solution for more details on how you can build, test and deploy automated headless process with CDK, ECS Fargate.

Jenkins CI/CD Pipeline

For designing and building IaC, we recommend separating the core infrastructure components such as networking, databases, and Cognito user pools into separate services with their CI/CD pipelines. Depending on your tech team's preference, either AWS CDK or Terraform can be used to provision and configure the infrastructure. For service components, co-locating them with the service repository streamlines build, test, and deployment processes. In the case of this solution, a single repository housed around 25 functions, each providing CRUD APIs for relevant entities/tables, and some functions for reporting, data replication, and Cognito user pool triggers. For systems with fewer services, consider using NX DevTool to build, test, and deploy Serverless services in a monorepo. You can also employ vendia/serverless-express, Middy, and Nest.js for deploying scalable REST APIs running on Lambda.

You might observe a significant aspect in our setup: the utilization of a Site-to-Site VPN. Initially, we employed this method to establish connectivity between AWS and our on-premise infrastructure. It served the purpose of creating a linked server for our MS SQL Database to communicate with the RDS MySQL instance on AWS. However, as time progressed, we encountered operational challenges that demanded a more stable connection. In response, we decided to revamp our approach for the data replication component, which ran on our on-premise servers. Instead of relying on VPN, we pivoted towards exposing REST APIs. To enhance security, we implemented IP address whitelisting and introduced basic authentication protocols.

Cost Breakdown

Now, let's delve into the cost breakdown of the architecture. Here's a snapshot of the AWS bills from the last six months:

Database Costs

The most substantial portion of the expenditure is attributed to the database. Our choice of an AWS RDS MySQL database in a fully managed capacity amounted to approximately $184 per month over the last six months. Notably, we opted not to utilize Multi-AZs. This decision was grounded in the understanding that our system did not necessitate a fully highly available configuration, aligning with our goal to optimize costs efficiently.

Lambda Costs

Lambda services, constituting the backbone for handling HTTP requests and powering custom integration services for data replication, emerged as the second most significant cost factor. The monthly expenditure hovered around $57. The uniqueness of Lambda lies in its adaptability to fluctuating workloads. While the volume of HTTP requests can be predicted, Lambda execution for data replication is less predictable. In the recent bills, the total amount of Lambda function was reduced a lot (around $20 per month), mostly because the amount of data that needs to be replicated is reduced. Lambda's scalability feature allowed us to dynamically respond to the varying data replication needs without the overhead of infrastructure management.

ECS (Fargate) Costs

ECS (Elastic Container Service) claimed the third spot in the list of expenses, accounting for around $29 per month. ECS served as a vital component for our headless automation processes. The system's scalability directly corresponded to the number of invoices requiring upload each month. ECS Fargate afforded us the flexibility to efficiently run these automation tasks, eliminating the need for the laborious setup and management of virtual machines, which would have been the case with a traditional architectural approach.

API Gateway Costs

Our system efficiently managed around 800,000 monthly requests. API Gateway's pricing structure is tiered, with the first 333 million requests billed at a rate of $3.5 per million, resulting in our average cost for API Gateway being $3.2 per month.

Other Services Costs

Beyond these major cost components, several other services contributed to the overall monthly expense. These included S3, KMS (Key Management Service), CloudFront, SES (Simple Email Service), and SNS (Simple Notification Service). Their collective costs amounted to roughly $5 per month.

Total Monthly Cost

When considering the combined expenses of all these components, the total monthly cost of operating the entire system averaged around $280. This cost-effective solution provided scalability and security while significantly reducing the operational burden for the client. This architecture allowed the client to focus on feature development, thus streamlining their operations and cutting down on operational expenses.

The cost breakdown presented focuses on the production environment. If you're curious about expenses in other settings like Staging or Development, here's the good news – in non-production environments, additional costs are minimal, and you can rest assured that these expenses won't be a significant concern. These environments typically handle low volumes of requests and user interactions. Furthermore, the database is shared among these instances for specific purposes. This showcases the efficiency of serverless architecture in meeting non-production needs while still adhering to the system's non-functional requirements. Now, the question that arises is: How can you achieve similar results with a traditional architecture while ensuring all non-functional prerequisites are met?

Optimizing Costs

You might see that based on the billing that we have captured here, we can do some actions to save costs, but, this is not the high priority that our team can focus on right now to save a few dollars. We still have a lot of feature development that we should focus on which increases ROI for the client. However, there are some tips or best practices we have applied according to our article Mastering AWS Cost Optimization with Best Practices and Hidden Costs as below:

VPC S3 Gateway Endpoint: For processing files on Lambda from a private subnet, establishing a VPC S3 gateway endpoint is highly recommended to mitigate potential cost overheads.
Lifecycle Policies for S3: Implementing lifecycle policies to automatically delete processed files from S3 after a specified time period is a cost-effective measure.
Right Retention for CloudWatch Logs: Configuring appropriate retention settings for CloudWatch log groups ensures efficient use of resources.
Lambda Power Tuning: Leveraging Lambda Power Tuning assists in identifying the optimal resource allocation (CPU and Memory) for Lambda functions, thus achieving cost savings while maintaining performance.

Summary

In conclusion, AWS Serverless services have proven to be a game-changer, especially for businesses seeking custom software solutions. By combining popular serverless services like Lambda, API Gateway, and DynamoDB, businesses can design, build, and deploy well-architected solutions that deliver a substantial return on investment (ROI).

Cost optimization, designing serverless application architectures, DevOps, and digital transformation are all key elements of this success story. This architecture not only streamlines complex processes but also reduces operational burdens, allowing businesses to focus on delivering core features while leaving the infrastructure management to the cloud provider. It's worth noting that while serverless architecture offers numerous advantages, it may involve vendor lock-in and reduced control over underlying infrastructure, aspects that organizations should carefully consider.

At INNOMIZE, our proficiency in cloud computing, digital transformation, and serverless development services has enabled us to craft a solution that not only streamlines billing processes but also paves the way for future scalability and innovation. Our track record of over five years of seamless operation stands as a testament to our commitment to excellence. We understand that every business is unique, and our tailored solutions reflect this understanding.

If you're grappling with similar challenges or seeking to embark on a digital transformation journey, INNOMIZE is here to help. Our team of experts is ready to collaborate with you, analyze your specific needs, and deliver solutions that not only solve your problems but also position your business for a dynamic, cloud-native future.

Ready to Embrace Digital Transformation? Contact INNOMIZE Today!

Your journey toward a more agile, cost-effective, and innovative future begins here. Don't hesitate to get in touch with INNOMIZE and unlock the power of cloud computing and serverless solutions.

Continuous Delivery - Deploying a Node.js app to AWS EC2 using Ansible

Hoang Le — Mon, 03 Feb 2020 02:50:35 +0000

The main key point for all our projects is trying to automate all things, it helps to reduce errors (e.g. human mistake), fast and easy to deploy / rollback, and improve customer satisfaction.

Images Sources

We do have some approaches to automate the deployment process that we have been applying to our projects such as using Amazon Elastic BeanStalk or Amazon Elastic Container Service. However, in some cases, our customers they don’t want to use those approaches because they don’t use AWS, they’ve already had the servers to run the app, they want to keep their app on their On-Prime Data Center and have their IT guy to manage it. It led us to have to find out another approach to automate our delivery process. And we have chosen Ansible as the tool to implement our Continuous Delivery pipeline.

So I would like to write this article to share with you and outline our workflow and provide a brief introduction along with sample code to build this flow.

The Tools

Jenkins: Jenkins is a self-contained, open-source automation server that can be used to automate all sorts of tasks related to building, testing, and delivering or deploying software.
Ansible: Ansible is an open-source automation platform. Ansible can help you with configuration management, application deployment, task automation. It can also do IT orchestration, where you have to run tasks in sequence and create a chain of events that must happen on several different servers or devices.
Slack: a cloud-based set of proprietary team collaboration tools and services.

Why Ansible?

As you know, there are a lot of tools for infrastructure automation (infrastructure as code) such as Terraform, Chef, Ansible, Juju, and more. But there are a few reasons that make me selected Ansible as below:

It is very, very simple to set up and yet powerful.
Ansible is a radically simple IT automation platform that makes your applications and systems easier to deploy. Avoid writing scripts or custom code to deploy and update your applications.
Automate in a language that approaches plain English, using SSH, with no agents (likes Puppet or Chef) to install on remote systems.
Easy to integrate with our CI/CD pipeline using the Jenkins server.

The Steps

What we are going to build is the following deployment flow:

Launch EC2 instance(s) (optional).
Update OS and install needed dependencies such as install Node.js.
Deploy the Node.js app to EC2 instance(s).
Configure DNS (optional).
Smock test.

Let’s coding…

Project Structure

When doing any coding stuff, the project structure is one of the most importing things I pay attention to. If you have experience in working with Ansible, you should know how to organize the Ansible project. I followed the alternative approach mentioned in this article, feel free to select your own approach.

Ansible Project Structure

My project contains the following root directories/files:

scripts: it contains a shell setup script to install Ansible, Ansible Galaxy, generate a self-signed certificate to run our Node.js app in secure mode (HTTPS).
cert: directory to keep a self-signed certificate.
ansible: put all Ansible code into this directory including playbooks, roles, var files, etc.
Jenkinsfile: the Jenkins pipeline code for CI/CD pipeline.

Role

In Ansible, the role is the primary mechanism for breaking a playbook into multiple files. This simplifies writing complex playbooks, and it makes them easier to reuse. The breaking of playbook allows you to logically break the playbook into reusable components.

Here are the following roles we will use on this project:

Ansible Roles

apache-server: we don’t use this role on this project but I wanted to keep it here. This role will configure the Apache using our own httpd.conf template file. To install Apache, we can use geerlingguy.apache role.
common: put all your comment stuff into this role such as update your OS to ensure your server(s) are up to date.
launch-ec2: this role to launch EC2 instance(s) to deploy our app.
node-server: contains all needed steps to deploy and run the Node.js app
nodejs: install the Node.js runtime.

Conclusion

Using Ansible, you can easy to provision and install software to run your application automatically, you can manage your infrastructure as code (IaC). However, running your software is not simple likes this demo, you will need to do more things such as setup network, configure firewall, auto-scaling. To do those things, you might need to combine with other tools. For example, you can use the AWS CDK framework to provide your AWS infrastructure (network, database, EC2 instances, and auto-scaling), then use Ansible to provision your EC2 instances and configure your applications.

I hope this post will bring some ideas to you to design and deploy your application. If you have any issues or need help, just let me know by adding comments on this post.

Thank you for reading!

This post is originally published on our blog.

Check list to avoid scam for Freelancer

Hoang Le — Thu, 26 Dec 2019 03:02:57 +0000

When I working as a Freelancer, I saw a job on a Freelancer site, they just say on the description that if anyone wants to work with them, then send them an email to get job descriptions. I emailed them and got their feedback on the next day, and we have a few more conversations, then they asked me for estimation, I replied then they accepted without asking for more details. After that, they ask me to send them the relative person in the USA in order to receive money for me.

They bring another person, they called them as their private consultant, he/she will send me the material (logo, image artwork to create a website). Not sure where he/she is living but they said he/she cannot receive money on PayPal or credit card.

I asked my friend in the USA and send them relative person info, but after that, I had a feeling, something wrong with this job, it is really odd. I discuss it with my US friend to get their advice. After some conversations, my friend asked some questions and help me realized that this is a SCAM.

Below are some messages from my friends asking me about the work.

This doesn’t sound legitimate. Why can’t they pay the other person directly? I’m not comfortable accepting and distributing money like this. Also, the email is very odd. You haven’t actually given her any details around the exact estimate and yet they are moving forward? Something isn’t right here.

Unfortunately, there are many people scamming out there. I feel like this could be some sort of scam or way for this person to pay someone without directly transferring the funds to them. Using us as a middle man. That could get us into a lot of trouble.

Finally, I replied back and said that I cancel this job and they disappeared, no response so far after weeks.

So when you try to bid any work, pay attention to some points as below, it will help recognize and avoid scam that will bring troubles, even lose your money:

Sender information: is the name spell correctly or check the email address, is that good? You can search by email address and you may see some posts about scam relates to that email address.
Project description: are they provide a valued description, deadlines or more specific information?
Check the payment option they prefer and never accept cashier's checks as payments

You can also read this post, it describes exactly my situation and helped me to avoid a SCAM. I hope it can help you too.

re:Invent 2019 announcements

Hoang Le — Fri, 20 Dec 2019 08:26:04 +0000

If you didn't have a chance to review all recent AWS announcements on their #awsreinvent2019, you can review this table.

aws #cloudcomputing #clouddevelopment #cloudconsulting #cloudengineer #news #announcements #serverless

A brief summary of JavaScript Learning Path

Hoang Le — Fri, 20 Dec 2019 06:00:51 +0000

This post is originally published on our blog.

JavaScript is a popular language among web developers which gave rise to several frameworks that simplify your code. According to the largest online developer communities, StackOverflow in their 2019 survey of the most sought-after programming languages shed some light on what to expect in the coming year:

Top Programming Languages in 2019

For the seventh year in a row, JavaScript is the most commonly used programming language. So how to learn JavaScript, how to up-level of your programming language skills, what is the path for learning JavaScript from the beginning? If you don't know how to get started, then you should read this post.

Step 1 - Learn basic and fundamentals

Firstly, you need to understand basic concepts and fundamental of JavaScript, you can learn from

You-Dont-Know-JS this is a series of books diving deep into the core mechanisms of the JavaScript language. You should read this to get more overview and advanced concepts of JavaScript such as Scope & Closures, Objects & Classes, Sync & Async, ES.Next & Beyond.

Step 2 - Learn modern JavaScript

Learning basic isn't enough, you need to learn more about advanced and modern JavaScript concepts/syntax, it helps you write code efficiently, quickly, and high performance

Step 3 - Coding style and Convention

Coding standards (also sometimes known as ‘Coding Conventions’ or ‘Coding Rules’) are a set of guidelines that a group of developers stick to, to ensure that they are all essentially follow the same style.

Read about prettier and see how to configure it in order to format your code and save your time.

Step 4 - Review your knowledge

Now is the time to review your understanding about JavaScript by answering interview questions

Step 5 - Practices

Learning JavaScript libraries (e.g. Lodash, ReactJS, Moment, jQuery, Ramda, and a lot more)
Learn JavaScript Frameworks such as Angular, VueJS, EmberJS)
Task runner (e.g. Gulp, Grunt) and module bunder (e.g. Webpack, Browserify)

Step 6 - Be an expert

Continue learning core concepts in JavaScript such as Closure, Hosting, Scope, Coercion, Prototypes
Learn OOP, Functional Programming
Read this book Learning JavaScript Design Patterns or design-patterns

If you want to become a web, backend or full-stack developer, JavaScript isn't enough, you need to learn more. Check out this roadmap and pick your path Web Developer Roadmap 2019 and keep yourself up to date.

Visit our blog for more interesting articles. If you have any questions or need help you can contact me via Twitter.

If you are looking for developers, offshore team, or need consulting about the AWS cloud, Serverless architecture, and so on, then hire us, we can help you!

Thank you for reading!

07 best practices when using AWS SSM Parameter Store

Hoang Le — Thu, 21 Nov 2019 03:28:41 +0000

This post appears first on our blog.

Cloud Encryption

Security is one of 5 pillars of the Well-Architected framework, it can archive by applying best practices and principals in IAM, Encryption, Complician, and Governance. Of course, best practices aren't enough, you need to learn more. In this post, I only share our best practices and tip when working with AWS SSM Parameter Store. By sharing our best practices, my hope is to encourage you to build and deploy secure and reliable applications and also giving us your feedback.

As you know, AWS Lambda supports native environment variables, you can easy to define and add any environment variables you want during deployment or change on the AWS Console Management. But using native environment variables contains some disadvantages:

It stores plaint-text variables which easy to see its value. You had an option to encrypt variables in the console using KMS, but it still fetching per innovation causes increase your bill.
Hard to share across projects and teams that add complexity to your applications and services. More complexity requires more time to operate and increase the cost, therefore, you won't meed the conditional for the Operational Excellence pillar of the Well-Architected framework.
As per Yan Cui, it hards for implementing fine-grained access to sensitive data.

What is AWS Systems Manager Parameter Store (aka SSM Parameter Store)?

AWS Systems Manager Parameter Store provides secure, hierarchical storage for configuration data management and secrets management. You can store data such as passwords, database strings, and license codes as parameter values. You can store values as plain text or encrypted data. You can then reference values by using the unique name that you specified when you created the parameter. Highly scalable, available, and durable, Parameter Store is backed by the AWS Cloud. ~AWS

What are the benefits?

There are a lot of benefits when using AWS SSM Parameter Store, I just copied those from the AWS documentation:

Use a secure, scalable, hosted secrets management service with no servers to manage.
Improve your security posture by separating your data from your code.
Store configuration data and secure strings in hierarchies and track versions.
Control and audit access at granular levels.
Configure change notifications and trigger automated actions for both parameters and parameter policies.
Tag parameters individually, and then secure access from different levels, including operational, parameter, Amazon EC2 tag, and path levels.
Reference AWS Secrets Manager secrets by using Parameter Store parameters.
Use Parameter Store parameters with other Systems Manager capabilities and AWS services to retrieve secrets and configuration data from a central store.
Configure integration with the AWS services for encryption, notification, monitoring, and auditing.

So now you understand what is the SSM parameter store and its challenges, let talk about how we use it by reviewing the following our best practices and tips:

#1 - Organizing parameters into hierarchies

AWS provides detailed instructions on how to organize your SSM Parameter Store to define and manage parameters easily. Following its best practices can help you and make your life easier. Below are a couple of formats/conventions that our team normally using:

/environment/service-name/type/application-name/parameter_name i.e. /prod/billing/databases/invoicing-portal/db_connection_string
You also can add your department name as well i.e. /prod/human-resource/employee/user_list

#2 - Consistent naming convention

Using a well-defined hierarchy helps you to manage and retrieve parameters more efficiently, but you also need to use a consistent naming convention across your AWS account, your departments, and your teams.

By archiving this best practice, it reduces your reviewing efforts by focusing on critical business logic rather than syntax and naming standards and then increase your productivity and quality which can increase your customer satisfaction.

#3 - Restrict IAM permission

AWS SSM Parameter Store normally keeps your sensitive information, so restrict permissions are required to improve your security of the application. Each Parameter Store has a unique Resource ARN per account and region, so you can easier to define role and policy base on the hierarchy of the parameter store.

Below is a sample code from the AWS official document shows how to define a policy to restrict access to the Parameter Store

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "ssm:*"
            ],
            "Resource": "arn:aws:ssm:us-east-2::parameter/*"
        },
        {
            "Effect": "Deny",
            "Action": [
                "ssm:GetParametersByPath"
            ],
            "Condition": {
                "StringEquals": {
                    "ssm:Recursive": [
                        "true"
                    ]
                }
            },
            "Resource": "arn:aws:ssm:us-east-2:123456789012:parameter/Dev/ERP/Oracle/*"
        },
        {
            "Effect": "Deny",
            "Action": [
                "ssm:PutParameter"
            ],
            "Condition": {
                "StringEquals": {
                    "ssm:Overwrite": [
                        "false"
                    ]
                }
            },
            "Resource": "arn:aws:ssm:us-east-2:123456789012:parameter/*"
        }
    ]
}

#4 - Combine into a single parameter likes database connection string and keep it all together (co-location)

By using consistency hierarchies and naming convention you can archive this idea. Keep all related parameters all together makes easy to find and retrieve. Using fewer parameters can reduce your bills.

Instead of using 4 separated parameters for the database connection string as below:

/{env}/{service}/databases/master/host = db.domain.com
                                 /user = username
                                 /password = password
                                 /port = 3306

We combine into a single parameter using a standard connection string format:

/{env}/{service}/databases/master/db_connection = jdbc://username:password@db_host:port/database_name

Using community library such as connection-string-parser, you can easy to parse the parameter values and use to open connection, see below code snippet:

import { createConnection as createConnectionPromise, Connection } from 'promise-mysql';
import { ConnectionStringParser } from 'connection-string-parser';

const parseConnectionString = (dialect: string, connectionUri: string) => {
  const connectionParser = new ConnectionStringParser({
    scheme: dialect || 'mysql',
    hosts: []
  });
  const connectionStrParams = connectionParser.parse(connectionUri);

  return {
    host: connectionStrParams.hosts[0].host,
    port: connectionStrParams.hosts[0].port || 3306,
    database: connectionStrParams.endpoint,
    user: connectionStrParams.username,
    password: connectionStrParams.password
  };
};

export const createConnection = (connectionUri: string): Promise<Connection> => {
  return createConnectionPromise(parseConnectionString('mysql', connectionUri));
};

#5 - Use tool/library to fetch, cache, and export to environment variables at runtime

You are charged for API integration to SSM parameters, every time you retrieve your parameter from the store, you will increase your bill. What could you do to reduce your billing?

By default, max throughput (transactions per second) to retrieve parameter via API is 1000, how do you manage and avoid exceed throughput error?

@Yan Cui wrote an article describes reasons why you should use AWS SSM Parameter Store over Lambda environment variables, he also mentioned approaches for caching and cache expiration using his custom client library.

Our team is using middy middleware to deal with some cross-cutting concerns outside business logic, like input parsing and validation, output serialization, error handling. Application configuration also an aspect that every developer needs to work out and manage to run business logic. Out of the box, middy provides ssm middleware support fetch and cache parameters from the AWS SSM Parameter Store, it also supports assign parameter values to environment variables.

Here is a sample code on how to use middy to fetch and cache parameter store

const middy = require('middy');
const { ssm } = require('middy/middlewares');

export const handler = middy((event, context, cb) => {
  // You can access the parameter value inside function handler
  console.log(process.env.HARVESTAR_PCMSS_DB_CONNECTION);

  // Your business logic here
}).use(
  ssm({
    cache: true,
    names: {
      // Should have a prefix that include this micro service i.e. pcmss
      HARVESTAR_PCMSS_DB_CONNECTION: '/dev/harvestar/pcmss/db_connection'
    }
  })
);

There are some alternative open-sourced libraries out there:

aws-parameter-cache

ssm-cache-python

Do you really trust the community package?

I head from some people, basically, they don't want to assign variables into environment variables (i.e. variables you can access through the process.env global object in Node.js runtime). If you do so, I have some advice as below:

Instead of assign variables to environment variables, you also have another option to assign the context object of the AWS Lambda when using middy/ssm middleware.
To avoid sending your sensitive information such as data credentials, accessing to the /tmp directory, or running a child process when executing your serverless functions. You can use @puresec/function-shield library. We are also using it in our production environment.

TIPS - Avoid fetching parameters at build/deploy time, fetch it at runtime instead. If you do so, you have to redeploy each time the parameter changed.

#6 - Using hardcoded environment variables for your local development

Do you need to run your function locally that fetches AWS SSM Parameter Store directly? The answer is it is optional, for your local environment, you might not need to use AWS SSM Parameter Store, you can use a .env file to keep your local variables. Below are some approaches you can use to archive that idea, note you still need to test your function with your desired approach on AWS environment:

Use the env-cmd library to load, extract and assign to process.env global object. By running env-cmd serverless offline command, you can access all variables defined in your .env file.
Using the serverless-secrets-plugin to define the environment variable in a secured manner, you can easier to share across the team and commit the encrypted file.

Using the same code as below with a modification, you can skip fetching parameter store from AWS and reduce your bill:

const middy = require('middy');
const { ssm } = require('middy/middlewares');

const isLocalEnv = process.env.IS_OFFLINE || process.env.IS_LOCAL;

export const handler = middy((event, context, cb) => {
  // You can access the parameter value inside function handler
  console.log(process.env.HARVESTAR_PCMSS_DB_CONNECTION);

  // Your business logic here
}).use(
  ssm({
    cache: true,
    // By setting the paramsLoaded, you tell the middleware to
    // not fetch it from AWS SSM
    paramsLoaded: isLocalEnv,
    names: {
      // Should have a prefix that include this micro service i.e. pcmss
      HARVESTAR_PCMSS_DB_CONNECTION: '/dev/harvestar/pcmss/db_connection'
    }
  })
);

#7 - Pay attention to services limits

Likes other AWS services, AWS SSM Parameter Store also has some limits, such as the maximum number of params per account and region, max param value size, max history. Understanding its limits help us design and build applications with high reliability. For example, avoid storing large items into the SSM parameter because of size limits (4KB for standard and 8KB for the advanced parameter). Refer to AWS service limits documentation for more others.

Resources

Conclusion

By applying best practices, you can implement your applications more reliable, secure, efficient, and cost-effective software on the cloud.

I hope this post brings some ideas to you and save your time. There are more interesting and useful articles, so find and read them to get more information. Feel free to let me know your recommendations or suggestions by adding comments below.

Thank you for reading!

What are the best ways to explore Singapore IT startup projects?

Hoang Le — Tue, 19 Nov 2019 10:50:08 +0000

I am trying to find a way to explore the Singapore IT startup industry to find more clients and help startups to build and bootstrap their development to bring their product to the market.

Suggesting ideas and approaches are very appreciated.

Thank you in advance!

My Cloud computing journey

Hoang Le — Sat, 16 Nov 2019 14:27:34 +0000

Cloud computing is a buzzing word nowadays. It appeared as early as 1996, with the first known mention in a Compaq internal document (per Wiki). Many companies are joining the cloud computing journey and archived a lot of benefits.

I have been working on various cloud computing projects from start to finish; involved in the full project lifecycle, including design, planning, development, deployment, testing, maintenance, and support. Some clients prefer to use their on-premise data centers, but most of my projects used the cloud - Amazon Web Services, Microsoft Azure and Heroku. So, I wanted to write this post to share my cloud computing journey. My hope is to encourage you to build secure, scalable, highly available and cost-effective cloud applications. Learn to share and share to learn will help us all grow, don't you agree?

If you are new to the cloud and wanted to explore more information, check out some articles as below:

Cloud Computing from Wiki

This is a lengthy article. So, if you prefer to simply view a particular topic, you can use the table of contents below.

Who am I?
Starting my career as a software developer
The beginning of my cloud computing journey
My startup journey is just started
My own company is officially established
We got the first AWS certification
A few more serverless projects
Our most recent cloud computing projects
Other works
My future plan

Who am I?

Firstly, I'd like to introduce myself. My name is Hoang, I am the Co-founder and CTO of InnomizeTech. My title is CTO but I am a full-stack developer and software architect, passionate about Cloud Computing, Serverless, DevOps, Machine Learning, and IoT.

With 7+ years of experience in software development (web, desktop, and mobile), systems integration, Cloud computing, DevOps engineering, project management, and agile methodology, I thrive working both independently and collaboratively in a team, and have strong leadership and communication skills from working directly with both technical and non-technical stakeholders.

Starting my career as a software developer

Before transitioning to Cloud computing, I worked for an outsourcing company in Vietnam for four years after I graduated from University.

For the first two years, the primary programming language I used was C# (AWS .NET WebForm and AWS .NET MVC).
For the next year, I transitioned to another role, leading a team working with system integration stuff using WSO2 and BizTalk, among other tools. I had a chance to work with our client's CTO. We had a lot of fun and we did a great job.

The beginning of my cloud computing journey

In the third year, I started my first Cloud computing project. We had a project from a US-based client to build an application for activating mobile devices for various carriers. I was the technical lead of that project. After studying the requirements, we selected the following technology stack:

Node.js for back-end, using the Sails.js framework
MySQL database and using Sequelize for ORM. We used Liquibase generate, migrate database schema using source control.
Angular for the front-end application

Actually, it isn't a real cloud-based application, since we used only a few cloud services to build the app. We have built the app and used the Elastic Beanstalk service for hosting. I have studied Jenkins and built a Jenkins job for analyzing code, building, testing, and deploying to the Elastic Beanstalk app. Here is the diagram of the CI/CD I've built at that time:

CI/CD with Jenkins CI

We initially hosted the application on Elastic Beanstalk apps in order to apply our CI/CD script using AWS Beanstalk plugin on Jenkins for deployment. However, our client wanted to host it on their data center. Then we still use Beanstalk for our SIT/UAT environments, but we had to write documentation around 10 pages to describe how to set up Web Servers, App Servers, NGINX, Node.js, RDS, etc. A lot of back and forth communications between our development team and infra team because of issues and errors. But finally, we have successfully deployed, the app is up and running.

We kept working on that project for around a year. While developing new features, I also learned Ansible and built some playbooks to allow my client to deploy the app automatically without manual steps.

Continuous Deployment - Deploying A Node.Js App To AWS EC2 Using Ansible

At that time, I and my co-workers have established a technical club to study and share about cloud computing. We learning cloud computing concepts, doing some hand-on exercises with AWS and Microsoft Azure. You can visit our Facebook page to see our activities and great memories.

Before leaving the company, I have studied and shared my CI/CD experiences. I have presented this presentation for my promotion to the Senior Engineer position (I have changed a few things to share with everyone).

My startup journey is just started

In August 2017, I have started my own job. We had a project for building a core/shared backend API for various client applications including SPA, Mobile App, Integration services. After clarifying requirements with our clients, we have selected the below technology stack:

AWS Lambda using Node.js runtime and Amazon API Gateway that utilize Auth, Caching, Logging, Monitoring Throttling, Bursting Elastic features for backend services.
We use the Serverless framework for building API, deploy and configure AWS resources such as S3 bucket, SNS topic, Queue, DynamoDB. At this time, we used JavaScript as the primary programming language.
Amazon Cognito User Pool for authentication and authorization.
Amazon Aurora for the relational database. We keep using Sequelize and Liquibase.
Amazon Elastic Cache (Redis) for in-memory caching.
Other AWS services we have used such as VPC, S3, SNS, SQS, CloudWatch, Route53, Certification Manager, AWS System Manager, AWS Parameter Store, X-Ray
Angular for the frontend app. Static hosting with S3 and CloudFront.
Jasmine, Protractor for unit test, integration test and E2E test.
We still use Jenkins for our DevOps server. But instead of defining the job manually on the admin console page, we wrote a Jenkinsfile that using the Mutliplebranch pipeline.
And more...

Jenkins Pipeline Execution

We kept applying the same stack to a few projects for other clients and continuously improve and refactoring our core modules/libraries. We have delivered and received positive feedback from our clients. The applications were running well, lower cost, high availability, secured.

In relation to DevOps, we have all the stuff scripted and provided CLIs to allow build and deploy applications automatically. We've used CloudFormation to provision and configure infrastructure components such as VPC, RDS. We outlined steps to deploy/remove and send it over to our client, DevOps engineers, they can deploy a new environment using a few commands then remove completely without manually step. Automation reduces human mistakes and increases productivity and quality.

My own company is officially established

In October 2018, we officially established InnomizeTech, our mission is to create awesome products that will make your everyday life easier. My aim is to build a professional team with passion, enthusiasm, and talent that can help bring more value to our customers, help them moving fast and right direction.

Experienced engineering and development team at Innomize are committed, passionate, and continually challenge themselves on the advancements and changes of the technologies we utilize to deliver the most up-to-date and innovative solutions.

We got the first AWS certification

I and my team have been learning, building a lot of real-world cloud-based applications for many years. To prove with our clients that we had experiences to do their jobs, we have studied and taken some AWS exams and finally, our team got AWS certifications:

02 AWS Certified Solution Architecture - Associate
02 AWS Cloud Practioner
01 AWS Certified Developer - Associate

We have been with The Coral Edge our partner and primary client. Recently, they officially became the Select Consulting Partner with AWS and we are their Development Team.

A few more serverless projects

Early 2019, we had a contract with another client from the US, we will help their development team maintenance and build new features for their micro-services (the integration component for their e-commerce system). They used Scala as their programming language, Gradle for the build tool, their developer implemented some Gradle tasks to deploy Lambda function, create the API gateway, and other required AWS resources. Here is the technology stack I can summary:

Each microservice has its own Lambda function and API Gateway as the original design. They had an NGINX server as a reserved proxy server that will route traffic to AWS API gateways based on the request path.
http4s as the interfaces for Restful API.
They used the Postgres database and also using Liquibase to manage their database schema and versioning. They used Doobie for functional JDBC for Scala.
Sentry for error monitoring and reporting.
They do use AWS CodeBuild, AWS CodeDeploy, and AWS CodePipeline to implement CI/CD but it isn't fully implemented when we join their team.

When our developers working with their team, we have proposed some approaches to enhance and refactor their design:

To remove their NGINX server and cost, we suggest they apply the Robust API pattern.
We helped design and built a project template in order to add a new microservice easily. Here are some of our ideas:
- Defining a project template that uses AWS CodeStart which will allow us creating CodeCommit repository, define CodeBuild projects, CodePipelines automatically when creating a new project. New microservices can be up and running by using a single command.
- We suggest using AWS SAM framework for building Lambda function and defining required AWS resources (i.e. Queue, Bucket, Topic, etc.). Previously, they deployed Lambda and other resources by implementing custom Gradle tasks.

Our most recent cloud computing projects

Recently, we have been working on some projects that apply Micro-service architecture. We continue the above technology stacks with some changes/improvements:

We used NX Dev Tool to define our workspace on a Monorepo. We built our CI/CD pipeline that supports deploy only affected apps by a change instead of redeploying all services. Thanks to the NX team for their idea and awesome tool.
Step Functions for our micro-service communication, automated workflow.
NestJS for our Restful API that running on AWS Lambda.
AWS CDK to provision and configure our infrastructure as code (IoC)
Jest, Cypress for unit test, integration test, and E2E testing.
Along with using Cognito we also using Okta, AuthO for authentication and authorization.
CI/CD with AWS Develop Tools, Jenkins, Azure DevOps.
Along with using Node.js runtime, we also using Java/Scala, .NET Core runtime.

Other works

We focused on AWS as the primary cloud vendor and wanted to become an active development service as partnering with them. Along with cloud computing, our team also has experience in other technologies. We work with the latest technologies specializing in Web Development, Mobile Development, Cloud Computing (AWS, Azure), DevOps, and Automation Testing. Refer to our Services section on our website for more information.

We continue working on some other projects such as:

DevOps using CloudFormation, AWS CDK, Ansible
Amplify framework: hosting, authentication, Mobile App with AppSync.
Ruby on Rails app on AWS ECS.
Headless upload bills to various sites with AWS ECS and Protractor (End to end automation test with Protractor).
Serverless web crawler with Puppeteer on AWS Fargate that I recently wrote that post.
React, Amplify, AppSync.
Work with other AWS services such as AWS WAF, AWS Shield, Glue, ElasticSearch, Kenesis, Athena.
We helped some clients design and define DevOps, CI/CD pipeline using AWS Developer Tools, Azure DevOps.

My future plan

My quote is "Learn to share and share to learn", I am continually challenging myself on the advancements and changes of the technologies. I wanted to meet more people, talk, work and help them. I do have a few things in my list to I am going to do:

Build INNOMIZE stronger, work with more clients and collaborate with more people.
Learn more things such as Machine Learning, Deep Learning, Transfer Learning, Convolution Neural Network, IoT.
Build my own product.

I cannot describe anything about myself on this post, if you wanted to know more about me and our team, then you can get in touch with me via:

If you are looking for developers, offshore team, or need consulting about the AWS cloud, Serverless architecture, and so on, then hire us, we can help you!

Thank you for reading!

Top reasons why we use AWS CDK over CloudFormation

Hoang Le — Tue, 12 Nov 2019 02:58:59 +0000

If you're working with AWS Infrastructure, you may know that currently there are some tools/frameworks support to implement your AWS infrastructure such as CloudFormation, Terraform, AWS CDK. What is the best tool that your team can rely on and use? What is the best tool can help you increase productivity and quality? Do you have the answer? If not, read this post, I will give you our answer and reasons.

Our team has been working with AWS for a couple of years. We have designed and implemented infrastructure components including networking, server, storage, and other AWS services either manually or automation. Before using AWS CDK, we have used AWS CloudFormation, but working with it is a challenge, hard to define and get an overview for a complex template in either JSON or YAML format.

In this post, I will not provide specific detail of AWS CDK. If you don't know what it is, then check out the AWS CDK home page to get some overviews and see how it works.

Sample code how to create a VPC on AWS with AWS CDK

We select AWS CDK as the primary tool for our Infrastructure as Code (IaC) because of the following reasons:

With AWS CDK we "WRITE LESS AND DO MORE"

We first impressed AWS CDK, when we try to find alternative ways to reduce complexity and overcome challenges of using CloudFormation. For example, we have a CloudFormation template to define and configure networking resources including VPC, subnets, route tables, security groups, bastion hosts, integrate gateway, nat gateways, etc. We need to write around 1000 lines of code in JSON format using CloudFormation. When we try with AWS CDK, we only need to write around 50 lines of code. As you can see, AWS CDK can do the same thing, even add more features such as conditional number of NAT gateway, number of subnets and availability zones.

The following code will create all required resources that allow you to create a new VPC on AWS in minutes, you can then modify or add more resources depends on your requirements:

import * as cdk from '@aws-cdk/core';
import * as ec2 from '@aws-cdk/aws-ec2';

export class NetworkStack extends cdk.Stack {
  public readonly vpc: ec2.Vpc;

  constructor(scope: cdk.Construct, id: string, props?: cdk.StackProps) {
    super(scope, id, props);

    this.vpc = new ec2.Vpc(this, 'VPC', {
      cidr: '10.0.0.0/16',
      natGatewaySubnets: {
        subnetName: 'Public'
      },
      subnetConfiguration: [
        {
          cidrMask: 26,
          name: 'Public',
          subnetType: ec2.SubnetType.PUBLIC
        },
        {
          name: 'Application',
          subnetType: ec2.SubnetType.PRIVATE
        },
        {
          cidrMask: 27,
          name: 'Database',
          subnetType: ec2.SubnetType.ISOLATED
        }
      ]
    });

    const vpcSecurityGroup = new ec2.SecurityGroup(this, 'SecurityGroup', {
      vpc: this.vpc,
      description: 'Allow ssh access to ec2 instances',
      allowAllOutbound: true
    });

    vpcSecurityGroup.addIngressRule(
      ec2.Peer.anyIpv4(),
      ec2.Port.tcp(22),
      'allow ssh access from the world'
    );
  }
}

Easy to share and reuse your infrastructure as a library

Declarative infrastructure as code tends to be in languages that we don’t use every day like HCL, YAML, or JSON. AWS CDK uses object-oriented, provides abstraction techniques to create a model of our system, we can define modules to share across projects. We can use built-in construct libraries or from the community that can increase our productivity.

CDK is a developer-friendly version of Cloud Formation

AWS CDK is an imperative programming language, supporting Java, JavaScript, Python, TypeScript and .NET. We can utilize our developer programming skills to reduce the time for learning a new syntax like Terraform. Think about a project that we use TypeScript as the primary programming language for Front-end, Back-end, CI/CD and IoC.

Easy to use logic (if statements, for-loops, etc) when defining your infrastructure

We know how to write an if or for-loops, right? How hard is it to write a condition when using CloudFormation? If you know the answer, then you will know why love AWS CDK.

Condition logic with AWS CDK

Easy to get and integrate with our coding review workflow

AWS CDK provides commands to generate the CloudFormation template, so we can still review the generated CloudFormation template before applying, it also can generate dif that makes it easy to review and make the decision.

Code completion within your IDE

Does anyone remember how to define a full detail of an AWS resource using CloudFormation? Is there any schema or intelligent for the CloudFormation template that you can use for code completion to speed up your coding time?

Code completion with AWS CDK

Allow us to test our code

AWS CDK allowing us to perform snapshot tests, fine-grained assertions, help us build reliable and testable code, here is the post that provides step by step guide to write and run unit tests using Jest. With CloudFormation, you also can write some tests, but it requires another tool, check out this post more details.

Easy to integrate with our CI/CD process

Deployment can be done with one single command requiring installing any additional tool. We also use AWS CDK to define our CI/CD pipeline using AWS DevOps tools such as CodeBuild, CodeDeploy, which is much better than writing a lot of CloudFormation code.

The AWS CDK just a high level of CloudFormation, so if you want to how exactly it works, my suggestion is to have a try with CloudFormation, do some works around it, then try to convert it into AWS CDK, you will find out whether you will continue with CloudFormation or change to use AWS, or maybe you can change to other tools such as Terraform.

Nothings is the best, it depends on your requirement, team, project and your company. Have a nice coding!

This post is originally published on our blog.

Hoang Le

Learn to Share and Share to Learn.

DEV Community: Hoang Le

12 Hidden AWS Costs That Silently Drain Your Budget

Data transfer: the invisible cost

Snapshot accumulation: the backup nobody deletes

Dormant resources: the zombies in your account

Case study: when "copy-paste infrastructure" costs $8,000/month extra

Case study: when instance sprawl blocks your Savings Plans

NAT Gateway charges: the set-and-forget tax

CloudWatch logs: infinite retention, infinite costs

The EKS Control Plane logging trap

Consider self-hosted observability

The small charges that add up

Cross-region replication: enabled once, runs forever

Third-party marketplace licensing: the hidden multiplier

Request-based pricing at scale

Reserved capacity underutilization: paying for commitments you don't use

The pattern behind hidden costs

What to do next

Related reading

The Builds That Last Manifesto

The gap between posts and reality

The iceberg problem

The speed trap

Maintenance costs more than building

The AI paradox

What this means for engineers

What this means for leaders

The bottom line

AWS re:Invent 2025: 3 Announcements That Matter for Your Architecture

1. Lambda gets Durable Functions and Managed Instances

2. EKS Capabilities: managed ArgoCD, ACK, and KRO

3. Database Savings Plans and SQL Server Developer Edition

The pattern across all three

What's next

Exploring Serverless Billing Management: Architecture and Cost

Overview of the System

System Architecture

Serverless Compute Services with Lambda, API Gateway

Datastore with Amazon RDS MySQL and S3

Authentication with Amazon Cognito

Event Sources with S3 and HTTPS Requests from API Gateway

Billing Portal with Angular, S3, and CloudFront

Mobile Application with React Native

Headless Automation for LEDES Uploads with CDK, AWS Fargate, Selenium, and Protractor

Monitoring and Logging with AWS CloudWatch, Sentry

IaC with Serverless Framework and AWS CDK

Cost Breakdown

Database Costs

Lambda Costs

ECS (Fargate) Costs

API Gateway Costs

Other Services Costs

Total Monthly Cost

Optimizing Costs

Summary

Ready to Embrace Digital Transformation? Contact INNOMIZE Today!

Continuous Delivery - Deploying a Node.js app to AWS EC2 using Ansible

The Tools

Why Ansible?

The Steps

Let’s coding…

Conclusion

Check list to avoid scam for Freelancer

re:Invent 2019 announcements

aws #cloudcomputing #clouddevelopment #cloudconsulting #cloudengineer #news #announcements #serverless

A brief summary of JavaScript Learning Path

Step 1 - Learn basic and fundamentals

Step 2 - Learn modern JavaScript

Step 3 - Coding style and Convention

Step 4 - Review your knowledge

Step 5 - Practices

Step 6 - Be an expert

07 best practices when using AWS SSM Parameter Store

What is AWS Systems Manager Parameter Store (aka SSM Parameter Store)?

What are the benefits?

#1 - Organizing parameters into hierarchies

#2 - Consistent naming convention

#3 - Restrict IAM permission

#4 - Combine into a single parameter likes database connection string and keep it all together (co-location)

#5 - Use tool/library to fetch, cache, and export to environment variables at runtime