DEV Community: Mursal Furqan Kumbhar

I Let a Bedrock Agent Watch My AWS Bill for 30 Days. Here Is Everything It Caught, Missed, and Made Up

Mursal Furqan Kumbhar — Sun, 05 Jul 2026 20:07:54 +0000

Ciao Amici 👋

Pull up a chair, because I have a story for you. It involves a forgotten GPU endpoint, a robot accountant I built over two evenings, a NAT gateway that quietly ate my wallet, and one genuinely absurd moment where my own creation reported itself to me as suspicious activity. For thirty days I let an Amazon Bedrock Agent read my AWS bill every morning and tell me what it thought, and by the end of this story you will have every line of code, every chart, every dollar figure, and every embarrassing mistake I made along the way. Some of those mistakes were mine. Two of them, delightfully, were the agent's. Make some chai, this is a long one, and I promise it earns its length.

The morning it all started

Last November I opened my AWS billing dashboard and felt that very specific cold feeling in the stomach that every cloud engineer knows. A SageMaker real-time endpoint I had spun up for a quick demo had been running for eleven days. Eleven days of me paying for a GPU-backed instance to serve exactly zero requests.

The endpoint cost me about $38. Not catastrophic. But here is the thing: it was the third time in a year this movie had played. Before that, an Elastic IP sat unattached for a month. Before that, a NAT gateway I had completely forgotten was part of a VPC I built for a workshop. Each time, the same script: I discover the charge weeks later, feel briefly stupid, delete the thing, and promise myself I will check the console more often. I never do, because manually checking a billing console is exactly the kind of chore human brains are engineered to avoid.

And yes, I had CloudWatch billing alerts configured, like a responsible adult. But a threshold alert only tells you that you are spending money. It never tells you why, and it definitely never tells you what to do about it. An alert saying "your bill crossed $80" at 2 AM is not insight. It is anxiety with a timestamp.

So, staring at that dead endpoint, an idea landed. What if I stopped being my own billing alarm entirely? What if, every morning, someone else read the bill for me, someone tireless, slightly paranoid, and immune to the phrase "I will deal with it after the weekend"?

That someone became a Bedrock Agent. For 30 days I gave it read-only access to Cost Explorer and a handful of describe APIs, scheduled it to run every morning, and told it to behave like a cautious FinOps consultant. Every day at 9 AM Karachi time it would analyze my spend, flag anything weird, estimate savings, and email me a report with the subject line "The Accountant," because if a robot is going to lecture me about money, it should at least have a title.

This is the full story of that month. Spoiler: I am still running it. Longer spoiler: it once found a database that does not exist.

Why an agent, and not just budgets and alarms?

A colleague asked me this on day two, with a raised eyebrow that suggested he already knew part of my answer would be "because it sounded fun."

It did sound fun. But there is a real gap here, and it decides whether this whole pattern is a toy or a tool, so let me make the case properly.

AWS Budgets, billing alarms, even Cost Anomaly Detection: these are threshold and statistics tools. They compare numbers to lines. They are excellent at "did I spend more than X" and reasonable at "is today statistically unusual." What they cannot do is answer the questions that actually eat your time when a bill goes wrong:

Why did my bill go up 40% this week, in plain language?
Is this NAT gateway data processing charge normal for my workload, or a routing misconfiguration?
Which of my resources are technically running but functionally idle?
If I keep going at this rate, what does month-end look like, and which line items are driving it?
Is this spike a problem, or the training run I kicked off on purpose yesterday?

That last one matters more than people admit. Anomaly detection without context produces alerts you learn to ignore, and an ignored alert is worse than no alert. The difference between "spend increased" and "spend is wrong" is a judgment call, and judgment calls require reasoning over context. That is exactly the shape of problem LLM agents are decent at, provided you feed them structured data and keep them on a very short leash.

The short leash part is not a figure of speech. It is the entire safety story of this project, and you will see it earn its keep on day 16.

There was a quieter motivation too. I spend a lot of my community time around ML tooling, and I kept seeing agent demos that book flights or order pizza, tasks nobody actually wants automated by a nondeterministic system. Billing analysis felt like the opposite case: tedious, recurring, data-rich, and low-stakes as long as the agent can only read. If agents are useful anywhere in a personal AWS account, they should be useful here. Consider the next five thousand words the test of that hypothesis.

Meet The Accountant

Here is the whole system. It is deliberately boring, and every boring choice was on purpose.

Five pieces:

EventBridge Scheduler fires once a day at 04:00 UTC, which is 9 AM for me in Karachi. Daily is the right cadence for a personal account. Hourly would multiply the watcher's own cost by 24 for almost no extra insight, because Cost Explorer data lags a few hours anyway.
A small trigger Lambda invokes the Bedrock Agent with a fixed daily prompt, collects the streamed response, and forwards the final report.
The Bedrock Agent (I used Claude via Bedrock, though the pattern works with any agent-capable model on the platform) reads its instructions, decides which tools it needs, calls them, and reasons over the results.
An action group Lambda is the only component with AWS API access. It exposes a small menu of tools: get cost and usage, get cost forecast, list SageMaker endpoints, list EC2 instances, describe NAT gateways, find unattached EBS volumes and Elastic IPs, and describe RDS instances.
The agent's final report lands in my inbox via Amazon SES, formatted as HTML with a color-coded status banner, and impossible for me to claim I did not see. Email was a deliberate choice over chat tools: I check my inbox every morning anyway, it keeps the whole stack inside AWS, and a month of reports becomes a searchable archive for free.

Now the tattoo-worthy design decision, the one I would print inside the cover of every agent tutorial if publishers let me: the agent has zero write permissions. It cannot stop, terminate, resize, delete, or modify anything. It can only look and complain. I wanted a consultant, not an intern with root access.

Hold that thought. It becomes the hero of this story later.

Building my robot accountant

Let me walk you through the build in the order I actually did it, including the parts that wasted an evening, because the wasted evenings are where the useful knowledge lives.

Step 1: The IAM policy

Everything the action group Lambda can do fits in one tight policy. Read it, and notice what is absent:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "CostVisibility",
      "Effect": "Allow",
      "Action": [
        "ce:GetCostAndUsage",
        "ce:GetCostForecast",
        "ce:GetDimensionValues"
      ],
      "Resource": "*"
    },
    {
      "Sid": "ResourceVisibility",
      "Effect": "Allow",
      "Action": [
        "ec2:DescribeInstances",
        "ec2:DescribeVolumes",
        "ec2:DescribeNatGateways",
        "ec2:DescribeAddresses",
        "sagemaker:ListEndpoints",
        "sagemaker:DescribeEndpoint",
        "rds:DescribeDBInstances"
      ],
      "Resource": "*"
    }
  ]
}

No ec2:StopInstances. No sagemaker:DeleteEndpoint. Nothing with a verb that changes state. Whenever the agent recommended killing something, a human (me, holding coffee) did the killing after a ten-second sanity check.

One small subtlety so you do not repeat my twenty wasted minutes: ce:* actions only work against Resource: "*" because Cost Explorer is account-scoped, so do not bother trying to narrow it.

Step 2: The action group Lambda

Bedrock Agents call your Lambda with a structured event describing which tool the model wants and with what parameters. Your job is to run the right boto3 calls and hand back JSON. Here is a trimmed version of my handler; the full version just has more entries in the tool table and better logging.

import boto3
import json
from datetime import datetime, timedelta

ce = boto3.client("ce")
sm = boto3.client("sagemaker")
ec2 = boto3.client("ec2")

def get_daily_costs(days=14):
    """Cost per service per day for the last N days, one API call."""
    end = datetime.utcnow().date()
    start = end - timedelta(days=days)
    resp = ce.get_cost_and_usage(
        TimePeriod={"Start": str(start), "End": str(end)},
        Granularity="DAILY",
        Metrics=["UnblendedCost"],
        GroupBy=[{"Type": "DIMENSION", "Key": "SERVICE"}],
    )
    out = []
    for day in resp["ResultsByTime"]:
        for grp in day["Groups"]:
            amount = float(grp["Metrics"]["UnblendedCost"]["Amount"])
            if amount > 0.01:  # skip fractional-cent noise
                out.append({
                    "date": day["TimePeriod"]["Start"],
                    "service": grp["Keys"][0],
                    "usd": round(amount, 2),
                })
    return out

def get_forecast():
    start = datetime.utcnow().date() + timedelta(days=1)
    end = start.replace(day=28) + timedelta(days=4)
    end = end - timedelta(days=end.day)  # last day of month
    resp = ce.get_cost_forecast(
        TimePeriod={"Start": str(start), "End": str(end)},
        Metric="UNBLENDED_COST",
        Granularity="MONTHLY",
    )
    return {"projected_remaining_usd": round(float(resp["Total"]["Amount"]), 2)}

def list_sagemaker_endpoints():
    eps = sm.list_endpoints()["Endpoints"]
    return [
        {
            "name": e["EndpointName"],
            "status": e["EndpointStatus"],
            "created": e["CreationTime"].isoformat(),
        }
        for e in eps
    ]

def find_unattached_volumes():
    vols = ec2.describe_volumes(
        Filters=[{"Name": "status", "Values": ["available"]}]
    )["Volumes"]
    return [
        {"id": v["VolumeId"], "size_gb": v["Size"], "type": v["VolumeType"]}
        for v in vols
    ]

def find_unattached_eips():
    addrs = ec2.describe_addresses()["Addresses"]
    return [
        {"ip": a["PublicIp"]}
        for a in addrs if "AssociationId" not in a
    ]

TOOLS = {
    "get_daily_costs": lambda p: get_daily_costs(int(p.get("days", 14))),
    "get_forecast": lambda p: get_forecast(),
    "list_sagemaker_endpoints": lambda p: list_sagemaker_endpoints(),
    "find_unattached_volumes": lambda p: find_unattached_volumes(),
    "find_unattached_eips": lambda p: find_unattached_eips(),
}

def lambda_handler(event, context):
    func = event["function"]
    params = {p["name"]: p["value"] for p in event.get("parameters", [])}
    body = TOOLS.get(func, lambda p: {"error": f"unknown function {func}"})(params)

    return {
        "messageVersion": "1.0",
        "response": {
            "actionGroup": event["actionGroup"],
            "function": func,
            "functionResponse": {
                "responseBody": {"TEXT": {"body": json.dumps(body)}}
            },
        },
    }

Two lessons from this file, each purchased with part of an evening:

Cost Explorer calls cost money. Each GetCostAndUsage request bills at $0.01. That sounds like nothing until your agent decides to be thorough and makes forty calls in one session, fetching one day at a time. I added an explicit instruction telling it to fetch 14 days in a single call. After that, the system settled at six to eight Cost Explorer calls per day.

Empty results need to say they are empty. Early on, my tools returned a bare [] when nothing was found. Later I changed them to return {"found": 0, "items": []} with an explicit count. Remember this detail. It is Chekhov's empty list, and it goes off on day 16.

Step 3: The agent itself

In the Bedrock console: create an agent, pick a model, paste instructions, attach the Lambda as an action group, and define each function with its parameters. The console generates the function schema for you these days, which removed most of the friction this setup used to have.

The instruction prompt matters more than any other component in this system. Mine went through five versions, and the diffs between them tell the real story of the month. This is where it ended up, lightly condensed:

You are a cautious FinOps analyst reviewing a personal AWS account.
The owner is a solo developer running ML experiments, a portfolio
site, and occasional demos. Typical monthly spend is 60 to 120 USD.

Every day you will:
1. Fetch the last 14 days of daily costs grouped by service,
   in a single call.
2. Compare today's trajectory against the trailing average.
3. Check idle-prone resources: SageMaker endpoints, unattached
   EBS volumes and Elastic IPs, NAT gateways, and stopped EC2
   instances with attached storage.
4. Fetch the month-end forecast.
5. Report findings in this exact structure:
   STATUS: green / yellow / red
   HEADLINE: one sentence
   ANOMALIES: bullet list, or "none"
   RECOMMENDATIONS: bullet list with estimated monthly savings
   as a range for each, or "none"
   FORECAST: projected month-end total
   WATCHER OVERHEAD: today's estimated cost of running this
   analysis (Bedrock + Cost Explorer calls)

Rules:
- Never recommend an action you cannot support with data you
  fetched in this session. If a tool returned zero items, there
  is nothing to recommend about those items.
- If you are unsure whether a resource is intentional, say so
  and ask. Do not assume.
- Distinguish between "spend increased" and "spend is anomalous".
  A planned training run is not an anomaly.
- State every savings estimate as a range, never a point figure.
- Your own Bedrock and Lambda usage is expected. Report it under
  WATCHER OVERHEAD, never under ANOMALIES.

If that prompt reads like it was written by someone who got burned, good instincts. Every rule after the numbered list was added mid-experiment in response to a specific failure. Scar tissue, in prompt form. The stories behind those scars are coming.

Step 4: The trigger Lambda and email delivery

The trigger Lambda is short. It invokes the agent, collects the streamed chunks, and sends the report through Amazon SES as an HTML email with a status-colored banner at the top:

import boto3
import os

agent_rt = boto3.client("bedrock-agent-runtime")
ses = boto3.client("ses")

COLORS = {"green": "#2eb67d", "yellow": "#ecb22e", "red": "#e01e5a"}

def lambda_handler(event, context):
    resp = agent_rt.invoke_agent(
        agentId=os.environ["AGENT_ID"],
        agentAliasId=os.environ["AGENT_ALIAS_ID"],
        sessionId=context.aws_request_id,
        inputText="Run the daily billing review.",
    )

    report = ""
    for chunk in resp["completion"]:
        if "chunk" in chunk:
            report += chunk["chunk"]["bytes"].decode()

    status = "yellow"
    for line in report.splitlines():
        if line.strip().upper().startswith("STATUS:"):
            status = line.split(":", 1)[1].strip().lower()
            break

    html = f"""
    <div style="font-family:Menlo,Consolas,monospace;max-width:640px">
      <div style="background:{COLORS.get(status, '#ecb22e')};
                  color:#fff;padding:10px 16px;border-radius:6px;
                  font-weight:bold;font-size:16px">
        The Accountant &middot; {status.upper()}
      </div>
      <pre style="white-space:pre-wrap;font-size:13px;
                  line-height:1.55;padding:8px 4px">{report}</pre>
    </div>
    """

    ses.send_email(
        Source=os.environ["FROM_ADDRESS"],
        Destination={"ToAddresses": [os.environ["TO_ADDRESS"]]},
        Message={
            "Subject": {
                "Data": f"The Accountant · {status.upper()} · daily review"
            },
            "Body": {
                "Html": {"Data": html},
                "Text": {"Data": report},
            },
        },
    )
    return {"status": status}

The Lambda's execution role needs one extra permission, ses:SendEmail, and that is the only write-adjacent permission in the whole system. It writes to my inbox, nothing else.

One SES gotcha before you build: new SES accounts start in sandbox mode, meaning both sender and recipient addresses must be verified identities. For this project that is actually fine, even pleasant. Verify your own address once in the SES console, click the confirmation link, done. You never need production access because you are only ever emailing yourself. The rare case where the sandbox is a feature.

The status word in the subject line turned out to be an underrated detail. After a week my inbox itself became a dashboard: a column of GREEN subjects I could skim past, with the occasional YELLOW or RED that actually earned a tap. And unlike a chat message, the reports pile up into a searchable archive. Mid-experiment I searched "NAT" in my mail app and had the full paper trail of an incident in three seconds.

Note the sessionId: fresh one per day, so each morning is a blank slate. I considered giving it memory of previous reports and decided against it for the experiment, because I wanted every finding to be reproducible from that day's data alone. Version two revisits this, and I will tell you why at the end.

Prefer WhatsApp instead?

Two people asked me this before the experiment even ended, which around here is not surprising; WhatsApp is where life happens. It works, with one honesty caveat: the AWS-native route, AWS End User Messaging Social, requires linking a WhatsApp Business Account, and Meta's business verification is the tedious part, not the code. Once linked, the swap is a few lines:

social = boto3.client("socialmessaging")

social.send_whatsapp_message(
    originationPhoneNumberId=os.environ["WA_PHONE_ID"],
    metaApiVersion="v20.0",
    message=json.dumps({
        "messaging_product": "whatsapp",
        "to": os.environ["MY_NUMBER"],
        "type": "text",
        "text": {"body": f"The Accountant · {status.upper()}\n\n{report}"},
    }).encode(),
)

If you just want it working in ten minutes for personal use, the Twilio WhatsApp sandbox is the pragmatic shortcut: one POST request from the same Lambda, no Meta paperwork. I stayed with email because it kept the entire stack inside one AWS account and gave me the archive, but a RED alert buzzing your phone on WhatsApp is a legitimately better interrupt for genuine emergencies. The architecture does not care; delivery is the last five lines of one Lambda, so pick the channel you actually look at.

Total build time, including the prompt fiddling and the email formatting I was unreasonably fussy about: around six hours across two evenings. Then I scheduled it, went to sleep, and waited for my new employee's first day.

The diary

Now the part you actually came for. I kept notes every day of the experiment. Here are the highlights, week by week. All figures come from my account during the experiment window; your account will tell its own story, which is rather the point.

Week 1: The agent finds my dirty laundry immediately

Day 1. First report lands in my inbox at 9:02 AM, YELLOW glaring from the subject line. It flagged a SageMaker endpoint named vitb16-demo-endpoint that had been running for six days. Estimated waste: $28 to $34 per month. I knew this one. It was serving a palmprint recognition demo I had shown a colleague exactly once, weeks after building the model, and I had told myself I would take it down "after the weekend." The agent did not care about my intentions. I killed the endpoint before finishing my tea. First blood to the robot, on its first morning, in under two minutes of compute.

Day 2. Green. The report was three lines long. I felt an entirely irrational sense of approval, like passing a surprise inspection.

Day 3. Green status, but with a line I did not expect: "Note: 3 unattached gp2 volumes totaling 240 GB, approximately $24/month. These may be remnants of terminated training instances." They were exactly that. Leftovers from my thesis experiments, volumes from terminated GPU instances where DeleteOnTermination had been false. I genuinely did not know these existed. They do not appear anywhere you casually look; they just sit in the EC2 console under a tab nobody visits, quietly billing. I deleted two immediately and snapshotted the third before deleting it, in case it held a dataset I cared about. It held nothing. It never does.

Day 5. The agent noted my Route 53 hosted zone and domain charges as "consistent and expected for a portfolio site" and then never mentioned them again all month. Small thing, but this is the moment I started trusting it. It had built a sense of what normal looked like and stopped crying wolf about it. Threshold alarms cannot do this. They have no concept of "expected."

Day 7. End of week one. Total recurring waste identified: roughly $55 to $60 per month, on an account averaging around $107. Sit with that for a second. More than half my baseline spend was waste, and an LLM found essentially all of it in its first three mornings. I had audited this account myself two months earlier and missed the volumes completely. My pride recovered. Eventually.

Week 2: The NAT gateway incident

Day 8. Status yellow, trending toward alarmed. "EC2-Other charges increased significantly day-over-day."

Day 9. Status red. Headline: "EC2-Other charges increased 6x over the trailing average, driven by NAT gateway data processing."

Here is the real-life situation behind it, and it is the kind of mistake that is obvious only in hindsight. I had deployed a scraper into a private subnet for a dataset collection job, pulling Sindhi news articles for a language benchmark project I have been building on the side. Private subnet felt like the responsible default. But private subnet means every byte of outbound traffic transits the NAT gateway, and NAT gateways charge $0.045 per GB processed on top of their hourly rate. My scraper was pulling gigabytes of HTML through that gateway like the meter did not exist, because I did not know the meter existed.

The agent did not just flag the spike. It named the mechanism and proposed the fix: "If a workload in a private subnet is downloading large volumes of external data, evaluate whether it requires private networking. For S3-bound traffic, a gateway VPC endpoint processes data at no charge."

I moved the scraper to a public subnet with a security group locked to egress only, since it held no credentials or sensitive data, and added the S3 gateway endpoint for the upload path. Charges collapsed the next day. Estimated savings for the rest of the scraping job: about $19. More valuable than the money was the education; I now check the network path of every batch workload before launching it, a habit purchased for the price of two red mornings.

Day 11. Green. The agent noted the EC2-Other line had "returned to baseline following the routing change," meaning it connected this morning's data to the anomaly it flagged two days earlier, within a single fresh session, purely because the 14-day window contained both. Emergent context from a dumb design decision. I will take it.

Day 12. A quiet day, but the forecast line caught my eye: "Projected month-end: $71 to $79, versus $107 trailing-month baseline." Watching that forecast walk downward over the month became weirdly motivating, like a fitness tracker for infrastructure.

Week 3: The agent lies to me twice

Every honest story about an LLM experiment needs this chapter, so here is mine, in full, no flattering edits.

Day 16. Status yellow. The agent claimed my RDS instance was "idle with minimal connection activity" and recommended stopping it, estimated savings $12 to $15 per month. Sensible advice with one flaw: I do not have an RDS instance. Never did, in this account.

I pulled the session trace to see what happened. The describe_db_instances tool had returned an empty list, my early-version bare []. Somewhere between that empty array and the final report, the model's reasoning turned "no instances found" into "an idle instance was found." It is a fascinating failure if you stare at it long enough: the concepts of idle and absent live close together in the model's semantic space, and nothing in my tooling or prompt forced the distinction. Made worse, of course, by the confident dollar range attached to the phantom.

Two fixes went in that afternoon. The tools started returning explicit counts ({"found": 0, "items": []}, Chekhov's empty list finally fixed), and the instructions gained the rule you saw earlier: never recommend an action unsupported by data fetched in this session; if a tool returned zero items, there is nothing to recommend about those items. The phantom never returned. Eleven subsequent days, zero fabricated resources.

Day 19. It happened again, in a form I find funnier every time I think about it. The agent reported Amazon Bedrock charges as an anomaly and recommended I "review unexpected Bedrock usage in the account."

The Bedrock usage was the agent itself.

It had discovered its own operating costs in the billing data and reported itself to me as suspicious activity. I laughed out loud, screenshotted the email, and then realized this is a genuine and slightly deep problem: a watcher built on metered infrastructure will eventually observe its own footprint, and unless it has a concept of self, it will classify itself as an intruder. My fix was prosaic rather than philosophical: a WATCHER OVERHEAD line in the report format, and an instruction that its own Bedrock and Lambda charges are expected and belong there. Somewhere in that incident there is a lovely essay about self-models that I am choosing not to write today.

The generalizable lesson from both failures, and the one thing I hope you quote from this whole piece: agent reliability is mostly a tool design and prompt engineering problem, not a model problem. Every failure I saw in 30 days traced back to an ambiguity I had left lying around, either in what the tools returned or in what the instructions permitted. Fix the ambiguity and the failure mode dies. I never once had to change the model.

Day 21. Green, uneventful, and the first day the new report format ran clean end to end. WATCHER OVERHEAD: $0.17. The accountant now accounts for itself.

Week 4: Steady state

The final week was almost boring, which is the whole point of the system.

Day 24. Yellow, correctly. An Elastic IP had been sitting unattached for two days after I terminated a test instance, at the newer per-hour idle pricing. About $3.60 a month. I released it from my phone while waiting for a haircut, which is exactly the level of effort billing hygiene should require.

Day 25. Yellow, and this one made me smile: "Elevated SageMaker training charges, consistent with an intentional training workload. Not classified as anomalous." I had kicked off a fine-tuning run the night before. The agent saw a spike, checked its character (training jobs, not endpoints; bounded, not recurring), and filed it exactly as instructed under the distinction between increased and anomalous. This is the report that alarms and statistical detectors cannot write, and it is the single best argument for the whole approach.

Day 28. The agent noticed something I never asked it to look for: nonzero inter-region data transfer, on an account where everything nominally lives in one region. It suggested a cross-region S3 access, and it was right; an old script of mine was reading a bucket in another region, a leftover from an experiment two years ago. Savings of maybe a dollar a month. The dollar is irrelevant. The fact that it went looking is not.

Day 30. Green. Final forecast: $76 to $78. Actual month-end: $76.10. I bought myself a decent lunch with a fraction of the difference and considered it a performance bonus paid to the wrong party.

The receipts

The month in one table. "Baseline" is my average across the previous three months.

Metric	Baseline (avg)	Experiment month	Change
Total monthly spend	$107.40	$76.10	-29%
Idle/waste spend (est.)	~$58	~$6	-90%
Billing surprises	1 to 2 per month	0	gone
Time I spent on billing	~45 min/month	~10 min/month	-78%
Watcher overhead	$0	$4.87	new cost

And the full recommendations log, verdicts included:

Day	Finding	Est. monthly impact	Verdict
1	Idle SageMaker endpoint	$28 to $34	Real. Killed it.
3	3 unattached EBS volumes	~$24	Real. Deleted.
9	NAT gateway data processing spike	~$19 (one-off)	Real. Rearchitected.
16	"Idle RDS instance"	$12 to $15	Hallucinated. No RDS exists.
19	"Unexpected Bedrock usage"	n/a	The agent found itself.
24	Unattached Elastic IP	~$3.60	Real. Released.
25	SageMaker training spike	n/a	Correctly labeled intentional.
28	Cross-region S3 access	~$1	Real. Fixed the script.

Six correct findings, one correct non-finding, two false ones. Both false ones were eliminated by tool and prompt fixes and never recurred. I will take that ratio in any system I run.

What the watcher itself costs

Full transparency, because a cost optimization story that hides its own costs would be a comedy:

Component	Monthly cost
Bedrock model invocations (30 daily sessions)	~$3.90
Cost Explorer API calls (~220 total)	~$2.20
Lambda, EventBridge, SES, CloudWatch Logs	under $0.30
Total	~$4.87

Call it five dollars to find fifty-five.

On a personal account the return is comfortable. On a company account with real spend and real sprawl, this is not even a conversation. The same architecture pointed at an organization's consolidated billing, with per-team tags in the grouping dimension, would pay for itself before lunch on day one.

Questions friends asked me along the way

A few of these came from colleagues, a couple from the AWS Community Builders Slack when I mentioned what I was up to.

Why not Cost Anomaly Detection? It is free.
Use both; they are not competitors. Anomaly Detection is a smoke detector: statistical, fast, and context-free. The agent is the person who walks into the kitchen, sees the smoke, and tells you it is the toaster and not the house. On day 25 Anomaly Detection would have flagged my training run. The agent explained it and moved on.

Why fresh sessions daily instead of giving it memory?
For the experiment, reproducibility. Every claim in every report had to be derivable from that day's tool calls alone, or the failures would have been impossible to debug. In steady-state operation, memory of past findings is clearly useful (it could track whether recommendations were acted on), and that is the headline feature of my version two.

Could the model just be pattern-matching generic FinOps advice rather than analyzing my data?
The NAT gateway diagnosis and the cross-region catch were both specific to numbers it fetched in-session, with the mechanism named correctly. The RDS hallucination is actually the counter-evidence in my favor: when it did drift into generic pattern-matching ("accounts usually have an underused database"), it produced garbage, and the fix was forcing every claim to bind to fetched data. The discipline is enforceable.

What about a multi-account org?
Same shape, bigger tools. Point Cost Explorer at the payer account, group by linked account and tag, and give the report a per-team section. The read-only constraint becomes more important there, not less.

Did you worry about sending billing data to a model?
The data stays inside my own AWS account boundary; Bedrock invocations are not used to train the underlying models, per AWS's service terms. For a personal account I was comfortable. A regulated enterprise should read those terms themselves rather than take a blog post's word for it, including mine.

So, should you hire a robot accountant?

The honest answer depends on what kind of AWS user you are.

Build it if you run experiments, demos, side projects, or anything with a lifecycle shorter than your attention span. The idle-resource problem is fundamentally a problem of forgetting, and machines do not forget. Six hours of building bought me a permanent employee who works for five dollars a month and has no opinion about weekends.

Skip it if your account runs one stable production workload that never changes. A static bill needs a budget alert, not an analyst. This tool earns its keep in accounts with churn.

And absolutely do not give the agent write permissions, no matter how elegant the fully autonomous version looks in your head. My agent hallucinated a database on day 16 and priced its imaginary savings with a straight face. Imagine that reasoning chain ending in a stop_db_instance call aimed at whatever resource it decided best fit its imaginary finding. Read-only is not a limitation of this design. It is the design. The human approves; the human executes; the agent never touches anything.

Three smaller lessons I would press into your hands before you go build:

Make the agent state ranges, not points. "Savings of $28 to $34" invites verification. "Savings of $31.20" invites false trust. Precision is a costume that confidence wears.
Teach it your normal in the instructions, including typical spend and known steady charges. Anomaly detection without a baseline is just complaining with extra steps.
Log every session and read the traces when something is off. The trace of which tools were called and what they returned was the only way to tell whether a bad report came from bad data or bad reasoning. In my month it was never the model and always my instructions. Debug yourself first.

Wrapping up

So, amici, that is the story of my month with a robot accountant. It began with a dead GPU endpoint and a familiar sting of embarrassment, and it ended with a 29% smaller bill, an inbox full of color-coded morning reports, and a considerably sharper understanding of where agents genuinely earn their keep. The Accountant found real money, explained real mechanisms, correctly shrugged at my intentional spikes, and got measurably smarter every time I tightened its instructions.

It also invented a database out of thin air and once reported its own existence to me as a suspicious charge, which is why it will remain, forever, a colleague I listen to carefully and never hand the keys. Version two is already half-built: weekly deep-dives, per-project cost attribution through tags, and persistence so it can nag me about recommendations I have ignored.

If you build your own version, and I hope this post leaves you no excuse not to, I would genuinely love to hear what your agent catches in its first week. Find me on Dev.to or LinkedIn and tell me the story. The weirdest finding gets a shout-out in the follow-up post. Until then, keep your endpoints deleted and your subnets public where they can afford to be.

#HappyCoding 👋

30 Tiny React Hooks That Punch Way Above Their Weight

Mursal Furqan Kumbhar — Fri, 26 Jun 2026 04:22:27 +0000

Hello devs 👋

Ever notice how the same ten lines of React keep showing up in every project you touch? You copy them from some old repo, paste them into the new one, rename a variable or two, and quietly pretend you wrote them fresh. Be honest, we all do it. Half of frontend work is just remembering which folder you left that one perfect hook in.

So I stopped pretending and collected the ones I actually reach for. This article is a stash of 30 tiny, battle tested React snippets you can drop straight into your codebase. No extra libraries, no bloat, no npm install marathon. Just copy, paste, and ship.

Bookmark this one. Future you is going to be very grateful to present you.

1. useDebounce

const useDebounce = (value, delay = 500) => {
  const [debounced, setDebounced] = useState(value);
  useEffect(() => {
    const t = setTimeout(() => setDebounced(value), delay);
    return () => clearTimeout(t);
  }, [value, delay]);
  return debounced;
};

Delays rapid state updates so you are not firing on every keystroke. Perfect for search bars and typeahead inputs where you only want to hit the API once the user stops typing. Your backend will send you a thank you card.

2. useClickOutside

const useClickOutside = (ref, handler) => {
  useEffect(() => {
    const listener = (e) => {
      if (!ref.current || ref.current.contains(e.target)) return;
      handler();
    };
    document.addEventListener("mousedown", listener);
    return () => document.removeEventListener("mousedown", listener);
  }, [ref, handler]);
};

Closes a modal, dropdown, or popover when the user clicks anywhere outside of it. Attach the ref to your menu, pass a close function as the handler, and you are done. Ten lines that quietly power half the dropdowns on the internet.

3. usePrevious

const usePrevious = (value) => {
  const ref = useRef();
  useEffect(() => { ref.current = value; }, [value]);
  return ref.current;
};

Gives you the value a prop or state had on the last render. Handy when you want to compare old versus new, like triggering an animation only when a count actually goes up. Tiny hook, surprisingly clutch.

4. useLocalStorage

const useLocalStorage = (key, initial) => {
  const [value, setValue] = useState(() => {
    const stored = localStorage.getItem(key);
    return stored ? JSON.parse(stored) : initial;
  });
  useEffect(() => {
    localStorage.setItem(key, JSON.stringify(value));
  }, [key, value]);
  return [value, setValue];
};

Works exactly like useState but survives a page refresh. Great for theme choices, draft form data, or a dismissed banner that should stay dismissed. Same API you already know, zero new mental overhead.

5. useFetch

const useFetch = (url) => {
  const [data, setData] = useState(null);
  const [loading, setLoading] = useState(true);
  useEffect(() => {
    fetch(url)
      .then((r) => r.json())
      .then(setData)
      .finally(() => setLoading(false));
  }, [url]);
  return { data, loading };
};

A pocket sized React Query. Fetches data and tracks the loading state in under fifteen lines. Not built for retries or caching, but it covers a solid 90 percent of simple screens.

6. useToggle

const useToggle = (initial = false) => {
  const [value, setValue] = useState(initial);
  const toggle = useCallback(() => setValue((v) => !v), []);
  return [value, toggle];
};

Stop writing setOpen(prev => !prev) for the hundredth time. This gives you a boolean and a one word toggle function. Modals, sidebars, accordions, password visibility, all of it gets cleaner.

7. useIsMobile

const useIsMobile = (breakpoint = 768) => {
  const [isMobile, setIsMobile] = useState(window.innerWidth <= breakpoint);
  useEffect(() => {
    const onResize = () => setIsMobile(window.innerWidth <= breakpoint);
    window.addEventListener("resize", onResize);
    return () => window.removeEventListener("resize", onResize);
  }, [breakpoint]);
  return isMobile;
};

Tells you whether the user is on a phone or a bigger screen. Use it to swap a desktop sidebar for a hamburger menu, or to render a completely different layout. CSS handles most of this, but sometimes you need the answer in JavaScript.

8. scrollToTop

const scrollToTop = () =>
  window.scrollTo({ top: 0, behavior: "smooth" });

Smoothly glides the page back to the top. Wire it up to a floating "back to top" button on long blog posts or endless product lists. One line, instant polish.

9. useInterval

const useInterval = (callback, delay) => {
  const savedCallback = useRef(callback);
  useEffect(() => { savedCallback.current = callback; }, [callback]);
  useEffect(() => {
    if (delay === null) return;
    const id = setInterval(() => savedCallback.current(), delay);
    return () => clearInterval(id);
  }, [delay]);
};

Runs a function on a repeating timer the React way. The saved ref means your callback always sees fresh state instead of a stale closure, which is the classic setInterval trap. Reach for it on live clocks, auto refreshing dashboards, and carousels that slide on their own.

10. useTimeout

const useTimeout = (callback, delay) => {
  const savedCallback = useRef(callback);
  useEffect(() => { savedCallback.current = callback; }, [callback]);
  useEffect(() => {
    if (delay === null) return;
    const id = setTimeout(() => savedCallback.current(), delay);
    return () => clearTimeout(id);
  }, [delay]);
};

The one shot cousin of useInterval. Fires your callback once after a delay and cleans itself up properly on unmount. Nice for auto hiding a toast or showing a "still loading" message after a few seconds.

11. useWindowSize

const useWindowSize = () => {
  const [size, setSize] = useState({
    width: window.innerWidth,
    height: window.innerHeight,
  });
  useEffect(() => {
    const onResize = () =>
      setSize({ width: window.innerWidth, height: window.innerHeight });
    window.addEventListener("resize", onResize);
    return () => window.removeEventListener("resize", onResize);
  }, []);
  return size;
};

Gives you live width and height that update as the window resizes. Useful for canvas rendering, virtualized lists, or any chart that needs exact pixel dimensions. The component re renders the moment the viewport changes.

12. useMediaQuery

const useMediaQuery = (query) => {
  const [matches, setMatches] = useState(
    () => window.matchMedia(query).matches
  );
  useEffect(() => {
    const mql = window.matchMedia(query);
    const onChange = () => setMatches(mql.matches);
    mql.addEventListener("change", onChange);
    return () => mql.removeEventListener("change", onChange);
  }, [query]);
  return matches;
};

Your CSS media queries, now available in JavaScript. Pass something like "(min-width: 1024px)" and get a boolean back. Cleaner than a manual resize listener when you only care about a specific breakpoint.

13. useCopyToClipboard

const useCopyToClipboard = () => {
  const [copied, setCopied] = useState(false);
  const copy = async (text) => {
    await navigator.clipboard.writeText(text);
    setCopied(true);
    setTimeout(() => setCopied(false), 2000);
  };
  return { copied, copy };
};

Copies any text and flips a "copied" flag for two seconds so you can show a little checkmark. Every code block, share link, and API key field wants this. Small touch, big "wow this app feels finished" energy.

14. useHover

const useHover = () => {
  const ref = useRef(null);
  const [hovered, setHovered] = useState(false);
  useEffect(() => {
    const node = ref.current;
    if (!node) return;
    const enter = () => setHovered(true);
    const leave = () => setHovered(false);
    node.addEventListener("mouseenter", enter);
    node.addEventListener("mouseleave", leave);
    return () => {
      node.removeEventListener("mouseenter", enter);
      node.removeEventListener("mouseleave", leave);
    };
  }, []);
  return [ref, hovered];
};

Tracks hover state without sprinkling onMouseEnter and onMouseLeave everywhere. Attach the ref, read the boolean, render a tooltip or a preview card. Great when hover needs to drive logic and not just a CSS color change.

15. useOnlineStatus

const useOnlineStatus = () => {
  const [online, setOnline] = useState(navigator.onLine);
  useEffect(() => {
    const goOnline = () => setOnline(true);
    const goOffline = () => setOnline(false);
    window.addEventListener("online", goOnline);
    window.addEventListener("offline", goOffline);
    return () => {
      window.removeEventListener("online", goOnline);
      window.removeEventListener("offline", goOffline);
    };
  }, []);
  return online;
};

Tells you in real time whether the user just lost their connection. Show a "you are offline" banner, pause auto saves, or queue requests for later. Essential for anything that pretends to be a serious web app.

16. useEventListener

const useEventListener = (event, handler, element = window) => {
  const savedHandler = useRef(handler);
  useEffect(() => { savedHandler.current = handler; }, [handler]);
  useEffect(() => {
    const listener = (e) => savedHandler.current(e);
    element.addEventListener(event, listener);
    return () => element.removeEventListener(event, listener);
  }, [event, element]);
};

One hook to rule all your addEventListener boilerplate. It handles the subscribe and cleanup dance for you and always calls the latest handler. Build most of the other hooks in this list on top of it if you want to get fancy.

17. useDocumentTitle

const useDocumentTitle = (title) => {
  useEffect(() => {
    const previous = document.title;
    document.title = title;
    return () => { document.title = previous; };
  }, [title]);
};

Updates the browser tab title and politely restores the old one when the component unmounts. Perfect for per page titles in a single page app, or showing an unread count like "(3) Inbox". Tiny SEO and UX win in three lines.

18. useKeyPress

const useKeyPress = (targetKey) => {
  const [pressed, setPressed] = useState(false);
  useEffect(() => {
    const down = (e) => e.key === targetKey && setPressed(true);
    const up = (e) => e.key === targetKey && setPressed(false);
    window.addEventListener("keydown", down);
    window.addEventListener("keyup", up);
    return () => {
      window.removeEventListener("keydown", down);
      window.removeEventListener("keyup", up);
    };
  }, [targetKey]);
  return pressed;
};

Returns true while a given key is held down. Wire up Escape to close a modal, Enter to submit, or arrow keys to move through a list. Keyboard shortcuts make your app feel like it was built by people who actually use it.

19. useDarkMode

const useDarkMode = () => {
  const [dark, setDark] = useState(
    () => localStorage.getItem("theme") === "dark"
  );
  useEffect(() => {
    document.documentElement.classList.toggle("dark", dark);
    localStorage.setItem("theme", dark ? "dark" : "light");
  }, [dark]);
  return [dark, setDark];
};

A complete dark mode toggle that remembers the choice across reloads. It flips a class on the html element so Tailwind or your CSS variables do the rest. Ship the feature everyone immediately looks for in settings.

20. useThrottle

const useThrottle = (value, limit = 500) => {
  const [throttled, setThrottled] = useState(value);
  const lastRan = useRef(Date.now());
  useEffect(() => {
    const handler = setTimeout(() => {
      if (Date.now() - lastRan.current >= limit) {
        setThrottled(value);
        lastRan.current = Date.now();
      }
    }, limit - (Date.now() - lastRan.current));
    return () => clearTimeout(handler);
  }, [value, limit]);
  return throttled;
};

Debounce waits for things to stop, throttle keeps a steady pace. This caps how often a value updates, which is exactly what you want for scroll position, mouse tracking, or window resize math. Smooth performance without melting the main thread.

21. useIntersectionObserver

const useIntersectionObserver = (options) => {
  const ref = useRef(null);
  const [isVisible, setIsVisible] = useState(false);
  useEffect(() => {
    const observer = new IntersectionObserver(
      ([entry]) => setIsVisible(entry.isIntersecting),
      options
    );
    if (ref.current) observer.observe(ref.current);
    return () => observer.disconnect();
  }, [options]);
  return [ref, isVisible];
};

Tells you when an element scrolls into view. This is the engine behind lazy loaded images, fade in on scroll animations, and infinite feeds that load more as you reach the bottom. Way cheaper than listening to the scroll event yourself.

22. useUpdateEffect

const useUpdateEffect = (effect, deps) => {
  const isFirst = useRef(true);
  useEffect(() => {
    if (isFirst.current) {
      isFirst.current = false;
      return;
    }
    return effect();
  }, deps);
};

Like useEffect, but it skips the very first render and only runs on updates. Saves you from that annoying duplicate API call or analytics event that fires the moment a component mounts. The "only when something actually changes" hook.

23. useEffectOnce

const useEffectOnce = (effect) => {
  useEffect(effect, []);
};

A tiny wrapper that runs an effect exactly once on mount. Mostly it just signals your intent clearly so the next dev does not stare at an empty dependency array wondering if you forgot something. Readability counts.

24. useCounter

const useCounter = (initial = 0, step = 1) => {
  const [count, setCount] = useState(initial);
  const increment = () => setCount((c) => c + step);
  const decrement = () => setCount((c) => c - step);
  const reset = () => setCount(initial);
  return { count, increment, decrement, reset };
};

A ready made counter with increment, decrement, and reset baked in. Drop it into quantity selectors, pagination, or step by step wizards. Less wiring, fewer bugs, more time for the fun stuff.

25. useArray

const useArray = (initial = []) => {
  const [array, setArray] = useState(initial);
  const push = (item) => setArray((a) => [...a, item]);
  const remove = (i) => setArray((a) => a.filter((_, idx) => idx !== i));
  const clear = () => setArray([]);
  return { array, set: setArray, push, remove, clear };
};

Managing a list in state with clean helper methods instead of spread syntax soup. Add, remove by index, or wipe it clean without rewriting the same map and filter logic. Great for to do lists, tag inputs, and cart items.

26. useGeolocation

const useGeolocation = () => {
  const [coords, setCoords] = useState(null);
  useEffect(() => {
    const id = navigator.geolocation.watchPosition((pos) =>
      setCoords({
        lat: pos.coords.latitude,
        lng: pos.coords.longitude,
      })
    );
    return () => navigator.geolocation.clearWatch(id);
  }, []);
  return coords;
};

Watches the user location and updates as they move. Use it for "stores near me", live delivery tracking, or auto filling a city field. Just remember the browser will ask for permission first, so handle the null case gracefully.

27. useIdle

const useIdle = (timeout = 3000) => {
  const [idle, setIdle] = useState(false);
  useEffect(() => {
    let timer = setTimeout(() => setIdle(true), timeout);
    const reset = () => {
      clearTimeout(timer);
      setIdle(false);
      timer = setTimeout(() => setIdle(true), timeout);
    };
    window.addEventListener("mousemove", reset);
    window.addEventListener("keydown", reset);
    return () => {
      clearTimeout(timer);
      window.removeEventListener("mousemove", reset);
      window.removeEventListener("keydown", reset);
    };
  }, [timeout]);
  return idle;
};

Detects when the user has gone quiet for a while. Pause a video, dim the screen, or auto log out of a banking dashboard after inactivity. Any reset of mouse or keyboard activity wakes it back up.

28. useSessionStorage

const useSessionStorage = (key, initial) => {
  const [value, setValue] = useState(() => {
    const stored = sessionStorage.getItem(key);
    return stored ? JSON.parse(stored) : initial;
  });
  useEffect(() => {
    sessionStorage.setItem(key, JSON.stringify(value));
  }, [key, value]);
  return [value, setValue];
};

Same idea as useLocalStorage, but the data clears when the tab closes. Ideal for a multi step checkout or a wizard where you want progress saved during the visit but gone afterward. Privacy friendly persistence.

29. useScrollLock

const useScrollLock = (locked) => {
  useEffect(() => {
    document.body.style.overflow = locked ? "hidden" : "";
    return () => { document.body.style.overflow = ""; };
  }, [locked]);
};

Freezes background scrolling while a modal or drawer is open. We have all rage scrolled a popup and watched the page behind it slide around. This one line fix kills that jank and restores scroll on close.

30. useRenderCount

const useRenderCount = () => {
  const count = useRef(0);
  count.current += 1;
  return count.current;
};

Counts how many times a component has rendered. Pure debugging gold when you are hunting down why something re renders forty times when you clicked once. Pop it in, log the number, find the culprit, take it out.

Conclusion

And that is the stash. Thirty little snippets that save you from rewriting the same logic in every single project from now until forever. Copy what you need, tweak the ones you love, and quietly become the teammate whose pull requests are weirdly clean.

I will be back soon with another round, probably something on performance or the patterns nobody warns you about until it is too late. Until then, go build something cool and stop reinventing the debounce.

happycoding

The Invisible Duct Tape of the Internet: Backend Tools You Hear About But Never Fully Get

Mursal Furqan Kumbhar — Mon, 22 Jun 2026 03:14:44 +0000

Hi 👋 fellow devs

Sorry for such a big gap since my last article...... Life got a bit hectic, but I am finally back in action!

You know how it goes. We spend so much of our energy obsessing over the flashy side of tech. We talk about gorgeous UI designs, smooth animations, and whatever frontend framework is trending on GitHub this week. But let’s be completely real for a second. What actually keeps your favorite apps from melting down when millions of people hit the refresh button at the exact same moment?

That is exactly what we are going to unpack today. We are pulling back the curtain on the quiet, brilliant backstage crew of infrastructure tools. You see their logos all over tech Twitter and hear senior engineers drop their names in meetings like secret handshakes, but today, we are stripping away the corporate fluff. We will break down eight legendary backend technologies using conversational paragraphs and quick bullet points so you can finally master what they actually do.
Let’s dive right in.

1. Redis

Traditional databases live on hard drives. They are fantastic for keeping your data safe and organized permanently, but pulling data off a physical drive takes time. If your application has to wander deep into those database aisles to fetch the exact same piece of information every single second, your entire system starts to stall.
To understand how Redis fixes this, imagine you are studying for a brutal exam. Your massive, 1,000-page textbook represents your main database. It holds every single answer, but flipping through the pages continuously is incredibly slow. Redis is the digital equivalent of writing the core formulas you need on a neon sticky note and taping it directly to your monitor. It keeps critical data sitting directly inside the system's lightning-fast short-term memory.
You will typically find Redis stepping in to handle operations like:

Session Management: Keeping users logged into an application without checking the main database on every click.
API Caching: Storing the responses of heavy, repetitive queries so they render instantly.
Live Leaderboards: Tracking high-velocity, real-time score updates during live events or games.

2. Kafka

Modern applications generate an overwhelming, continuous stream of mini-events. Every single click, profile update, and background notification triggers a piece of data. If every single microservice in your software tries to communicate directly with every other service to share these updates, your architecture quickly devolves into a tangled, fragile spiderweb of digital chaos.
Think of Kafka like a wildly busy fast-food kitchen. Instead of the cashiers running into the back to scream individual orders directly at the face of every cook, they hang receipts on a long, central ticket line. The burger station, the fry station, and the drinks manager all pull tickets from that central line at their own pace. Kafka acts as that indestructible, high-speed central ticket line for your data.
Engineers rely on Kafka to manage heavy data pipelines because it excels at:

Decoupling Services: Allowing different parts of an app to function independently without relying on direct connections.
Activity Tracking: Processing millions of user interactions across a massive platform simultaneously.
Real-time Data Streams: Managing live GPS coordinates from a fleet of delivery drivers without dropping a single packet.

3. Nginx

When a massive wave of users hits a website simultaneously, your core application servers can easily experience a traffic jam, get overwhelmed, and crash. Nginx sits right at the edge of your infrastructure to act as an incredibly efficient shield, managing thousands of incoming connections before they ever touch your actual code.
To picture this, imagine a highly exclusive apartment building. Instead of letting random delivery drivers wander the hallways looking for individual doors, everyone is forced to stop at a central front desk. Nginx is that ultimate receptionist. It checks security certificates, hands out public brochures instantly so people do not even have to step inside, and cleanly directs visitors to the exact elevator they need.
In everyday setups, Nginx is deployed to handle:

Load Balancing: Distributing incoming user traffic evenly across multiple backend servers.
Reverse Proxying: Routing requests smoothly to different applications running on different ports.
Static File Serving: Instantly handing over images, HTML, and CSS directly to the browser to save server power.

4. Kubernetes

Managing a couple of servers by hand is completely fine, but managing hundreds of isolated microservices scattered across global cloud networks is a total nightmare. If a server goes down in the middle of the night, someone has to wake up, diagnose the issue, and manually restart it. Kubernetes exists to automate that entire headache away.
Think of Kubernetes as the manager of a massive, fully automated shipping port. You do not manually tell every truck where to drive or check if a crane operator fell asleep. You program your desired layout into a central management system. If a cargo truck randomly breaks down, the system instantly detects it, dispatches a identical replacement truck, and moves the cargo along without stalling operations for a single second.
Kubernetes keeps complex cloud applications running smoothly by providing:

Self-Healing: Automatically restarting containers that crash and replacing unresponsive servers.
Horizontal Scaling: Spatially spinning up new instances of an app during traffic spikes and scaling them down when quiet.
Automated Rollouts: Gradually deploying new code updates across a network without causing a single second of downtime.

5. Elasticsearch

Standard database lookups are incredibly rigid. They work perfectly when you are searching for an exact user ID number, but if you try to perform a complex, messy search across millions of rows of unstructured text, a normal database will grind to a painful halt.
Imagine trying to find every single mention of the word "coffee" inside a massive university library. A standard database approach would involve opening every single book and scanning every single page word by word. Elasticsearch works by building a massive, highly optimized index ahead of time. It acts exactly like the index at the back of a textbook, but it is smart enough to find the right page even if you make a typo.
It is the industry standard tool when you need to build:

Advanced E-commerce Search: Powering filters, categories, and instant results across massive product catalogs.
Fuzzy Matching: Predicting what a user actually meant even when they misspell words in a search bar.
Log Analysis: Sifting through gigabytes of system error logs to find a single needle in a haystack.

6. GraphQL

Traditional REST APIs can sometimes feel like a restaurant that only serves rigid, pre-packaged combo meals. If you only want a side of fries, you are still forced to order the entire burger and drink combo, which wastes data and bandwidth. Alternatively, you might have to hit three different endpoints just to gather enough information to build a single profile page.
GraphQL fundamentally changes this dynamic by turning your API into an a la carte menu. It allows the client application to hand the backend a specific checklist of requirements. If your mobile app only needs a user's first name and profile photo for a tiny header card, GraphQL ensures the server sends back exactly those two data points and absolutely nothing extra.
This precision makes GraphQL incredibly valuable for:

Mobile App Optimization: Saving precious cellular data and battery life for users on slower networks.
Consolidating Requests: Fetching data from multiple distinct sources using a single, unified network call.
Rapid Frontend Development: Allowing frontend devs to change visual layouts without needing backend devs to constantly rewrite API endpoints.

7. RabbitMQ

When a user clicks a button to upload a file or make a purchase, they expect an instant confirmation response. They do not want to stare at a frozen loading spinner for ten minutes while your backend server crunches a heavy file, updates inventory, creates a PDF invoice, and sends out confirmation emails.
RabbitMQ fixes this by acting exactly like a post office drop box. You do not stand at the counter waiting for the clerk to drive your letter across the country. You drop it in the slot and walk away, confident it will be handled. RabbitMQ accepts these heavy, time-consuming tasks from your main application and holds them safely in a queue, letting your user interface stay incredibly snappy and responsive.
You should implement RabbitMQ whenever you need to offload tasks like:

Background File Processing: Resizing high-resolution images or transcoding videos after an upload.
Notification Blasts: Dispatching transactional emails or push notifications to thousands of users without blocking the main app.
Asynchronous Workflows: Delaying heavy calculations or third-party API syncs until your server has free resources.

8. Docker

Every developer has lived through the absolute nightmare where a feature runs flawlessly on their local laptop, but completely shatters the moment it gets deployed to a live production server due to a missing file or a conflicting software version.
Docker completely eliminates this issue by introducing the concept of digital shipping containers. In the physical world, a standard shipping container fits perfectly onto any cargo ship, train, or truck regardless of what is inside it. Docker applies this exact logic to software, wrapping your code, your runtime, your tools, and your specific configurations into an isolated, standardized digital box.
By containerizing your applications, Docker guarantees:

Environment Consistency: Ensuring your code behaves identically on a Mac laptop, a Windows desktop, or a Linux cloud instance.
Isolation: Running multiple applications with completely conflicting dependencies on the exact same computer without any interference.
Simplified Onboarding: Allowing new developers to spin up a complete, complex project environment using one single command.

Quick Reference Guide

The Tool	Its Core Superpower	In Plain English
Redis	In-memory speed	The sticky note on your monitor.
Kafka	Event streaming	The kitchen ticket line.
Nginx	Reverse proxy and web server	The gatekeeper at the front desk.
Kubernetes	Container orchestration	The automated shipping port manager.
Elasticsearch	Distributed search engine	The textbook index on steroids.
GraphQL	Precise API queries	The custom sandwich checklist.
RabbitMQ	Message broker	The post office drop box.
Docker	Containerization	The standardized shipping container.

Conclusion

At the end of the day, you do not need to be an absolute wizard in all eight of these tools to be a fantastic developer, but understanding where they fit in the puzzle changes how you think about software. Writing code that functions locally is just step one. True engineering happens when you learn how these infrastructure pieces coordinate behind the scenes to architect systems capable of handling millions of users without breaking a sweat.
Drop a comment below and let me know which of these backend heroes you are planning to deploy in your next project!

VS Code Extensions That Code For You

Mursal Furqan Kumbhar — Tue, 14 Apr 2026 10:35:13 +0000

Hello Devs 👋

If your VSCode setup still looks like it did before the AI boom, you may be coding with a bicycle in a Formula 1 race. 🏎️ Modern AI extensions are no longer novelty gadgets, they’re force multipliers. From ghost-writing boilerplate to understanding sprawling codebases and debugging at 2 AM when your brain has blue-screened, these tools can turn your editor into a caffeinated co-pilot. Here are 6 of the best AI extensions helping developers ship faster, smarter, and with fewer “why is this null?” moments.

Why does this matter?

Your editor should work as hard as you do. Most devs use VSCode, but only 10% use AI extensions. In 2026, that gap is the difference between a slow dev and a 10x dev.

1. Continue by Continue.dev

Open-Source AI Assistant. Plug in any model, like GPT-4, Claude, Llama or Mistral directly into VSCode. It:

Uses your own API Key (GPT/Claude/etc.)
Run local models via Llama
Custom slash commands & context
Full-Chat + Inline edit supports Rating: 5/5 Cost: Free | Open-Source

2. Supermaven by Supermaven Inc.

Blazing fast AI autocomplete with a 1 million context window. Its usage feels almost instant, doesn't lag and doesn't make you wait.

1,000,000 Token context.
Sees your entire repository and codebase
Sub-100ms suggestions. Truly feels instant
Understands cross-file dependencies
Free tier available Rating: 5/5 Cost: Free Plan | Pro is $10/Month

3. Pieces for Devs by Pieces.app

Your AI-Powered developer memory. It saves code snippets, context and links, and retrieves them as and when needed. It also:

Auto-saves snippets with AI Tags
Long-term memory across projects
AI Explains saved snippets on demand
Works offline with on-device AI Rating: 5/5 Cost: Free for Personal Use

4. Codeium by Exafunction

The best free Copilot alternative. It's fast, accurate and supports 70+ languages. Codeium facilitates its users in many ways, such as:

It's completely free for individual devs
Provides in-editor AI Chat for debugging
Autocompletes across 70+ programming languages
Also comes with Codeium Search and Codebase Q&A Rating: 5/5 Cost: 100% FREE | No Limits

5. Tabine by Tabnine

AI Code Completor, trained on your own codebase. This is best for teams that can't share code with external servers (mainly proprietary developers). Its main features include, but are not limited to:

Runs locally, your code stays private
Learns your codebase patterns
Works offline, no internet needed
Team model training on your repository Rating: 5/5 Cost: Pro is $12/Month

6. GitHub Copilot by GitHub (Microsoft)

The OG AI Pair Programmer. This extension auto-completes entire functions, writes tests, and explains code, all inside VSCode. Its main features include:

Inline code suggestions as you type
Copilot Chat, ask anything about your code
Multi-file context awareness
Generate unit tests automatically Rating: 5/5 Cost: $10/Month | Free for Students

Conclucion

The best developers in 2026 are not just writing code, they’re orchestrating tools. Whether you want lightning-fast autocomplete, local privacy-first AI, or a full-blown coding sidekick living in your editor, there’s an extension here for every workflow. Try a few, mix and match, and build the AI-powered setup that makes your keyboard feel slightly unfair. ⚡

Giving AI Agents “Live Memory” with Aurora Zero-ETL and Redshift Vector Search

Mursal Furqan Kumbhar — Wed, 25 Mar 2026 05:38:18 +0000

Hey everyone! Honestly, it feels so good to be sitting down and writing to you all again. I want to start with a massive apology for the radio silence and the long gap between my last few posts. Life has a funny way of getting busy, but I’ve missed this community and our deep dives into the future of tech.

We’re talking about a problem that has been bugging me all through 2025 and into 2026: AI situational awareness. Let’s dive in.

The "Goldfish" Problem: Why Your AI is Gaslighting Your Users 🐠🌀

Look, we need to have a genuine heart-to-heart about your AI agents. We’re well into 2026, and yet, most AI “agents” still have the situational awareness of a goldfish in a blender. It’s a bit heartbreaking, isn't it?

You know the feeling: You’ve spent weeks perfecting your system prompts, the UI is looking like a work of art, and your LLM is sharp. But the second a user actually does something—like upgrading a subscription, changing a delivery address, or updating their profile—the AI is the absolute last one to know. It’s like the “memory” of your agent is stuck in a permanent time lag, trapped in a 2024 batch-processing nightmare.

Imagine you’re building the ultimate AI concierge for a luxury travel app.

2:00:00 PM: Your user, Sarah, realizes she’s over the cold and cancels her flight to Paris because she’d rather sip espresso in Rome. This change hits your Amazon Aurora database instantly. Success!
2:00:10 PM: Feeling productive, Sarah asks the AI Agent, “Hey, what time does my flight leave?”
The Disaster: Because your old-school data sync (ETL) only runs every 30 minutes, the AI looks at the stale warehouse data and says, “Your flight to Paris departs at 6 PM! Don't forget your beret!” 🥖

Sarah is now officially being gaslit by an algorithm. 🤦‍♂️ The immersion? Totally gone. The trust? Shattered. This happens because the "Transactional Brain" (Aurora) isn't talking to the "Analytical Memory" (Redshift) fast enough. We need a way to move that data so quickly it feels like magic. Today, we are fixing that with Live Memory using Amazon Aurora Zero-ETL and Redshift Vector Search.

Step 1: The “Wormhole” (Zero-ETL) 🚀

In the "old days" (way back in 2024), moving data from Aurora to Redshift meant building a "Glue" pipeline. It was like trying to build a bridge between two islands using nothing but popsicle sticks, duct tape, and a whole lot of hope. It broke constantly, it cost a fortune, and it was painfully slow.

Zero-ETL is the modern way to handle this. Think of it as a native “wormhole” between Aurora and Redshift. When you enable it, AWS handles all the heavy lifting and plumbing behind the scenes. No Python scripts to debug, no Lambda triggers to monitor, and absolutely zero drama.

Why this is the “Secret Sauce” for your Agent:

Near-Zero Latency: We’re talking seconds, not minutes. By the time Sarah finishes typing her question, the data is already there.
Auto-Scaling: As your app goes viral, the teleporter grows with it.
Total Peace of Mind: You can finally stop monitoring failed Glue jobs at 3 AM. If it’s in Aurora, it’s in Redshift.

Step 2: From “Rows” to “Recall” (Vector Search) 🧠

Okay, so the data has successfully teleported into Redshift. But here’s the catch: Redshift is a warehouse full of structured tables, and AI Agents don’t really "think" in tables—they think in Vectors (mathematical representations of meaning).

In the past, you’d have to ship this data again to a dedicated vector database. But that just adds more lag! Instead, we use Redshift’s Native Vector Search. This keeps your “Live Memory” in one single, high-performance place. To make this work, we set up a Materialized View in Redshift that automatically calls Amazon Bedrock to turn those new rows into vectors on the fly.

-- The "I don't have a goldfish memory anymore" Query
CREATE MATERIALIZED VIEW live_user_context AS
SELECT 
    user_id, 
    event_description,
    -- This calls Bedrock to create a vector on the fly!
    amazon_bedrock_embed(event_description) as semantic_vector
FROM aurora_synced_data.user_logs;

Step 3: The “Aha!” Moment 🔍

Now, let's replay the Sarah scenario. When she asks about her flight, the AI Agent sends a vector query to Redshift. Redshift looks at the most recent events—including the one that landed just 10 seconds ago—and finds the “Canceled Paris / Booked Rome” entry.

The Agent responds: “I see you switched to the Rome trip—great choice! That flight leaves at 8 PM. Should I book a car to the airport?”

Boom. You just saved the user experience. 🥳 You've moved from a "chatbot" to a genuine "agent" that understands the present moment.

By The Numbers: Is it worth your time? 📈

We ran some benchmarks to see just how much of a difference this "Live Memory" architecture actually makes. When you compare the old "Batch" world to the new "Zero-ETL" world, the latency gap is massive.

Average “Brain Lag” (Latency):
- Standard S3/Glue Sync: ~480 seconds
- Zero-ETL + Redshift: ~12 seconds (A 97% improvement!)
Developer Happiness:
- Measured in “How many times I had to fix a pipeline this week”: Dropped from an average of 4 down to 0.

Pro-Tips for the Modern ML Engineer 🛠️

Keep it Serverless: Use Redshift Serverless. It scales down to zero when your agents are sleeping.
Hybrid Search is King: Don’t rely only on vectors. Combine them with regular SQL filters (like WHERE user_id = '123') to make your search lightning fast.
Capture the Vibe: Remember, this isn’t just for logs. If a user tells the AI, “I love my better half a lot,” ❤️ that sentiment gets synced and embedded instantly. The next time they ask for anniversary ideas, the agent immediately knows the priority is “High Romance.”

Final Thoughts

Anyway, it feels amazing to be back in the flow and sharing these breakthroughs with you all again. The leap from “static” AI to “live” agents is one of the most exciting shifts we’re seeing this year, and I can’t wait to see how you all implement these “wormholes” in your own stacks. If you run into any snags or just want to geek out over vector search optimizations, drop a comment below or find me over on Dev.to! I promise the next article won’t take nearly as long to reach your screens. Until next time — stay curious, keep building, and let’s make those agents a whole lot smarter. 🚀✨

RIP 12-Digit IDs: The AWS Matrix Has Finally Been Decoded

Mursal Furqan Kumbhar — Mon, 09 Feb 2026 09:41:12 +0000

Y'all... pull up a chair. We need to talk about a literal miracle. 🥂

It was 2:41 AM. PagerDuty was screaming like a banshee in my ear. My eyes were blurry, my coffee was cold, and Slack was silent in that heavy, "something-is-very-wrong" way. I logged into the AWS Console to “quickly restart a service” and get back to sleep.

Five clicks later—just as I was about to hit "Delete"—my stomach dropped. My heart hit the floor.

Wrong account. I was in Production. Nothing blew up that night. I caught myself at the last millisecond. I got lucky. But for 21+ years, millions of engineers weren’t so lucky. We’ve been squinting at 12-digit account IDs like we’re trying to decode the Matrix while the world burns around us.

But the wait is over. AWS finally added the Account Name to the top bar. It’s a "high-tech" feature that only took two decades to arrive, and it is GLORIOUS.

🏆 Welcome to the "Almost Nuked Prod" Hall of Fame

For over two decades, AWS UX followed the "Extreme Stealth" philosophy. The top bar gave you the region and your username, but the actual account context? Hidden behind a click. Pure chaos. It was like driving a car where the speedometer is inside the glove box.

This led to the Three Stages of AWS Grief:

The Over-Confidence: “I definitely logged into Sandbox. I'm a pro, I don't need to check.”
The Sudden Realization: “Wait... why does this S3 bucket have 4PB of data and a 'Do Not Delete' tag? Why is this instance type an x1e.32xlarge?”
The Panic: Aggressively smashing Ctrl+W to close 42 Chrome tabs before your shaking finger accidentally clicks 'Terminate Instance'.

Why This is Low-Key Life-Changing

This isn’t just cosmetic; it’s cognitive safety. In a multi-account world, you’re juggling dozens of environments. This update assumes that engineers are human—that we get tired, stressed, and caffeine-deprived. It’s a sanity check for your career. Good UX doesn’t assume perfection; it assumes you're human.

🛠️ Hands-On: Build Your Safety Net Today

Since you're likely rocking .ipynb files in VSCode (the ultimate dev setup, IMO), you’re used to seeing your environment in the kernel picker or the status bar. Let's get that same "don't-get-fired" energy in your browser.

1. Make it Pretty with the AWS CLI

If your account name is company-billing-prod-final-v2, it’s going to get truncated. Don't click through menus like it's 2004—we're builders! Pop this into your terminal to set a punchy, unmistakable Alias:

# Give your account a name that SCREAMS at you in the top bar
aws iam create-account-alias --account-alias PROD-SENSITIVE-ZONE

Boom. Now your top bar says PROD-SENSITIVE-ZONE in clear text instead of a random string of numbers. 💅

2. The "Visual Fire Alarm" (Layered Security)

Text is great, but color-coding saves careers. Since AWS hasn't given us a "Red Header" mode for Prod yet (maybe in the year 2047?), we have to be clever:

Unified Settings: Click your Name (top right) → Settings → Visual Mode. Choose a distinct theme for your main accounts.
Browser Profiles: This is the real pro-tip. Use different Browser Profiles (Chrome/Brave/Firefox) for different environments. I keep my Prod profile themed in "Emergency Red" with a massive warning icon. If the window isn't red, it isn't Prod.

3. The "Don't Fire Me" Python Snippet

Before running any destructive boto3 code in your Jupyter Notebook, run this check. It’s the programmatic version of "looking both ways before crossing the street."

import boto3

def safety_check():
    # Grab the current identity
    sts = boto3.client('sts')
    identity = sts.get_caller_identity()
    account_id = identity['Account']

    # Put your real Prod ID here
    PROD_ID = "123456789012" 

    print("-" * 30)
    if account_id == PROD_ID:
        print("🚨 ALERT: YOU ARE IN PRODUCTION! 🚨")
        print("Step away from the Shift+Enter key unless you're sure!")
    else:
        print(f"✅ Connected to: {account_id}")
        print("Environment: Non-Prod (Safe to play, fam)")
    print("-" * 30)

safety_check()

🚀 The Future of AWS UX

While we celebrate this "giant leap for engineer-kind," we’re still looking toward the future. Maybe one day the "Delete Database" button won't be the exact same shade of blue as the "Save" button? Maybe we'll get a "High Stakes" mode that requires a physical key turn?

Until then, appreciate the label. It might just be the thing that keeps you from becoming a "Why We Had an Outage" post-mortem.

Final Thought: The most dangerous bugs don’t live in your code; they live in interfaces that assume humans never slip. AWS finally acknowledged that we’re human. Sometimes, the biggest cloud innovation... is simply knowing exactly where you are before you hit 'Enter'. ☁️

From Monoliths to Multitaskers: Building Your AWS AI Dream Team! 🚀🤖

Mursal Furqan Kumbhar — Sat, 07 Feb 2026 07:47:02 +0000

After being away from active tech ecosystem for about a month, I have decided to shed some light on the current landscape of AI agents on AWS. And let me tell you, things have gotten wild while I was gone. We aren't just talking to single chatbots anymore; we're building entire autonomous squads.

If you’ve been building with AI lately, you’ve probably hit the "Monolith Wall." You know the one: you try to get one Large Language Model (LLM) to handle your database, write your emails, troubleshoot your network, and maybe make a decent cup of coffee. It gets confused, starts hallucinating, and suddenly your "smart" app is telling users that the server is down because it's "feeling sleepy."

The 2025 vibe is different. We are moving away from "User → LLM → Response" and toward "User → Agent Network → Coordinated Action." Think of it like a D&D party: you wouldn't ask the Barbarian to pick a lock, and you wouldn't ask the Rogue to tank a dragon. You need specialists.

Today, we’re going to learn how to orchestrate a Multi-Agent System (MAS) on AWS Bedrock. Grab your coffee (or mana potion), and let’s dive in!

🛠 The Secret Sauce: What is a "Bedrock Agent"?

Before we build the team, we need to understand the individual "hero." A Bedrock Agent isn't just an LLM with a fancy name. It's a system with three core parts:

The Brain (Model Provider): Usually something like Claude 3.5 Sonnet or the zippy new Amazon Nova series.
The Manual (System Prompt): The instructions that tell the agent, "You are a world-class SRE," and give it boundaries (like "Don't delete production, please").
The Toolbelt (Action Groups): These are the superpowers! This is where the agent calls AWS Lambda functions to actually do things in the real world.

🎭 The Multi-Agent Party: Meet the Supervisor

Why have one agent when you can have five? In a Supervisor-Agent Pattern, you have one "Dungeon Master" (the Supervisor) who listens to the user and decides which specialist to call.

Imagine we're building a Telco Network Operations Assistant. Our party looks like this:

Role	Specialty	Data Source
The Supervisor	Orchestration & Planning	Routes the query to the right specialist.
The Alchemist (KPI Agent)	Performance Metrics	Checks throughput and latency.
The Scout (Alarm Agent)	Real-time Status	Looks for active site failures.
The Chronicler (Log Agent)	History & Patterns	Scans CloudWatch logs for anomalies.

🏗 Hands-On Walkthrough: Building the "Site Ops Hero"

Let's get our hands dirty. We want to ask: "What's the status of site_dallas_001?" and get a correlated answer.

Step 1: The Magic Lambda (Your Action Group)
Your agent needs a bridge to the real world. We’ll use a Lambda function. When the agent calls it, it sends a JSON event that looks like this:

{
  "actionGroup": "NetworkTools",
  "apiPath": "/get-site-alarms",
  "parameters": [{"name": "site_id", "value": "site_dallas_001"}]
}

Tip: Bedrock events aren't the same as API Gateway events! You’ll need to parse the apiPath and parameters directly.

Step 2: Set up the Specialist Agents
In the Bedrock Console (or via Boto3), create your specialist agents. Give them Knowledge Bases so they can read your documentation stored in S3.

Pro-tip: Use Titan Text Embeddings v2 to turn your PDFs into "vector gold" that the agent can search in milliseconds.

Step 3: Summon the Supervisor
Create a new agent and enable Collaboration Mode. Associate your specialist agents as "collaborators." Now, when you talk to the Supervisor, it creates a plan:

Ask the Scout for active alarms.
Ask the Alchemist if performance is dropping.
Combine the answers and tell the human exactly what’s broken.

💰 Budgeting Your "Mana" (Cost Optimization)

Running a whole team of AI agents can get pricey if you're not careful. Here’s how to keep your AWS bill from going "Critical Hit" on your wallet:

Tiered Intelligence: Use the "big brain" (Claude 3.5 Sonnet or Nova Premier) for the Supervisor who needs to plan. Use the "fast brain" (Nova Lite or Micro) for simple sub-tasks.
Prompt Caching: If you’re sending the same 50-page manual to your agent every time, Prompt Caching can save you up to $90\%$ on input tokens.
Batch Inference: If you’re doing something that doesn’t need an answer right now (like summarizing yesterday's logs), use Batch Mode for a 50% discount.

🛡 Staying Safe (Guardrails)

You don't want your agent going rogue. Use Amazon Bedrock Guardrails to set "Denied Topics." For example, you can explicitly forbid the agent from giving out employee home addresses or venting about its boss. You can even set up Contextual Grounding Checks to make sure the agent only answers based on your data, not its own imagination (aka hallucinations).

🏁 Conclusion: Your Adventure Awaits!

Building multi-agent systems on AWS is like moving from playing a single-instrument solo to conducting a full philharmonic orchestra. It’s more complex, sure, but the results are magical.

Whether you're automating SRE tasks or building a Pokémon battle advisor, the tools are all there in Bedrock.

What are you waiting for? Go build something awesome! 🚀✨

P.S. When you're done, remember to delete your S3 buckets and Lambda functions so you don't wake up to a "Surprise Invoice" Boss Fight.

The AWS Lambda Setting I Wish I’d Changed Earlier

Mursal Furqan Kumbhar — Tue, 30 Dec 2025 09:10:50 +0000

I've been using AWS Lambda for quite a few years, and like many people, I left the architecture on its default settings, i.e. x86 (Without thinking 2x). It worked really well, it scaled as needed, but the truth unfolded when invoices showed up. End of Story. Fine Della Storia.

Upon closer examination of the situation, I paid more attention to the execution time & cost, and then I thought to experiment with Arm64, i.e. AWS Graviton. And the rare cloud optimisations that actually delivered, followed. 🎇 A VISIBLE WIN 🎇
.
.
.
Now the main Question

What led me to try Arm64?

The answer's as simple as it gets: A few Lambda handling API Requests & Background jobs were being called 1,000,000s of times per month and frankly, each one looked quite cheap, individually. But when bunched together, they burned a steamy hole in my pocket every month.

Now being a lazy_developer who uses snake cases while programming, I said NO to refactoring the logic or the code. I just wanted a clean performance win. And that's when... Arm64 looked like a low-risk bet to me.

Time taken towards the CHANGE!!!

Now, when I say I moved from x86 to Arm64, the first thing to pop in your head would be, "Nah! It's gonna take forever to change." Right? WRONG. I changed the lambda architecture from x86 to Arm64, redeployed and tested it within a matter of hours (And that's because I chose NOT to read the documentation first 🥲. It could have been sooner). That was it. No code was changed, no dependencies were broken, and no surprises. This alone was... unexpected.

What were my ACTUAL Observations

The moment I started moving only a few Lambdas functions to Arm64, the execution time dropped noticeably, especially for CPU-based logic. And guess what... the Cold Start felt snappier on lightweight Node.js & Python functions. In the end, the monthly lambda cost dip without touching memory allocation felt like a cherry on top.

So, nothing dramatic for every invocation, but at scale, the difference was truly impossible to ignore.

Reality Check: The Cost

Lambda billing isn't rocket science. It's simple maths:

Execution time * Memory * Invocations

If your function runs just 20-30 milliseconds faster, and it is executed millions of times, that tiny change in the execution speed turns into a new Apple Watch (Yes, no kidneys needed 😜). This was the moment Arm64 stopped feeling like an optimisation experiment & started feeling like a worthy replacement to x86 as default choice

Why did Arm64 work the best for ME?

In my small but meaningful experience, Arm64 shone brighter than star Sirius, when I used it for API Gateway-backed Lambdas, Event Driven pipelines, background jobs, schedulers, data processing & automation or even for lightweight ML inference. In short, I opted for Arm64, where I used modern serverless workloads with updated runtimes.

Where I would never use Arm64?

It's not like I am completely against x86 now. I still used it in my project for some scenarios. For example:

I kept x86 for older functions with legacy native binaries
Or for rare dependencies that weren't Arm-ready, yet
Or for the code, that I didn't want to touch, being that close to the deadline. (Programmers, IYKYK). For those who don't, we don't want the project go

What changes in the future?

In the future, if I am starting a new serverless project (Or today, who knows):

I will definitely go for Arm64, by default
I am gonna only move back to x86, only & only, if something explicitly breaks (Or my client doesn't love their 💵)
Will benchmark early, instead of assuming

Conclusion

Arm64 on AWS Lambda isn't a future feature, it's just a teeny tiny and quiet upgrade that's already paying off its cost in real-life systems. It's not theoretical. IT's PRACTICAL. Well, if you care about factors such as performance, cost & efficiency and you're running modern hsuper heavy serverless workloads, listen to me (Or read this carefully): TRY IT ONCE. 9/10 Chances? Like me, you won't turn back 😉.

Once again, this was my personal experience.

Buon Anno

HappyCoding

How I Automated Document Insights Using AWS Textract, Bedrock, and QuickSight

Mursal Furqan Kumbhar — Fri, 12 Sep 2025 17:35:28 +0000

Hello Everyone 👋

So I had a bunch of scanned invoices and PDFs lying around from an old freelance gig. Manually reading them and extracting data felt like a punishment. Instead of going through them one by one, I decided to try something smarter (I know, common Developer Syndrome 😜), to build a small AWS workflow that could:

Read scanned docs,
Summarise key info,
Convert it to structured data, and
Visualise everything inside a dashboard.

Here’s how I did it, step by step. You can follow the same process and tweak it for contracts, reports, receipts, whatever you’ve got.

✅ Step 1: Set Up Your AWS Environment (Production-ready)

1) Pick a single region that supports everything

Goal: Keep S3, Textract, Bedrock, and QuickSight in the same region to avoid cross-region latency/fees.
How to choose: Check that your chosen region offers Bedrock (model availability varies) and Textract. If QuickSight isn’t in that region, use Athena + Glue as a bridge (S3 anywhere → Athena in QuickSight’s region).
Edge cases
- Bedrock not in your favorite region: run Textract + S3 where your data lives, then send plain text only to Bedrock in a supported region (privacy + cost trade-off).
- Corporate constraint on regions: document the exception and add an S3 Replication rule or use Athena.

2) Create an S3 bucket with the right defaults

Name: doc-insights-<env>-<region>-<account-id> (e.g., doc-insights-prod-eu-central-1-123456789012)
Settings to enable right away
- Block Public Access: ON (all four toggles).
- Versioning: ON (lets you recover bad parses).
- Default encryption: SSE-KMS (customer-managed key) for auditability.
- Object Ownership: Bucket owner enforced (disables ACLs, simplifies perms).
- Lifecycle rules:
- Move raw/ to infrequent access after N days.
- Expire tmp/ after M days.
- Keep processed/ longer for audits.
Folder layout (simple + scalable)

  s3://doc-insights-bucket/
    raw/               # uploads: PDFs, images
    textract-json/     # raw JSON from Textract (per doc)
    extracted/         # normalized tables/kv (CSV/Parquet)
    summaries/         # Bedrock text summaries (JSON/MD)
    processed/         # final CSVs for BI
    logs/              # app / pipeline logs

Edge cases
- Big uploads / flaky networks: use multi-part upload (SDK/CLI handles this).
- PII/PHI compliance: restrict bucket to VPC endpoints (below), rotate KMS keys, and enable CloudTrail data events for S3.
- Cross-account access: prefer bucket policy + role assumption over sharing long-lived keys.

3) IAM roles & least-privilege policies

Create three roles (one per workload) and scope them tightly:

a) Textract role (used by your Lambda/ETL or notebook)

Trusted entity: lambda.amazonaws.com (or your compute service)
Inline policy (example)

{
  "Version": "2012-10-17",
  "Statement": [
    {"Effect":"Allow","Action":["textract:AnalyzeDocument","textract:StartDocumentAnalysis","textract:GetDocumentAnalysis"],"Resource":"*"},
    {"Effect":"Allow","Action":["s3:GetObject","s3:PutObject"],"Resource":["arn:aws:s3:::doc-insights-bucket/*"]},
    {"Effect":"Allow","Action":["s3:ListBucket"],"Resource":["arn:aws:s3:::doc-insights-bucket"]},
    {"Effect":"Allow","Action":["kms:Encrypt","kms:Decrypt","kms:GenerateDataKey","kms:DescribeKey"],"Resource":"arn:aws:kms:REGION:ACCOUNT:key/KEY_ID"}
  ]
}

Use Start/Get APIs for async jobs (multi-page PDFs); AnalyzeDocument is fine for small/sync runs.

b) Bedrock role (for summarization)

{
  "Version":"2012-10-17",
  "Statement":[
    {"Effect":"Allow","Action":["bedrock:InvokeModel","bedrock:InvokeModelWithResponseStream"],"Resource":"*"},
    {"Effect":"Allow","Action":["s3:GetObject","s3:PutObject"],"Resource":["arn:aws:s3:::doc-insights-bucket/*"]},
    {"Effect":"Allow","Action":["kms:Encrypt","kms:Decrypt","kms:GenerateDataKey","kms:DescribeKey"],"Resource":"arn:aws:kms:REGION:ACCOUNT:key/KEY_ID"}
  ]
}

Limit Resource to the specific model ARNs once you pick them (Claude, Mistral, etc.).

c) QuickSight service role access to S3

Inside QuickSight → Admin → Security & permissions, grant access to your bucket. Backing IAM should allow:

{
  "Version":"2012-10-17",
  "Statement":[
    {"Effect":"Allow","Action":["s3:GetObject"],"Resource":["arn:aws:s3:::doc-insights-bucket/processed/*"]},
    {"Effect":"Allow","Action":["s3:ListBucket"],"Resource":["arn:aws:s3:::doc-insights-bucket"]}
  ]
}

Edge cases
- SSE-KMS blocks reads: add the QuickSight role principal to your KMS key policy.
- Cross-account S3: add a bucket policy that allows the QuickSight account role + use a manifest file.

4) KMS key policy that won’t bite you later

Add all compute roles and the QuickSight role to the key policy (not just IAM perms), e.g.:

{
  "Version":"2012-10-17",
  "Statement":[
    {"Sid":"EnableRoot","Effect":"Allow","Principal":{"AWS":"arn:aws:iam::ACCOUNT:root"},"Action":"kms:*","Resource":"*"},
    {"Sid":"AllowUseOfKey","Effect":"Allow","Principal":{"AWS":[
      "arn:aws:iam::ACCOUNT:role/TextractRole",
      "arn:aws:iam::ACCOUNT:role/BedrockRole",
      "arn:aws:iam::ACCOUNT:role/service-role/QuickSight-ROLE"
    ]},"Action":["kms:Encrypt","kms:Decrypt","kms:GenerateDataKey","kms:DescribeKey"],"Resource":"*"}
  ]
}

Edge cases
- AccessDenied (kms:Decrypt) in QuickSight/Athena: 99% of the time it’s the key policy missing the service role.

5) Networking (private, compliant, and fast)

Create VPC interface endpoints (PrivateLink) for:
- S3 (Gateway endpoint), Textract, Bedrock, STS, Logs, etc.
Lock your compute (Lambda/ECS/EC2) to no-internet and route AWS traffic via endpoints.
Edge cases
- QuickSight can’t sit in your VPC: it’s managed; keep the S3 bucket policy open to QuickSight role (not public) and allow KMS for that role.
- Corporate egress proxy: verify Bedrock/Textract endpoints are reachable or use endpoints.

6) Turn on Amazon Bedrock access (once per account/region)

In the Bedrock console, enable model access (Claude/Mistral etc.). Some models require an opt-in.
Edge cases
- Model not visible: you’re in a region without that model, or access not granted yet.
- Throughput errors: request service quota increases for InvokeModel if you batch summaries.

7) Enable Amazon QuickSight

Choose the same region as your S3 if possible.
In Security & permissions, tick your bucket and (if needed) Athena access.
Decide SPICE vs Direct Query:
- SPICE (in-memory) is faster; watch capacity limits.
- Direct is fine for small CSVs; for large data, prefer Parquet in S3 via Athena.
Edge cases
- CSV > size limit / too many rows: convert to Parquet and query via Athena.
- Timezone mismatches: set dataset time zone explicitly; normalize date formats (ISO-8601).

8) Budget, cost alerts, and audit

AWS Budgets: create a monthly budget with email alerts (S3 + Textract + Bedrock + QuickSight).
CloudTrail data events: ON for S3 (per-object audit).
CloudWatch Logs: centralize Lambda/app logs to logs/.
Edge cases
- Sudden spikes: enable S3 object-level CloudWatch metrics and add alarms on object count/size in raw/.

9) File formats & naming that make your life easy

Uploads (raw): accept PDF/JPEG/PNG/TIFF; normalize on ingest.
Outputs: prefer CSV (QuickSight-friendly) and/or Parquet (Athena-friendly).
Names: raw/{yyyy}/{mm}/{dd}/{vendor}/{fileid}.pdf processed/{yyyy}/{mm}/{dd}/invoices.csv
Edge cases
- Weird encodings: force UTF-8, escape delimiters, include header row.
- Duplicate filenames: use a UUID suffix or checksum in the key.

10) Optional: IaC and CLI quick start

Create bucket (CLI)

aws s3api create-bucket --bucket doc-insights-bucket --create-bucket-configuration LocationConstraint=$AWS_REGION
aws s3api put-bucket-encryption --bucket doc-insights-bucket --server-side-encryption-configuration '{
  "Rules":[{"ApplyServerSideEncryptionByDefault":{"SSEAlgorithm":"aws:kms","KMSMasterKeyID":"arn:aws:kms:REGION:ACCOUNT:key/KEY_ID"}}]}'
aws s3api put-bucket-versioning --bucket doc-insights-bucket --versioning-configuration Status=Enabled
aws s3api put-public-access-block --bucket doc-insights-bucket --public-access-block-configuration \
'{"BlockPublicAcls":true,"IgnorePublicAcls":true,"BlockPublicPolicy":true,"RestrictPublicBuckets":true}'

Example lifecycle (CLI)

aws s3api put-bucket-lifecycle-configuration --bucket doc-insights-bucket --lifecycle-configuration '{
  "Rules":[
    {"ID":"raw-to-ia","Filter":{"Prefix":"raw/"},"Status":"Enabled","Transitions":[{"Days":30,"StorageClass":"STANDARD_IA"}]},
    {"ID":"expire-tmp","Filter":{"Prefix":"tmp/"},"Status":"Enabled","Expiration":{"Days":7}}
  ]}'

📄 Step 2: Use Amazon Textract to Extract Text from Documents (Production-grade)

Textract can do three things you’ll care about:

OCR (lines/words) → DetectDocumentText
Generic forms & tables → AnalyzeDocument with FORMS, TABLES, optionally QUERIES
Invoice/receipt specialization → AnalyzeExpense (often best for invoices)

Rule of thumb

If it’s an invoice/receipt, try AnalyzeExpense first (it returns normalized fields like VENDOR_NAME, INVOICE_TOTAL).

If it’s a contract/form/report, use AnalyzeDocument with FORMS and TABLES.

Use QUERIES to target specific fields (e.g., “What is the due date?”) when layout varies wildly.

1) Sync vs Async: choose the right API

Synchronous (single call; immediate result)
- AnalyzeDocument (good for single-page or small docs; great for tests)
- DetectDocumentText (basic OCR)
Asynchronous (submit job → poll or page through results)
- StartDocumentAnalysis + GetDocumentAnalysis
- Required for multi-page and larger PDFs/images
- Supports pagination via NextToken

Guidance: Use sync for quick console/dev tests or one-pagers; use async for anything real (multi-page, batch).

2) Pre-processing (massively improves accuracy)

Input formats: PDF, PNG, JPEG, TIFF work well; keep scans clean.
Quality: Aim for readable text (good contrast, minimal blur). If possible:
- Deskew, denoise, increase contrast, convert to grayscale/PNG.
- Fix rotation (many scanners rotate by 90° or 180°).
Encrypted or password-protected PDFs: decrypt before upload.
Color: Grayscale often beats noisy full-color scans.
Multi-page splits: If you hit page/file limits, split before upload.
File names: Include vendor/date hints in the key to simplify downstream routing (e.g., raw/2025/09/11/vendorX/invoice_123.pdf).

3) Minimal but robust code paths (Python/boto3)

A. Synchronous: quick test for forms + tables

import boto3

textract = boto3.client("textract", region_name="eu-central-1")

resp = textract.analyze_document(
    Document={"S3Object": {"Bucket": "doc-insights-bucket", "Name": "invoice1.pdf"}},
    FeatureTypes=["FORMS", "TABLES"]
)

blocks = resp.get("Blocks", [])
# → Save raw JSON, parse KEY_VALUE_SET/TABLE blocks later

B. Asynchronous: production for multi-page docs (handles pagination + retries)

import boto3, time, itertools

textract = boto3.client("textract", region_name="eu-central-1")

job = textract.start_document_analysis(
    DocumentLocation={"S3Object": {"Bucket": "doc-insights-bucket", "Name": "long-invoice.pdf"}},
    FeatureTypes=["FORMS","TABLES"]
)
job_id = job["JobId"]

def get_all_pages(job_id, sleep=2, max_wait_s=900):
    start = time.time()
    pages = []
    next_token = None

    # Wait for completion (simple poll)
    status = "IN_PROGRESS"
    while status == "IN_PROGRESS":
        res = textract.get_document_analysis(JobId=job_id, NextToken=next_token) if next_token \
              else textract.get_document_analysis(JobId=job_id)
        status = res["JobStatus"]
        if status == "IN_PROGRESS":
            # If first page says in progress, wait; otherwise collect & keep paging
            if "Blocks" in res:
                pages.append(res)
                next_token = res.get("NextToken")
                if not next_token:
                    time.sleep(sleep)
            else:
                time.sleep(sleep)
            if time.time() - start > max_wait_s:
                raise TimeoutError("Textract job took too long.")
        elif status == "SUCCEEDED":
            pages.append(res)
            while res.get("NextToken"):
                res = textract.get_document_analysis(JobId=job_id, NextToken=res["NextToken"])
                pages.append(res)
        else:
            raise RuntimeError(f"Textract job failed: {res.get('StatusMessage', status)}")
    return pages

pages = get_all_pages(job_id)
# Combine blocks across pages for parsing
all_blocks = list(itertools.chain.from_iterable(p["Blocks"] for p in pages))

C. Invoices/Receipts: specialized `AnalyzeExpense` (often higher accuracy on totals/dates)

import boto3

textract = boto3.client("textract", region_name="eu-central-1")

resp = textract.analyze_expense(
    Document={"S3Object": {"Bucket": "doc-insights-bucket", "Name": "invoice1.pdf"}}
)

# resp["ExpenseDocuments"][0]["SummaryFields"] includes types like:
# VENDOR_NAME, INVOICE_RECEIPT_DATE, INVOICE_RECEIPT_ID, TOTAL, DUE_DATE

D. Targeted fields with `QUERIES` (when layouts vary)

import boto3
textract = boto3.client("textract", region_name="eu-central-1")

resp = textract.analyze_document(
    Document={"S3Object": {"Bucket": "doc-insights-bucket", "Name": "invoice1.pdf"}},
    FeatureTypes=["QUERIES","TABLES"],  # you can mix with FORMS too
    QueriesConfig={
        "Queries": [
            {"Text": "What is the invoice number?", "Alias": "invoice_no"},
            {"Text": "What is the total amount?", "Alias": "total_amount"},
            {"Text": "What is the due date?", "Alias": "due_date"}
        ]
    }
)

You can run QUERIES + TABLES together to pull exact fields and still capture line items.

4) How to parse Textract blocks safely (just enough for Step 2)

Textract represents a graph of Blocks with Relationships. The key ones:

PAGE, LINE, WORD
KEY_VALUE_SET (with EntityTypes: KEY or VALUE)
TABLE → CELL (cell has row/col indices)
QUERY → ANSWER

Parsing outline (KV):

Build a dict of blocks by Id.
Collect all KEY_VALUE_SET blocks.
For each KEY block, follow Relationships to its VALUE block.
Extract text by concatenating child WORD blocks; track Confidence.
Normalize keys (vendor, invoice no, etc.) and keep the BoundingBox for traceability.

Parsing outline (TABLE):

Find TABLE blocks, get their child CELLs.
Use (RowIndex, ColumnIndex) to reconstruct rows.
Heuristically detect headers (first row or bold-ish/upper-case hints if present in text).
Output as CSV; preserve page number.

We’ll do the full CSV/JSON conversion in Step 3. Here we only ensure you’ve got clean raw JSON saved and a clear parser plan.

5) Storage & idempotency

Always save raw responses (one file per source doc):
- textract-json/<doc-id>.json (for AnalyzeDocument/DetectDocumentText)
- textract-json/<doc-id>.expense.json (for AnalyzeExpense)
Add metadata to input S3 objects (e.g., processed=true, textract_job_id=...) to avoid reprocessing.
Use S3 object ETag/size as a cheap duplicate detector.

6) Confidence, validation & fallbacks

Confidence thresholds: treat fields under a threshold (e.g., < 0.8) as “needs review.”
Numeric sanity checks: parse amounts with a currency regex, verify:
- sum(line_items) ≈ total (± a few cents, taxes/discounts considered)
- date is valid & plausible (e.g., due date ≥ invoice date)
Disambiguation: fight common OCR confusions:
- O↔0, l/I↔1, S↔5, B↔8
Fallback order for invoices:

AnalyzeExpense
AnalyzeDocument (FORMS+TABLES)
DetectDocumentText + regex heuristics (as a last resort)
- Human-in-the-loop: flag low-confidence docs to a review queue (CSV row with a needs_review=1 flag).

7) Permissions & security edge cases (most common causes of failure)

S3: role needs s3:GetObject, s3:ListBucket, and if encrypted, kms:Decrypt on the CMK.
KMS key policy: include the calling role principal (not just IAM permissions).
Cross-account: use bucket policy + role assumption; ensure KMS allows the assumable role’s principal.
Private networking: if using VPC-only compute, add VPC endpoints for Textract/S3/STS/Logs.

8) Scaling, throughput, and cost control

Batching: drive Textract from a queue (SQS) or Step Functions; cap concurrency to avoid throttling.
Retries: exponential backoff on ThrottlingException and transient network errors.
De-dup: check if a doc was already processed (S3 metadata) before calling Textract again.
Specialization saves money: AnalyzeExpense or QUERIES can reduce post-processing and re-runs.

9) Troubleshooting cheat-sheet

Symptom	Likely cause	Fast fix
`AccessDenied` on S3 or KMS	Missing `s3:GetObject` / `kms:Decrypt` or KMS key policy excludes role	Add actions to role and role principal to KMS key policy
Job stuck “IN_PROGRESS” forever	Not polling pages / ignoring `NextToken`	Implement full pagination loop until `NextToken` is empty
Empty or low-quality text	Bad scans (blur, skew), wrong rotation, low contrast	Preprocess image: deskew, increase contrast, convert to PNG, rotate correctly
Fields missing on invoices	Using generic forms instead of expense API	Switch to `AnalyzeExpense` or add `QUERIES`
Dates/amounts parsed wrong	Locale/regex issues or OCR confusions	Normalize formats (ISO-8601), add currency/number regex, apply character-swap heuristics
QuickSight can’t see outputs	Using KMS + missing QS access	Add QS role to bucket policy and KMS key policy

10) Console workflow (great for validation)

Upload a sample to S3 → Textract Console → Analyze.
Toggle Forms + Tables, confirm highlights match expectations.
For invoices, test Expense analysis and compare field names (often cleaner).
Save the result JSON and move on to parsing (Step 3).

✏️ Step 3: Convert Textract Output to Structured Data (CSV or JSON)

1) Clean & normalize the extracted data

Keep originals: store the raw Textract JSON (read-only) so you can re-parse later without re-running Textract.
Whitespace & casing: trim, collapse double spaces; keep a _raw copy for each value if you’ll audit later.
Key normalization (FORMS/QUERIES): map variants to canonical keys:
- invoice no, inv #, invoice number → invoice_number
- total, grand total, amount due → total
- supplier, vendor, billed by → vendor_name
- invoice date, date → invoice_date; due date → due_date
Dates: convert to ISO-8601 (YYYY-MM-DD). If ambiguous/parsing fails, leave blank and flag needs_review.
Numbers & currency: strip thousands separators, unify decimals (.), extract currency symbol/code ($→USD, €→EUR, £→GBP, ₨→PKR, ₹→INR). If none, set a default and flag for review.
Confidence tracking: carry Textract confidence (0–100). Anything below your threshold (e.g., 80) sets needs_review = 1.
OCR confusions: apply gentle corrections when context demands numeric values (O↔0, I/l↔1, S↔5, B↔8) but still keep needs_review if confidence was low.

2) Choose your outputs

CSV (analytics-ready):
- KV (long format): doc_id, page, key_norm, value, confidence, needs_review
- Tables (per page/table): doc_id, page, row, col, text, confidence, is_header
- Invoice summary (if using Expense API): one row per document with normalized fields
JSON (traceability): keep structured JSON when you need bounding boxes/page indices for audits.
(Optional) Parquet: for Athena/QuickSight at scale.

3) Dump key–value pairs to CSV

What to capture: each detected pair as one row in a long table:
- doc_id, page, key_raw, key_norm, value, conf_key, conf_val, needs_review
Duplicates: keep all candidates. Later, pick the highest confidence (ties → closest to the label “Total”, latest on the page, or appears in a summary box).
Missing values: if a KEY has no VALUE, record value = "" and set needs_review = 1.
Multi-line values: join lines with a single space; preserve original in value_raw if you keep audits.

4) Convert tables from JSON to CSV

Reconstruct using cells: one row per cell with indexes:
- doc_id, page, row, col, text, confidence, span_rows, span_cols, is_header
Headers: default to row 1 = header; mark is_header = 1. If row 1 has poor confidence but contains many alphabetic tokens vs numeric rows below, still treat as header.
Merged cells: repeat the merged text in the spanned area and record span_rows/span_cols for fidelity.
Multi-page tables: keep separate by page; your business logic decides whether to stitch them later.

5) Build the simple “wide” table for your dashboard

Target columns from your outline:

Invoice No (`invoice_number`)	Date (`invoice_date`)	Vendor (`vendor_name`)	Amount (`total`)	Due Date (`due_date`)
INV-123	2025-08-03	Alpha Supplies	423.90	2025-09-02

Population priority:

Expense API (if used): INVOICE_RECEIPT_ID, INVOICE_RECEIPT_DATE, VENDOR_NAME, TOTAL, DUE_DATE.
FORMS/QUERIES: after key normalization and numeric/date parsing.
TABLES/OCR fallback: totals row or summary box heuristics (e.g., last column named Amount, Total, Balance).

Also add helpful derived fields:

currency, source (expense/forms/ocr), needs_review, month, year.

6) Quality & integrity checks

Arithmetic sanity: if you have subtotal and tax, verify subtotal + tax ≈ total (small tolerance for rounding). If not, needs_review = 1.
Date logic: due_date ≥ invoice_date; otherwise needs_review = 1.
Negatives & zeros: total must be > 0 for invoices; credits/notes may be negative tag with doc_type = credit_note.
Duplicates: same invoice_number appearing across different doc_ids → keep the one with higher composite score (weighted confidence of critical fields) and mark others as superseded.
Two documents in one file: if you detect two distinct vendors or invoice numbers in one doc_id, set multi_doc_suspected = 1 and route to review.
Locale drift: month names/currency symbols outside your default locale → still parse but set locale_detected and needs_review if parsing relied on heuristics.
Handwritten/low DPI scans: values may be fragmented; prefer QUERIES for targeted fields and flag low confidence.

7) Foldering

textract-json/            # raw API outputs (never edited)
extracted/
  kv/                     # <doc_id>_kv.csv
  tables/                 # <doc_id>_table_<page>.csv
  expense/                # <doc_id>_summary.csv, <doc_id>_line_items.csv (if used)
processed/
  invoices_wide.csv       # your dashboard input (or Parquet partitioned by year/month)

🧠 Step 4: Summarise the Document Using Bedrock

1) What you’ll summarise

Feed Bedrock only what it needs:

Facts you already extracted in Step 3 (invoice_number, vendor_name, total, due_date, currency, etc.).
Top signal text: page headers, totals area, vendor/address block, and (for invoices) the line-items table flattened to a few lines (don’t paste thousands of rows).
Optional full text: chunked by page if you need narrative context.

Redact or truncate PII that isn’t needed (addresses beyond city/country, phone, full account numbers). Keep the doc_id so you can trace back.

2) Output format

Ask Bedrock for structured JSON + a short narrative. Example JSON schema:

{
  "doc_id": "string",
  "summary": "1–3 sentence human summary",
  "key_fields": {
    "vendor_name": "string|null",
    "invoice_number": "string|null",
    "invoice_date": "YYYY-MM-DD|null",
    "due_date": "YYYY-MM-DD|null",
    "currency": "string|null",
    "total_amount": "number|null"
  },
  "observations": [
    "string"  // anomalies, discounts, late fees, notes
  ],
  "data_quality": {
    "missing_fields": ["string"],
    "suspected_errors": ["string"],
    "confidence_note": "string"
  }
}

This slots straight into your processed/ layer or QuickSight as a secondary dataset.

3) Prompting

Use a system + user style prompt; instruct the model not to invent data.

System

You are a careful financial document assistant. Use ONLY the supplied text and fields.
If a field is missing or unclear, return null and add a helpful note in data_quality.
Output must be valid JSON matching the provided schema. Do not include any other text.

User

DOC_ID: {{doc_id}}

FACTS (from extraction):
- vendor_name: {{vendor_name}}
- invoice_number: {{invoice_number}}
- invoice_date: {{invoice_date}}
- due_date: {{due_date}}
- currency: {{currency}}
- total_amount: {{total}}

EXCERPTS (trimmed; page-tagged):
[PAGE 1]
{{top_of_page_text}}
[PAGE 2 - totals box]
{{totals_area_text}}
[OPTIONAL line items]
{{few_key_lines}}

TASK:
1) Produce a JSON object exactly matching this schema:
{{paste schema}}
2) Base fields on FACTS when available; only use EXCERPTS to confirm or fill blanks.
3) If numbers disagree, prefer FACTS but note the discrepancy in `data_quality.suspected_errors`.
4) Do not guess. Use `null` for unknowns.

Why this works

It anchors the model to your extracted fields (reduces hallucinations).
It forces a predictable JSON payload for downstream use.
It adds a quality channel for “known unknowns”.

4) Model choice & params

Claude (Anthropic): strong on instruction following & JSON fidelity.
Mistral: concise, fast; good when payloads are small and schema is simple.
Parameters: keep temperature low (≈0.0–0.2) for factual stability; set max tokens tight (e.g., 600–900) for cost control.

Tip: If JSON validation fails sometimes, wrap with a “fix to valid JSON” pass (or enable JSON mode where available).

5) Long documents or Map/Reduce

For multi-page or very verbose docs:

Map step: summarize each page (fixed schema, plus page-level anomalies).
Reduce step: feed all page summaries (+ your Step-3 facts) into a second prompt that merges into a single final JSON, breaking ties in favor of Step-3 facts.

This avoids token limits and preserves page context.

6) Robust invocation

Use bedrock-runtime with the model you enabled (ids vary by region/provider). Two common patterns:

Non-streaming (simple)

import json, boto3, time
brt = boto3.client("bedrock-runtime", region_name="YOUR_REGION")

def invoke_bedrock(model_id, system_prompt, user_prompt, max_tokens=800, temperature=0.1):
    body = {
        "anthropic_version": "bedrock-2023-05-31",
        "messages": [
            {"role": "system", "content": system_prompt},
            {"role": "user", "content": user_prompt}
        ],
        "max_tokens": max_tokens,
        "temperature": temperature
    }
    resp = brt.invoke_model(modelId=model_id, body=json.dumps(body))
    out = json.loads(resp["body"].read())
    # Extract the text depending on provider; for Claude:
    text = "".join([c.get("text","") for m in out.get("content",[]) for c in [m]])
    return text

Streaming

Stream when you want low latency for UI, otherwise skip (adds complexity).

Note: Exact request/response shapes differ by model family. Keep the modelId and payload format configurable and test in the Bedrock console first.

7) Validation & safety rails

JSON validation: run a quick schema check (pydantic/jsonschema). If invalid, re-prompt the model with: “Return only valid JSON conforming to this schema. Do not add commentary.”
No-invention rule: instruct “If unsure, use null and add a data_quality note.” Penalize hallucinations by rejecting outputs that add fields you didn’t request.
Discrepancy handling: when EXCERPTS disagree with FACTS, keep FACTS and add a note like: “Totals area shows 432.90 but extracted total is 423.90 (difference 9.00).”
PII minimization: don’t pass line-item text unless needed; mask account numbers (e.g., ****1234).
Cost control: truncate excerpts, avoid full dumps; cap max_tokens; batch summarize.

8) Edge cases

Edge case	What to do in prompt / output
Missing total or due date	Return `"total_amount": null` / `"due_date": null`; add a `data_quality.missing_fields` entry.
Multiple totals (invoice vs balance)	Prefer FACTS; if absent, pick the one labeled “Total” near summary; note the alternative.
Conflicting currency symbols vs codes	Prefer the currency from FACTS; if excerpt differs, note discrepancy.
Handwritten / low DPI	Expect nulls and notes; don’t interpolate from context.
Multi-currency line items	Set currency to document currency; add observation like “Line items include mixed currency not used for total.”
Multi-doc PDF	If two vendors or two invoice numbers appear, say so in `observations` and add `suspected_errors`.
Non-English docs	Add “Output in English.” Keep dates numeric (YYYY-MM-DD).
Extremely long inputs	Use Map/Reduce; only pass per-page top areas and final consolidation.

9) Example “good” output

{
  "doc_id": "INV-2025-0045",
  "summary": "Invoice 0045 from Beta Traders for 89.50 EUR, due on 2025-08-28.",
  "key_fields": {
    "vendor_name": "Beta Traders",
    "invoice_number": "0045",
    "invoice_date": "2025-08-14",
    "due_date": "2025-08-28",
    "currency": "EUR",
    "total_amount": 89.50
  },
  "observations": [
    "Early-payment discount mentioned (2%)."
  ],
  "data_quality": {
    "missing_fields": [],
    "suspected_errors": [],
    "confidence_note": "All fields aligned with extracted facts."
  }
}

10) Where to store results

Write JSON summaries to summaries/ and (optionally) a compact CSV view processed/invoice_summaries.csv with doc_id, vendor_name, total_amount, due_date, needs_review, source='bedrock'.
Join this dataset with invoices_wide.csv for dashboard overlays (e.g., a tooltip with the narrative summary).

🗂️ Step 5: Store Final Data Back in S3

1) Folder layout

Use a layout that separates raw vs extracted vs consumable and supports BI tools:

s3://doc-insights-bucket/
  processed/
    wide/                 # final, dashboard-ready “wide” tables (CSV/Parquet)
    kv/                   # merged KV outputs across docs
    tables/               # flattened tables across docs
    expense/              # invoice summaries & line items (if using AnalyzeExpense)
    summaries/            # Bedrock JSON summaries
    parquet/              # same datasets in Parquet (for Athena/QuickSight at scale)
    _staging/             # temp writes (see atomic pattern)
    manifests/            # optional manifest/index files

Partitioning (for Athena/scale):

Prefer hive-style partitions for Parquet:
- processed/parquet/invoices_wide/year=2025/month=09/day=12/part-0001.parquet
Keep CSVs unpartitioned or lightly partitioned by month to avoid too many small files.

2) Naming strategy

Document-scoped files: processed/expense/summary/<doc_id>.csv processed/summaries/<doc_id>.json
Batch/append files: processed/wide/invoices_wide_YYYYMMDD.csv (or append into a single table and de-dupe by doc_id)
Timestamped artifacts: add an ISO timestamp or a monotonically increasing batch id:
- .../2025-09-12T18-40-12Z_invoices_wide.csv
Avoid collisions: include doc_id (or a UUID) in per-doc artifacts; never rely only on original filenames.

3) Atomic write pattern

S3 has strong read-after-write consistency, but clients can still read half-written objects if your writer crashes mid-upload.

Pattern:

Write to processed/_staging/<uuid>/<file.tmp> (or status=writing tag).
Verify checksum/row count.
Copy to final key (e.g., processed/wide/invoices_wide_20250912.csv).
Optionally delete the staging file, or keep with tag status=archived.

Edge cases:

Large files (>5 GB): use multipart upload; always send Content-MD5 for integrity when feasible.
Concurrent writers: add a lightweight lock (S3 object processed/_locks/invoices_wide.lock with short TTL) or write per-writer partitions then compact later.

4) Metadata, tags, and headers

Server-side encryption: SSE-KMS with your CMK (consistent with earlier steps).
Content-Type: set correctly (text/csv, application/json, application/parquet).
Cache-Control: not crucial for data lakes; you can omit or set no-cache for rapidly changing CSVs.
Object tags (helpful for governance/search):
- dataset=invoices, stage=processed, doc_id=<id>, status=ready, source=expense/forms/ocr/bedrock.
Custom metadata (e.g., x-amz-meta-rowcount, x-amz-meta-hash): useful for quick validation.

5) Security & access

Bucket policy: already denies insecure transport; also allow the QuickSight service role to ListBucket and GetObject on processed/*.
KMS key policy: include the QuickSight role principal (not just IAM permissions) so it can decrypt.
Cross-account reads (optional): prefer role assumption + bucket policy + KMS grants.
VPC endpoints: if compute is private, ensure endpoints for S3/KMS are present.

Edge cases:

QuickSight can list but can’t read → key policy is missing the role principal.
Using SPICE: watch dataset size limits; switch to Athena over Parquet when CSVs grow.

6) Data integrity & quality gates

Before moving from _staging → processed:

Schema check: headers exist and match expected column set.
Row count > 0 (or allow zero with a note=empty_batch tag).
Type validation: dates parse (ISO-8601), amounts numeric, currency present.
Sanity checks: if files include subtotal, tax, total, verify subtotal + tax ≈ total (tolerance).
Checksum: compute SHA-256 of the file and store as sidecar:
- processed/wide/invoices_wide_20250912.csv.sha256

7) Lifecycle & retention

Versioning: keep enabled to roll back erroneous overwrites.
Lifecycle rules:
- Transition processed/_staging/ to Expire after 7 days.
- Transition processed/*.csv to STANDARD_IA after 30 days.
- Keep Parquet in STANDARD (or GLACIER if archived) based on access.
S3 Inventory (optional): enable for daily listings & encryption audit.

8) Manifests & discovery (helps BI tools)

Manifest file lists the latest batch outputs:

  {
    "dataset": "invoices_wide",
    "updated_at": "2025-09-12T18:40:12Z",
    "files": [
      "s3://.../processed/wide/invoices_wide_20250912.csv"
    ],
    "row_count": 1243,
    "schema": ["doc_id","vendor_name","invoice_number","invoice_date","due_date","total","currency","needs_review","source","month","year"]
  }

Save as processed/manifests/invoices_wide.latest.json.

Glue Catalog (for Athena): create/update a table pointing to your Parquet path with partitions year, month, day.

9) Eventing

S3 → EventBridge rule on processed/ prefix:
- Trigger a Lambda to refresh QuickSight (or kick an ingestion SPICE refresh).
- Or publish a Slack/Email notification with the manifest summary.
Idempotency: include run_id or batch_id in event detail; ignore duplicates in the consumer.

📊 Step 6: Create a Dashboard in QuickSight

1) Pick your connection path

Path A | S3 CSV (fast start)

Best for small/medium datasets (≤ a few million rows per file).
In QuickSight: Datasets → New dataset → S3.
Point to a single CSV or a manifest JSON (recommended if you have many files).

Path B | Athena + Parquet (scales)

Best for large data, partitions, or frequent refresh.
Store Parquet at processed/parquet/invoices_wide/year=YYYY/month=MM/day=DD/...
In QuickSight: Datasets → New dataset → Athena → select database & table (from Glue Catalog).

Edge cases

If QuickSight is in a different region than your S3 bucket, prefer Athena to avoid cross-region pain.

Using SSE-KMS? Ensure the QuickSight service role is in your KMS key policy and bucket policy.

2) Connect to your data

A) S3 with a manifest (many files)

Create processed/manifests/invoices_wide.latest.json:

{
  "fileLocations": [
    {"URIPrefixes": ["s3://doc-insights-bucket/processed/wide/"]}
  ],
  "globalUploadSettings": {
    "format": "CSV",
    "delimiter": ",",
    "textqualifier": "\"",
    "containsHeader": true
  }
}

Then in QuickSight S3 connector, paste this manifest file’s S3 URL.

B) Athena (Parquet partitions)

Make sure your Glue table points to processed/parquet/invoices_wide/ and has partitions year, month, day.
In QuickSight, choose Direct Query (live) or Import to SPICE (in-memory; faster but size-limited).

Edge cases

“AccessDenied” when previewing = missing KMS permission or bucket grant.

No new data after partition upload = run crawler/add partition OR enable partition projection.

3) Prepare the dataset

In the data prep screen:

Map columns

invoice_number → Text
invoice_date, due_date → Date (format YYYY-MM-DD)
total → Decimal
vendor_name → Text
currency → Text
needs_review → Integer/Boolean
(Optional) ingest_date or processed_at → Date/Time

Create calculated fields (QuickSight expressions)

Month (bucket) truncDate('MM', {invoice_date})
Year extract('YYYY', {invoice_date})
Overdue (flag) ifelse({due_date} < now(), 1, 0)
Processing time (days) – pick the source you actually have:
- If you kept an ingest timestamp: dateDiff('DD', {invoice_date}, {ingest_date})
- Or from S3 key date (if you saved it): dateDiff('DD', {invoice_date}, {s3_ingest_date})
Clean total (fallback) – if some totals are text:

parseDecimal({total})

Edge cases

Mixed date formats? Normalize upstream; otherwise use a calculated field with parseDate() per pattern and coalesce.

Multiple currencies? Keep a currency dimension; optional FX table join later.

4) Build the visuals

A) Total amount by vendor (Top-N bar)

Visual: Horizontal bar
X: sum({total})
Y: vendor_name
Sort: by sum(total) desc
Filter: Top 10 vendors (add a Top N filter)
Optional: color by needs_review to spot messy vendors

B) Monthly invoice totals (trend)

Visual: Line chart
X: truncDate('MM', {invoice_date})
Y: sum({total})
Forecast (optional): enable forecast to project next months
Tooltip: add countDistinct({invoice_number}) for volume context

C) Average processing time (days)

Visual: KPI or bar by vendor
Field: avg(dateDiff('DD', {invoice_date}, {ingest_date}))
Group by: vendor_name (or monthly bucket)
Filter: exclude needs_review = 1 if you want only clean rows

D) Bonus quick wins

Overdue heatmap: vendor_name × month with color sum(ifelse({due_date} < now(), {total}, 0))
Quality gate: table filtered to needs_review = 1 (your manual review queue)

5) Filters, controls, and parameters

Date range control: Filter on {invoice_date} with “Relative dates” (e.g., Last 12 months).
Vendor control: Dropdown on vendor_name (multi-select).
Confidence/quality toggle: needs_review = 0 by default (checkbox control).
Currency control (if applicable): filter or parameter to select a single currency.

Edge cases

If your dataset mixes currencies, don’t sum across different codes in one visual unless you convert first.

6) SPICE vs Direct Query

SPICE (Import): fastest visuals, schedule refreshes (e.g., hourly). Watch capacity limits.
Direct Query: always live; best with Athena/Parquet; slightly slower per query.

Recommended

Start with SPICE for S3 CSVs; switch to Athena + SPICE as you scale.

7) Refresh strategy

S3 CSV path
- If you overwrite one key (e.g., invoices_wide_latest.csv), set a refresh schedule (hourly/daily).
- If you append new files, use a manifest and schedule refresh.
Athena path
- After adding partitions, either run a crawler or use partition projection.
- In QuickSight, schedule SPICE refresh or stay on Direct Query.

Edge cases

Refresh fails with KMS errors → add QuickSight role to CMK key policy.

Stale data → you’re writing to new paths; either point QS at a stable key or use manifests.

8) Sharing, RLS, and governance

Share analysis → Publish dashboard; grant users/groups access.
Row-Level Security (RLS): upload an RLS mapping dataset (user, vendor_name) to restrict rows per viewer.
Data dictionary tab: add field descriptions so the team knows what “processing time” means.

10) Minimal recipe

Connect

S3: dataset → S3 → manifest → validate preview.
Athena: dataset → Athena → select table.

Model

Types: map date/number fields.
Calcs: month = truncDate('MM', {invoice_date}), processing_days = dateDiff('DD', {invoice_date}, {ingest_date}).

Visualize

Bar: sum(total) by vendor_name (Top 10).
Line: sum(total) by month.
KPI/Bar: avg(processing_days) by vendor.

Control

Date range (last 12 months).
Vendor multi-select.
needs_review toggle (default 0).

Operationalize

SPICE refresh schedule (hourly/daily).
If Athena: ensure partitions & permissions.

💭 Conclusion

This was honestly a fun little fortnight project.
It saved me hours of manual work and gave me clean, visual summaries of everything I needed. By combining S3, Textract, Bedrock, and QuickSight, you can turn unstructured, scanned documents into analysis-ready data and clear dashboards. This pipeline handles extraction, normalisation, summarisation, governed storage, and visualisation, built with security (SSE-KMS), least-privilege IAM, and scalable patterns (async Textract, Parquet/Athena). It replaces manual review with consistent, auditable automation and surfaces vendor spend, monthly trends, and processing KPIs in minutes. From here, you can harden orchestration with Step Functions, add Row-Level Security in QuickSight, and extend the same approach to receipts, contracts, or reports. Small lift-big leverage.

You could easily apply this to:

Legal contracts
Academic papers
Medical prescriptions
Tax docs

If you try it out and get stuck, drop a comment, and I'd be happy to help.

HappyHacking 👋

🚀 Scaling React Like Big Tech: Folder Structures, Clean Code & Beyond

Mursal Furqan Kumbhar — Sat, 30 Aug 2025 18:55:22 +0000

Hey Everyone 👋

I know it's been quite a long gap since my last article. But we are finally back with a bang.

So if you’ve ever wondered how Airbnb, Shopify, or Meta keep their codebases sane while thousands of engineers ship features daily — the answer is simple: structure + discipline + automation.

This isn’t about writing code that just works. It’s about writing code that:

Scales with your team.
Survives onboarding of 10 new devs.
Passes reviews without a hundred nitpicks.
And doesn’t make you cry when you open it six months later.

Let’s break down what the big companies do differently — and how you can bring those practices into your own projects.

📂 Folder Structure that Scales

Top companies don’t just throw files into a random components/ folder and hope for the best. They use feature-driven architecture, which grows naturally with the product.

❌ Traditional (Bad) Structure

src/
├── components/
│   ├── Button.jsx
│   ├── Card.jsx
│   ├── Form.jsx
├── services/
├── hooks/
└── App.jsx

This looks fine at first… until your app has 300+ components and no one knows which belongs to what.

✅ Scalable (Feature-Based) Structure

src/
├── features/
│   └── auth/
│       ├── components/
│       ├── hooks/
│       └── services/
├── shared/
│   ├── ui/
│   └── utils/
└── App.jsx

📌 Key Difference: Everything related to a single feature (Auth, Products, Cart, etc.) is grouped. Shared logic/components live in shared/.

Approach	Pros	Cons
Traditional (by type)	Simple to start	Becomes chaos at scale
Feature-Based	Easy ownership, scalable, testable	Slightly more setup upfront

🎨 Design Systems & UI Libraries

Big companies don’t reinvent the button every Tuesday. They use design systems and UI libraries.

🚀 Airbnb → Lona
🚀 Shopify → Polaris
🚀 Meta → Reusable UI + strict guidelines

👉 Tools you can use: Chakra UI, Tailwind + Headless UI, Radix UI

Visual Example

❌ Without Design System:

<Button variant="danger" color="red" bg="red" />

✅ With Design System:

<DeleteButton />

Why Use Design Systems?	Benefits
Consistency	UI looks the same across app
Speed	Devs move faster
Accessibility	Built-in ally support
Brand Identity	Product feels cohesive

🧑‍💻 Hooks = Where Logic Lives

Business logic doesn’t belong inside components. Custom hooks make code reusable, testable, and clean.

Example:

// useProduct.js
export function useProducts() {
   const [data, setData] = useState([])
   useEffect(() => {
      fetchProducts().then(setData)
   }, [])
   return data
}

✅ UI stays declarative.
✅ Logic stays modular.

📊 Diagram: Components vs Hooks

[Component] ---> UI (render)
       |
       v
 [Custom Hook] ---> Data Fetching, State, Business Logic

🧼 Clean Code = Scalable Code

Big tech engineers follow clean code principles:

SRP → Single Responsibility Principle
KISS → Keep It Simple, Stupid
DRY → Don’t Repeat Yourself

❌ Bad	✅ Good
`<Button variant="danger" color="red" bg="red" />`	`<DeleteButton />`

✅ Abstraction = Fewer Bugs + Cleaner Reviews

🔍 Real-Time Code Reviews + Linting

At big companies, automation + reviews are mandatory.

✅ Code Reviews → shared knowledge
✅ Pre-Commit Hooks (Husky) → catch errors early
✅ ESLint + Prettier → consistent style

CI/CD Pipeline Visual

[Developer Pushes Code] 
       |
       v
[Pre-Commit Checks: Lint + Tests]
       |
       v
[Pull Request Review]
       |
       v
[CI/CD: Build + Deploy + Monitor]

🔐 Security & ♿ Accessibility First

Security isn’t optional. Accessibility isn’t either.

Security Checklist

Prevent XSS and CSRF
Proper token handling
Input validation

Accessibility Checklist

Keyboard navigation works
Screen reader support
High color contrast

👉 Use tools like axe-core, eslint-plugin-jsx-ally, lighthouse.

⚡ Automation + Monitoring

Big teams don’t just ship code — they ship pipelines + monitoring.

✅ CI/CD Pipelines (GitHub Actions, Vercel, CircleCI)
✅ Bundle Size Alerts
✅ Error Tracking (Sentry, Datadog)
✅ Performance Monitoring (Lighthouse CI)

📊 Example:

Commit → Build → Test → Deploy → Monitor

📊 Documentation & Knowledge Sharing

What separates side projects from real companies?
👉 Documentation.

Good README.md = contributor-friendly.
Internal wikis = no lost knowledge.
ADRs (Architecture Decision Records) = explain why choices were made.

🧠 Culture = The Invisible Architecture

Finally — even the best code structure won’t save a bad culture. Big companies win because:

PRs = conversations, not fights.
Automation is embraced, not skipped.
Docs + testing = part of “done”.

At the end of the day, how your team codes together matters more than folder names.

🚦 Startup vs Big Tech Codebases

Here’s how things usually look in a scrappy startup vs how big tech organizes for scale:

┌────────────────────┐           ┌─────────────────────────┐
│   Startup Style    │           │     Big Tech Style      │
├────────────────────┤           ├─────────────────────────┤
│ 📂 components/     │           │ 📂 features/auth/       │
│ 📂 hooks/          │           │   ├── components/       │
│ 📂 services/       │           │   ├── hooks/            │
│ App.jsx            │           │   └── services/         │
│                    │           │ 📂 shared/ui/           │
│ Easy to start ✅   │           │ 📂 shared/utils/        │
│ Chaos later ❌      │           │ Scales across teams ✅  │
└────────────────────┘           └─────────────────────────┘

📊 Side-by-Side Table

Aspect	Startup-Style	Big Tech-Style
Folder Structure	Flat, all components in one folder	Feature-based, modular
UI	Custom each time	Design system + UI library
Logic	Inside components	Extracted into hooks
Code Quality	Quick fixes, less abstraction	Clean code, reusable patterns
Reviews	Ad-hoc, sometimes skipped	Mandatory PRs + linting + tests
Security	Later concern	Built-in from day one
Docs	README only	Wikis + ADRs + onboarding guides
Culture	“Move fast, break things”	“Move fast, don’t break prod”

🎯 Final Takeaway

Big companies scale not because their engineers are magical unicorns 🦄, but because they follow battle-tested practices:

Organize code by features, not file types.
Use design systems for UI.
Extract logic into hooks.
Follow clean code principles.
Automate reviews + pipelines.
Prioritize security & accessibility.
Write documentation.
Build a team culture around consistency.
Startups often optimize for speed, but end up with technical debt.
Big Tech optimizes for scalability, making onboarding, reviews, and growth smooth.
The sweet spot? Start with startup speed, but gradually adopt Big Tech practices before chaos hits.

Happy Hacking 👋

WSL2 TensorFlow GPU Setup – RTX 4060 + Ubuntu 22.04 + CUDA 12.2 + cuDNN

Mursal Furqan Kumbhar — Wed, 16 Apr 2025 16:36:12 +0000

🛠 Prereqs:

WSL2 enabled on Windows
NVIDIA GPU driver (≥ v535) installed on Windows
Ubuntu 22.04 installed via WSL
Download cuDNN manually while logged in to NVIDIA Developer

🧱 1. Setup Python environment in Ubuntu (WSL2)

sudo apt update && sudo apt install python3.11 python3.11-venv gcc-11 g++-11 -y
python3.11 -m venv ~/tf-gpu-311
source ~/tf-gpu-311/bin/activate
pip install --upgrade pip
pip install tensorflow==2.15.0

⚙️ 2. Install CUDA 12.2 (runfile method)

wget https://developer.download.nvidia.com/compute/cuda/12.2.0/local_installers/cuda_12.2.0_535.54.03_linux.run
chmod +x cuda_12.2.0_535.54.03_linux.run
sudo ./cuda_12.2.0_535.54.03_linux.run

❌ Uncheck driver install (you're using Windows one)
✅ Install toolkit only

Then set up environment:

echo 'export PATH=/usr/local/cuda-12.2/bin:$PATH' >> ~/.bashrc
echo 'export LD_LIBRARY_PATH=/usr/local/cuda-12.2/lib64:$LD_LIBRARY_PATH' >> ~/.bashrc
source ~/.bashrc

📦 3. Install cuDNN 8.9.4 (for CUDA 12.x)

Go to cuDNN Downloads
Login → Choose:
- Version: cuDNN 8.9.4
- OS: Linux
- Installer: Local Installer for Ubuntu22.04 x86_64 (.deb)
Download .deb file and move it into Ubuntu:

cp /mnt/c/Users/<yourname>/Downloads/cudnn-local-repo-ubuntu2204-8.9.4.25_1.0-1_amd64.deb ~/
cd ~
sudo dpkg -i cudnn-local-repo-ubuntu2204-8.9.4.25_1.0-1_amd64.deb
sudo cp /var/cudnn-local-repo-ubuntu2204-8.9.4.25/cudnn-*-keyring.gpg /usr/share/keyrings/
sudo apt update
sudo apt install libcudnn8 libcudnn8-dev -y

🔄 4. Restart WSL2

In PowerShell (Windows):

wsl --shutdown

Then open Ubuntu again:

source ~/tf-gpu-311/bin/activate

✅ 5. Test TensorFlow GPU

import tensorflow as tf
print("TensorFlow version:", tf.__version__)
print("GPUs available:", tf.config.list_physical_devices('GPU'))
print("CUDA enabled:", tf.test.is_built_with_cuda())
print("GPU name:", tf.test.gpu_device_name())

💚 Expected Output

GPUs available: [PhysicalDevice(name='/physical_device:GPU:0', device_type='GPU')]
CUDA enabled: True
GPU name: /device:GPU:0

Happy TensorFlowing, GPU wizard 🧙‍♂️⚡

Smart Image Tagging on AWS - The Finale

Mursal Furqan Kumbhar — Fri, 04 Apr 2025 12:10:00 +0000

Hello and Greetings Everyone 👋

I am hoping that you are having fun in our journey of creating the auto-tagging application for images like a pro using Amazon Rekognition, AWS Lambda and S3.

Recap

Let's do a quick recap of what we did in the last two parts.

1. Auto-Tag Images on AWS using Amazon Rekognition + Lambda + S3

Created an S3 bucket to store uploaded images
Created a Lambda function to handle image upload events
Set up IAM roles and permissions for Rekognition and S3 access
Used Amazon Rekognition to detect objects/scenes in images
Logged detected labels to CloudWatch Logs
Connected S3 and Lambda using event notifications

2. Level Up Your Auto-Tagging Pipeline on AWS

Uploaded test images and successfully triggered Lambda
Viewed detection results in CloudWatch logs
Improved the Lambda logic to store Rekognition labels as .json files inside a tags/ folder in S3
Validated that the full pipeline was working end-to-end

I guess that would have jogged your memory. In this article, we are going to discuss:

Sending real-time email notifications using Amazon SNS
Wrapping up the entire series with final thoughts and next steps

Let's Go 🚀

Send Real-Time notifications via Amazon SNS

As discussed, we already have created the pipeline for auto-tagging the image and saving the data in a JSON file. Now what can be cooler than getting an email whenever it happens? In this step, we will modify and enhance our pipeline by integrating Amazon SNS to send notifications everytime an image is tagged.

Create an SNS Topic

Like every other AWS Service, we need to first go to the AWS Console and find SNS (Simple Notification Service). On the SNS Homepage, write your topic's name, ImageTaggingNotifications, in the Create Topic textbox (As shown in the figure below). Once done, click on Next Step button. In the next screen, select Standard as the type and click Create Topic at the bottom of the screen leaving everything default.

Seeing our friend, the green notification box would tell us that our topic is created. Once done, click on orange Create Substription button and fill in Email under Protocol and type in your email under Endpoint.

All done? Click on Create Subscription. As soon as subscription is created, check your email for the confirmation from AWS. Click on the link to confirm the subscription.

Remember: Don't skip/miss this step or you won't get any emails.

The email would look something like this:

Give Lambda Permission to Publish to SNS

After successfully creating the topic and subscribing to it with your email, we need to go to IAM > Roles > RekognitionLambdaRole > Add Permissions > Create Inline Policy.

In the next screen, click on JSON and copy paste the following code:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "sns:Publish",
      "Resource": "arn:aws:sns:your-region:your-account-id:ImageTaggingNotifications"
    }
  ]
}

Don't forget to replace your-region (e.g. us-east-1) and your-account-id (e.g. 123456789012) accordingly

Click next and Give it a good name such as AllowSnsPublishPolicy in the next screen. After doing so, click on Create Policy to finish the policy creation process.

Update the Lambda Code to Send Notifications

After creating the policy, we need to let the Lambda function know when and how to send the notification. To do so, we need to the below given code right after saving the .json file in S3:

import boto3

sns = boto3.client('sns')
TOPIC_ARN = 'arn:aws:sns:us-east-1:896415180455:ImageTaggingNotifications'  # Replace with your actual ARN

# Get top 3 labels
top_labels = label_data[:3]
labels_text = ', '.join([f"{l['Name']} ({l['Confidence']}%)" for l in top_labels])
message = f"Image '{key}' was tagged with:\n\n{labels_text}"

# Send SNS notification
sns.publish(
    TopicArn=TOPIC_ARN,
    Message=message,
    Subject='[Image Tagging Alert] New Image Tagged on S3'
)

Or if you didn't get it, replace the complete code for Lambda Function with:

import json
import boto3
import urllib.parse
import datetime

rekognition = boto3.client('rekognition')
s3 = boto3.client('s3')
sns = boto3.client('sns')

# Replace with your actual SNS topic ARN
TOPIC_ARN = '<TOPIC-ARN>'

def lambda_handler(event, context):
    print("Lambda Triggered at:", str(datetime.datetime.now()))
    print("Received event:", json.dumps(event, indent=2))

    bucket = event['Records'][0]['s3']['bucket']['name']
    key = urllib.parse.unquote_plus(event['Records'][0]['s3']['object']['key'])

    # ✅ Skip non-image files
    if not key.lower().endswith(('.jpg', '.jpeg', '.png')):
        print(f"Skipping file {key} — not a supported image format.")
        return {
            'statusCode': 200,
            'body': json.dumps('Skipped non-image file.')
        }

    try:
        # 🔍 Step 1: Detect labels using Rekognition
        response = rekognition.detect_labels(
            Image={
                'S3Object': {
                    'Bucket': bucket,
                    'Name': key
                }
            },
            MaxLabels=10,
            MinConfidence=80
        )

        # 📋 Step 2: Format label data
        label_data = []
        for label in response['Labels']:
            label_data.append({
                'Name': label['Name'],
                'Confidence': round(label['Confidence'], 2)
            })
            print(f" - {label['Name']} ({label['Confidence']:.2f}%)")

        # 💾 Step 3: Save labels as JSON to /tags/
        image_name = key.rsplit('/', 1)[-1].rsplit('.', 1)[0]
        tags_key = f"tags/{image_name}.json"

        s3.put_object(
            Bucket=bucket,
            Key=tags_key,
            Body=json.dumps(label_data, indent=2),
            ContentType='application/json'
        )

        print(f"Saved tags JSON to: {tags_key}")

        # 📬 Step 4: Send SNS notification with top labels
        top_labels = label_data[:3]
        labels_text = ', '.join([f"{l['Name']} ({l['Confidence']}%)" for l in top_labels])
        message = f"Image '{key}' was tagged with:\n\n{labels_text}"

        sns.publish(
            TopicArn=TOPIC_ARN,
            Message=message,
            Subject='[Image Tagging Alert] New Image Tagged on S3'
        )

        print("Notification sent via SNS!")

        return {
            'statusCode': 200,
            'body': json.dumps('Labels detected, saved to S3, and notification sent!')
        }

    except Exception as e:
        print(f"Error processing {key}: {str(e)}")
        raise e

Replace TOPIC_ARN with your actual SNS Topic ARN.

And it's done!

After this, our entire email notification pipeline is complete. Let's test this. To test, go to your AWS S3 bucket and upload the image like we did in the previous article. If the entire pipeline is working, you will immediately receive an email with the top 3 tags of the image in the email address you gave while creating the SNS Topic. The email would look something like this:

If you don't get the email, go through all the process and see what you missed. And in case you don't find what you did wrong, write me in the comments below.

🎉 Conclusion

And that’s a wrap, folks! 🎬
We’ve come a long way — from uploading images to S3 and detecting labels with Rekognition, to storing results, sending real-time notifications, and polishing our code like true builders 💪

This serverless, scalable, and smart image tagging workflow is just one example of how you can combine AWS Lambda, Amazon S3, Rekognition, and SNS to automate powerful tasks with ease.

Whether you’re building this for fun, a portfolio project, or the foundation of a real-world app — I hope this series inspired you to explore what’s possible on AWS.

Thanks for sticking around till the end — and as always, if you enjoyed this, drop a ❤️, share it, or leave a comment.

Until next time... Ciao! 👋