DEV Community: Mursal Furqan Kumbhar

Same AWS Error, Two Answers: I Built a Tool That Stops Guessing at IAM

Mursal Furqan Kumbhar — Mon, 10 Aug 2026 09:22:55 +0000

Ciao Amici 👋

Let me start with a confession. Every time I hit an AccessDenied in AWS, I do the same lazy thing you do. I copy the wall of text, paste it into a chat model, and ask "why?".

And every time, I get back a tidy little list of maybes. Maybe it is your identity policy. Maybe it is an SCP. Maybe a permissions boundary. Maybe a resource policy. All plausible. All confident. None of them actually checked against my account.

That bothered me for a long time. The model was not lying to me exactly. It was doing the only thing it could do with the words I gave it. But "here are five things it might be" is not an answer. It is a starting point for an afternoon of clicking through the IAM console.

So I built a tool that stops guessing and starts checking. It is called Rosetta, and this post is the full story of building it, including the bug I found in my own ranking logic and the Bedrock wall I walked straight into.

Here is the whole thing in one screenshot, before I explain a single line of code.

The money shot

Same error string. One extra flag. Two completely different answers.

First, text only, no credentials at all:

  AWS Rosetta  [text-only]
  code=AccessDenied  op=GetObject  action=s3:GetObject
  principal=arn:aws:iam::896415180455:user/rosetta-victim
  resource=arn:aws:s3:::rosetta-demo-secret/report.txt

  1. [########............] 40%  No identity-based policy allows the action
  2. [########............] 40%  Explicit deny in a Service Control Policy
  3. [########............] 40%  Explicit deny in a permission boundary
  4. [########............] 40%  Resource policy denies or omits the principal
  5. [########............] 40%  Missing KMS grant on an encrypted resource
  note: Text-only mode. Pass a read-only role ARN for authoritative causes.

Five causes. All at 40 percent. The tool is shrugging politely.

Now the same error, with one --role-arn flag pointing at a read only role:

  AWS Rosetta  [enriched]
  code=AccessDenied  op=GetObject  action=s3:GetObject
  principal=arn:aws:iam::896415180455:user/rosetta-victim
  resource=arn:aws:s3:::rosetta-demo-secret/report.txt

  1. [###################.] 96%  Explicit Deny in an attached policy
     fix: An explicit Deny matched: user_rosetta-victim_rosetta-victim-policy.
          Remove or scope down that statement.
     - iam:SimulatePrincipalPolicy (authoritative)

One cause. 96 percent. The exact policy named. The exact statement named. Evidence attached.

That collapse, from five-way uncertainty to one cited answer, is the entire point of this tool. Everything below is how it works and how you can run it against your own account.

Why the guess is not the tool's fault

Here is the uncomfortable truth. From the error text alone, nobody can do better than guessing. Not a model, not a regex, not me.

The string AccessDenied when calling GetObject is genuinely ambiguous. It can mean any of these, and the message often will not tell you which one:

No Allow in the identity policy
An explicit Deny in the identity policy
An explicit Deny in a Service Control Policy above the account
A permissions boundary that excludes the action
A resource policy on the bucket that omits the principal
A missing KMS grant on an encrypted object

Six very different fixes. One identical error message. The words on your screen simply do not carry enough information to choose between them.

Every "paste it into AI" tool tops out right here. It reads the words and pattern matches to the most common causes. That is exactly what my text only mode does, and it is honest about it. Five causes, flat confidence, a note telling you it is guessing.

The fix is not a smarter prompt. You cannot prompt your way to information that is not in the input. The fix is to stop reading the words and start querying the account that produced them.

The turn: SimulatePrincipalPolicy

AWS has one API that answers the question authoritatively: iam:SimulatePrincipalPolicy. You hand it a principal, an action, and a resource, and it evaluates the full policy chain the way IAM itself does at request time. It returns an allow or deny decision, and crucially, the exact statement that made the call, plus separate signals for whether an SCP or a boundary was the thing that blocked you.

This is the heart of Rosetta's enrichment. Here is the real code:

def simulate(session, principal_arn, action, resource):
    if not (principal_arn and action):
        return []
    iam = session.client("iam")
    resp = iam.simulate_principal_policy(
        PolicySourceArn=principal_arn,
        ActionNames=[action],
        ResourceArns=[resource] if resource and resource.startswith("arn:aws") else ["*"],
    )

    out = []
    for r in resp.get("EvaluationResults", []):
        detail = {
            "action": r.get("EvalActionName"),
            "decision": r.get("EvalDecision"),
            "allowed_by_organizations": r.get("OrganizationsDecisionDetail", {}).get("AllowedByOrganizations"),
            "allowed_by_boundary": r.get("PermissionsBoundaryDecisionDetail", {}).get("AllowedByPermissionsBoundary"),
            "matched_statements": [m.get("SourcePolicyId") for m in r.get("MatchedStatements", [])],
        }
        out.append(Evidence(
            kind="simulate",
            source="iam:SimulatePrincipalPolicy",
            detail=json.dumps(detail),
            authoritative=True,
        ))
    return out

Notice what comes back. Not just explicitDeny, but allowed_by_organizations and allowed_by_boundary as separate booleans. That is how Rosetta distinguishes an SCP deny from a boundary deny from a plain policy deny, three completely different fixes that share one error message. The API already knows all of this. My job was just to read it correctly, which, spoiler, I did not do on the first try.

The three fields work like a decision tree. If allowed_by_organizations is false, an SCP is your problem and no amount of editing the user's policy will help. If allowed_by_boundary is false, the permissions boundary is clamping you. Otherwise the decision field tells you whether it was an explicit deny or simply the absence of any allow. Rosetta turns each of those into a distinct, correctly worded cause.

The architecture, in four moves

Rosetta is a pipeline. Each stage is a small pure function, which meant I could test the whole thing without touching AWS until the very end.

parse    error string -> structured fields
enrich   assume read-only role, run simulate, fetch resource policy, match CloudTrail
reason   optional Bedrock pass over the evidence
rank     authoritative evidence beats model guesses beats corpus fallback

The design decision I am happiest with is the hybrid model. Text only mode needs zero credentials and runs anywhere, which makes the tool useful even when you are looking at someone else's pasted error with no access to their account. Enrichment needs exactly one thing: a read only role you let Rosetta assume. Nothing else. Your credentials never leave your machine, and the role only ever has read permissions.

The parser is boring regex, and that is fine. Boring is reliable:

_CODE = re.compile(r"An error occurred \(([A-Za-z0-9.]+)\)")
_OP = re.compile(r"when calling the (\w+) operation")
_PRINCIPAL = re.compile(r"(?:User|Role):\s*(arn:aws[^\s]+)")
_ACTION = re.compile(r"perform:\s*([a-zA-Z0-9]+:[A-Za-z0-9*]+)")
_RESOURCE = re.compile(r"on resource:\s*(arn:aws[^\s]+|\*|[^\s]+)")

Feed it a botocore error string, get back a clean struct with the service, action, principal, and resource pulled out. That struct is the contract every downstream stage depends on. If parsing gets the principal ARN wrong, the simulate targets the wrong identity, so this unglamorous regex is load bearing.

Setting up the read only role

The authoritative mode needs a role Rosetta can assume, and that role needs exactly the read permissions the enrichment uses and nothing more. Here is the policy, straight from the repo at infra/readonly-role-policy.json:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "RosettaSimulate",
      "Effect": "Allow",
      "Action": [
        "iam:SimulatePrincipalPolicy",
        "iam:GetPolicy",
        "iam:GetPolicyVersion",
        "iam:ListAttachedRolePolicies",
        "iam:ListAttachedUserPolicies"
      ],
      "Resource": "*"
    },
    {
      "Sid": "RosettaResourcePolicies",
      "Effect": "Allow",
      "Action": ["s3:GetBucketPolicy", "kms:GetKeyPolicy"],
      "Resource": "*"
    },
    {
      "Sid": "RosettaCloudTrail",
      "Effect": "Allow",
      "Action": ["cloudtrail:LookupEvents"],
      "Resource": "*"
    }
  ]
}

Read only, top to bottom. Nothing here can change your account. The role's trust policy points at whichever principal runs Rosetta, your CLI identity locally or a Lambda execution role if you deploy it. Once the role exists, you pass its ARN and Rosetta assumes it with STS, does its read only work, and throws the session away.

Reproduce the demo yourself

I did not want to test enrichment against my real infrastructure, so I built a deliberately broken IAM setup as a throwaway CloudFormation stack. One stack up, one stack down, no leftover mess to hunt for later. The core of it is a victim user carrying an Allow on S3 reads, then an explicit Deny on one secret bucket:

VictimUser:
  Type: AWS::IAM::User
  Properties:
    UserName: rosetta-victim
    Policies:
      - PolicyName: rosetta-victim-policy
        PolicyDocument:
          Version: "2012-10-17"
          Statement:
            - Sid: AllowS3Read
              Effect: Allow
              Action: s3:GetObject
              Resource: "*"
            - Sid: DenySecretBucket
              Effect: Deny
              Action: s3:GetObject
              Resource: arn:aws:s3:::rosetta-demo-secret/*

That explicit Deny is the exact thing the money shot catches. Deploy the stack, run Rosetta against it, get your own before and after, then delete the stack and everything vanishes in one action. The full template is in the repo under scenario/.

The one trick worth stealing

If you take one technical thing from this post, take this.

I wanted Rosetta to find the actual failed call in CloudTrail, so it could show you the real request that got denied, with the source IP and the resources involved. My first instinct was to look it up by request ID. That is the natural key, right? Every AWS error has one.

Wrong. CloudTrail's LookupEvents does not accept RequestID as a lookup attribute. The allowed keys are things like EventName, Username, and EventSource. Request ID is simply not on the list, and if you pass it you get a ValidationException.

So the trick is: look up by EventName, pull a small batch of recent events, then filter on the requestID field inside the returned event JSON. The ID is in the payload, just not in the index you can query on.

def cloudtrail_match(session, operation, principal_arn, request_id):
    ct = session.client("cloudtrail")
    resp = ct.lookup_events(
        LookupAttributes=[{"AttributeKey": "EventName", "AttributeValue": operation}],
        MaxResults=20,
    )
    for ev in resp.get("Events", []):
        record = json.loads(ev.get("CloudTrailEvent", "{}"))
        if request_id and record.get("requestID") != request_id:
            continue
        # matched the exact failing call
        ...

One more honest detail. CloudTrail has a delivery delay, so a very fresh error may not have a matching event yet. Rosetta says so rather than inventing one. Pretending you found the event when you did not is worse than admitting the delay.

The bug I found by testing my own tool

Here is the part every reviewer actually cares about, because it is the part most posts quietly leave out.

I pointed Rosetta at my broken stack. The enrichment fired, the simulate ran, and Rosetta confidently told me:

No policy allows this action (implicit deny)

Except it was not an implicit deny. It was a screaming explicit Deny, sitting right there in the policy I had written thirty seconds earlier. My ranking code was wrong.

The cause was dumb and instructive. My rank.py was checking the decision but had brittle branch ordering, and worse, it was hiding the raw simulate decision, so I could not even see what the API had actually returned. The tool was making an authoritative claim while concealing the evidence for it. That is the most dangerous kind of bug, the confident one.

I rewrote the ranking to read the decision properly and to always surface the real EvalDecision and matched statement. The branch that matters now reads:

if decision == "explicitDeny":
    return Cause(
        "Explicit Deny in an attached policy",
        0.96,
        f"An explicit Deny matched: {label}. Remove or scope down that statement.",
        [evidence],
    )

To confirm the fix, I added a --json flag that dumps the raw pipeline output, every piece of evidence included. Running it showed the truth the pretty output had been hiding:

"decision": "explicitDeny",
"matched_statements": ["user_rosetta-victim_rosetta-victim-policy"]

The API had been telling me explicitDeny the whole time. My code was just not listening. The lesson stuck with me: when your tool makes an authoritative claim, make it show its work, or you will end up trusting a bug with a confidence bar next to it.

The wall: Bedrock, Marketplace, and a payment instrument

I wanted an optional LLM layer on top, using Bedrock to phrase the causes more naturally and catch anything my rules missed. This is where AWS taught me something genuinely new.

I picked a Claude model, set my BEDROCK_MODEL_ID, and invoked. Access denied. But not an IAM denial, a Marketplace one:

Model access is denied due to INVALID_PAYMENT_INSTRUMENT:
A valid payment instrument must be provided.

Here is what is going on, because it took me a while to piece together. AWS retired the old manual model access page. Models now auto enable on first invocation, which sounds convenient. But that auto enable quietly creates an AWS Marketplace subscription behind the scenes, and that subscription needs a valid payment method to complete. My account did not have one on file. AWS even emailed me a Marketplace agreement that started and expired at the exact same second, because the billing failed instantly.

This is undocumented tribal knowledge, so let me save you the afternoon I lost: if you get INVALID_PAYMENT_INSTRUMENT from Bedrock, it is not your IAM policy, it is not your region, and it is not the model ID. It is billing. Add a payment method, or use a model that does not route through Marketplace.

I chose to ship without the reasoning layer entirely, and here is why it does not matter one bit. The authoritative answer, the 96 percent explicit deny with the matched statement, comes from SimulatePrincipalPolicy, not from any model. The LLM was only ever going to reword what the deterministic check already knew for certain. Reasoning is a garnish. The simulate is the meal.

So I made the model layer truly optional. No BEDROCK_MODEL_ID set means Rosetta runs parse plus enrich and says nothing at all about missing models:

if os.environ.get("BEDROCK_MODEL_ID"):
    try:
        model_output = reason(parsed, evidence, region=region)
    except Exception as e:
        notes.append(f"Model reasoning unavailable: {e}")
        model_output = {"causes": []}
else:
    model_output = {"causes": []}

Graceful failure is a feature

One more thing the build taught me. After I tore down the test stack, I ran the enriched command again out of habit. Rosetta could no longer assume the role, because the role was gone with the stack. It did not crash. It fell back to text only and told me exactly why:

note: Enrichment skipped: AccessDenied when calling the AssumeRole operation:
      User aws-rosetta is not authorized to perform sts:AssumeRole on
      resource role/rosetta-readonly

That is precisely the behavior you want from a tool that reaches into live accounts. When the privileged path is unavailable, degrade to the safe path and explain the drop in plain language. Never pretend, never crash, never leave the user wondering whether the answer they got was the good one or the fallback. The mode badge at the top, [enriched] or [text-only], tells you which brain answered.

Four ways to run it

The whole tool is one engine with four front doors, so you can reach for whichever fits the moment:

CLI: aws-decode "<error>" is the primary interface, and the one I use daily.
Web: web/index.html is a paste box in the same off white and orange I use everywhere. Run python web/local_server.py for a local backend and it works with zero AWS setup.
API: api/handler.py is a Lambda Function URL handler, ready to deploy with the SAM template in infra/.
Library: from rosetta import resolve if you want to fold it into something bigger.

All four call the same resolve function, so they cannot drift apart. Fix a bug in the engine and every surface gets it.

A note on cost and safety

Text only mode is free. It makes no AWS calls at all. Enrichment makes a handful of read only API calls per error, SimulatePrincipalPolicy, maybe a GetBucketPolicy, a LookupEvents, all of which are effectively free at any sane volume. The optional Bedrock layer is the only thing that costs money, and it is per token and tiny, and it is off by default.

On safety, the important line is that Rosetta reads your private IAM state, so it is designed to run against your own account with your own role, on your own machine. That is why the product is the repo, not a hosted service. A public endpoint that assumed roles in my account would be a gift to strangers. You run it, you own the credentials, they never leave your laptop.

Try it yourself

Rosetta is open source, MIT licensed, and runs against your own account with your own read only role.

Repo: https://github.com/mursalfk/aws-rosetta

Text only mode needs nothing but Python:

git clone https://github.com/mursalfk/aws-rosetta.git
cd aws-rosetta
python -m venv .venv
source .venv/bin/activate        # Windows: source .venv/Scripts/activate
pip install -e .

aws-decode "An error occurred (AccessDenied) when calling the GetObject operation: ..."

For the authoritative mode, create a read only role from infra/readonly-role-policy.json, trust the principal that runs Rosetta, and pass it with --role-arn. The scenario stack under scenario/ lets you reproduce the exact before and after from this post.

What I actually learned

Four things stuck with me.

Text alone can only ever guess, because the same error genuinely has many causes. The account holds the answer, and one API, SimulatePrincipalPolicy, will hand it to you authoritatively if you bother to ask.

Make your tools show their work. My ranking bug survived exactly until I forced the raw decision into the open with a --json flag. An authoritative claim you cannot inspect is a bug waiting to be trusted.

Degrade loudly, not silently. The moment the privileged path disappears, say so and fall back. A tool that quietly gives you a worse answer without telling you is worse than one that crashes.

And when AWS says INVALID_PAYMENT_INSTRUMENT, go check your billing, not your policies. You are welcome.

Ciao! 👋

My Robot Accountant Got an Eviction Notice. So We Moved: Migrating from Bedrock Agents Classic to AgentCore

Mursal Furqan Kumbhar — Fri, 31 Jul 2026 07:40:04 +0000

Ciao Amici 👋

Grab your chai, because the robot accountant is back, and this time we have a moving-day story. If you read my last article, you know I built "The Accountant," a Bedrock Agent that reads my AWS bill every morning, emails me a color-coded report, and once famously reported its own existence to me as suspicious activity.

That article did well. It did so well, in fact, that it reached exactly the right person to tell me my beloved architecture was standing on borrowed ground. I remember the moment precisely: phone buzzing on the desk, a notification from Dev.to, me expecting another nice comment about the day 16 hallucination. Instead, a community manager from AWS, polite as anything: Bedrock Agents is heading into maintenance mode, consider rewriting this with AgentCore. A link to the announcement sat under it like a small landmine.

My stomach did the same thing it did the day I found the idle GPU endpoint. So I did what any of us would do: I read the announcement twice, once fast and once slowly, made more chai, and moved the whole thing. This is the complete story of that migration, box by box, with every line of code, what got easier, what got weirder, and what it cost. By the end, you will be able to move your own Classic agent in an afternoon.

The eviction notice, in plain language

First, let us be precise about what is actually happening, because "maintenance mode" causes more panic than it should. I know, because I panicked for about ten minutes before actually reading the page.

Amazon Bedrock Agents, the service my first article was built on, is now officially called Amazon Bedrock Agents Classic, and as of July 30, 2026, it is closed to new customers. If your AWS account has never called CreateAgent, that API will simply refuse you with an AccessDeniedException after that date.

What it does not mean:

Existing agents do not stop working. There is no announced end-of-life date.
Nobody is deleting your agents or your action groups.
Your bills do not change.

What it does mean, and this is the part that matters:

The service is frozen. No new features, and critically, a frozen model catalog. The models available to Classic agents today are the models it will have forever. Every new model generation from here on will skip Classic entirely.
Any tutorial that says "go to the Bedrock console and create an agent" is now a historical document for anyone with a fresh account. Including, painfully, mine.

That second bullet is the real eviction notice. My Accountant would have kept running for years. But an agent that can never get a better brain is an agent on a slowly sinking ship, and an article that new readers cannot follow is not much of an article. Sitting there with my cooling chai, I had to admit it: the community manager was right. Time to pack.

The destination AWS points everyone toward is Amazon Bedrock AgentCore, and after living in it for a few weeks, I can report that the move is less scary than it sounds, and in a couple of ways it is a genuine upgrade.

Meet the new neighbourhood: AgentCore in ninety seconds

AgentCore is not "Bedrock Agents 2." It is a different philosophy. Classic was a managed agent: you configured an agent in the console, AWS owned the orchestration loop, and your code only existed at the edges, inside action group Lambdas. AgentCore flips this. Your agent is your code, written in any framework you like (Strands Agents, LangGraph, CrewAI, plain Python), and AgentCore provides the production infrastructure around it:

Runtime: a serverless home that runs your agent in an isolated microVM per session, scales it, and charges you only for what you use.
Gateway: turns Lambdas and APIs into MCP tools your agent can call.
Memory: managed short-term and long-term memory.
Identity: OAuth and credential handling for agents that act on someone's behalf.
Observability: tracing for every step and tool call.

For The Accountant, a solitary creature with five tools and no users, I needed exactly one of these: Runtime. That is the pleasant surprise of AgentCore for small projects. It is a menu, not a prix fixe. You take the piece you need and ignore the rest until you grow into it.

The framework I chose for the agent itself is Strands Agents, AWS's open-source agent SDK, mostly because it is the path of least resistance on AgentCore and its tool model is, as you are about to see, almost comically convenient for this migration.

The moving map

Before touching code, I sat down one evening with a notebook, the old architecture diagram on one screen and the AgentCore docs on the other, and mapped every piece of the old build to its new home. I suggest you do the same for your agent; the whole migration falls out of this one table.

Bedrock Agents Classic	AgentCore + Strands	Effort
Agent + instructions in the console	Strands `Agent` with a system prompt, in code	Copy and paste
Action group Lambda + function schemas	Plain Python functions with `@tool`, same process	Delete code, mostly
`InvokeAgent` API in trigger Lambda	`InvokeAgentRuntime` API	Twenty lines
Console configuration	`agentcore configure` + `agentcore launch` CLI	New, pleasant
Bedrock trace console	CloudWatch + AgentCore Observability	Different, fine
IAM read-only policy	Unchanged	Zero
SES email delivery	Unchanged	Zero
EventBridge schedule	Unchanged	Zero

Read that Effort column again. I did, three times, waiting for the catch. The two things I was most protective of, the read-only IAM philosophy and the email delivery my mornings now depend on, moved without a single edit. The heart of the migration is really just one move: the action group Lambda dissolves, and its functions walk across the street to live inside the agent itself.

Packing the boxes: the new agent

Here is the entire agent, one file, main.py. If you read the first article, you will recognize every boto3 call, because they are the same functions that used to live in the action group Lambda. The difference: they are now decorated with @tool and run in the same process as the agent's reasoning. No Lambda packaging, no function schemas to declare in a console, no messageVersion response envelopes. The docstring is the schema; Strands hands it to the model as the tool description.

from bedrock_agentcore.runtime import BedrockAgentCoreApp
from strands import Agent, tool
import boto3
import json
from datetime import datetime, timedelta

ce = boto3.client("ce")
sm = boto3.client("sagemaker")
ec2 = boto3.client("ec2")

@tool
def get_daily_costs(days: int = 14) -> str:
    """Fetch daily AWS costs grouped by service for the last N days,
    in a single Cost Explorer call. Returns JSON."""
    end = datetime.utcnow().date()
    start = end - timedelta(days=days)
    resp = ce.get_cost_and_usage(
        TimePeriod={"Start": str(start), "End": str(end)},
        Granularity="DAILY",
        Metrics=["UnblendedCost"],
        GroupBy=[{"Type": "DIMENSION", "Key": "SERVICE"}],
    )
    out = []
    for day in resp["ResultsByTime"]:
        for grp in day["Groups"]:
            amount = float(grp["Metrics"]["UnblendedCost"]["Amount"])
            if amount > 0.01:
                out.append({
                    "date": day["TimePeriod"]["Start"],
                    "service": grp["Keys"][0],
                    "usd": round(amount, 2),
                })
    return json.dumps({"found": len(out), "items": out})

@tool
def get_forecast() -> str:
    """Fetch the Cost Explorer forecast for the remainder of the
    current month. Returns JSON."""
    start = datetime.utcnow().date() + timedelta(days=1)
    end = start.replace(day=28) + timedelta(days=4)
    end = end - timedelta(days=end.day)
    resp = ce.get_cost_forecast(
        TimePeriod={"Start": str(start), "End": str(end)},
        Metric="UNBLENDED_COST",
        Granularity="MONTHLY",
    )
    return json.dumps(
        {"projected_remaining_usd": round(float(resp["Total"]["Amount"]), 2)}
    )

@tool
def list_sagemaker_endpoints() -> str:
    """List all SageMaker inference endpoints with status and
    creation time. Returns JSON with an explicit count."""
    eps = sm.list_endpoints()["Endpoints"]
    items = [
        {
            "name": e["EndpointName"],
            "status": e["EndpointStatus"],
            "created": e["CreationTime"].isoformat(),
        }
        for e in eps
    ]
    return json.dumps({"found": len(items), "items": items})

@tool
def find_unattached_volumes() -> str:
    """Find EBS volumes in 'available' state (attached to nothing,
    still billing). Returns JSON with an explicit count."""
    vols = ec2.describe_volumes(
        Filters=[{"Name": "status", "Values": ["available"]}]
    )["Volumes"]
    items = [
        {"id": v["VolumeId"], "size_gb": v["Size"], "type": v["VolumeType"]}
        for v in vols
    ]
    return json.dumps({"found": len(items), "items": items})

@tool
def find_unattached_eips() -> str:
    """Find Elastic IPs not associated with any instance.
    Returns JSON with an explicit count."""
    addrs = ec2.describe_addresses()["Addresses"]
    items = [{"ip": a["PublicIp"]} for a in addrs
             if "AssociationId" not in a]
    return json.dumps({"found": len(items), "items": items})

SYSTEM_PROMPT = """You are a cautious FinOps analyst reviewing a
personal AWS account. The owner is a solo developer running ML
experiments, a portfolio site, and occasional demos. Typical
monthly spend is 60 to 120 USD.

Every day you will:
1. Fetch the last 14 days of daily costs grouped by service,
   in a single call.
2. Compare today's trajectory against the trailing average.
3. Check idle-prone resources: SageMaker endpoints, unattached
   EBS volumes and Elastic IPs.
4. Fetch the month-end forecast.
5. Report findings in this exact structure:
   STATUS: green / yellow / red
   HEADLINE: one sentence
   ANOMALIES: bullet list, or "none"
   RECOMMENDATIONS: bullet list with estimated monthly savings
   as a range for each, or "none"
   FORECAST: projected month-end total
   WATCHER OVERHEAD: today's estimated cost of running this
   analysis

Rules:
- Never recommend an action you cannot support with data you
  fetched in this session. If a tool returned found: 0, there
  is nothing to recommend about those items.
- If you are unsure whether a resource is intentional, say so
  and ask. Do not assume.
- Distinguish between "spend increased" and "spend is anomalous".
  A planned training run is not an anomaly.
- State every savings estimate as a range, never a point figure.
- Your own model and runtime usage is expected. Report it under
  WATCHER OVERHEAD, never under ANOMALIES.
"""

app = BedrockAgentCoreApp()

agent = Agent(
    system_prompt=SYSTEM_PROMPT,
    tools=[
        get_daily_costs,
        get_forecast,
        list_sagemaker_endpoints,
        find_unattached_volumes,
        find_unattached_eips,
    ],
)

@app.entrypoint
def invoke(payload):
    prompt = payload.get("prompt", "Run the daily billing review.")
    result = agent(prompt)
    return {"report": str(result.message)}

if __name__ == "__main__":
    app.run()

Notice three things while the tape gun is still warm.

First, the system prompt is my old Classic instruction prompt, nearly verbatim. All five versions of scar tissue from the first experiment, the ranges rule, the "found: 0 means nothing to recommend" rule, the watcher-overhead rule born the day the agent reported itself, all of it transferred untouched. Prompts are luggage. They fly free.

Second, Chekhov's empty list is now structural. In the first article, the day 16 hallucination (the agent inventing an idle RDS instance from an empty response) taught me that tools must return explicit counts. In the new build, every tool returns {"found": N, "items": [...]} by construction. Some lessons you write in a prompt. Better lessons you write in the return type.

Third, the @app.entrypoint wrapper is the entire integration with AgentCore. Three lines. BedrockAgentCoreApp handles the HTTP server, the session plumbing, and the container contract so that this file runs identically on my laptop and in the cloud.

The moving truck: deploying with the starter toolkit

This is where AgentCore genuinely out-classes Classic. With Classic, "development" meant clicking through the console, saving, preparing, testing in the chat panel, and squinting at traces. With AgentCore, my agent is a Python file I can run locally like any other Python file:

pip install bedrock-agentcore strands-agents bedrock-agentcore-starter-toolkit

python main.py
# in another terminal:
curl -X POST http://localhost:8080/invocations \
  -H "Content-Type: application/json" \
  -d '{"prompt": "Run the daily billing review."}'

I want you to picture this moment, because it is the moment the migration stopped feeling like a chore. Two terminals side by side on my screen, close to midnight. In the left one, my agent, running as an ordinary Python process. In the right one, a curl command. I pressed enter, and thirty seconds later I was watching my agent call Cost Explorer from my own machine, with my own credentials, printing its reasoning line by line as it thought. The development loop went from minutes per iteration to seconds. This alone justified the move for me emotionally, whatever the maintenance-mode situation.

When it behaves locally, deployment is two commands:

agentcore configure -e main.py
agentcore launch

configure asks a few questions, writes a .bedrock_agentcore.yaml, and, helpfully, creates the IAM execution role for you. launch packages the agent and hosts it on AgentCore Runtime, returning an agent runtime ARN. One important errand before celebrating: go attach the read-only billing policy from the first article to that execution role, since the agent's boto3 calls now run under it instead of under an action group Lambda's role. Same policy, new badge holder. The zero-write-permissions philosophy survives the move fully intact:

A quick sanity check from the CLI:

agentcore invoke '{"prompt": "Run the daily billing review."}'

And there it was: STATUS: green, in a brand-new house, first try.

Well. Second try. The first try taught me the quirk below.

The one box that rattled: sessions

Every migration has one. Mine was the runtime session ID.

InvokeAgentRuntime requires a runtimeSessionId, and it must be at least 33 characters long. My first invocation passed the same short daily ID string the old trigger Lambda used, and the API rejected it with a validation error that took me an embarrassing amount of squinting to parse. I read it four times, convinced the problem was my ARN, my region, my IAM, anything but the humble session string. It was the session string. The fix is a UUID4, which is 36 characters and therefore sails through.

There is also a conceptual shift hiding in that parameter. In Classic, a session was a conversational thread the service tracked for you. In AgentCore Runtime, the session maps to an isolated microVM; same ID means same warm sandbox, new ID means a fresh one. For The Accountant this is a feature: a fresh UUID every morning gives me the same blank-slate reproducibility I insisted on in the first experiment, now enforced by actual infrastructure isolation rather than by my own discipline.

Here is the migrated trigger Lambda, the only other code that changed:

import boto3
import json
import os
import uuid

agent_core = boto3.client("bedrock-agentcore")
ses = boto3.client("ses")

COLORS = {"green": "#2eb67d", "yellow": "#ecb22e", "red": "#e01e5a"}

def lambda_handler(event, context):
    payload = json.dumps({"prompt": "Run the daily billing review."}).encode()

    resp = agent_core.invoke_agent_runtime(
        agentRuntimeArn=os.environ["AGENT_RUNTIME_ARN"],
        runtimeSessionId=str(uuid.uuid4()),  # 33+ chars required
        qualifier="DEFAULT",
        payload=payload,
    )

    chunks = []
    for chunk in resp.get("response", []):
        chunks.append(chunk.decode("utf-8"))
    report = json.loads("".join(chunks)).get("report", "")

    status = "yellow"
    for line in report.splitlines():
        if line.strip().upper().startswith("STATUS:"):
            status = line.split(":", 1)[1].strip().lower()
            break

    html = f"""
    <div style="font-family:Menlo,Consolas,monospace;max-width:640px">
      <div style="background:{COLORS.get(status, '#ecb22e')};
                  color:#fff;padding:10px 16px;border-radius:6px;
                  font-weight:bold;font-size:16px">
        The Accountant &middot; {status.upper()}
      </div>
      <pre style="white-space:pre-wrap;font-size:13px;
                  line-height:1.55;padding:8px 4px">{report}</pre>
    </div>
    """

    ses.send_email(
        Source=os.environ["FROM_ADDRESS"],
        Destination={"ToAddresses": [os.environ["TO_ADDRESS"]]},
        Message={
            "Subject": {
                "Data": f"The Accountant · {status.upper()} · daily review"
            },
            "Body": {"Html": {"Data": html}, "Text": {"Data": report}},
        },
    )
    return {"status": status}

Swap bedrock:InvokeAgent for bedrock-agentcore:InvokeAgentRuntime in this Lambda's role, point the EventBridge schedule at it, and the morning ritual continues without missing a day. The SES block is character-for-character the one from the first article. My inbox never even noticed the agent changed houses.

The shadow week

I did not flip a switch and hope. For seven days I ran both accountants in parallel, old and new, each reviewing the same account every morning, reports arriving side by side in my inbox. Two subject lines, same status word, a minute apart. There was something quietly funny about it, like watching an employee train their own replacement without knowing it. I recommend this to anyone migrating an agent that they actually rely on; it costs a few dollars and buys real confidence.

Results of the parity check, honestly reported:

Six of seven days: functionally identical findings. Same status, same anomalies or lack of them, forecasts within a dollar or two of each other. Different sentence rhythms, since the orchestration loops differ, but the same accountant's judgment.
One divergence, and it favored the new build. Mid-week I terminated a test instance and its Elastic IP sat unattached for a day. Both agents caught it. The v2 agent, though, reported it as "found: 1 unattached Elastic IP" with the address, quoting the tool's count directly in its reasoning. Watching the explicit-count design pay off in the wild, in the exact failure category that produced the day 16 hallucination last time, was the single most satisfying moment of the migration. I may have said "yes" out loud to an empty room.
Zero hallucinated resources on either side all week. The scar tissue holds.

On day eight I deleted the Classic agent. A small pang, honestly, my cursor hovering over the confirmation button a beat longer than necessary. It found me fifty-five dollars a month and once accused itself of fraud. But its model catalog was frozen in time, and mine is not. Delete.

The receipts, moving-day edition

What did the move cost in time and money?

Time: one honest afternoon. About four hours end to end, and half of that was me reading AgentCore docs out of curiosity rather than necessity. The mechanical work, converting the Lambda functions to @tool functions, deploying, updating the trigger, was maybe ninety minutes. Your first migration will be slower than your second; the second is nearly mechanical.

Money: a rounding error.

Component	v1 · Classic	v2 · AgentCore
Model tokens (Bedrock)	~$3.62	~$3.55
Cost Explorer API calls	~$1.05	~$1.05
Agent runtime compute	~$0.08 (Lambda)	~$0.31 (Runtime, per-second)
Lambda, EventBridge, SES	~$0.12	~$0.11
Monthly total	~$4.87	~$5.02

AgentCore Runtime bills for consumed CPU and memory per second while the agent is actually working, and a sixty-second daily run is simply not much of either. Fifteen cents a month is what my accountant's new apartment costs over the old one. The dominant cost remains, as it always was, the model tokens, and those are identical either way. The economics of the whole idea, five dollars to find fifty-five, are untouched by the migration.

What got better, what got weirder

The honest ledger, because a migration story that reads like a brochure is worthless.

Genuinely better:

The local dev loop. Running the agent as a plain Python process and curling it on localhost changed how it feels to work on this project. Prompt iteration that used to mean console round-trips now happens at the speed of saving a file.
Tools as functions. The action group ceremony (Lambda packaging, function schemas, response envelopes) is gone. A docstring and a decorator. I deleted more code than I wrote in this migration, which is my favorite genre of engineering.
The frozen-catalog problem is solved by design. The model is a string in my code now. When a better one ships, I change one line and relaunch.
Structural guardrails. Explicit counts in tool returns, enforced by construction rather than by prompt discipline.

Weirder, or at least new:

The 33-character session ID rule. Documented, but it will get you once. It got me at 11:40 PM.
A new mental model. Classic made the agent feel like an AWS resource you configure. AgentCore makes it a program you own. I prefer the second, but it is a real shift, and teams who liked the console-first workflow will feel it.
One more CLI and one more hidden YAML (.bedrock_agentcore.yaml) in your life. Pleasant, but it is another thing.
Observability lives in a different place. The Classic trace console was crude but one click away. Now I read traces through CloudWatch and AgentCore's observability tooling, which is more powerful and slightly more assembly-required.

And unchanged, by design: the read-only IAM policy, the SES delivery, the EventBridge schedule, the system prompt, and the entire philosophy. The agent looks and complains. The human acts. No maintenance-mode announcement will ever change that rule.

Should you move yours?

If you have a Bedrock Agents Classic agent running today, here is my honest triage:

Move soon if your agent is something you are still actively improving, or if your content, docs, or team onboarding depend on others being able to recreate it. New accounts cannot build Classic agents after July 30, 2026, and the frozen model catalog means your agent's ceiling is already set. The migration is an afternoon; the mapping table above is most of the thinking.

Move eventually if your agent is stable and internal. Nothing breaks tomorrow. But put it on the roadmap, because every quarter you wait, the gap between the frozen catalog and the current model frontier widens, and that gap is your agent quietly getting relatively dumber.

And if you are starting fresh: there is no decision to make. Start on AgentCore, pick a framework, and enjoy the local dev loop from day one. The first article's ideas (read-only tools, explicit counts, ranges not points, teach the agent your normal) all apply verbatim; only the plumbing in it is now historical.

Wrapping up

So, guys, that is the story of moving day. It started with a comment on my last article that briefly ruined my evening and ended, one afternoon of work later, with the same accountant in a better house: the same paranoid morning reports, the same read-only leash, the same five-dollar salary, but now with a dev loop I actually enjoy, tools that are just Python functions, and a brain that can be upgraded with a one-line change instead of being frozen in 2026 forever.

I owe genuine thanks to the community manager who nudged me, because "your architecture is deprecated" is a gift when it arrives early enough to do something about it, and I would much rather write a migration story than a post-mortem. If you have a Classic agent of your own, take the moving map, block out an afternoon, and give your robot its new keys; then come tell me how it went, because the follow-up to this article will feature the best migration stories I hear. Find me on Dev.to or LinkedIn. Until then: keep your permissions read-only, your tool counts explicit, and your model catalogs unfrozen.

Ciao! 👋

I Let a Bedrock Agent Watch My AWS Bill for 30 Days. Here Is Everything It Caught, Missed, and Made Up

Mursal Furqan Kumbhar — Sun, 05 Jul 2026 20:07:54 +0000

Ciao Amici 👋

Pull up a chair, because I have a story for you. It involves a forgotten GPU endpoint, a robot accountant I built over two evenings, a NAT gateway that quietly ate my wallet, and one genuinely absurd moment where my own creation reported itself to me as suspicious activity. For thirty days I let an Amazon Bedrock Agent read my AWS bill every morning and tell me what it thought, and by the end of this story you will have every line of code, every chart, every dollar figure, and every embarrassing mistake I made along the way. Some of those mistakes were mine. Two of them, delightfully, were the agent's. Make some chai, this is a long one, and I promise it earns its length.

The morning it all started

Last November I opened my AWS billing dashboard and felt that very specific cold feeling in the stomach that every cloud engineer knows. A SageMaker real-time endpoint I had spun up for a quick demo had been running for eleven days. Eleven days of me paying for a GPU-backed instance to serve exactly zero requests.

The endpoint cost me about $38. Not catastrophic. But here is the thing: it was the third time in a year this movie had played. Before that, an Elastic IP sat unattached for a month. Before that, a NAT gateway I had completely forgotten was part of a VPC I built for a workshop. Each time, the same script: I discover the charge weeks later, feel briefly stupid, delete the thing, and promise myself I will check the console more often. I never do, because manually checking a billing console is exactly the kind of chore human brains are engineered to avoid.

And yes, I had CloudWatch billing alerts configured, like a responsible adult. But a threshold alert only tells you that you are spending money. It never tells you why, and it definitely never tells you what to do about it. An alert saying "your bill crossed $80" at 2 AM is not insight. It is anxiety with a timestamp.

So, staring at that dead endpoint, an idea landed. What if I stopped being my own billing alarm entirely? What if, every morning, someone else read the bill for me, someone tireless, slightly paranoid, and immune to the phrase "I will deal with it after the weekend"?

That someone became a Bedrock Agent. For 30 days I gave it read-only access to Cost Explorer and a handful of describe APIs, scheduled it to run every morning, and told it to behave like a cautious FinOps consultant. Every day at 9 AM Karachi time it would analyze my spend, flag anything weird, estimate savings, and email me a report with the subject line "The Accountant," because if a robot is going to lecture me about money, it should at least have a title.

This is the full story of that month. Spoiler: I am still running it. Longer spoiler: it once found a database that does not exist.

Why an agent, and not just budgets and alarms?

A colleague asked me this on day two, with a raised eyebrow that suggested he already knew part of my answer would be "because it sounded fun."

It did sound fun. But there is a real gap here, and it decides whether this whole pattern is a toy or a tool, so let me make the case properly.

AWS Budgets, billing alarms, even Cost Anomaly Detection: these are threshold and statistics tools. They compare numbers to lines. They are excellent at "did I spend more than X" and reasonable at "is today statistically unusual." What they cannot do is answer the questions that actually eat your time when a bill goes wrong:

Why did my bill go up 40% this week, in plain language?
Is this NAT gateway data processing charge normal for my workload, or a routing misconfiguration?
Which of my resources are technically running but functionally idle?
If I keep going at this rate, what does month-end look like, and which line items are driving it?
Is this spike a problem, or the training run I kicked off on purpose yesterday?

That last one matters more than people admit. Anomaly detection without context produces alerts you learn to ignore, and an ignored alert is worse than no alert. The difference between "spend increased" and "spend is wrong" is a judgment call, and judgment calls require reasoning over context. That is exactly the shape of problem LLM agents are decent at, provided you feed them structured data and keep them on a very short leash.

The short leash part is not a figure of speech. It is the entire safety story of this project, and you will see it earn its keep on day 16.

There was a quieter motivation too. I spend a lot of my community time around ML tooling, and I kept seeing agent demos that book flights or order pizza, tasks nobody actually wants automated by a nondeterministic system. Billing analysis felt like the opposite case: tedious, recurring, data-rich, and low-stakes as long as the agent can only read. If agents are useful anywhere in a personal AWS account, they should be useful here. Consider the next five thousand words the test of that hypothesis.

Meet The Accountant

Here is the whole system. It is deliberately boring, and every boring choice was on purpose.

Five pieces:

EventBridge Scheduler fires once a day at 04:00 UTC, which is 9 AM for me in Karachi. Daily is the right cadence for a personal account. Hourly would multiply the watcher's own cost by 24 for almost no extra insight, because Cost Explorer data lags a few hours anyway.
A small trigger Lambda invokes the Bedrock Agent with a fixed daily prompt, collects the streamed response, and forwards the final report.
The Bedrock Agent (I used Claude via Bedrock, though the pattern works with any agent-capable model on the platform) reads its instructions, decides which tools it needs, calls them, and reasons over the results.
An action group Lambda is the only component with AWS API access. It exposes a small menu of tools: get cost and usage, get cost forecast, list SageMaker endpoints, list EC2 instances, describe NAT gateways, find unattached EBS volumes and Elastic IPs, and describe RDS instances.
The agent's final report lands in my inbox via Amazon SES, formatted as HTML with a color-coded status banner, and impossible for me to claim I did not see. Email was a deliberate choice over chat tools: I check my inbox every morning anyway, it keeps the whole stack inside AWS, and a month of reports becomes a searchable archive for free.

Now the tattoo-worthy design decision, the one I would print inside the cover of every agent tutorial if publishers let me: the agent has zero write permissions. It cannot stop, terminate, resize, delete, or modify anything. It can only look and complain. I wanted a consultant, not an intern with root access.

Hold that thought. It becomes the hero of this story later.

Building my robot accountant

Let me walk you through the build in the order I actually did it, including the parts that wasted an evening, because the wasted evenings are where the useful knowledge lives.

Step 1: The IAM policy

Everything the action group Lambda can do fits in one tight policy. Read it, and notice what is absent:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "CostVisibility",
      "Effect": "Allow",
      "Action": [
        "ce:GetCostAndUsage",
        "ce:GetCostForecast",
        "ce:GetDimensionValues"
      ],
      "Resource": "*"
    },
    {
      "Sid": "ResourceVisibility",
      "Effect": "Allow",
      "Action": [
        "ec2:DescribeInstances",
        "ec2:DescribeVolumes",
        "ec2:DescribeNatGateways",
        "ec2:DescribeAddresses",
        "sagemaker:ListEndpoints",
        "sagemaker:DescribeEndpoint",
        "rds:DescribeDBInstances"
      ],
      "Resource": "*"
    }
  ]
}

No ec2:StopInstances. No sagemaker:DeleteEndpoint. Nothing with a verb that changes state. Whenever the agent recommended killing something, a human (me, holding coffee) did the killing after a ten-second sanity check.

One small subtlety so you do not repeat my twenty wasted minutes: ce:* actions only work against Resource: "*" because Cost Explorer is account-scoped, so do not bother trying to narrow it.

Step 2: The action group Lambda

Bedrock Agents call your Lambda with a structured event describing which tool the model wants and with what parameters. Your job is to run the right boto3 calls and hand back JSON. Here is a trimmed version of my handler; the full version just has more entries in the tool table and better logging.

import boto3
import json
from datetime import datetime, timedelta

ce = boto3.client("ce")
sm = boto3.client("sagemaker")
ec2 = boto3.client("ec2")

def get_daily_costs(days=14):
    """Cost per service per day for the last N days, one API call."""
    end = datetime.utcnow().date()
    start = end - timedelta(days=days)
    resp = ce.get_cost_and_usage(
        TimePeriod={"Start": str(start), "End": str(end)},
        Granularity="DAILY",
        Metrics=["UnblendedCost"],
        GroupBy=[{"Type": "DIMENSION", "Key": "SERVICE"}],
    )
    out = []
    for day in resp["ResultsByTime"]:
        for grp in day["Groups"]:
            amount = float(grp["Metrics"]["UnblendedCost"]["Amount"])
            if amount > 0.01:  # skip fractional-cent noise
                out.append({
                    "date": day["TimePeriod"]["Start"],
                    "service": grp["Keys"][0],
                    "usd": round(amount, 2),
                })
    return out

def get_forecast():
    start = datetime.utcnow().date() + timedelta(days=1)
    end = start.replace(day=28) + timedelta(days=4)
    end = end - timedelta(days=end.day)  # last day of month
    resp = ce.get_cost_forecast(
        TimePeriod={"Start": str(start), "End": str(end)},
        Metric="UNBLENDED_COST",
        Granularity="MONTHLY",
    )
    return {"projected_remaining_usd": round(float(resp["Total"]["Amount"]), 2)}

def list_sagemaker_endpoints():
    eps = sm.list_endpoints()["Endpoints"]
    return [
        {
            "name": e["EndpointName"],
            "status": e["EndpointStatus"],
            "created": e["CreationTime"].isoformat(),
        }
        for e in eps
    ]

def find_unattached_volumes():
    vols = ec2.describe_volumes(
        Filters=[{"Name": "status", "Values": ["available"]}]
    )["Volumes"]
    return [
        {"id": v["VolumeId"], "size_gb": v["Size"], "type": v["VolumeType"]}
        for v in vols
    ]

def find_unattached_eips():
    addrs = ec2.describe_addresses()["Addresses"]
    return [
        {"ip": a["PublicIp"]}
        for a in addrs if "AssociationId" not in a
    ]

TOOLS = {
    "get_daily_costs": lambda p: get_daily_costs(int(p.get("days", 14))),
    "get_forecast": lambda p: get_forecast(),
    "list_sagemaker_endpoints": lambda p: list_sagemaker_endpoints(),
    "find_unattached_volumes": lambda p: find_unattached_volumes(),
    "find_unattached_eips": lambda p: find_unattached_eips(),
}

def lambda_handler(event, context):
    func = event["function"]
    params = {p["name"]: p["value"] for p in event.get("parameters", [])}
    body = TOOLS.get(func, lambda p: {"error": f"unknown function {func}"})(params)

    return {
        "messageVersion": "1.0",
        "response": {
            "actionGroup": event["actionGroup"],
            "function": func,
            "functionResponse": {
                "responseBody": {"TEXT": {"body": json.dumps(body)}}
            },
        },
    }

Two lessons from this file, each purchased with part of an evening:

Cost Explorer calls cost money. Each GetCostAndUsage request bills at $0.01. That sounds like nothing until your agent decides to be thorough and makes forty calls in one session, fetching one day at a time. I added an explicit instruction telling it to fetch 14 days in a single call. After that, the system settled at six to eight Cost Explorer calls per day.

Empty results need to say they are empty. Early on, my tools returned a bare [] when nothing was found. Later I changed them to return {"found": 0, "items": []} with an explicit count. Remember this detail. It is Chekhov's empty list, and it goes off on day 16.

Step 3: The agent itself

In the Bedrock console: create an agent, pick a model, paste instructions, attach the Lambda as an action group, and define each function with its parameters. The console generates the function schema for you these days, which removed most of the friction this setup used to have.

The instruction prompt matters more than any other component in this system. Mine went through five versions, and the diffs between them tell the real story of the month. This is where it ended up, lightly condensed:

You are a cautious FinOps analyst reviewing a personal AWS account.
The owner is a solo developer running ML experiments, a portfolio
site, and occasional demos. Typical monthly spend is 60 to 120 USD.

Every day you will:
1. Fetch the last 14 days of daily costs grouped by service,
   in a single call.
2. Compare today's trajectory against the trailing average.
3. Check idle-prone resources: SageMaker endpoints, unattached
   EBS volumes and Elastic IPs, NAT gateways, and stopped EC2
   instances with attached storage.
4. Fetch the month-end forecast.
5. Report findings in this exact structure:
   STATUS: green / yellow / red
   HEADLINE: one sentence
   ANOMALIES: bullet list, or "none"
   RECOMMENDATIONS: bullet list with estimated monthly savings
   as a range for each, or "none"
   FORECAST: projected month-end total
   WATCHER OVERHEAD: today's estimated cost of running this
   analysis (Bedrock + Cost Explorer calls)

Rules:
- Never recommend an action you cannot support with data you
  fetched in this session. If a tool returned zero items, there
  is nothing to recommend about those items.
- If you are unsure whether a resource is intentional, say so
  and ask. Do not assume.
- Distinguish between "spend increased" and "spend is anomalous".
  A planned training run is not an anomaly.
- State every savings estimate as a range, never a point figure.
- Your own Bedrock and Lambda usage is expected. Report it under
  WATCHER OVERHEAD, never under ANOMALIES.

If that prompt reads like it was written by someone who got burned, good instincts. Every rule after the numbered list was added mid-experiment in response to a specific failure. Scar tissue, in prompt form. The stories behind those scars are coming.

Step 4: The trigger Lambda and email delivery

The trigger Lambda is short. It invokes the agent, collects the streamed chunks, and sends the report through Amazon SES as an HTML email with a status-colored banner at the top:

import boto3
import os

agent_rt = boto3.client("bedrock-agent-runtime")
ses = boto3.client("ses")

COLORS = {"green": "#2eb67d", "yellow": "#ecb22e", "red": "#e01e5a"}

def lambda_handler(event, context):
    resp = agent_rt.invoke_agent(
        agentId=os.environ["AGENT_ID"],
        agentAliasId=os.environ["AGENT_ALIAS_ID"],
        sessionId=context.aws_request_id,
        inputText="Run the daily billing review.",
    )

    report = ""
    for chunk in resp["completion"]:
        if "chunk" in chunk:
            report += chunk["chunk"]["bytes"].decode()

    status = "yellow"
    for line in report.splitlines():
        if line.strip().upper().startswith("STATUS:"):
            status = line.split(":", 1)[1].strip().lower()
            break

    html = f"""
    <div style="font-family:Menlo,Consolas,monospace;max-width:640px">
      <div style="background:{COLORS.get(status, '#ecb22e')};
                  color:#fff;padding:10px 16px;border-radius:6px;
                  font-weight:bold;font-size:16px">
        The Accountant &middot; {status.upper()}
      </div>
      <pre style="white-space:pre-wrap;font-size:13px;
                  line-height:1.55;padding:8px 4px">{report}</pre>
    </div>
    """

    ses.send_email(
        Source=os.environ["FROM_ADDRESS"],
        Destination={"ToAddresses": [os.environ["TO_ADDRESS"]]},
        Message={
            "Subject": {
                "Data": f"The Accountant · {status.upper()} · daily review"
            },
            "Body": {
                "Html": {"Data": html},
                "Text": {"Data": report},
            },
        },
    )
    return {"status": status}

The Lambda's execution role needs one extra permission, ses:SendEmail, and that is the only write-adjacent permission in the whole system. It writes to my inbox, nothing else.

One SES gotcha before you build: new SES accounts start in sandbox mode, meaning both sender and recipient addresses must be verified identities. For this project that is actually fine, even pleasant. Verify your own address once in the SES console, click the confirmation link, done. You never need production access because you are only ever emailing yourself. The rare case where the sandbox is a feature.

The status word in the subject line turned out to be an underrated detail. After a week my inbox itself became a dashboard: a column of GREEN subjects I could skim past, with the occasional YELLOW or RED that actually earned a tap. And unlike a chat message, the reports pile up into a searchable archive. Mid-experiment I searched "NAT" in my mail app and had the full paper trail of an incident in three seconds.

Note the sessionId: fresh one per day, so each morning is a blank slate. I considered giving it memory of previous reports and decided against it for the experiment, because I wanted every finding to be reproducible from that day's data alone. Version two revisits this, and I will tell you why at the end.

Prefer WhatsApp instead?

Two people asked me this before the experiment even ended, which around here is not surprising; WhatsApp is where life happens. It works, with one honesty caveat: the AWS-native route, AWS End User Messaging Social, requires linking a WhatsApp Business Account, and Meta's business verification is the tedious part, not the code. Once linked, the swap is a few lines:

social = boto3.client("socialmessaging")

social.send_whatsapp_message(
    originationPhoneNumberId=os.environ["WA_PHONE_ID"],
    metaApiVersion="v20.0",
    message=json.dumps({
        "messaging_product": "whatsapp",
        "to": os.environ["MY_NUMBER"],
        "type": "text",
        "text": {"body": f"The Accountant · {status.upper()}\n\n{report}"},
    }).encode(),
)

If you just want it working in ten minutes for personal use, the Twilio WhatsApp sandbox is the pragmatic shortcut: one POST request from the same Lambda, no Meta paperwork. I stayed with email because it kept the entire stack inside one AWS account and gave me the archive, but a RED alert buzzing your phone on WhatsApp is a legitimately better interrupt for genuine emergencies. The architecture does not care; delivery is the last five lines of one Lambda, so pick the channel you actually look at.

Total build time, including the prompt fiddling and the email formatting I was unreasonably fussy about: around six hours across two evenings. Then I scheduled it, went to sleep, and waited for my new employee's first day.

The diary

Now the part you actually came for. I kept notes every day of the experiment. Here are the highlights, week by week. All figures come from my account during the experiment window; your account will tell its own story, which is rather the point.

Week 1: The agent finds my dirty laundry immediately

Day 1. First report lands in my inbox at 9:02 AM, YELLOW glaring from the subject line. It flagged a SageMaker endpoint named vitb16-demo-endpoint that had been running for six days. Estimated waste: $28 to $34 per month. I knew this one. It was serving a palmprint recognition demo I had shown a colleague exactly once, weeks after building the model, and I had told myself I would take it down "after the weekend." The agent did not care about my intentions. I killed the endpoint before finishing my tea. First blood to the robot, on its first morning, in under two minutes of compute.

Day 2. Green. The report was three lines long. I felt an entirely irrational sense of approval, like passing a surprise inspection.

Day 3. Green status, but with a line I did not expect: "Note: 3 unattached gp2 volumes totaling 240 GB, approximately $24/month. These may be remnants of terminated training instances." They were exactly that. Leftovers from my thesis experiments, volumes from terminated GPU instances where DeleteOnTermination had been false. I genuinely did not know these existed. They do not appear anywhere you casually look; they just sit in the EC2 console under a tab nobody visits, quietly billing. I deleted two immediately and snapshotted the third before deleting it, in case it held a dataset I cared about. It held nothing. It never does.

Day 5. The agent noted my Route 53 hosted zone and domain charges as "consistent and expected for a portfolio site" and then never mentioned them again all month. Small thing, but this is the moment I started trusting it. It had built a sense of what normal looked like and stopped crying wolf about it. Threshold alarms cannot do this. They have no concept of "expected."

Day 7. End of week one. Total recurring waste identified: roughly $55 to $60 per month, on an account averaging around $107. Sit with that for a second. More than half my baseline spend was waste, and an LLM found essentially all of it in its first three mornings. I had audited this account myself two months earlier and missed the volumes completely. My pride recovered. Eventually.

Week 2: The NAT gateway incident

Day 8. Status yellow, trending toward alarmed. "EC2-Other charges increased significantly day-over-day."

Day 9. Status red. Headline: "EC2-Other charges increased 6x over the trailing average, driven by NAT gateway data processing."

Here is the real-life situation behind it, and it is the kind of mistake that is obvious only in hindsight. I had deployed a scraper into a private subnet for a dataset collection job, pulling Sindhi news articles for a language benchmark project I have been building on the side. Private subnet felt like the responsible default. But private subnet means every byte of outbound traffic transits the NAT gateway, and NAT gateways charge $0.045 per GB processed on top of their hourly rate. My scraper was pulling gigabytes of HTML through that gateway like the meter did not exist, because I did not know the meter existed.

The agent did not just flag the spike. It named the mechanism and proposed the fix: "If a workload in a private subnet is downloading large volumes of external data, evaluate whether it requires private networking. For S3-bound traffic, a gateway VPC endpoint processes data at no charge."

I moved the scraper to a public subnet with a security group locked to egress only, since it held no credentials or sensitive data, and added the S3 gateway endpoint for the upload path. Charges collapsed the next day. Estimated savings for the rest of the scraping job: about $19. More valuable than the money was the education; I now check the network path of every batch workload before launching it, a habit purchased for the price of two red mornings.

Day 11. Green. The agent noted the EC2-Other line had "returned to baseline following the routing change," meaning it connected this morning's data to the anomaly it flagged two days earlier, within a single fresh session, purely because the 14-day window contained both. Emergent context from a dumb design decision. I will take it.

Day 12. A quiet day, but the forecast line caught my eye: "Projected month-end: $71 to $79, versus $107 trailing-month baseline." Watching that forecast walk downward over the month became weirdly motivating, like a fitness tracker for infrastructure.

Week 3: The agent lies to me twice

Every honest story about an LLM experiment needs this chapter, so here is mine, in full, no flattering edits.

Day 16. Status yellow. The agent claimed my RDS instance was "idle with minimal connection activity" and recommended stopping it, estimated savings $12 to $15 per month. Sensible advice with one flaw: I do not have an RDS instance. Never did, in this account.

I pulled the session trace to see what happened. The describe_db_instances tool had returned an empty list, my early-version bare []. Somewhere between that empty array and the final report, the model's reasoning turned "no instances found" into "an idle instance was found." It is a fascinating failure if you stare at it long enough: the concepts of idle and absent live close together in the model's semantic space, and nothing in my tooling or prompt forced the distinction. Made worse, of course, by the confident dollar range attached to the phantom.

Two fixes went in that afternoon. The tools started returning explicit counts ({"found": 0, "items": []}, Chekhov's empty list finally fixed), and the instructions gained the rule you saw earlier: never recommend an action unsupported by data fetched in this session; if a tool returned zero items, there is nothing to recommend about those items. The phantom never returned. Eleven subsequent days, zero fabricated resources.

Day 19. It happened again, in a form I find funnier every time I think about it. The agent reported Amazon Bedrock charges as an anomaly and recommended I "review unexpected Bedrock usage in the account."

The Bedrock usage was the agent itself.

It had discovered its own operating costs in the billing data and reported itself to me as suspicious activity. I laughed out loud, screenshotted the email, and then realized this is a genuine and slightly deep problem: a watcher built on metered infrastructure will eventually observe its own footprint, and unless it has a concept of self, it will classify itself as an intruder. My fix was prosaic rather than philosophical: a WATCHER OVERHEAD line in the report format, and an instruction that its own Bedrock and Lambda charges are expected and belong there. Somewhere in that incident there is a lovely essay about self-models that I am choosing not to write today.

The generalizable lesson from both failures, and the one thing I hope you quote from this whole piece: agent reliability is mostly a tool design and prompt engineering problem, not a model problem. Every failure I saw in 30 days traced back to an ambiguity I had left lying around, either in what the tools returned or in what the instructions permitted. Fix the ambiguity and the failure mode dies. I never once had to change the model.

Day 21. Green, uneventful, and the first day the new report format ran clean end to end. WATCHER OVERHEAD: $0.17. The accountant now accounts for itself.

Week 4: Steady state

The final week was almost boring, which is the whole point of the system.

Day 24. Yellow, correctly. An Elastic IP had been sitting unattached for two days after I terminated a test instance, at the newer per-hour idle pricing. About $3.60 a month. I released it from my phone while waiting for a haircut, which is exactly the level of effort billing hygiene should require.

Day 25. Yellow, and this one made me smile: "Elevated SageMaker training charges, consistent with an intentional training workload. Not classified as anomalous." I had kicked off a fine-tuning run the night before. The agent saw a spike, checked its character (training jobs, not endpoints; bounded, not recurring), and filed it exactly as instructed under the distinction between increased and anomalous. This is the report that alarms and statistical detectors cannot write, and it is the single best argument for the whole approach.

Day 28. The agent noticed something I never asked it to look for: nonzero inter-region data transfer, on an account where everything nominally lives in one region. It suggested a cross-region S3 access, and it was right; an old script of mine was reading a bucket in another region, a leftover from an experiment two years ago. Savings of maybe a dollar a month. The dollar is irrelevant. The fact that it went looking is not.

Day 30. Green. Final forecast: $76 to $78. Actual month-end: $76.10. I bought myself a decent lunch with a fraction of the difference and considered it a performance bonus paid to the wrong party.

The receipts

The month in one table. "Baseline" is my average across the previous three months.

Metric	Baseline (avg)	Experiment month	Change
Total monthly spend	$107.40	$76.10	-29%
Idle/waste spend (est.)	~$58	~$6	-90%
Billing surprises	1 to 2 per month	0	gone
Time I spent on billing	~45 min/month	~10 min/month	-78%
Watcher overhead	$0	$4.87	new cost

And the full recommendations log, verdicts included:

Day	Finding	Est. monthly impact	Verdict
1	Idle SageMaker endpoint	$28 to $34	Real. Killed it.
3	3 unattached EBS volumes	~$24	Real. Deleted.
9	NAT gateway data processing spike	~$19 (one-off)	Real. Rearchitected.
16	"Idle RDS instance"	$12 to $15	Hallucinated. No RDS exists.
19	"Unexpected Bedrock usage"	n/a	The agent found itself.
24	Unattached Elastic IP	~$3.60	Real. Released.
25	SageMaker training spike	n/a	Correctly labeled intentional.
28	Cross-region S3 access	~$1	Real. Fixed the script.

Six correct findings, one correct non-finding, two false ones. Both false ones were eliminated by tool and prompt fixes and never recurred. I will take that ratio in any system I run.

What the watcher itself costs

Full transparency, because a cost optimization story that hides its own costs would be a comedy:

Component	Monthly cost
Bedrock model invocations (30 daily sessions)	~$3.90
Cost Explorer API calls (~220 total)	~$2.20
Lambda, EventBridge, SES, CloudWatch Logs	under $0.30
Total	~$4.87

Call it five dollars to find fifty-five.

On a personal account the return is comfortable. On a company account with real spend and real sprawl, this is not even a conversation. The same architecture pointed at an organization's consolidated billing, with per-team tags in the grouping dimension, would pay for itself before lunch on day one.

Questions friends asked me along the way

A few of these came from colleagues, a couple from the AWS Community Builders Slack when I mentioned what I was up to.

Why not Cost Anomaly Detection? It is free.
Use both; they are not competitors. Anomaly Detection is a smoke detector: statistical, fast, and context-free. The agent is the person who walks into the kitchen, sees the smoke, and tells you it is the toaster and not the house. On day 25 Anomaly Detection would have flagged my training run. The agent explained it and moved on.

Why fresh sessions daily instead of giving it memory?
For the experiment, reproducibility. Every claim in every report had to be derivable from that day's tool calls alone, or the failures would have been impossible to debug. In steady-state operation, memory of past findings is clearly useful (it could track whether recommendations were acted on), and that is the headline feature of my version two.

Could the model just be pattern-matching generic FinOps advice rather than analyzing my data?
The NAT gateway diagnosis and the cross-region catch were both specific to numbers it fetched in-session, with the mechanism named correctly. The RDS hallucination is actually the counter-evidence in my favor: when it did drift into generic pattern-matching ("accounts usually have an underused database"), it produced garbage, and the fix was forcing every claim to bind to fetched data. The discipline is enforceable.

What about a multi-account org?
Same shape, bigger tools. Point Cost Explorer at the payer account, group by linked account and tag, and give the report a per-team section. The read-only constraint becomes more important there, not less.

Did you worry about sending billing data to a model?
The data stays inside my own AWS account boundary; Bedrock invocations are not used to train the underlying models, per AWS's service terms. For a personal account I was comfortable. A regulated enterprise should read those terms themselves rather than take a blog post's word for it, including mine.

So, should you hire a robot accountant?

The honest answer depends on what kind of AWS user you are.

Build it if you run experiments, demos, side projects, or anything with a lifecycle shorter than your attention span. The idle-resource problem is fundamentally a problem of forgetting, and machines do not forget. Six hours of building bought me a permanent employee who works for five dollars a month and has no opinion about weekends.

Skip it if your account runs one stable production workload that never changes. A static bill needs a budget alert, not an analyst. This tool earns its keep in accounts with churn.

And absolutely do not give the agent write permissions, no matter how elegant the fully autonomous version looks in your head. My agent hallucinated a database on day 16 and priced its imaginary savings with a straight face. Imagine that reasoning chain ending in a stop_db_instance call aimed at whatever resource it decided best fit its imaginary finding. Read-only is not a limitation of this design. It is the design. The human approves; the human executes; the agent never touches anything.

Three smaller lessons I would press into your hands before you go build:

Make the agent state ranges, not points. "Savings of $28 to $34" invites verification. "Savings of $31.20" invites false trust. Precision is a costume that confidence wears.
Teach it your normal in the instructions, including typical spend and known steady charges. Anomaly detection without a baseline is just complaining with extra steps.
Log every session and read the traces when something is off. The trace of which tools were called and what they returned was the only way to tell whether a bad report came from bad data or bad reasoning. In my month it was never the model and always my instructions. Debug yourself first.

Wrapping up

So, amici, that is the story of my month with a robot accountant. It began with a dead GPU endpoint and a familiar sting of embarrassment, and it ended with a 29% smaller bill, an inbox full of color-coded morning reports, and a considerably sharper understanding of where agents genuinely earn their keep. The Accountant found real money, explained real mechanisms, correctly shrugged at my intentional spikes, and got measurably smarter every time I tightened its instructions.

It also invented a database out of thin air and once reported its own existence to me as a suspicious charge, which is why it will remain, forever, a colleague I listen to carefully and never hand the keys. Version two is already half-built: weekly deep-dives, per-project cost attribution through tags, and persistence so it can nag me about recommendations I have ignored.

If you build your own version, and I hope this post leaves you no excuse not to, I would genuinely love to hear what your agent catches in its first week. Find me on Dev.to or LinkedIn and tell me the story. The weirdest finding gets a shout-out in the follow-up post. Until then, keep your endpoints deleted and your subnets public where they can afford to be.

#HappyCoding 👋

30 Tiny React Hooks That Punch Way Above Their Weight

Mursal Furqan Kumbhar — Fri, 26 Jun 2026 04:22:27 +0000

Hello devs 👋

Ever notice how the same ten lines of React keep showing up in every project you touch? You copy them from some old repo, paste them into the new one, rename a variable or two, and quietly pretend you wrote them fresh. Be honest, we all do it. Half of frontend work is just remembering which folder you left that one perfect hook in.

So I stopped pretending and collected the ones I actually reach for. This article is a stash of 30 tiny, battle tested React snippets you can drop straight into your codebase. No extra libraries, no bloat, no npm install marathon. Just copy, paste, and ship.

Bookmark this one. Future you is going to be very grateful to present you.

1. useDebounce

const useDebounce = (value, delay = 500) => {
  const [debounced, setDebounced] = useState(value);
  useEffect(() => {
    const t = setTimeout(() => setDebounced(value), delay);
    return () => clearTimeout(t);
  }, [value, delay]);
  return debounced;
};

Delays rapid state updates so you are not firing on every keystroke. Perfect for search bars and typeahead inputs where you only want to hit the API once the user stops typing. Your backend will send you a thank you card.

2. useClickOutside

const useClickOutside = (ref, handler) => {
  useEffect(() => {
    const listener = (e) => {
      if (!ref.current || ref.current.contains(e.target)) return;
      handler();
    };
    document.addEventListener("mousedown", listener);
    return () => document.removeEventListener("mousedown", listener);
  }, [ref, handler]);
};

Closes a modal, dropdown, or popover when the user clicks anywhere outside of it. Attach the ref to your menu, pass a close function as the handler, and you are done. Ten lines that quietly power half the dropdowns on the internet.

3. usePrevious

const usePrevious = (value) => {
  const ref = useRef();
  useEffect(() => { ref.current = value; }, [value]);
  return ref.current;
};

Gives you the value a prop or state had on the last render. Handy when you want to compare old versus new, like triggering an animation only when a count actually goes up. Tiny hook, surprisingly clutch.

4. useLocalStorage

const useLocalStorage = (key, initial) => {
  const [value, setValue] = useState(() => {
    const stored = localStorage.getItem(key);
    return stored ? JSON.parse(stored) : initial;
  });
  useEffect(() => {
    localStorage.setItem(key, JSON.stringify(value));
  }, [key, value]);
  return [value, setValue];
};

Works exactly like useState but survives a page refresh. Great for theme choices, draft form data, or a dismissed banner that should stay dismissed. Same API you already know, zero new mental overhead.

5. useFetch

const useFetch = (url) => {
  const [data, setData] = useState(null);
  const [loading, setLoading] = useState(true);
  useEffect(() => {
    fetch(url)
      .then((r) => r.json())
      .then(setData)
      .finally(() => setLoading(false));
  }, [url]);
  return { data, loading };
};

A pocket sized React Query. Fetches data and tracks the loading state in under fifteen lines. Not built for retries or caching, but it covers a solid 90 percent of simple screens.

6. useToggle

const useToggle = (initial = false) => {
  const [value, setValue] = useState(initial);
  const toggle = useCallback(() => setValue((v) => !v), []);
  return [value, toggle];
};

Stop writing setOpen(prev => !prev) for the hundredth time. This gives you a boolean and a one word toggle function. Modals, sidebars, accordions, password visibility, all of it gets cleaner.

7. useIsMobile

const useIsMobile = (breakpoint = 768) => {
  const [isMobile, setIsMobile] = useState(window.innerWidth <= breakpoint);
  useEffect(() => {
    const onResize = () => setIsMobile(window.innerWidth <= breakpoint);
    window.addEventListener("resize", onResize);
    return () => window.removeEventListener("resize", onResize);
  }, [breakpoint]);
  return isMobile;
};

Tells you whether the user is on a phone or a bigger screen. Use it to swap a desktop sidebar for a hamburger menu, or to render a completely different layout. CSS handles most of this, but sometimes you need the answer in JavaScript.

8. scrollToTop

const scrollToTop = () =>
  window.scrollTo({ top: 0, behavior: "smooth" });

Smoothly glides the page back to the top. Wire it up to a floating "back to top" button on long blog posts or endless product lists. One line, instant polish.

9. useInterval

const useInterval = (callback, delay) => {
  const savedCallback = useRef(callback);
  useEffect(() => { savedCallback.current = callback; }, [callback]);
  useEffect(() => {
    if (delay === null) return;
    const id = setInterval(() => savedCallback.current(), delay);
    return () => clearInterval(id);
  }, [delay]);
};

Runs a function on a repeating timer the React way. The saved ref means your callback always sees fresh state instead of a stale closure, which is the classic setInterval trap. Reach for it on live clocks, auto refreshing dashboards, and carousels that slide on their own.

10. useTimeout

const useTimeout = (callback, delay) => {
  const savedCallback = useRef(callback);
  useEffect(() => { savedCallback.current = callback; }, [callback]);
  useEffect(() => {
    if (delay === null) return;
    const id = setTimeout(() => savedCallback.current(), delay);
    return () => clearTimeout(id);
  }, [delay]);
};

The one shot cousin of useInterval. Fires your callback once after a delay and cleans itself up properly on unmount. Nice for auto hiding a toast or showing a "still loading" message after a few seconds.

11. useWindowSize

const useWindowSize = () => {
  const [size, setSize] = useState({
    width: window.innerWidth,
    height: window.innerHeight,
  });
  useEffect(() => {
    const onResize = () =>
      setSize({ width: window.innerWidth, height: window.innerHeight });
    window.addEventListener("resize", onResize);
    return () => window.removeEventListener("resize", onResize);
  }, []);
  return size;
};

Gives you live width and height that update as the window resizes. Useful for canvas rendering, virtualized lists, or any chart that needs exact pixel dimensions. The component re renders the moment the viewport changes.

12. useMediaQuery

const useMediaQuery = (query) => {
  const [matches, setMatches] = useState(
    () => window.matchMedia(query).matches
  );
  useEffect(() => {
    const mql = window.matchMedia(query);
    const onChange = () => setMatches(mql.matches);
    mql.addEventListener("change", onChange);
    return () => mql.removeEventListener("change", onChange);
  }, [query]);
  return matches;
};

Your CSS media queries, now available in JavaScript. Pass something like "(min-width: 1024px)" and get a boolean back. Cleaner than a manual resize listener when you only care about a specific breakpoint.

13. useCopyToClipboard

const useCopyToClipboard = () => {
  const [copied, setCopied] = useState(false);
  const copy = async (text) => {
    await navigator.clipboard.writeText(text);
    setCopied(true);
    setTimeout(() => setCopied(false), 2000);
  };
  return { copied, copy };
};

Copies any text and flips a "copied" flag for two seconds so you can show a little checkmark. Every code block, share link, and API key field wants this. Small touch, big "wow this app feels finished" energy.

14. useHover

const useHover = () => {
  const ref = useRef(null);
  const [hovered, setHovered] = useState(false);
  useEffect(() => {
    const node = ref.current;
    if (!node) return;
    const enter = () => setHovered(true);
    const leave = () => setHovered(false);
    node.addEventListener("mouseenter", enter);
    node.addEventListener("mouseleave", leave);
    return () => {
      node.removeEventListener("mouseenter", enter);
      node.removeEventListener("mouseleave", leave);
    };
  }, []);
  return [ref, hovered];
};

Tracks hover state without sprinkling onMouseEnter and onMouseLeave everywhere. Attach the ref, read the boolean, render a tooltip or a preview card. Great when hover needs to drive logic and not just a CSS color change.

15. useOnlineStatus

const useOnlineStatus = () => {
  const [online, setOnline] = useState(navigator.onLine);
  useEffect(() => {
    const goOnline = () => setOnline(true);
    const goOffline = () => setOnline(false);
    window.addEventListener("online", goOnline);
    window.addEventListener("offline", goOffline);
    return () => {
      window.removeEventListener("online", goOnline);
      window.removeEventListener("offline", goOffline);
    };
  }, []);
  return online;
};

Tells you in real time whether the user just lost their connection. Show a "you are offline" banner, pause auto saves, or queue requests for later. Essential for anything that pretends to be a serious web app.

16. useEventListener

const useEventListener = (event, handler, element = window) => {
  const savedHandler = useRef(handler);
  useEffect(() => { savedHandler.current = handler; }, [handler]);
  useEffect(() => {
    const listener = (e) => savedHandler.current(e);
    element.addEventListener(event, listener);
    return () => element.removeEventListener(event, listener);
  }, [event, element]);
};

One hook to rule all your addEventListener boilerplate. It handles the subscribe and cleanup dance for you and always calls the latest handler. Build most of the other hooks in this list on top of it if you want to get fancy.

17. useDocumentTitle

const useDocumentTitle = (title) => {
  useEffect(() => {
    const previous = document.title;
    document.title = title;
    return () => { document.title = previous; };
  }, [title]);
};

Updates the browser tab title and politely restores the old one when the component unmounts. Perfect for per page titles in a single page app, or showing an unread count like "(3) Inbox". Tiny SEO and UX win in three lines.

18. useKeyPress

const useKeyPress = (targetKey) => {
  const [pressed, setPressed] = useState(false);
  useEffect(() => {
    const down = (e) => e.key === targetKey && setPressed(true);
    const up = (e) => e.key === targetKey && setPressed(false);
    window.addEventListener("keydown", down);
    window.addEventListener("keyup", up);
    return () => {
      window.removeEventListener("keydown", down);
      window.removeEventListener("keyup", up);
    };
  }, [targetKey]);
  return pressed;
};

Returns true while a given key is held down. Wire up Escape to close a modal, Enter to submit, or arrow keys to move through a list. Keyboard shortcuts make your app feel like it was built by people who actually use it.

19. useDarkMode

const useDarkMode = () => {
  const [dark, setDark] = useState(
    () => localStorage.getItem("theme") === "dark"
  );
  useEffect(() => {
    document.documentElement.classList.toggle("dark", dark);
    localStorage.setItem("theme", dark ? "dark" : "light");
  }, [dark]);
  return [dark, setDark];
};

A complete dark mode toggle that remembers the choice across reloads. It flips a class on the html element so Tailwind or your CSS variables do the rest. Ship the feature everyone immediately looks for in settings.

20. useThrottle

const useThrottle = (value, limit = 500) => {
  const [throttled, setThrottled] = useState(value);
  const lastRan = useRef(Date.now());
  useEffect(() => {
    const handler = setTimeout(() => {
      if (Date.now() - lastRan.current >= limit) {
        setThrottled(value);
        lastRan.current = Date.now();
      }
    }, limit - (Date.now() - lastRan.current));
    return () => clearTimeout(handler);
  }, [value, limit]);
  return throttled;
};

Debounce waits for things to stop, throttle keeps a steady pace. This caps how often a value updates, which is exactly what you want for scroll position, mouse tracking, or window resize math. Smooth performance without melting the main thread.

21. useIntersectionObserver

const useIntersectionObserver = (options) => {
  const ref = useRef(null);
  const [isVisible, setIsVisible] = useState(false);
  useEffect(() => {
    const observer = new IntersectionObserver(
      ([entry]) => setIsVisible(entry.isIntersecting),
      options
    );
    if (ref.current) observer.observe(ref.current);
    return () => observer.disconnect();
  }, [options]);
  return [ref, isVisible];
};

Tells you when an element scrolls into view. This is the engine behind lazy loaded images, fade in on scroll animations, and infinite feeds that load more as you reach the bottom. Way cheaper than listening to the scroll event yourself.

22. useUpdateEffect

const useUpdateEffect = (effect, deps) => {
  const isFirst = useRef(true);
  useEffect(() => {
    if (isFirst.current) {
      isFirst.current = false;
      return;
    }
    return effect();
  }, deps);
};

Like useEffect, but it skips the very first render and only runs on updates. Saves you from that annoying duplicate API call or analytics event that fires the moment a component mounts. The "only when something actually changes" hook.

23. useEffectOnce

const useEffectOnce = (effect) => {
  useEffect(effect, []);
};

A tiny wrapper that runs an effect exactly once on mount. Mostly it just signals your intent clearly so the next dev does not stare at an empty dependency array wondering if you forgot something. Readability counts.

24. useCounter

const useCounter = (initial = 0, step = 1) => {
  const [count, setCount] = useState(initial);
  const increment = () => setCount((c) => c + step);
  const decrement = () => setCount((c) => c - step);
  const reset = () => setCount(initial);
  return { count, increment, decrement, reset };
};

A ready made counter with increment, decrement, and reset baked in. Drop it into quantity selectors, pagination, or step by step wizards. Less wiring, fewer bugs, more time for the fun stuff.

25. useArray

const useArray = (initial = []) => {
  const [array, setArray] = useState(initial);
  const push = (item) => setArray((a) => [...a, item]);
  const remove = (i) => setArray((a) => a.filter((_, idx) => idx !== i));
  const clear = () => setArray([]);
  return { array, set: setArray, push, remove, clear };
};

Managing a list in state with clean helper methods instead of spread syntax soup. Add, remove by index, or wipe it clean without rewriting the same map and filter logic. Great for to do lists, tag inputs, and cart items.

26. useGeolocation

const useGeolocation = () => {
  const [coords, setCoords] = useState(null);
  useEffect(() => {
    const id = navigator.geolocation.watchPosition((pos) =>
      setCoords({
        lat: pos.coords.latitude,
        lng: pos.coords.longitude,
      })
    );
    return () => navigator.geolocation.clearWatch(id);
  }, []);
  return coords;
};

Watches the user location and updates as they move. Use it for "stores near me", live delivery tracking, or auto filling a city field. Just remember the browser will ask for permission first, so handle the null case gracefully.

27. useIdle

const useIdle = (timeout = 3000) => {
  const [idle, setIdle] = useState(false);
  useEffect(() => {
    let timer = setTimeout(() => setIdle(true), timeout);
    const reset = () => {
      clearTimeout(timer);
      setIdle(false);
      timer = setTimeout(() => setIdle(true), timeout);
    };
    window.addEventListener("mousemove", reset);
    window.addEventListener("keydown", reset);
    return () => {
      clearTimeout(timer);
      window.removeEventListener("mousemove", reset);
      window.removeEventListener("keydown", reset);
    };
  }, [timeout]);
  return idle;
};

Detects when the user has gone quiet for a while. Pause a video, dim the screen, or auto log out of a banking dashboard after inactivity. Any reset of mouse or keyboard activity wakes it back up.

28. useSessionStorage

const useSessionStorage = (key, initial) => {
  const [value, setValue] = useState(() => {
    const stored = sessionStorage.getItem(key);
    return stored ? JSON.parse(stored) : initial;
  });
  useEffect(() => {
    sessionStorage.setItem(key, JSON.stringify(value));
  }, [key, value]);
  return [value, setValue];
};

Same idea as useLocalStorage, but the data clears when the tab closes. Ideal for a multi step checkout or a wizard where you want progress saved during the visit but gone afterward. Privacy friendly persistence.

29. useScrollLock

const useScrollLock = (locked) => {
  useEffect(() => {
    document.body.style.overflow = locked ? "hidden" : "";
    return () => { document.body.style.overflow = ""; };
  }, [locked]);
};

Freezes background scrolling while a modal or drawer is open. We have all rage scrolled a popup and watched the page behind it slide around. This one line fix kills that jank and restores scroll on close.

30. useRenderCount

const useRenderCount = () => {
  const count = useRef(0);
  count.current += 1;
  return count.current;
};

Counts how many times a component has rendered. Pure debugging gold when you are hunting down why something re renders forty times when you clicked once. Pop it in, log the number, find the culprit, take it out.

Conclusion

And that is the stash. Thirty little snippets that save you from rewriting the same logic in every single project from now until forever. Copy what you need, tweak the ones you love, and quietly become the teammate whose pull requests are weirdly clean.

I will be back soon with another round, probably something on performance or the patterns nobody warns you about until it is too late. Until then, go build something cool and stop reinventing the debounce.

happycoding

The Invisible Duct Tape of the Internet: Backend Tools You Hear About But Never Fully Get

Mursal Furqan Kumbhar — Mon, 22 Jun 2026 03:14:44 +0000

Hi 👋 fellow devs

Sorry for such a big gap since my last article...... Life got a bit hectic, but I am finally back in action!

You know how it goes. We spend so much of our energy obsessing over the flashy side of tech. We talk about gorgeous UI designs, smooth animations, and whatever frontend framework is trending on GitHub this week. But let’s be completely real for a second. What actually keeps your favorite apps from melting down when millions of people hit the refresh button at the exact same moment?

That is exactly what we are going to unpack today. We are pulling back the curtain on the quiet, brilliant backstage crew of infrastructure tools. You see their logos all over tech Twitter and hear senior engineers drop their names in meetings like secret handshakes, but today, we are stripping away the corporate fluff. We will break down eight legendary backend technologies using conversational paragraphs and quick bullet points so you can finally master what they actually do.
Let’s dive right in.

1. Redis

Traditional databases live on hard drives. They are fantastic for keeping your data safe and organized permanently, but pulling data off a physical drive takes time. If your application has to wander deep into those database aisles to fetch the exact same piece of information every single second, your entire system starts to stall.
To understand how Redis fixes this, imagine you are studying for a brutal exam. Your massive, 1,000-page textbook represents your main database. It holds every single answer, but flipping through the pages continuously is incredibly slow. Redis is the digital equivalent of writing the core formulas you need on a neon sticky note and taping it directly to your monitor. It keeps critical data sitting directly inside the system's lightning-fast short-term memory.
You will typically find Redis stepping in to handle operations like:

Session Management: Keeping users logged into an application without checking the main database on every click.
API Caching: Storing the responses of heavy, repetitive queries so they render instantly.
Live Leaderboards: Tracking high-velocity, real-time score updates during live events or games.

2. Kafka

Modern applications generate an overwhelming, continuous stream of mini-events. Every single click, profile update, and background notification triggers a piece of data. If every single microservice in your software tries to communicate directly with every other service to share these updates, your architecture quickly devolves into a tangled, fragile spiderweb of digital chaos.
Think of Kafka like a wildly busy fast-food kitchen. Instead of the cashiers running into the back to scream individual orders directly at the face of every cook, they hang receipts on a long, central ticket line. The burger station, the fry station, and the drinks manager all pull tickets from that central line at their own pace. Kafka acts as that indestructible, high-speed central ticket line for your data.
Engineers rely on Kafka to manage heavy data pipelines because it excels at:

Decoupling Services: Allowing different parts of an app to function independently without relying on direct connections.
Activity Tracking: Processing millions of user interactions across a massive platform simultaneously.
Real-time Data Streams: Managing live GPS coordinates from a fleet of delivery drivers without dropping a single packet.

3. Nginx

When a massive wave of users hits a website simultaneously, your core application servers can easily experience a traffic jam, get overwhelmed, and crash. Nginx sits right at the edge of your infrastructure to act as an incredibly efficient shield, managing thousands of incoming connections before they ever touch your actual code.
To picture this, imagine a highly exclusive apartment building. Instead of letting random delivery drivers wander the hallways looking for individual doors, everyone is forced to stop at a central front desk. Nginx is that ultimate receptionist. It checks security certificates, hands out public brochures instantly so people do not even have to step inside, and cleanly directs visitors to the exact elevator they need.
In everyday setups, Nginx is deployed to handle:

Load Balancing: Distributing incoming user traffic evenly across multiple backend servers.
Reverse Proxying: Routing requests smoothly to different applications running on different ports.
Static File Serving: Instantly handing over images, HTML, and CSS directly to the browser to save server power.

4. Kubernetes

Managing a couple of servers by hand is completely fine, but managing hundreds of isolated microservices scattered across global cloud networks is a total nightmare. If a server goes down in the middle of the night, someone has to wake up, diagnose the issue, and manually restart it. Kubernetes exists to automate that entire headache away.
Think of Kubernetes as the manager of a massive, fully automated shipping port. You do not manually tell every truck where to drive or check if a crane operator fell asleep. You program your desired layout into a central management system. If a cargo truck randomly breaks down, the system instantly detects it, dispatches a identical replacement truck, and moves the cargo along without stalling operations for a single second.
Kubernetes keeps complex cloud applications running smoothly by providing:

Self-Healing: Automatically restarting containers that crash and replacing unresponsive servers.
Horizontal Scaling: Spatially spinning up new instances of an app during traffic spikes and scaling them down when quiet.
Automated Rollouts: Gradually deploying new code updates across a network without causing a single second of downtime.

5. Elasticsearch

Standard database lookups are incredibly rigid. They work perfectly when you are searching for an exact user ID number, but if you try to perform a complex, messy search across millions of rows of unstructured text, a normal database will grind to a painful halt.
Imagine trying to find every single mention of the word "coffee" inside a massive university library. A standard database approach would involve opening every single book and scanning every single page word by word. Elasticsearch works by building a massive, highly optimized index ahead of time. It acts exactly like the index at the back of a textbook, but it is smart enough to find the right page even if you make a typo.
It is the industry standard tool when you need to build:

Advanced E-commerce Search: Powering filters, categories, and instant results across massive product catalogs.
Fuzzy Matching: Predicting what a user actually meant even when they misspell words in a search bar.
Log Analysis: Sifting through gigabytes of system error logs to find a single needle in a haystack.

6. GraphQL

Traditional REST APIs can sometimes feel like a restaurant that only serves rigid, pre-packaged combo meals. If you only want a side of fries, you are still forced to order the entire burger and drink combo, which wastes data and bandwidth. Alternatively, you might have to hit three different endpoints just to gather enough information to build a single profile page.
GraphQL fundamentally changes this dynamic by turning your API into an a la carte menu. It allows the client application to hand the backend a specific checklist of requirements. If your mobile app only needs a user's first name and profile photo for a tiny header card, GraphQL ensures the server sends back exactly those two data points and absolutely nothing extra.
This precision makes GraphQL incredibly valuable for:

Mobile App Optimization: Saving precious cellular data and battery life for users on slower networks.
Consolidating Requests: Fetching data from multiple distinct sources using a single, unified network call.
Rapid Frontend Development: Allowing frontend devs to change visual layouts without needing backend devs to constantly rewrite API endpoints.

7. RabbitMQ

When a user clicks a button to upload a file or make a purchase, they expect an instant confirmation response. They do not want to stare at a frozen loading spinner for ten minutes while your backend server crunches a heavy file, updates inventory, creates a PDF invoice, and sends out confirmation emails.
RabbitMQ fixes this by acting exactly like a post office drop box. You do not stand at the counter waiting for the clerk to drive your letter across the country. You drop it in the slot and walk away, confident it will be handled. RabbitMQ accepts these heavy, time-consuming tasks from your main application and holds them safely in a queue, letting your user interface stay incredibly snappy and responsive.
You should implement RabbitMQ whenever you need to offload tasks like:

Background File Processing: Resizing high-resolution images or transcoding videos after an upload.
Notification Blasts: Dispatching transactional emails or push notifications to thousands of users without blocking the main app.
Asynchronous Workflows: Delaying heavy calculations or third-party API syncs until your server has free resources.

8. Docker

Every developer has lived through the absolute nightmare where a feature runs flawlessly on their local laptop, but completely shatters the moment it gets deployed to a live production server due to a missing file or a conflicting software version.
Docker completely eliminates this issue by introducing the concept of digital shipping containers. In the physical world, a standard shipping container fits perfectly onto any cargo ship, train, or truck regardless of what is inside it. Docker applies this exact logic to software, wrapping your code, your runtime, your tools, and your specific configurations into an isolated, standardized digital box.
By containerizing your applications, Docker guarantees:

Environment Consistency: Ensuring your code behaves identically on a Mac laptop, a Windows desktop, or a Linux cloud instance.
Isolation: Running multiple applications with completely conflicting dependencies on the exact same computer without any interference.
Simplified Onboarding: Allowing new developers to spin up a complete, complex project environment using one single command.

Quick Reference Guide

The Tool	Its Core Superpower	In Plain English
Redis	In-memory speed	The sticky note on your monitor.
Kafka	Event streaming	The kitchen ticket line.
Nginx	Reverse proxy and web server	The gatekeeper at the front desk.
Kubernetes	Container orchestration	The automated shipping port manager.
Elasticsearch	Distributed search engine	The textbook index on steroids.
GraphQL	Precise API queries	The custom sandwich checklist.
RabbitMQ	Message broker	The post office drop box.
Docker	Containerization	The standardized shipping container.

Conclusion

At the end of the day, you do not need to be an absolute wizard in all eight of these tools to be a fantastic developer, but understanding where they fit in the puzzle changes how you think about software. Writing code that functions locally is just step one. True engineering happens when you learn how these infrastructure pieces coordinate behind the scenes to architect systems capable of handling millions of users without breaking a sweat.
Drop a comment below and let me know which of these backend heroes you are planning to deploy in your next project!

VS Code Extensions That Code For You

Mursal Furqan Kumbhar — Tue, 14 Apr 2026 10:35:13 +0000

Hello Devs 👋

If your VSCode setup still looks like it did before the AI boom, you may be coding with a bicycle in a Formula 1 race. 🏎️ Modern AI extensions are no longer novelty gadgets, they’re force multipliers. From ghost-writing boilerplate to understanding sprawling codebases and debugging at 2 AM when your brain has blue-screened, these tools can turn your editor into a caffeinated co-pilot. Here are 6 of the best AI extensions helping developers ship faster, smarter, and with fewer “why is this null?” moments.

Why does this matter?

Your editor should work as hard as you do. Most devs use VSCode, but only 10% use AI extensions. In 2026, that gap is the difference between a slow dev and a 10x dev.

1. Continue by Continue.dev

Open-Source AI Assistant. Plug in any model, like GPT-4, Claude, Llama or Mistral directly into VSCode. It:

Uses your own API Key (GPT/Claude/etc.)
Run local models via Llama
Custom slash commands & context
Full-Chat + Inline edit supports Rating: 5/5 Cost: Free | Open-Source

2. Supermaven by Supermaven Inc.

Blazing fast AI autocomplete with a 1 million context window. Its usage feels almost instant, doesn't lag and doesn't make you wait.

1,000,000 Token context.
Sees your entire repository and codebase
Sub-100ms suggestions. Truly feels instant
Understands cross-file dependencies
Free tier available Rating: 5/5 Cost: Free Plan | Pro is $10/Month

3. Pieces for Devs by Pieces.app

Your AI-Powered developer memory. It saves code snippets, context and links, and retrieves them as and when needed. It also:

Auto-saves snippets with AI Tags
Long-term memory across projects
AI Explains saved snippets on demand
Works offline with on-device AI Rating: 5/5 Cost: Free for Personal Use

4. Codeium by Exafunction

The best free Copilot alternative. It's fast, accurate and supports 70+ languages. Codeium facilitates its users in many ways, such as:

It's completely free for individual devs
Provides in-editor AI Chat for debugging
Autocompletes across 70+ programming languages
Also comes with Codeium Search and Codebase Q&A Rating: 5/5 Cost: 100% FREE | No Limits

5. Tabine by Tabnine

AI Code Completor, trained on your own codebase. This is best for teams that can't share code with external servers (mainly proprietary developers). Its main features include, but are not limited to:

Runs locally, your code stays private
Learns your codebase patterns
Works offline, no internet needed
Team model training on your repository Rating: 5/5 Cost: Pro is $12/Month

6. GitHub Copilot by GitHub (Microsoft)

The OG AI Pair Programmer. This extension auto-completes entire functions, writes tests, and explains code, all inside VSCode. Its main features include:

Inline code suggestions as you type
Copilot Chat, ask anything about your code
Multi-file context awareness
Generate unit tests automatically Rating: 5/5 Cost: $10/Month | Free for Students

Conclucion

The best developers in 2026 are not just writing code, they’re orchestrating tools. Whether you want lightning-fast autocomplete, local privacy-first AI, or a full-blown coding sidekick living in your editor, there’s an extension here for every workflow. Try a few, mix and match, and build the AI-powered setup that makes your keyboard feel slightly unfair. ⚡

Giving AI Agents “Live Memory” with Aurora Zero-ETL and Redshift Vector Search

Mursal Furqan Kumbhar — Wed, 25 Mar 2026 05:38:18 +0000

Hey everyone! Honestly, it feels so good to be sitting down and writing to you all again. I want to start with a massive apology for the radio silence and the long gap between my last few posts. Life has a funny way of getting busy, but I’ve missed this community and our deep dives into the future of tech.

We’re talking about a problem that has been bugging me all through 2025 and into 2026: AI situational awareness. Let’s dive in.

The "Goldfish" Problem: Why Your AI is Gaslighting Your Users 🐠🌀

Look, we need to have a genuine heart-to-heart about your AI agents. We’re well into 2026, and yet, most AI “agents” still have the situational awareness of a goldfish in a blender. It’s a bit heartbreaking, isn't it?

You know the feeling: You’ve spent weeks perfecting your system prompts, the UI is looking like a work of art, and your LLM is sharp. But the second a user actually does something—like upgrading a subscription, changing a delivery address, or updating their profile—the AI is the absolute last one to know. It’s like the “memory” of your agent is stuck in a permanent time lag, trapped in a 2024 batch-processing nightmare.

Imagine you’re building the ultimate AI concierge for a luxury travel app.

2:00:00 PM: Your user, Sarah, realizes she’s over the cold and cancels her flight to Paris because she’d rather sip espresso in Rome. This change hits your Amazon Aurora database instantly. Success!
2:00:10 PM: Feeling productive, Sarah asks the AI Agent, “Hey, what time does my flight leave?”
The Disaster: Because your old-school data sync (ETL) only runs every 30 minutes, the AI looks at the stale warehouse data and says, “Your flight to Paris departs at 6 PM! Don't forget your beret!” 🥖

Sarah is now officially being gaslit by an algorithm. 🤦‍♂️ The immersion? Totally gone. The trust? Shattered. This happens because the "Transactional Brain" (Aurora) isn't talking to the "Analytical Memory" (Redshift) fast enough. We need a way to move that data so quickly it feels like magic. Today, we are fixing that with Live Memory using Amazon Aurora Zero-ETL and Redshift Vector Search.

Step 1: The “Wormhole” (Zero-ETL) 🚀

In the "old days" (way back in 2024), moving data from Aurora to Redshift meant building a "Glue" pipeline. It was like trying to build a bridge between two islands using nothing but popsicle sticks, duct tape, and a whole lot of hope. It broke constantly, it cost a fortune, and it was painfully slow.

Zero-ETL is the modern way to handle this. Think of it as a native “wormhole” between Aurora and Redshift. When you enable it, AWS handles all the heavy lifting and plumbing behind the scenes. No Python scripts to debug, no Lambda triggers to monitor, and absolutely zero drama.

Why this is the “Secret Sauce” for your Agent:

Near-Zero Latency: We’re talking seconds, not minutes. By the time Sarah finishes typing her question, the data is already there.
Auto-Scaling: As your app goes viral, the teleporter grows with it.
Total Peace of Mind: You can finally stop monitoring failed Glue jobs at 3 AM. If it’s in Aurora, it’s in Redshift.

Step 2: From “Rows” to “Recall” (Vector Search) 🧠

Okay, so the data has successfully teleported into Redshift. But here’s the catch: Redshift is a warehouse full of structured tables, and AI Agents don’t really "think" in tables—they think in Vectors (mathematical representations of meaning).

In the past, you’d have to ship this data again to a dedicated vector database. But that just adds more lag! Instead, we use Redshift’s Native Vector Search. This keeps your “Live Memory” in one single, high-performance place. To make this work, we set up a Materialized View in Redshift that automatically calls Amazon Bedrock to turn those new rows into vectors on the fly.

-- The "I don't have a goldfish memory anymore" Query
CREATE MATERIALIZED VIEW live_user_context AS
SELECT 
    user_id, 
    event_description,
    -- This calls Bedrock to create a vector on the fly!
    amazon_bedrock_embed(event_description) as semantic_vector
FROM aurora_synced_data.user_logs;

Step 3: The “Aha!” Moment 🔍

Now, let's replay the Sarah scenario. When she asks about her flight, the AI Agent sends a vector query to Redshift. Redshift looks at the most recent events—including the one that landed just 10 seconds ago—and finds the “Canceled Paris / Booked Rome” entry.

The Agent responds: “I see you switched to the Rome trip—great choice! That flight leaves at 8 PM. Should I book a car to the airport?”

Boom. You just saved the user experience. 🥳 You've moved from a "chatbot" to a genuine "agent" that understands the present moment.

By The Numbers: Is it worth your time? 📈

We ran some benchmarks to see just how much of a difference this "Live Memory" architecture actually makes. When you compare the old "Batch" world to the new "Zero-ETL" world, the latency gap is massive.

Average “Brain Lag” (Latency):
- Standard S3/Glue Sync: ~480 seconds
- Zero-ETL + Redshift: ~12 seconds (A 97% improvement!)
Developer Happiness:
- Measured in “How many times I had to fix a pipeline this week”: Dropped from an average of 4 down to 0.

Pro-Tips for the Modern ML Engineer 🛠️

Keep it Serverless: Use Redshift Serverless. It scales down to zero when your agents are sleeping.
Hybrid Search is King: Don’t rely only on vectors. Combine them with regular SQL filters (like WHERE user_id = '123') to make your search lightning fast.
Capture the Vibe: Remember, this isn’t just for logs. If a user tells the AI, “I love my better half a lot,” ❤️ that sentiment gets synced and embedded instantly. The next time they ask for anniversary ideas, the agent immediately knows the priority is “High Romance.”

Final Thoughts

Anyway, it feels amazing to be back in the flow and sharing these breakthroughs with you all again. The leap from “static” AI to “live” agents is one of the most exciting shifts we’re seeing this year, and I can’t wait to see how you all implement these “wormholes” in your own stacks. If you run into any snags or just want to geek out over vector search optimizations, drop a comment below or find me over on Dev.to! I promise the next article won’t take nearly as long to reach your screens. Until next time — stay curious, keep building, and let’s make those agents a whole lot smarter. 🚀✨

RIP 12-Digit IDs: The AWS Matrix Has Finally Been Decoded

Mursal Furqan Kumbhar — Mon, 09 Feb 2026 09:41:12 +0000

Y'all... pull up a chair. We need to talk about a literal miracle. 🥂

It was 2:41 AM. PagerDuty was screaming like a banshee in my ear. My eyes were blurry, my coffee was cold, and Slack was silent in that heavy, "something-is-very-wrong" way. I logged into the AWS Console to “quickly restart a service” and get back to sleep.

Five clicks later—just as I was about to hit "Delete"—my stomach dropped. My heart hit the floor.

Wrong account. I was in Production. Nothing blew up that night. I caught myself at the last millisecond. I got lucky. But for 21+ years, millions of engineers weren’t so lucky. We’ve been squinting at 12-digit account IDs like we’re trying to decode the Matrix while the world burns around us.

But the wait is over. AWS finally added the Account Name to the top bar. It’s a "high-tech" feature that only took two decades to arrive, and it is GLORIOUS.

🏆 Welcome to the "Almost Nuked Prod" Hall of Fame

For over two decades, AWS UX followed the "Extreme Stealth" philosophy. The top bar gave you the region and your username, but the actual account context? Hidden behind a click. Pure chaos. It was like driving a car where the speedometer is inside the glove box.

This led to the Three Stages of AWS Grief:

The Over-Confidence: “I definitely logged into Sandbox. I'm a pro, I don't need to check.”
The Sudden Realization: “Wait... why does this S3 bucket have 4PB of data and a 'Do Not Delete' tag? Why is this instance type an x1e.32xlarge?”
The Panic: Aggressively smashing Ctrl+W to close 42 Chrome tabs before your shaking finger accidentally clicks 'Terminate Instance'.

Why This is Low-Key Life-Changing

This isn’t just cosmetic; it’s cognitive safety. In a multi-account world, you’re juggling dozens of environments. This update assumes that engineers are human—that we get tired, stressed, and caffeine-deprived. It’s a sanity check for your career. Good UX doesn’t assume perfection; it assumes you're human.

🛠️ Hands-On: Build Your Safety Net Today

Since you're likely rocking .ipynb files in VSCode (the ultimate dev setup, IMO), you’re used to seeing your environment in the kernel picker or the status bar. Let's get that same "don't-get-fired" energy in your browser.

1. Make it Pretty with the AWS CLI

If your account name is company-billing-prod-final-v2, it’s going to get truncated. Don't click through menus like it's 2004—we're builders! Pop this into your terminal to set a punchy, unmistakable Alias:

# Give your account a name that SCREAMS at you in the top bar
aws iam create-account-alias --account-alias PROD-SENSITIVE-ZONE

Boom. Now your top bar says PROD-SENSITIVE-ZONE in clear text instead of a random string of numbers. 💅

2. The "Visual Fire Alarm" (Layered Security)

Text is great, but color-coding saves careers. Since AWS hasn't given us a "Red Header" mode for Prod yet (maybe in the year 2047?), we have to be clever:

Unified Settings: Click your Name (top right) → Settings → Visual Mode. Choose a distinct theme for your main accounts.
Browser Profiles: This is the real pro-tip. Use different Browser Profiles (Chrome/Brave/Firefox) for different environments. I keep my Prod profile themed in "Emergency Red" with a massive warning icon. If the window isn't red, it isn't Prod.

3. The "Don't Fire Me" Python Snippet

Before running any destructive boto3 code in your Jupyter Notebook, run this check. It’s the programmatic version of "looking both ways before crossing the street."

import boto3

def safety_check():
    # Grab the current identity
    sts = boto3.client('sts')
    identity = sts.get_caller_identity()
    account_id = identity['Account']

    # Put your real Prod ID here
    PROD_ID = "123456789012" 

    print("-" * 30)
    if account_id == PROD_ID:
        print("🚨 ALERT: YOU ARE IN PRODUCTION! 🚨")
        print("Step away from the Shift+Enter key unless you're sure!")
    else:
        print(f"✅ Connected to: {account_id}")
        print("Environment: Non-Prod (Safe to play, fam)")
    print("-" * 30)

safety_check()

🚀 The Future of AWS UX

While we celebrate this "giant leap for engineer-kind," we’re still looking toward the future. Maybe one day the "Delete Database" button won't be the exact same shade of blue as the "Save" button? Maybe we'll get a "High Stakes" mode that requires a physical key turn?

Until then, appreciate the label. It might just be the thing that keeps you from becoming a "Why We Had an Outage" post-mortem.

Final Thought: The most dangerous bugs don’t live in your code; they live in interfaces that assume humans never slip. AWS finally acknowledged that we’re human. Sometimes, the biggest cloud innovation... is simply knowing exactly where you are before you hit 'Enter'. ☁️

From Monoliths to Multitaskers: Building Your AWS AI Dream Team! 🚀🤖

Mursal Furqan Kumbhar — Sat, 07 Feb 2026 07:47:02 +0000

After being away from active tech ecosystem for about a month, I have decided to shed some light on the current landscape of AI agents on AWS. And let me tell you, things have gotten wild while I was gone. We aren't just talking to single chatbots anymore; we're building entire autonomous squads.

If you’ve been building with AI lately, you’ve probably hit the "Monolith Wall." You know the one: you try to get one Large Language Model (LLM) to handle your database, write your emails, troubleshoot your network, and maybe make a decent cup of coffee. It gets confused, starts hallucinating, and suddenly your "smart" app is telling users that the server is down because it's "feeling sleepy."

The 2025 vibe is different. We are moving away from "User → LLM → Response" and toward "User → Agent Network → Coordinated Action." Think of it like a D&D party: you wouldn't ask the Barbarian to pick a lock, and you wouldn't ask the Rogue to tank a dragon. You need specialists.

Today, we’re going to learn how to orchestrate a Multi-Agent System (MAS) on AWS Bedrock. Grab your coffee (or mana potion), and let’s dive in!

🛠 The Secret Sauce: What is a "Bedrock Agent"?

Before we build the team, we need to understand the individual "hero." A Bedrock Agent isn't just an LLM with a fancy name. It's a system with three core parts:

The Brain (Model Provider): Usually something like Claude 3.5 Sonnet or the zippy new Amazon Nova series.
The Manual (System Prompt): The instructions that tell the agent, "You are a world-class SRE," and give it boundaries (like "Don't delete production, please").
The Toolbelt (Action Groups): These are the superpowers! This is where the agent calls AWS Lambda functions to actually do things in the real world.

🎭 The Multi-Agent Party: Meet the Supervisor

Why have one agent when you can have five? In a Supervisor-Agent Pattern, you have one "Dungeon Master" (the Supervisor) who listens to the user and decides which specialist to call.

Imagine we're building a Telco Network Operations Assistant. Our party looks like this:

Role	Specialty	Data Source
The Supervisor	Orchestration & Planning	Routes the query to the right specialist.
The Alchemist (KPI Agent)	Performance Metrics	Checks throughput and latency.
The Scout (Alarm Agent)	Real-time Status	Looks for active site failures.
The Chronicler (Log Agent)	History & Patterns	Scans CloudWatch logs for anomalies.

🏗 Hands-On Walkthrough: Building the "Site Ops Hero"

Let's get our hands dirty. We want to ask: "What's the status of site_dallas_001?" and get a correlated answer.

Step 1: The Magic Lambda (Your Action Group)
Your agent needs a bridge to the real world. We’ll use a Lambda function. When the agent calls it, it sends a JSON event that looks like this:

{
  "actionGroup": "NetworkTools",
  "apiPath": "/get-site-alarms",
  "parameters": [{"name": "site_id", "value": "site_dallas_001"}]
}

Tip: Bedrock events aren't the same as API Gateway events! You’ll need to parse the apiPath and parameters directly.

Step 2: Set up the Specialist Agents
In the Bedrock Console (or via Boto3), create your specialist agents. Give them Knowledge Bases so they can read your documentation stored in S3.

Pro-tip: Use Titan Text Embeddings v2 to turn your PDFs into "vector gold" that the agent can search in milliseconds.

Step 3: Summon the Supervisor
Create a new agent and enable Collaboration Mode. Associate your specialist agents as "collaborators." Now, when you talk to the Supervisor, it creates a plan:

Ask the Scout for active alarms.
Ask the Alchemist if performance is dropping.
Combine the answers and tell the human exactly what’s broken.

💰 Budgeting Your "Mana" (Cost Optimization)

Running a whole team of AI agents can get pricey if you're not careful. Here’s how to keep your AWS bill from going "Critical Hit" on your wallet:

Tiered Intelligence: Use the "big brain" (Claude 3.5 Sonnet or Nova Premier) for the Supervisor who needs to plan. Use the "fast brain" (Nova Lite or Micro) for simple sub-tasks.
Prompt Caching: If you’re sending the same 50-page manual to your agent every time, Prompt Caching can save you up to $90\%$ on input tokens.
Batch Inference: If you’re doing something that doesn’t need an answer right now (like summarizing yesterday's logs), use Batch Mode for a 50% discount.

🛡 Staying Safe (Guardrails)

You don't want your agent going rogue. Use Amazon Bedrock Guardrails to set "Denied Topics." For example, you can explicitly forbid the agent from giving out employee home addresses or venting about its boss. You can even set up Contextual Grounding Checks to make sure the agent only answers based on your data, not its own imagination (aka hallucinations).

🏁 Conclusion: Your Adventure Awaits!

Building multi-agent systems on AWS is like moving from playing a single-instrument solo to conducting a full philharmonic orchestra. It’s more complex, sure, but the results are magical.

Whether you're automating SRE tasks or building a Pokémon battle advisor, the tools are all there in Bedrock.

What are you waiting for? Go build something awesome! 🚀✨

P.S. When you're done, remember to delete your S3 buckets and Lambda functions so you don't wake up to a "Surprise Invoice" Boss Fight.

The AWS Lambda Setting I Wish I’d Changed Earlier

Mursal Furqan Kumbhar — Tue, 30 Dec 2025 09:10:50 +0000

I've been using AWS Lambda for quite a few years, and like many people, I left the architecture on its default settings, i.e. x86 (Without thinking 2x). It worked really well, it scaled as needed, but the truth unfolded when invoices showed up. End of Story. Fine Della Storia.

Upon closer examination of the situation, I paid more attention to the execution time & cost, and then I thought to experiment with Arm64, i.e. AWS Graviton. And the rare cloud optimisations that actually delivered, followed. 🎇 A VISIBLE WIN 🎇
.
.
.
Now the main Question

What led me to try Arm64?

The answer's as simple as it gets: A few Lambda handling API Requests & Background jobs were being called 1,000,000s of times per month and frankly, each one looked quite cheap, individually. But when bunched together, they burned a steamy hole in my pocket every month.

Now being a lazy_developer who uses snake cases while programming, I said NO to refactoring the logic or the code. I just wanted a clean performance win. And that's when... Arm64 looked like a low-risk bet to me.

Time taken towards the CHANGE!!!

Now, when I say I moved from x86 to Arm64, the first thing to pop in your head would be, "Nah! It's gonna take forever to change." Right? WRONG. I changed the lambda architecture from x86 to Arm64, redeployed and tested it within a matter of hours (And that's because I chose NOT to read the documentation first 🥲. It could have been sooner). That was it. No code was changed, no dependencies were broken, and no surprises. This alone was... unexpected.

What were my ACTUAL Observations

The moment I started moving only a few Lambdas functions to Arm64, the execution time dropped noticeably, especially for CPU-based logic. And guess what... the Cold Start felt snappier on lightweight Node.js & Python functions. In the end, the monthly lambda cost dip without touching memory allocation felt like a cherry on top.

So, nothing dramatic for every invocation, but at scale, the difference was truly impossible to ignore.

Reality Check: The Cost

Lambda billing isn't rocket science. It's simple maths:

Execution time * Memory * Invocations

If your function runs just 20-30 milliseconds faster, and it is executed millions of times, that tiny change in the execution speed turns into a new Apple Watch (Yes, no kidneys needed 😜). This was the moment Arm64 stopped feeling like an optimisation experiment & started feeling like a worthy replacement to x86 as default choice

Why did Arm64 work the best for ME?

In my small but meaningful experience, Arm64 shone brighter than star Sirius, when I used it for API Gateway-backed Lambdas, Event Driven pipelines, background jobs, schedulers, data processing & automation or even for lightweight ML inference. In short, I opted for Arm64, where I used modern serverless workloads with updated runtimes.

Where I would never use Arm64?

It's not like I am completely against x86 now. I still used it in my project for some scenarios. For example:

I kept x86 for older functions with legacy native binaries
Or for rare dependencies that weren't Arm-ready, yet
Or for the code, that I didn't want to touch, being that close to the deadline. (Programmers, IYKYK). For those who don't, we don't want the project go

What changes in the future?

In the future, if I am starting a new serverless project (Or today, who knows):

I will definitely go for Arm64, by default
I am gonna only move back to x86, only & only, if something explicitly breaks (Or my client doesn't love their 💵)
Will benchmark early, instead of assuming

Conclusion

Arm64 on AWS Lambda isn't a future feature, it's just a teeny tiny and quiet upgrade that's already paying off its cost in real-life systems. It's not theoretical. IT's PRACTICAL. Well, if you care about factors such as performance, cost & efficiency and you're running modern hsuper heavy serverless workloads, listen to me (Or read this carefully): TRY IT ONCE. 9/10 Chances? Like me, you won't turn back 😉.

Once again, this was my personal experience.

Buon Anno

HappyCoding

How I Automated Document Insights Using AWS Textract, Bedrock, and QuickSight

Mursal Furqan Kumbhar — Fri, 12 Sep 2025 17:35:28 +0000

Hello Everyone 👋

So I had a bunch of scanned invoices and PDFs lying around from an old freelance gig. Manually reading them and extracting data felt like a punishment. Instead of going through them one by one, I decided to try something smarter (I know, common Developer Syndrome 😜), to build a small AWS workflow that could:

Read scanned docs,
Summarise key info,
Convert it to structured data, and
Visualise everything inside a dashboard.

Here’s how I did it, step by step. You can follow the same process and tweak it for contracts, reports, receipts, whatever you’ve got.

✅ Step 1: Set Up Your AWS Environment (Production-ready)

1) Pick a single region that supports everything

Goal: Keep S3, Textract, Bedrock, and QuickSight in the same region to avoid cross-region latency/fees.
How to choose: Check that your chosen region offers Bedrock (model availability varies) and Textract. If QuickSight isn’t in that region, use Athena + Glue as a bridge (S3 anywhere → Athena in QuickSight’s region).
Edge cases
- Bedrock not in your favorite region: run Textract + S3 where your data lives, then send plain text only to Bedrock in a supported region (privacy + cost trade-off).
- Corporate constraint on regions: document the exception and add an S3 Replication rule or use Athena.

2) Create an S3 bucket with the right defaults

Name: doc-insights-<env>-<region>-<account-id> (e.g., doc-insights-prod-eu-central-1-123456789012)
Settings to enable right away
- Block Public Access: ON (all four toggles).
- Versioning: ON (lets you recover bad parses).
- Default encryption: SSE-KMS (customer-managed key) for auditability.
- Object Ownership: Bucket owner enforced (disables ACLs, simplifies perms).
- Lifecycle rules:
- Move raw/ to infrequent access after N days.
- Expire tmp/ after M days.
- Keep processed/ longer for audits.
Folder layout (simple + scalable)

  s3://doc-insights-bucket/
    raw/               # uploads: PDFs, images
    textract-json/     # raw JSON from Textract (per doc)
    extracted/         # normalized tables/kv (CSV/Parquet)
    summaries/         # Bedrock text summaries (JSON/MD)
    processed/         # final CSVs for BI
    logs/              # app / pipeline logs

Edge cases
- Big uploads / flaky networks: use multi-part upload (SDK/CLI handles this).
- PII/PHI compliance: restrict bucket to VPC endpoints (below), rotate KMS keys, and enable CloudTrail data events for S3.
- Cross-account access: prefer bucket policy + role assumption over sharing long-lived keys.

3) IAM roles & least-privilege policies

Create three roles (one per workload) and scope them tightly:

a) Textract role (used by your Lambda/ETL or notebook)

Trusted entity: lambda.amazonaws.com (or your compute service)
Inline policy (example)

{
  "Version": "2012-10-17",
  "Statement": [
    {"Effect":"Allow","Action":["textract:AnalyzeDocument","textract:StartDocumentAnalysis","textract:GetDocumentAnalysis"],"Resource":"*"},
    {"Effect":"Allow","Action":["s3:GetObject","s3:PutObject"],"Resource":["arn:aws:s3:::doc-insights-bucket/*"]},
    {"Effect":"Allow","Action":["s3:ListBucket"],"Resource":["arn:aws:s3:::doc-insights-bucket"]},
    {"Effect":"Allow","Action":["kms:Encrypt","kms:Decrypt","kms:GenerateDataKey","kms:DescribeKey"],"Resource":"arn:aws:kms:REGION:ACCOUNT:key/KEY_ID"}
  ]
}

Use Start/Get APIs for async jobs (multi-page PDFs); AnalyzeDocument is fine for small/sync runs.

b) Bedrock role (for summarization)

{
  "Version":"2012-10-17",
  "Statement":[
    {"Effect":"Allow","Action":["bedrock:InvokeModel","bedrock:InvokeModelWithResponseStream"],"Resource":"*"},
    {"Effect":"Allow","Action":["s3:GetObject","s3:PutObject"],"Resource":["arn:aws:s3:::doc-insights-bucket/*"]},
    {"Effect":"Allow","Action":["kms:Encrypt","kms:Decrypt","kms:GenerateDataKey","kms:DescribeKey"],"Resource":"arn:aws:kms:REGION:ACCOUNT:key/KEY_ID"}
  ]
}

Limit Resource to the specific model ARNs once you pick them (Claude, Mistral, etc.).

c) QuickSight service role access to S3

Inside QuickSight → Admin → Security & permissions, grant access to your bucket. Backing IAM should allow:

{
  "Version":"2012-10-17",
  "Statement":[
    {"Effect":"Allow","Action":["s3:GetObject"],"Resource":["arn:aws:s3:::doc-insights-bucket/processed/*"]},
    {"Effect":"Allow","Action":["s3:ListBucket"],"Resource":["arn:aws:s3:::doc-insights-bucket"]}
  ]
}

Edge cases
- SSE-KMS blocks reads: add the QuickSight role principal to your KMS key policy.
- Cross-account S3: add a bucket policy that allows the QuickSight account role + use a manifest file.

4) KMS key policy that won’t bite you later

Add all compute roles and the QuickSight role to the key policy (not just IAM perms), e.g.:

{
  "Version":"2012-10-17",
  "Statement":[
    {"Sid":"EnableRoot","Effect":"Allow","Principal":{"AWS":"arn:aws:iam::ACCOUNT:root"},"Action":"kms:*","Resource":"*"},
    {"Sid":"AllowUseOfKey","Effect":"Allow","Principal":{"AWS":[
      "arn:aws:iam::ACCOUNT:role/TextractRole",
      "arn:aws:iam::ACCOUNT:role/BedrockRole",
      "arn:aws:iam::ACCOUNT:role/service-role/QuickSight-ROLE"
    ]},"Action":["kms:Encrypt","kms:Decrypt","kms:GenerateDataKey","kms:DescribeKey"],"Resource":"*"}
  ]
}

Edge cases
- AccessDenied (kms:Decrypt) in QuickSight/Athena: 99% of the time it’s the key policy missing the service role.

5) Networking (private, compliant, and fast)

Create VPC interface endpoints (PrivateLink) for:
- S3 (Gateway endpoint), Textract, Bedrock, STS, Logs, etc.
Lock your compute (Lambda/ECS/EC2) to no-internet and route AWS traffic via endpoints.
Edge cases
- QuickSight can’t sit in your VPC: it’s managed; keep the S3 bucket policy open to QuickSight role (not public) and allow KMS for that role.
- Corporate egress proxy: verify Bedrock/Textract endpoints are reachable or use endpoints.

6) Turn on Amazon Bedrock access (once per account/region)

In the Bedrock console, enable model access (Claude/Mistral etc.). Some models require an opt-in.
Edge cases
- Model not visible: you’re in a region without that model, or access not granted yet.
- Throughput errors: request service quota increases for InvokeModel if you batch summaries.

7) Enable Amazon QuickSight

Choose the same region as your S3 if possible.
In Security & permissions, tick your bucket and (if needed) Athena access.
Decide SPICE vs Direct Query:
- SPICE (in-memory) is faster; watch capacity limits.
- Direct is fine for small CSVs; for large data, prefer Parquet in S3 via Athena.
Edge cases
- CSV > size limit / too many rows: convert to Parquet and query via Athena.
- Timezone mismatches: set dataset time zone explicitly; normalize date formats (ISO-8601).

8) Budget, cost alerts, and audit

AWS Budgets: create a monthly budget with email alerts (S3 + Textract + Bedrock + QuickSight).
CloudTrail data events: ON for S3 (per-object audit).
CloudWatch Logs: centralize Lambda/app logs to logs/.
Edge cases
- Sudden spikes: enable S3 object-level CloudWatch metrics and add alarms on object count/size in raw/.

9) File formats & naming that make your life easy

Uploads (raw): accept PDF/JPEG/PNG/TIFF; normalize on ingest.
Outputs: prefer CSV (QuickSight-friendly) and/or Parquet (Athena-friendly).
Names: raw/{yyyy}/{mm}/{dd}/{vendor}/{fileid}.pdf processed/{yyyy}/{mm}/{dd}/invoices.csv
Edge cases
- Weird encodings: force UTF-8, escape delimiters, include header row.
- Duplicate filenames: use a UUID suffix or checksum in the key.

10) Optional: IaC and CLI quick start

Create bucket (CLI)

aws s3api create-bucket --bucket doc-insights-bucket --create-bucket-configuration LocationConstraint=$AWS_REGION
aws s3api put-bucket-encryption --bucket doc-insights-bucket --server-side-encryption-configuration '{
  "Rules":[{"ApplyServerSideEncryptionByDefault":{"SSEAlgorithm":"aws:kms","KMSMasterKeyID":"arn:aws:kms:REGION:ACCOUNT:key/KEY_ID"}}]}'
aws s3api put-bucket-versioning --bucket doc-insights-bucket --versioning-configuration Status=Enabled
aws s3api put-public-access-block --bucket doc-insights-bucket --public-access-block-configuration \
'{"BlockPublicAcls":true,"IgnorePublicAcls":true,"BlockPublicPolicy":true,"RestrictPublicBuckets":true}'

Example lifecycle (CLI)

aws s3api put-bucket-lifecycle-configuration --bucket doc-insights-bucket --lifecycle-configuration '{
  "Rules":[
    {"ID":"raw-to-ia","Filter":{"Prefix":"raw/"},"Status":"Enabled","Transitions":[{"Days":30,"StorageClass":"STANDARD_IA"}]},
    {"ID":"expire-tmp","Filter":{"Prefix":"tmp/"},"Status":"Enabled","Expiration":{"Days":7}}
  ]}'

📄 Step 2: Use Amazon Textract to Extract Text from Documents (Production-grade)

Textract can do three things you’ll care about:

OCR (lines/words) → DetectDocumentText
Generic forms & tables → AnalyzeDocument with FORMS, TABLES, optionally QUERIES
Invoice/receipt specialization → AnalyzeExpense (often best for invoices)

Rule of thumb

If it’s an invoice/receipt, try AnalyzeExpense first (it returns normalized fields like VENDOR_NAME, INVOICE_TOTAL).

If it’s a contract/form/report, use AnalyzeDocument with FORMS and TABLES.

Use QUERIES to target specific fields (e.g., “What is the due date?”) when layout varies wildly.

1) Sync vs Async: choose the right API

Synchronous (single call; immediate result)
- AnalyzeDocument (good for single-page or small docs; great for tests)
- DetectDocumentText (basic OCR)
Asynchronous (submit job → poll or page through results)
- StartDocumentAnalysis + GetDocumentAnalysis
- Required for multi-page and larger PDFs/images
- Supports pagination via NextToken

Guidance: Use sync for quick console/dev tests or one-pagers; use async for anything real (multi-page, batch).

2) Pre-processing (massively improves accuracy)

Input formats: PDF, PNG, JPEG, TIFF work well; keep scans clean.
Quality: Aim for readable text (good contrast, minimal blur). If possible:
- Deskew, denoise, increase contrast, convert to grayscale/PNG.
- Fix rotation (many scanners rotate by 90° or 180°).
Encrypted or password-protected PDFs: decrypt before upload.
Color: Grayscale often beats noisy full-color scans.
Multi-page splits: If you hit page/file limits, split before upload.
File names: Include vendor/date hints in the key to simplify downstream routing (e.g., raw/2025/09/11/vendorX/invoice_123.pdf).

3) Minimal but robust code paths (Python/boto3)

A. Synchronous: quick test for forms + tables

import boto3

textract = boto3.client("textract", region_name="eu-central-1")

resp = textract.analyze_document(
    Document={"S3Object": {"Bucket": "doc-insights-bucket", "Name": "invoice1.pdf"}},
    FeatureTypes=["FORMS", "TABLES"]
)

blocks = resp.get("Blocks", [])
# → Save raw JSON, parse KEY_VALUE_SET/TABLE blocks later

B. Asynchronous: production for multi-page docs (handles pagination + retries)

import boto3, time, itertools

textract = boto3.client("textract", region_name="eu-central-1")

job = textract.start_document_analysis(
    DocumentLocation={"S3Object": {"Bucket": "doc-insights-bucket", "Name": "long-invoice.pdf"}},
    FeatureTypes=["FORMS","TABLES"]
)
job_id = job["JobId"]

def get_all_pages(job_id, sleep=2, max_wait_s=900):
    start = time.time()
    pages = []
    next_token = None

    # Wait for completion (simple poll)
    status = "IN_PROGRESS"
    while status == "IN_PROGRESS":
        res = textract.get_document_analysis(JobId=job_id, NextToken=next_token) if next_token \
              else textract.get_document_analysis(JobId=job_id)
        status = res["JobStatus"]
        if status == "IN_PROGRESS":
            # If first page says in progress, wait; otherwise collect & keep paging
            if "Blocks" in res:
                pages.append(res)
                next_token = res.get("NextToken")
                if not next_token:
                    time.sleep(sleep)
            else:
                time.sleep(sleep)
            if time.time() - start > max_wait_s:
                raise TimeoutError("Textract job took too long.")
        elif status == "SUCCEEDED":
            pages.append(res)
            while res.get("NextToken"):
                res = textract.get_document_analysis(JobId=job_id, NextToken=res["NextToken"])
                pages.append(res)
        else:
            raise RuntimeError(f"Textract job failed: {res.get('StatusMessage', status)}")
    return pages

pages = get_all_pages(job_id)
# Combine blocks across pages for parsing
all_blocks = list(itertools.chain.from_iterable(p["Blocks"] for p in pages))

C. Invoices/Receipts: specialized `AnalyzeExpense` (often higher accuracy on totals/dates)

import boto3

textract = boto3.client("textract", region_name="eu-central-1")

resp = textract.analyze_expense(
    Document={"S3Object": {"Bucket": "doc-insights-bucket", "Name": "invoice1.pdf"}}
)

# resp["ExpenseDocuments"][0]["SummaryFields"] includes types like:
# VENDOR_NAME, INVOICE_RECEIPT_DATE, INVOICE_RECEIPT_ID, TOTAL, DUE_DATE

D. Targeted fields with `QUERIES` (when layouts vary)

import boto3
textract = boto3.client("textract", region_name="eu-central-1")

resp = textract.analyze_document(
    Document={"S3Object": {"Bucket": "doc-insights-bucket", "Name": "invoice1.pdf"}},
    FeatureTypes=["QUERIES","TABLES"],  # you can mix with FORMS too
    QueriesConfig={
        "Queries": [
            {"Text": "What is the invoice number?", "Alias": "invoice_no"},
            {"Text": "What is the total amount?", "Alias": "total_amount"},
            {"Text": "What is the due date?", "Alias": "due_date"}
        ]
    }
)

You can run QUERIES + TABLES together to pull exact fields and still capture line items.

4) How to parse Textract blocks safely (just enough for Step 2)

Textract represents a graph of Blocks with Relationships. The key ones:

PAGE, LINE, WORD
KEY_VALUE_SET (with EntityTypes: KEY or VALUE)
TABLE → CELL (cell has row/col indices)
QUERY → ANSWER

Parsing outline (KV):

Build a dict of blocks by Id.
Collect all KEY_VALUE_SET blocks.
For each KEY block, follow Relationships to its VALUE block.
Extract text by concatenating child WORD blocks; track Confidence.
Normalize keys (vendor, invoice no, etc.) and keep the BoundingBox for traceability.

Parsing outline (TABLE):

Find TABLE blocks, get their child CELLs.
Use (RowIndex, ColumnIndex) to reconstruct rows.
Heuristically detect headers (first row or bold-ish/upper-case hints if present in text).
Output as CSV; preserve page number.

We’ll do the full CSV/JSON conversion in Step 3. Here we only ensure you’ve got clean raw JSON saved and a clear parser plan.

5) Storage & idempotency

Always save raw responses (one file per source doc):
- textract-json/<doc-id>.json (for AnalyzeDocument/DetectDocumentText)
- textract-json/<doc-id>.expense.json (for AnalyzeExpense)
Add metadata to input S3 objects (e.g., processed=true, textract_job_id=...) to avoid reprocessing.
Use S3 object ETag/size as a cheap duplicate detector.

6) Confidence, validation & fallbacks

Confidence thresholds: treat fields under a threshold (e.g., < 0.8) as “needs review.”
Numeric sanity checks: parse amounts with a currency regex, verify:
- sum(line_items) ≈ total (± a few cents, taxes/discounts considered)
- date is valid & plausible (e.g., due date ≥ invoice date)
Disambiguation: fight common OCR confusions:
- O↔0, l/I↔1, S↔5, B↔8
Fallback order for invoices:

AnalyzeExpense
AnalyzeDocument (FORMS+TABLES)
DetectDocumentText + regex heuristics (as a last resort)
- Human-in-the-loop: flag low-confidence docs to a review queue (CSV row with a needs_review=1 flag).

7) Permissions & security edge cases (most common causes of failure)

S3: role needs s3:GetObject, s3:ListBucket, and if encrypted, kms:Decrypt on the CMK.
KMS key policy: include the calling role principal (not just IAM permissions).
Cross-account: use bucket policy + role assumption; ensure KMS allows the assumable role’s principal.
Private networking: if using VPC-only compute, add VPC endpoints for Textract/S3/STS/Logs.

8) Scaling, throughput, and cost control

Batching: drive Textract from a queue (SQS) or Step Functions; cap concurrency to avoid throttling.
Retries: exponential backoff on ThrottlingException and transient network errors.
De-dup: check if a doc was already processed (S3 metadata) before calling Textract again.
Specialization saves money: AnalyzeExpense or QUERIES can reduce post-processing and re-runs.

9) Troubleshooting cheat-sheet

Symptom	Likely cause	Fast fix
`AccessDenied` on S3 or KMS	Missing `s3:GetObject` / `kms:Decrypt` or KMS key policy excludes role	Add actions to role and role principal to KMS key policy
Job stuck “IN_PROGRESS” forever	Not polling pages / ignoring `NextToken`	Implement full pagination loop until `NextToken` is empty
Empty or low-quality text	Bad scans (blur, skew), wrong rotation, low contrast	Preprocess image: deskew, increase contrast, convert to PNG, rotate correctly
Fields missing on invoices	Using generic forms instead of expense API	Switch to `AnalyzeExpense` or add `QUERIES`
Dates/amounts parsed wrong	Locale/regex issues or OCR confusions	Normalize formats (ISO-8601), add currency/number regex, apply character-swap heuristics
QuickSight can’t see outputs	Using KMS + missing QS access	Add QS role to bucket policy and KMS key policy

10) Console workflow (great for validation)

Upload a sample to S3 → Textract Console → Analyze.
Toggle Forms + Tables, confirm highlights match expectations.
For invoices, test Expense analysis and compare field names (often cleaner).
Save the result JSON and move on to parsing (Step 3).

✏️ Step 3: Convert Textract Output to Structured Data (CSV or JSON)

1) Clean & normalize the extracted data

Keep originals: store the raw Textract JSON (read-only) so you can re-parse later without re-running Textract.
Whitespace & casing: trim, collapse double spaces; keep a _raw copy for each value if you’ll audit later.
Key normalization (FORMS/QUERIES): map variants to canonical keys:
- invoice no, inv #, invoice number → invoice_number
- total, grand total, amount due → total
- supplier, vendor, billed by → vendor_name
- invoice date, date → invoice_date; due date → due_date
Dates: convert to ISO-8601 (YYYY-MM-DD). If ambiguous/parsing fails, leave blank and flag needs_review.
Numbers & currency: strip thousands separators, unify decimals (.), extract currency symbol/code ($→USD, €→EUR, £→GBP, ₨→PKR, ₹→INR). If none, set a default and flag for review.
Confidence tracking: carry Textract confidence (0–100). Anything below your threshold (e.g., 80) sets needs_review = 1.
OCR confusions: apply gentle corrections when context demands numeric values (O↔0, I/l↔1, S↔5, B↔8) but still keep needs_review if confidence was low.

2) Choose your outputs

CSV (analytics-ready):
- KV (long format): doc_id, page, key_norm, value, confidence, needs_review
- Tables (per page/table): doc_id, page, row, col, text, confidence, is_header
- Invoice summary (if using Expense API): one row per document with normalized fields
JSON (traceability): keep structured JSON when you need bounding boxes/page indices for audits.
(Optional) Parquet: for Athena/QuickSight at scale.

3) Dump key–value pairs to CSV

What to capture: each detected pair as one row in a long table:
- doc_id, page, key_raw, key_norm, value, conf_key, conf_val, needs_review
Duplicates: keep all candidates. Later, pick the highest confidence (ties → closest to the label “Total”, latest on the page, or appears in a summary box).
Missing values: if a KEY has no VALUE, record value = "" and set needs_review = 1.
Multi-line values: join lines with a single space; preserve original in value_raw if you keep audits.

4) Convert tables from JSON to CSV

Reconstruct using cells: one row per cell with indexes:
- doc_id, page, row, col, text, confidence, span_rows, span_cols, is_header
Headers: default to row 1 = header; mark is_header = 1. If row 1 has poor confidence but contains many alphabetic tokens vs numeric rows below, still treat as header.
Merged cells: repeat the merged text in the spanned area and record span_rows/span_cols for fidelity.
Multi-page tables: keep separate by page; your business logic decides whether to stitch them later.

5) Build the simple “wide” table for your dashboard

Target columns from your outline:

Invoice No (`invoice_number`)	Date (`invoice_date`)	Vendor (`vendor_name`)	Amount (`total`)	Due Date (`due_date`)
INV-123	2025-08-03	Alpha Supplies	423.90	2025-09-02

Population priority:

Expense API (if used): INVOICE_RECEIPT_ID, INVOICE_RECEIPT_DATE, VENDOR_NAME, TOTAL, DUE_DATE.
FORMS/QUERIES: after key normalization and numeric/date parsing.
TABLES/OCR fallback: totals row or summary box heuristics (e.g., last column named Amount, Total, Balance).

Also add helpful derived fields:

currency, source (expense/forms/ocr), needs_review, month, year.

6) Quality & integrity checks

Arithmetic sanity: if you have subtotal and tax, verify subtotal + tax ≈ total (small tolerance for rounding). If not, needs_review = 1.
Date logic: due_date ≥ invoice_date; otherwise needs_review = 1.
Negatives & zeros: total must be > 0 for invoices; credits/notes may be negative tag with doc_type = credit_note.
Duplicates: same invoice_number appearing across different doc_ids → keep the one with higher composite score (weighted confidence of critical fields) and mark others as superseded.
Two documents in one file: if you detect two distinct vendors or invoice numbers in one doc_id, set multi_doc_suspected = 1 and route to review.
Locale drift: month names/currency symbols outside your default locale → still parse but set locale_detected and needs_review if parsing relied on heuristics.
Handwritten/low DPI scans: values may be fragmented; prefer QUERIES for targeted fields and flag low confidence.

7) Foldering

textract-json/            # raw API outputs (never edited)
extracted/
  kv/                     # <doc_id>_kv.csv
  tables/                 # <doc_id>_table_<page>.csv
  expense/                # <doc_id>_summary.csv, <doc_id>_line_items.csv (if used)
processed/
  invoices_wide.csv       # your dashboard input (or Parquet partitioned by year/month)

🧠 Step 4: Summarise the Document Using Bedrock

1) What you’ll summarise

Feed Bedrock only what it needs:

Facts you already extracted in Step 3 (invoice_number, vendor_name, total, due_date, currency, etc.).
Top signal text: page headers, totals area, vendor/address block, and (for invoices) the line-items table flattened to a few lines (don’t paste thousands of rows).
Optional full text: chunked by page if you need narrative context.

Redact or truncate PII that isn’t needed (addresses beyond city/country, phone, full account numbers). Keep the doc_id so you can trace back.

2) Output format

Ask Bedrock for structured JSON + a short narrative. Example JSON schema:

{
  "doc_id": "string",
  "summary": "1–3 sentence human summary",
  "key_fields": {
    "vendor_name": "string|null",
    "invoice_number": "string|null",
    "invoice_date": "YYYY-MM-DD|null",
    "due_date": "YYYY-MM-DD|null",
    "currency": "string|null",
    "total_amount": "number|null"
  },
  "observations": [
    "string"  // anomalies, discounts, late fees, notes
  ],
  "data_quality": {
    "missing_fields": ["string"],
    "suspected_errors": ["string"],
    "confidence_note": "string"
  }
}

This slots straight into your processed/ layer or QuickSight as a secondary dataset.

3) Prompting

Use a system + user style prompt; instruct the model not to invent data.

System

You are a careful financial document assistant. Use ONLY the supplied text and fields.
If a field is missing or unclear, return null and add a helpful note in data_quality.
Output must be valid JSON matching the provided schema. Do not include any other text.

User

DOC_ID: {{doc_id}}

FACTS (from extraction):
- vendor_name: {{vendor_name}}
- invoice_number: {{invoice_number}}
- invoice_date: {{invoice_date}}
- due_date: {{due_date}}
- currency: {{currency}}
- total_amount: {{total}}

EXCERPTS (trimmed; page-tagged):
[PAGE 1]
{{top_of_page_text}}
[PAGE 2 - totals box]
{{totals_area_text}}
[OPTIONAL line items]
{{few_key_lines}}

TASK:
1) Produce a JSON object exactly matching this schema:
{{paste schema}}
2) Base fields on FACTS when available; only use EXCERPTS to confirm or fill blanks.
3) If numbers disagree, prefer FACTS but note the discrepancy in `data_quality.suspected_errors`.
4) Do not guess. Use `null` for unknowns.

Why this works

It anchors the model to your extracted fields (reduces hallucinations).
It forces a predictable JSON payload for downstream use.
It adds a quality channel for “known unknowns”.

4) Model choice & params

Claude (Anthropic): strong on instruction following & JSON fidelity.
Mistral: concise, fast; good when payloads are small and schema is simple.
Parameters: keep temperature low (≈0.0–0.2) for factual stability; set max tokens tight (e.g., 600–900) for cost control.

Tip: If JSON validation fails sometimes, wrap with a “fix to valid JSON” pass (or enable JSON mode where available).

5) Long documents or Map/Reduce

For multi-page or very verbose docs:

Map step: summarize each page (fixed schema, plus page-level anomalies).
Reduce step: feed all page summaries (+ your Step-3 facts) into a second prompt that merges into a single final JSON, breaking ties in favor of Step-3 facts.

This avoids token limits and preserves page context.

6) Robust invocation

Use bedrock-runtime with the model you enabled (ids vary by region/provider). Two common patterns:

Non-streaming (simple)

import json, boto3, time
brt = boto3.client("bedrock-runtime", region_name="YOUR_REGION")

def invoke_bedrock(model_id, system_prompt, user_prompt, max_tokens=800, temperature=0.1):
    body = {
        "anthropic_version": "bedrock-2023-05-31",
        "messages": [
            {"role": "system", "content": system_prompt},
            {"role": "user", "content": user_prompt}
        ],
        "max_tokens": max_tokens,
        "temperature": temperature
    }
    resp = brt.invoke_model(modelId=model_id, body=json.dumps(body))
    out = json.loads(resp["body"].read())
    # Extract the text depending on provider; for Claude:
    text = "".join([c.get("text","") for m in out.get("content",[]) for c in [m]])
    return text

Streaming

Stream when you want low latency for UI, otherwise skip (adds complexity).

Note: Exact request/response shapes differ by model family. Keep the modelId and payload format configurable and test in the Bedrock console first.

7) Validation & safety rails

JSON validation: run a quick schema check (pydantic/jsonschema). If invalid, re-prompt the model with: “Return only valid JSON conforming to this schema. Do not add commentary.”
No-invention rule: instruct “If unsure, use null and add a data_quality note.” Penalize hallucinations by rejecting outputs that add fields you didn’t request.
Discrepancy handling: when EXCERPTS disagree with FACTS, keep FACTS and add a note like: “Totals area shows 432.90 but extracted total is 423.90 (difference 9.00).”
PII minimization: don’t pass line-item text unless needed; mask account numbers (e.g., ****1234).
Cost control: truncate excerpts, avoid full dumps; cap max_tokens; batch summarize.

8) Edge cases

Edge case	What to do in prompt / output
Missing total or due date	Return `"total_amount": null` / `"due_date": null`; add a `data_quality.missing_fields` entry.
Multiple totals (invoice vs balance)	Prefer FACTS; if absent, pick the one labeled “Total” near summary; note the alternative.
Conflicting currency symbols vs codes	Prefer the currency from FACTS; if excerpt differs, note discrepancy.
Handwritten / low DPI	Expect nulls and notes; don’t interpolate from context.
Multi-currency line items	Set currency to document currency; add observation like “Line items include mixed currency not used for total.”
Multi-doc PDF	If two vendors or two invoice numbers appear, say so in `observations` and add `suspected_errors`.
Non-English docs	Add “Output in English.” Keep dates numeric (YYYY-MM-DD).
Extremely long inputs	Use Map/Reduce; only pass per-page top areas and final consolidation.

9) Example “good” output

{
  "doc_id": "INV-2025-0045",
  "summary": "Invoice 0045 from Beta Traders for 89.50 EUR, due on 2025-08-28.",
  "key_fields": {
    "vendor_name": "Beta Traders",
    "invoice_number": "0045",
    "invoice_date": "2025-08-14",
    "due_date": "2025-08-28",
    "currency": "EUR",
    "total_amount": 89.50
  },
  "observations": [
    "Early-payment discount mentioned (2%)."
  ],
  "data_quality": {
    "missing_fields": [],
    "suspected_errors": [],
    "confidence_note": "All fields aligned with extracted facts."
  }
}

10) Where to store results

Write JSON summaries to summaries/ and (optionally) a compact CSV view processed/invoice_summaries.csv with doc_id, vendor_name, total_amount, due_date, needs_review, source='bedrock'.
Join this dataset with invoices_wide.csv for dashboard overlays (e.g., a tooltip with the narrative summary).

🗂️ Step 5: Store Final Data Back in S3

1) Folder layout

Use a layout that separates raw vs extracted vs consumable and supports BI tools:

s3://doc-insights-bucket/
  processed/
    wide/                 # final, dashboard-ready “wide” tables (CSV/Parquet)
    kv/                   # merged KV outputs across docs
    tables/               # flattened tables across docs
    expense/              # invoice summaries & line items (if using AnalyzeExpense)
    summaries/            # Bedrock JSON summaries
    parquet/              # same datasets in Parquet (for Athena/QuickSight at scale)
    _staging/             # temp writes (see atomic pattern)
    manifests/            # optional manifest/index files

Partitioning (for Athena/scale):

Prefer hive-style partitions for Parquet:
- processed/parquet/invoices_wide/year=2025/month=09/day=12/part-0001.parquet
Keep CSVs unpartitioned or lightly partitioned by month to avoid too many small files.

2) Naming strategy

Document-scoped files: processed/expense/summary/<doc_id>.csv processed/summaries/<doc_id>.json
Batch/append files: processed/wide/invoices_wide_YYYYMMDD.csv (or append into a single table and de-dupe by doc_id)
Timestamped artifacts: add an ISO timestamp or a monotonically increasing batch id:
- .../2025-09-12T18-40-12Z_invoices_wide.csv
Avoid collisions: include doc_id (or a UUID) in per-doc artifacts; never rely only on original filenames.

3) Atomic write pattern

S3 has strong read-after-write consistency, but clients can still read half-written objects if your writer crashes mid-upload.

Pattern:

Write to processed/_staging/<uuid>/<file.tmp> (or status=writing tag).
Verify checksum/row count.
Copy to final key (e.g., processed/wide/invoices_wide_20250912.csv).
Optionally delete the staging file, or keep with tag status=archived.

Edge cases:

Large files (>5 GB): use multipart upload; always send Content-MD5 for integrity when feasible.
Concurrent writers: add a lightweight lock (S3 object processed/_locks/invoices_wide.lock with short TTL) or write per-writer partitions then compact later.

4) Metadata, tags, and headers

Server-side encryption: SSE-KMS with your CMK (consistent with earlier steps).
Content-Type: set correctly (text/csv, application/json, application/parquet).
Cache-Control: not crucial for data lakes; you can omit or set no-cache for rapidly changing CSVs.
Object tags (helpful for governance/search):
- dataset=invoices, stage=processed, doc_id=<id>, status=ready, source=expense/forms/ocr/bedrock.
Custom metadata (e.g., x-amz-meta-rowcount, x-amz-meta-hash): useful for quick validation.

5) Security & access

Bucket policy: already denies insecure transport; also allow the QuickSight service role to ListBucket and GetObject on processed/*.
KMS key policy: include the QuickSight role principal (not just IAM permissions) so it can decrypt.
Cross-account reads (optional): prefer role assumption + bucket policy + KMS grants.
VPC endpoints: if compute is private, ensure endpoints for S3/KMS are present.

Edge cases:

QuickSight can list but can’t read → key policy is missing the role principal.
Using SPICE: watch dataset size limits; switch to Athena over Parquet when CSVs grow.

6) Data integrity & quality gates

Before moving from _staging → processed:

Schema check: headers exist and match expected column set.
Row count > 0 (or allow zero with a note=empty_batch tag).
Type validation: dates parse (ISO-8601), amounts numeric, currency present.
Sanity checks: if files include subtotal, tax, total, verify subtotal + tax ≈ total (tolerance).
Checksum: compute SHA-256 of the file and store as sidecar:
- processed/wide/invoices_wide_20250912.csv.sha256

7) Lifecycle & retention

Versioning: keep enabled to roll back erroneous overwrites.
Lifecycle rules:
- Transition processed/_staging/ to Expire after 7 days.
- Transition processed/*.csv to STANDARD_IA after 30 days.
- Keep Parquet in STANDARD (or GLACIER if archived) based on access.
S3 Inventory (optional): enable for daily listings & encryption audit.

8) Manifests & discovery (helps BI tools)

Manifest file lists the latest batch outputs:

  {
    "dataset": "invoices_wide",
    "updated_at": "2025-09-12T18:40:12Z",
    "files": [
      "s3://.../processed/wide/invoices_wide_20250912.csv"
    ],
    "row_count": 1243,
    "schema": ["doc_id","vendor_name","invoice_number","invoice_date","due_date","total","currency","needs_review","source","month","year"]
  }

Save as processed/manifests/invoices_wide.latest.json.

Glue Catalog (for Athena): create/update a table pointing to your Parquet path with partitions year, month, day.

9) Eventing

S3 → EventBridge rule on processed/ prefix:
- Trigger a Lambda to refresh QuickSight (or kick an ingestion SPICE refresh).
- Or publish a Slack/Email notification with the manifest summary.
Idempotency: include run_id or batch_id in event detail; ignore duplicates in the consumer.

📊 Step 6: Create a Dashboard in QuickSight

1) Pick your connection path

Path A | S3 CSV (fast start)

Best for small/medium datasets (≤ a few million rows per file).
In QuickSight: Datasets → New dataset → S3.
Point to a single CSV or a manifest JSON (recommended if you have many files).

Path B | Athena + Parquet (scales)

Best for large data, partitions, or frequent refresh.
Store Parquet at processed/parquet/invoices_wide/year=YYYY/month=MM/day=DD/...
In QuickSight: Datasets → New dataset → Athena → select database & table (from Glue Catalog).

Edge cases

If QuickSight is in a different region than your S3 bucket, prefer Athena to avoid cross-region pain.

Using SSE-KMS? Ensure the QuickSight service role is in your KMS key policy and bucket policy.

2) Connect to your data

A) S3 with a manifest (many files)

Create processed/manifests/invoices_wide.latest.json:

{
  "fileLocations": [
    {"URIPrefixes": ["s3://doc-insights-bucket/processed/wide/"]}
  ],
  "globalUploadSettings": {
    "format": "CSV",
    "delimiter": ",",
    "textqualifier": "\"",
    "containsHeader": true
  }
}

Then in QuickSight S3 connector, paste this manifest file’s S3 URL.

B) Athena (Parquet partitions)

Make sure your Glue table points to processed/parquet/invoices_wide/ and has partitions year, month, day.
In QuickSight, choose Direct Query (live) or Import to SPICE (in-memory; faster but size-limited).

Edge cases

“AccessDenied” when previewing = missing KMS permission or bucket grant.

No new data after partition upload = run crawler/add partition OR enable partition projection.

3) Prepare the dataset

In the data prep screen:

Map columns

invoice_number → Text
invoice_date, due_date → Date (format YYYY-MM-DD)
total → Decimal
vendor_name → Text
currency → Text
needs_review → Integer/Boolean
(Optional) ingest_date or processed_at → Date/Time

Create calculated fields (QuickSight expressions)

Month (bucket) truncDate('MM', {invoice_date})
Year extract('YYYY', {invoice_date})
Overdue (flag) ifelse({due_date} < now(), 1, 0)
Processing time (days) – pick the source you actually have:
- If you kept an ingest timestamp: dateDiff('DD', {invoice_date}, {ingest_date})
- Or from S3 key date (if you saved it): dateDiff('DD', {invoice_date}, {s3_ingest_date})
Clean total (fallback) – if some totals are text:

parseDecimal({total})

Edge cases

Mixed date formats? Normalize upstream; otherwise use a calculated field with parseDate() per pattern and coalesce.

Multiple currencies? Keep a currency dimension; optional FX table join later.

4) Build the visuals

A) Total amount by vendor (Top-N bar)

Visual: Horizontal bar
X: sum({total})
Y: vendor_name
Sort: by sum(total) desc
Filter: Top 10 vendors (add a Top N filter)
Optional: color by needs_review to spot messy vendors

B) Monthly invoice totals (trend)

Visual: Line chart
X: truncDate('MM', {invoice_date})
Y: sum({total})
Forecast (optional): enable forecast to project next months
Tooltip: add countDistinct({invoice_number}) for volume context

C) Average processing time (days)

Visual: KPI or bar by vendor
Field: avg(dateDiff('DD', {invoice_date}, {ingest_date}))
Group by: vendor_name (or monthly bucket)
Filter: exclude needs_review = 1 if you want only clean rows

D) Bonus quick wins

Overdue heatmap: vendor_name × month with color sum(ifelse({due_date} < now(), {total}, 0))
Quality gate: table filtered to needs_review = 1 (your manual review queue)

5) Filters, controls, and parameters

Date range control: Filter on {invoice_date} with “Relative dates” (e.g., Last 12 months).
Vendor control: Dropdown on vendor_name (multi-select).
Confidence/quality toggle: needs_review = 0 by default (checkbox control).
Currency control (if applicable): filter or parameter to select a single currency.

Edge cases

If your dataset mixes currencies, don’t sum across different codes in one visual unless you convert first.

6) SPICE vs Direct Query

SPICE (Import): fastest visuals, schedule refreshes (e.g., hourly). Watch capacity limits.
Direct Query: always live; best with Athena/Parquet; slightly slower per query.

Recommended

Start with SPICE for S3 CSVs; switch to Athena + SPICE as you scale.

7) Refresh strategy

S3 CSV path
- If you overwrite one key (e.g., invoices_wide_latest.csv), set a refresh schedule (hourly/daily).
- If you append new files, use a manifest and schedule refresh.
Athena path
- After adding partitions, either run a crawler or use partition projection.
- In QuickSight, schedule SPICE refresh or stay on Direct Query.

Edge cases

Refresh fails with KMS errors → add QuickSight role to CMK key policy.

Stale data → you’re writing to new paths; either point QS at a stable key or use manifests.

8) Sharing, RLS, and governance

Share analysis → Publish dashboard; grant users/groups access.
Row-Level Security (RLS): upload an RLS mapping dataset (user, vendor_name) to restrict rows per viewer.
Data dictionary tab: add field descriptions so the team knows what “processing time” means.

10) Minimal recipe

Connect

S3: dataset → S3 → manifest → validate preview.
Athena: dataset → Athena → select table.

Model

Types: map date/number fields.
Calcs: month = truncDate('MM', {invoice_date}), processing_days = dateDiff('DD', {invoice_date}, {ingest_date}).

Visualize

Bar: sum(total) by vendor_name (Top 10).
Line: sum(total) by month.
KPI/Bar: avg(processing_days) by vendor.

Control

Date range (last 12 months).
Vendor multi-select.
needs_review toggle (default 0).

Operationalize

SPICE refresh schedule (hourly/daily).
If Athena: ensure partitions & permissions.

💭 Conclusion

This was honestly a fun little fortnight project.
It saved me hours of manual work and gave me clean, visual summaries of everything I needed. By combining S3, Textract, Bedrock, and QuickSight, you can turn unstructured, scanned documents into analysis-ready data and clear dashboards. This pipeline handles extraction, normalisation, summarisation, governed storage, and visualisation, built with security (SSE-KMS), least-privilege IAM, and scalable patterns (async Textract, Parquet/Athena). It replaces manual review with consistent, auditable automation and surfaces vendor spend, monthly trends, and processing KPIs in minutes. From here, you can harden orchestration with Step Functions, add Row-Level Security in QuickSight, and extend the same approach to receipts, contracts, or reports. Small lift-big leverage.

You could easily apply this to:

Legal contracts
Academic papers
Medical prescriptions
Tax docs

If you try it out and get stuck, drop a comment, and I'd be happy to help.

HappyHacking 👋

🚀 Scaling React Like Big Tech: Folder Structures, Clean Code & Beyond

Mursal Furqan Kumbhar — Sat, 30 Aug 2025 18:55:22 +0000

Hey Everyone 👋

I know it's been quite a long gap since my last article. But we are finally back with a bang.

So if you’ve ever wondered how Airbnb, Shopify, or Meta keep their codebases sane while thousands of engineers ship features daily — the answer is simple: structure + discipline + automation.

This isn’t about writing code that just works. It’s about writing code that:

Scales with your team.
Survives onboarding of 10 new devs.
Passes reviews without a hundred nitpicks.
And doesn’t make you cry when you open it six months later.

Let’s break down what the big companies do differently — and how you can bring those practices into your own projects.

📂 Folder Structure that Scales

Top companies don’t just throw files into a random components/ folder and hope for the best. They use feature-driven architecture, which grows naturally with the product.

❌ Traditional (Bad) Structure

src/
├── components/
│   ├── Button.jsx
│   ├── Card.jsx
│   ├── Form.jsx
├── services/
├── hooks/
└── App.jsx

This looks fine at first… until your app has 300+ components and no one knows which belongs to what.

✅ Scalable (Feature-Based) Structure

src/
├── features/
│   └── auth/
│       ├── components/
│       ├── hooks/
│       └── services/
├── shared/
│   ├── ui/
│   └── utils/
└── App.jsx

📌 Key Difference: Everything related to a single feature (Auth, Products, Cart, etc.) is grouped. Shared logic/components live in shared/.

Approach	Pros	Cons
Traditional (by type)	Simple to start	Becomes chaos at scale
Feature-Based	Easy ownership, scalable, testable	Slightly more setup upfront

🎨 Design Systems & UI Libraries

Big companies don’t reinvent the button every Tuesday. They use design systems and UI libraries.

🚀 Airbnb → Lona
🚀 Shopify → Polaris
🚀 Meta → Reusable UI + strict guidelines

👉 Tools you can use: Chakra UI, Tailwind + Headless UI, Radix UI

Visual Example

❌ Without Design System:

<Button variant="danger" color="red" bg="red" />

✅ With Design System:

<DeleteButton />

Why Use Design Systems?	Benefits
Consistency	UI looks the same across app
Speed	Devs move faster
Accessibility	Built-in ally support
Brand Identity	Product feels cohesive

🧑‍💻 Hooks = Where Logic Lives

Business logic doesn’t belong inside components. Custom hooks make code reusable, testable, and clean.

Example:

// useProduct.js
export function useProducts() {
   const [data, setData] = useState([])
   useEffect(() => {
      fetchProducts().then(setData)
   }, [])
   return data
}

✅ UI stays declarative.
✅ Logic stays modular.

📊 Diagram: Components vs Hooks

[Component] ---> UI (render)
       |
       v
 [Custom Hook] ---> Data Fetching, State, Business Logic

🧼 Clean Code = Scalable Code

Big tech engineers follow clean code principles:

SRP → Single Responsibility Principle
KISS → Keep It Simple, Stupid
DRY → Don’t Repeat Yourself

❌ Bad	✅ Good
`<Button variant="danger" color="red" bg="red" />`	`<DeleteButton />`

✅ Abstraction = Fewer Bugs + Cleaner Reviews

🔍 Real-Time Code Reviews + Linting

At big companies, automation + reviews are mandatory.

✅ Code Reviews → shared knowledge
✅ Pre-Commit Hooks (Husky) → catch errors early
✅ ESLint + Prettier → consistent style

CI/CD Pipeline Visual

[Developer Pushes Code] 
       |
       v
[Pre-Commit Checks: Lint + Tests]
       |
       v
[Pull Request Review]
       |
       v
[CI/CD: Build + Deploy + Monitor]

🔐 Security & ♿ Accessibility First

Security isn’t optional. Accessibility isn’t either.

Security Checklist

Prevent XSS and CSRF
Proper token handling
Input validation

Accessibility Checklist

Keyboard navigation works
Screen reader support
High color contrast

👉 Use tools like axe-core, eslint-plugin-jsx-ally, lighthouse.

⚡ Automation + Monitoring

Big teams don’t just ship code — they ship pipelines + monitoring.

✅ CI/CD Pipelines (GitHub Actions, Vercel, CircleCI)
✅ Bundle Size Alerts
✅ Error Tracking (Sentry, Datadog)
✅ Performance Monitoring (Lighthouse CI)

📊 Example:

Commit → Build → Test → Deploy → Monitor

📊 Documentation & Knowledge Sharing

What separates side projects from real companies?
👉 Documentation.

Good README.md = contributor-friendly.
Internal wikis = no lost knowledge.
ADRs (Architecture Decision Records) = explain why choices were made.

🧠 Culture = The Invisible Architecture

Finally — even the best code structure won’t save a bad culture. Big companies win because:

PRs = conversations, not fights.
Automation is embraced, not skipped.
Docs + testing = part of “done”.

At the end of the day, how your team codes together matters more than folder names.

🚦 Startup vs Big Tech Codebases

Here’s how things usually look in a scrappy startup vs how big tech organizes for scale:

┌────────────────────┐           ┌─────────────────────────┐
│   Startup Style    │           │     Big Tech Style      │
├────────────────────┤           ├─────────────────────────┤
│ 📂 components/     │           │ 📂 features/auth/       │
│ 📂 hooks/          │           │   ├── components/       │
│ 📂 services/       │           │   ├── hooks/            │
│ App.jsx            │           │   └── services/         │
│                    │           │ 📂 shared/ui/           │
│ Easy to start ✅   │           │ 📂 shared/utils/        │
│ Chaos later ❌      │           │ Scales across teams ✅  │
└────────────────────┘           └─────────────────────────┘

📊 Side-by-Side Table

Aspect	Startup-Style	Big Tech-Style
Folder Structure	Flat, all components in one folder	Feature-based, modular
UI	Custom each time	Design system + UI library
Logic	Inside components	Extracted into hooks
Code Quality	Quick fixes, less abstraction	Clean code, reusable patterns
Reviews	Ad-hoc, sometimes skipped	Mandatory PRs + linting + tests
Security	Later concern	Built-in from day one
Docs	README only	Wikis + ADRs + onboarding guides
Culture	“Move fast, break things”	“Move fast, don’t break prod”

🎯 Final Takeaway

Big companies scale not because their engineers are magical unicorns 🦄, but because they follow battle-tested practices:

Organize code by features, not file types.
Use design systems for UI.
Extract logic into hooks.
Follow clean code principles.
Automate reviews + pipelines.
Prioritize security & accessibility.
Write documentation.
Build a team culture around consistency.
Startups often optimize for speed, but end up with technical debt.
Big Tech optimizes for scalability, making onboarding, reviews, and growth smooth.
The sweet spot? Start with startup speed, but gradually adopt Big Tech practices before chaos hits.