DEV Community: Julia Salem

Least Privilege Is a Losing Race for Agents

Julia Salem — Wed, 30 Sep 2026 13:30:00 +0000

Disclosure: I work at Eon, on the content side, and we sell what control 5 describes. The other five need no vendor, and control 5 carries a build-it-yourself path. The sources are OWASP, Microsoft, Anthropic's own numbers, Hugging Face's incident timeline and a founder's postmortem of losing his own database.

Restricting what an AI agent can reach is a losing race

You strip a permission on Monday, and an engineer with a blocked ticket grants it back by Thursday. You put a human approval in front of the dangerous actions, and by Anthropic's own count, that human says yes 97% of the time. And the one control that catches rm -rf on sight cannot see the API call that wipes a customer table, because to anything reading the traffic, the two look nothing alike.

Every published guide to agent security says the opposite. OWASP, Microsoft, GitHub, and both companies at the center of the July intrusion landed on some version of scope it tighter and watch it closely. All of it is worth doing, as long as it is not where you stop.

Recovery is the control everyone left out

OWASP calls the problem Excessive Agency: an agent with more functionality, permissions, or autonomy than its task needs. In the 2026 edition of its Top 10 for LLM Applications, published August 4, 2026, Excessive Agency jumped from sixth place to third, the biggest move on the list. The reason is blunt. OWASP checked its practitioner survey against 7,714 real incidents this year, and agentic deployments are where the damage is landing.

Source	Published	What it recommends
OWASP GenAI, Top 10 for LLM Apps 2026	Aug 4, 2026	Minimize tools and permissions
Microsoft Security, least privilege for agents	Jul 16, 2026	Dedicated identity per agent
GitHub Well-Architected, governing agents	Apr 13, updated Sep 9, 2026	Enterprise agent and MCP policies
Hugging Face, July intrusion timeline	Jul 27, 2026	Stricter isolation, short-lived credentials
OpenAI, Hugging Face incident report	Aug 26, 2026	More isolated sandboxes

The fuller versions run longer. OWASP also says to avoid open-ended tools like a shell and to require human approval for high-impact actions. Microsoft adds task-scoped roles, tool allowlists, just-in-time elevation and end-to-end audit logs. GitHub adds audit log streaming to a SIEM. Hugging Face adds narrower trust boundaries, blocked metadata access and faster detection. OpenAI adds restricted internet access and more compute on chain-of-thought monitoring.

Read the right-hand column. It is one idea in five vocabularies: keep the agent away from things, and notice fast when it gets near them. Recovery does not appear in any row.

OWASP would fairly answer that resilience lives in its other documents and that the Top 10 is scoped to application risk. The claim here is about emphasis, not error. Five documents on what to do about agents, and not one of them budgets a line for the day the controls fail.

Every control in that column runs before the write, and when the last of them fails, the only question left is whether you can put the data back.

Not one of these documents tells you how to put the data back, and from what we hear in customer conversations, that gap is now the biggest risk in the room.

Least privilege only works for so long

To be fair, everything in that table shrinks the surface an agent can damage. A dedicated identity per agent means you revoke one thing instead of rotating a shared secret, and task-scoped roles cut the blast radius of an honest mistake. Allowlisted tools keep an agent from discovering a capability nobody meant to hand it.

Nobody should stop doing this work. The argument is narrower: restriction cannot be the finish line, and the guidance published so far treats it as one. The open question is what happens once you finish scoping and the agents are still writing to production.

Every permission you strip makes the agent worse at the job you hired it for, so the pressure to loosen comes from inside the team. Microsoft's guidance describes the pattern without flinching. A team grants a broad Reader role because the use case looks read-only, the workflow expands to fixing what it finds, and instead of redesigning the role, the team grants something broader and moves on.

Anthropic measured the same erosion in Claude Code. As of June 2026, 49.5% of active Claude Code CLI users had written a rule telling the tool to stop asking before it runs shell commands. The share was growing about 5 percentage points every 5 weeks. Beyond those rules, 62% had switched Bash prompts off entirely or clicked "don't ask again," and 25% of interactive sessions started with permissions bypassed.

Those are developers on their laptops, where a bad command usually costs a git reset. Now apply the same erosion to the service account an agent uses against a production database. The person deciding whether to widen that role is usually the engineer the agent is blocking, and the role does not get narrower after that.

Bash is also the one channel the built-in undo in these tools does not cover, which we took apart in Claude Code's /rewind Won't Bring Back What Bash Deleted.

So the permissions come back, and the last line of defense is the approval prompt.

Human approval decays to 5% inside one session

Claude Code users reject 3% of individual permission prompts and 39% of the plans put in front of them, per Anthropic's August 7, 2026 data. People do say no when they actually read the prompt, but when dozens of prompts come up a day, they may start clicking "Approve" without reading.

Anthropic measured the decline in a controlled study of 1,053 paid professional testers. A research vendor swapped one permission prompt in each session for a clearly dangerous command. Early in a session, testers caught about 17%. After 50 or more prior prompts, the catch rate fell to about 5%. Nobody reads the fortieth prompt of the hour.

Swapping the human for a model does not close the gap

The obvious fix is to replace the tired human with a model, and Anthropic made its classifier the default on August 14. The classifier is better than the human and, by Anthropic's own account, still not good enough for production.

The March 25 engineering write-up reports that the classifier let through 17% of the real cases where the agent went further than the user asked. The August post still recommends that people review the agent's actions themselves for high-stakes changes to production infrastructure. The review Anthropic recommends for production is the one that caught 13.6% of dangerous commands in its own study.

And even a gate that never tires has a built-in blind spot.

The dangerous part of an API call is the part no gate reads

A classifier can catch rm -rf because the command announces itself, and that is the whole reason a reviewer or a classifier stands a chance. A reviewer often won't catch a well-formed POST to a documented archive endpoint. Whether it is routine cleanup or a catastrophe depends entirely on the request body, the part of the call that names which records, and nothing in the request itself tells you which.

Anthropic's classifier authors say as much. Their classifier deliberately never sees the results of the agent's earlier steps. So if the user never named the target, it cannot tell whether the agent pulled that ID from a query or made it up. A row delete in production is a valid SQL statement or API call, identical to the thousands of harmless ones before it.

Hugging Face saw exactly this in July 2026, when OpenAI's models escaped an evaluation sandbox during an internal cyber-capability test and got into Hugging Face's production systems. Its dataset loader carried a URL allowlist, and the allowlist worked: it rejected every attempt to make the worker fetch a remote resource. So the agent stopped asking the worker to fetch things and instead acted on local ones, through a data file that pointed to a local path and a template field that ran code. In Hugging Face's words, neither is a URL fetch, so the allowlist never saw them.

A gate can only judge what it can see, and it never sees which records the call names.

July was a containment and exfiltration story, and recovery does nothing for exfiltration. The data-loss version had already happened in February.

One command took the database and its backups together

On February 26, 2026, the founder of DataTalks.Club used a coding agent to fold a static website into the Terraform setup that already ran his production course platform. The Terraform state file was still on his old laptop, so Terraform believed nothing existed and started creating duplicates of resources that were already running.

He stopped the apply and copied the old machine's Terraform archive over. The archive held the production state, and the agent unpacked it without him noticing. With the full picture in hand, the agent proposed terraform destroy as the cleaner way to remove the duplicates. Auto-approve was on. He did not stop it.

The destroy removed the VPC, RDS database, ECS cluster, load balancers, bastion host, and every nightly snapshot. Two and a half years of student submissions were gone: 1,943,200 rows in one table alone. The platform came back about 24 hours later, after he upgraded to AWS Business Support and an engineer found an internal snapshot that was not visible in his console.

Nothing in that sequence is specific to AI. Someone with the same state file and permissions could have done it, minus the pause a person usually notices.

His lesson, in his words: automated backups were deleted together with the database. Native snapshots and versioning live in the same account, under the same credentials, in the same lifecycle as the data. They cover an isolated mistake. Against a full-resource removal or an actor holding the account's credentials, the call that deletes the resource deletes them too.

6 controls to implement immediately

A VP of architecture at a payments company put it plainly to our team: the scenario that worries them is their own code, ahead of ransomware. Here are some controls to implement:

Give the agent its own identity. An agent on a developer's credentials inherits everything the developer can do. A dedicated identity forces the decision about what this agent may touch, makes every action attributable, and ends a session with one revoked role.
Put human review outside the agent's reach, and only on irreversible actions. Destroy, drop, replace, schema changes. A prompt inside the tool is the first thing a blocked engineer switches off, so put the review somewhere the agent cannot bypass. Gating everything is how you end up at 97%.
Treat state as production data. Terraform state and migration history decide what "cleanup" means. State belongs in versioned remote storage with deletion protection, moves between machines through that storage and never by hand, and the agent's identity reads it at most.
Protect the resource at the platform level. Deletion protection on the database, prevent_destroy on the module, Object Lock on the bucket. A rule in a config file is advice the model can override; these hold regardless of what the agent decides. They stop removal, do nothing about a valid write to the contents, and are not a backup.
Keep a copy that survives the loss of the thing it protects. A separate account with its own identities, so a credential that can destroy the infrastructure cannot reach the copy. Granular to the row or object, so you put back what the agent touched and nothing else. Native snapshots fail this test; DataTalks.Club proved it in one command.

(Eon, where I work, sells this as a managed service. Recovery copies land in isolated vaults, logically air-gapped and immutable, held in a separate account from your production environment, so they sit outside the blast radius of compromised production credentials. A restore comes back at the level of a single object or row, in minutes. You can build the same property without us. Replicate into an account whose credentials your production roles and your agents never hold. Turn on S3 Object Lock in compliance mode, or an AWS Backup vault lock, so the copies stay immutable for the retention period. Keep the state backend and the migration history under the same rule. Then rehearse one restore, so the recovery time is a number you have measured.)

Make every write attributable. Identity plus audit logging means every mutating call ties to a specific agent, resource, set of records, and time. When a valid credential makes a valid call against the wrong records, nothing alerts, and the first hour is spent figuring out what changed. The hour is short only if you recorded the answer before the incident.

Controls one through four decide how often a bad change reaches production. Five and six decide what happens when one does, and five is the one almost nobody has. Cloud data protection assessments Eon ran across 78 companies' cloud environments counted 53,709 production resources. 61.2% of them had no recovery copy at all.

Know, before the agent runs, exactly which records you could put back and how long it would take. DataTalks.Club got theirs back because an AWS support engineer found a copy their console never showed, and nobody should plan around that.

We're running a hands-on version of this on October 8 in New York, 4 to 7pm at LOFT39. You break an environment with an agent, then recover it, in a sandbox built on Bedrock AgentCore. Details are on the workshop page.

I'll answer what I can in the comments. If you've run terraform destroy against a stale state file, I'd like to know what you actually got back and how long it took.

Claude Code's /rewind Won't Bring Back What Bash Deleted

Julia Salem — Tue, 29 Sep 2026 13:30:00 +0000

Disclosure: I work at Eon, on the content side, and we sell cloud data protection, which is what the last two sections are about. Everything before them is Anthropic's and Huawei's own docs, four open-source READMEs and three public GitHub issues, all linked.

What Claude Code checkpointing tracks, and the 5 things it skips

As of September 14, 2026, the checkpointing page at code.claude.com says Claude Code captures a checkpoint before each user prompt and tracks changes made through its file editing tools. The Agent SDK page names them: Write, Edit, and NotebookEdit. /rewind (or Esc twice on an empty prompt) restores code, conversation, or both, from the 100 most recent checkpoints in a session.

The limitations section is short and worth reading in full. Its first line: "Checkpointing does not track files modified by bash commands." The docs list these examples:

rm file.txt
mv old.txt new.txt
cp source.txt dest.txt

The SDK page adds echo > file.txt and sed -i. Redirects, in-place edits, moves, and any script the agent wrote 30 seconds earlier all sit outside the checkpoint.

Four more limits follow. Rewind doesn't restore edits made by subagents, and the docs point you to git for those. A restore skips symlinked and hard-linked files and prints Restored the code, but skipped N files. It doesn't track edits made outside the session, by hand or by another process. And the snapshots expire: a retention sweep deletes a session's file snapshots about 30 days after the last save, after which rewinding to one of them fails with No files were restored.

The page ends on a heading Anthropic wrote itself: checkpoints are not a replacement for version control.

Try it in 60 seconds

In a scratch repo:

Ask Claude Code to create a.txt with its editor.
Ask it to create b.txt through the shell:

echo hi > b.txt

Ask it to add a line to each, the same way it created them.
Run /rewind and restore to the first checkpoint.

Per the docs, a.txt comes back, and b.txt stays exactly as the shell left it.

Where /rewind is the right tool

The limits don't make checkpointing useless. Most of what a coding agent does, most of the time, is edit source files through its own tools, and every one of those edits is restorable in a keystroke. For a refactor that went sideways, or a failing test the agent "fixed" by deleting the assertion, /rewind is the right tool and a good one.

"Just use git" doesn't cover it either. Git holds committed state, untracked files vanish under git clean, and the agent has the same shell you do, so git checkout . is one tool call away. (git-safepoint exists for exactly that reason.)

What falls outside the checkpoint is a short list of commands, and most of them delete or overwrite files. An agent that decides a directory is stale runs rm -rf, and a migration runs through a shell; neither goes anywhere near the Edit tool. Both can remove data that nothing else on the machine recorded.

4 open-source tools that cover the bash channel

Engineers hit this often enough that 4 open-source projects now exist to cover it.

bashward (Rust, v0.1.1) hooks Claude Code's PreToolUse event for the Bash tool and parses each command. Before it runs, bashward snapshots the paths that rm, mv, cp, dd, sed -i and shell redirects are about to touch, using APFS clones on macOS.
git-safepoint (Python, v0.0.1, June 23, 2026) captures tracked and untracked files before every destructive command using git plumbing, so you can restore a single file without touching HEAD.
stepback (PyPI, v0.1.2) wraps any agent, watches the filesystem instead of one tool's hooks, and can rewind the agent's on-disk conversation transcript along with the files.
doover (Rust, v0.2.3, September 2026) journals every shell action, classifies commands against a registry of 152 reversibility rules, and snapshots affected paths anywhere on disk, including outside the project.

All four are pre-1.0, and git-safepoint is at v0.0.1, so try them in a scratch repo before you put one in front of work you can't lose.

The ask reached Anthropic directly too. On January 9, 2026, a user opened issue #16976 on the Claude Code repository, asking Anthropic to expose checkpoint restore programmatically and calling the current feature "effectively UI-only." Its status: closed as not planned.

The patch tools are candid about their own ceiling. Doover's README has a section titled "What doover is not" that flags DROP TABLE, kubectl delete and force-pushes as unrecoverable, because no local snapshot can bring back remote state. By default, it keeps 7 days and 5 GiB of history on the same disk as the files it protects. The README's advice on that point comes down to 3 words: "Keep real backups."

Nor is the limit Anthropic's alone. Huawei's CodeArts Agent documentation (updated July 29, 2026) draws the same line. Checkpoints there don't track file changes made through bash commands or external editors, and they expire after 15 days. Two vendors document the same limit, for the same architectural reason. A checkpoint is a copy of the files the agent's own tools touched. It can't see what a shell did.

When the checkpoint restore itself deletes files

The sharpest version of the problem is a report where the undo did the damage. On February 25, 2026, a user of GitHub's Copilot CLI filed issue #1675 about what happened when they chose "restore to checkpoint." Per the report, the rollback ran git clean -fd on the repository root and deleted roughly a gigabyte of untracked evaluation output that no checkpoint had ever covered. It remains open as of September 9, 2026. It is a single user's account, with no confirmed root cause from GitHub in the public thread, and the mechanism is the same. The restore was itself a shell command, and nothing was checkpointing the shell.

A related bug report on the Claude Code repository (#34368, March 14, 2026) describes the rewind option appearing on a turn where the agent had deleted a file. Selecting it did nothing to restore the file. Read it as a UI question, not a scandal. The docs say the code-restore options appear only when a checkpoint has tracked changes to revert, and a checkpoint can have tracked edits without holding the file you want back.

Checkpoint vs recovery copy, side by side

A checkpoint answers one question: what did the agent change in my working tree since the last prompt. A recovery copy of a database or bucket (what most teams still call a backup) answers a different one: what did this row, object, or file look like at 02:13, before the agent's UPDATE ran. The first lives on the agent's machine for a bounded time. The second lives in a separate account under separate credentials, and you restore it to a point in time, regardless of which tool or person made the change.

Layer	File-tool edits	Shell changes	Rows and objects	Undo window
Claude Code checkpoints	Yes	No	No	Last 100 checkpoints
Huawei CodeArts checkpoints	Yes	No	No	15 days
git	Committed only	Committed only	No	Until history is rewritten
Shell-hook tools	Varies by tool	Yes, local disk	No	Bounded, same disk
Recovery copies, separate account	Not the job	Not the job	Yes	Set by retention policy

Two notes on that table. Claude Code's snapshots get swept about 30 days after the last save (docs read September 14, 2026), and Huawei's expire at 15 days (docs updated July 29, 2026). Git protects committed state only, and the agent can run git clean or git checkout . itself. Doover flags row and object operations as unrecoverable outright, and defaults to 7 days and 5 GiB on the same disk as the files it protects.

The same shell reaches your production database

On a developer machine, the worst outcome is a lost afternoon. The same shell channel the checkpoint can't see is also how an agent reaches everything that isn't a source file: psql, aws s3 rm, a migration script, kubectl. Anthropic's March 25, 2026 engineering post on auto mode lists, from its own internal incident log, an agent "attempting migrations against a production database." The only thing that changes is what the command can reach.

Managed databases add a twist. A nightly snapshot tells you what the table looked like at 02:00. It doesn't tell you which rows the agent touched at 02:14, or which writes since then were legitimate. And if the recovery copies sit in the same account, and the agent's credentials can delete them, they inherit the agent's mistakes. Eon's August 5, 2026 post on agent-driven data loss walks through the public incidents where they did.

What a real undo has to do at the data layer

Set the bar for an undo the way you'd set it for a restore, because it is one. It records the data state at the moment of the operation, so nobody has to reconstruct it from logs afterward. Only what changed comes back: a file, an object, or a row, in place (the rest of the system never notices). The copy lives outside the credentials that made the change, and the restore finishes in minutes.

Checkpoints in coding agents were never built for that, and their docs say so. Before the next agent session:

Start from a clean commit, ideally in a fresh worktree or container, so rm -rf has a small blast radius.
Put a shell hook in front of the Bash tool (bashward or doover), and treat it as local-disk protection only.
Give the agent its own database credentials, so the database's own logs name the agent and not you when something needs unwinding.
Time how long it takes to get 1 row back from your managed database's recovery copies. If the answer is "restore the instance to a side environment and export," write that number down.
Keep recovery copies in a separate account, with credentials the agent never holds.

(Eon, where I work, sells the last two as a managed service. Recovery copies land in isolated vaults, logically air-gapped and immutable, held in a separate account from your production environment, so they sit outside the blast radius of compromised production credentials. A restore comes back at the level of a single file, object or row, in minutes. You can build the same property without us. Replicate into an account whose credentials your production roles and your agents never hold. Turn on S3 Object Lock in compliance mode, or an AWS Backup vault lock, so the copies stay immutable for the retention period. Give the agent its own database user. Then rehearse one restore, so the recovery time is a number you have measured.)

Frequently asked questions

Can Claude Code's /rewind undo bash commands?

No. Anthropic's checkpointing documentation (read September 14, 2026) states that checkpointing does not track files modified by bash commands, with rm, mv, and cp as the listed examples. Rewind restores only edits made through Claude Code's file editing tools.

What does Claude Code checkpointing not track?

Files changed by bash commands, edits made by subagents, and edits made outside the session by hand or by another process. A restore also skips symlinked and hard-linked files, and file snapshots expire about 30 days after the session last saved one.

How do you undo changes an AI agent made outside the editor?

Open-source shell-hook tools snapshot files before destructive commands run: bashward and doover hook Claude Code's PreToolUse event, git-safepoint uses git plumbing to capture untracked files, and stepback watches the filesystem for any agent. All of them work on the local disk only.

Can you recover data an AI agent deleted from a database?

Only from a recovery copy of the database that lives outside the agent's credentials. Coding-agent checkpoints cover files on the agent's machine, and shell-hook tools such as doover flag DROP TABLE and other remote operations as unrecoverable.

What is the difference between agent checkpointing and a backup?

A checkpoint is a bounded, same-machine copy of files an agent's own tools edited during a session. A backup, or recovery copy, is a copy of the data itself, held in a separate account under separate credentials. You can restore it to a point in time, regardless of which tool or person made the change.

We're running a hands-on version of this on October 8 in New York, 4 to 7pm at LOFT39. You break an environment with an agent, then recover it, in a sandbox you don't have to clean up afterwards. Details are on the workshop page.

I'll answer what I can in the comments. If you're running one of these four on something real, I'd rather hear how the retention defaults held up than hear that they work, and if the 60-second test behaves differently on your setup than the docs describe, say so and I'll correct the post.

We checked 53,709 production resources for backups. 61% had none.

Julia Salem — Mon, 28 Sep 2026 18:00:00 +0000

Disclosure: I work at Eon, on the content side. Our team ran these assessments and this is our data. I didn't write the queries, but I sat with the engineer who did, and the methodology below includes what we couldn't see. Argue with it.

Most backup tools produce a coverage report, and most of those reports work from tags. They check the resources a policy already enrolled, which is usually everything tagged environment=production, and report how many of those have a recent recovery point. The number comes back green.

That number is real. It's just answering a smaller question than the one you asked.

We wanted to know how much smaller. Across 78 companies' cloud environments on AWS, Google Cloud, and Azure, our team connected through a read-only role, discovered every resource, classified which ones were production, and checked each one for a recovery copy. 101,340 live resources, 53,709 of them production. Pulled September 15, 2026.

Only 14% of AWS production resources had a production tag

This is the finding that explains all the others.

Of about 47,000 AWS production resources:

14% carried a tag saying production
66% had no environment tag at all
27% had no tags of any kind
19% had an environment tag containing something that isn't an environment

That last group is the one you'll recognize. Real values found in the environment field: base, stable, research, corporate.

Somebody typed those. The field was required, it was 6pm, and they put something in it. Then a backup policy read that field to decide what to protect.

A coverage report keyed to environment=production would have found about 1 in 7 of these resources. The rest never enter the count, so they can't show up as unprotected. They don't show up at all.

61.2% had no recovery copy

Of the 53,709 production resources, 61.2% had no backup policy, no recovery point, and no native cloud snapshot the assessment could detect.

Three definitions, in case you think we picked the flattering one:

Definition	Share
No policy, no recovery point, no native snapshot detected	61.2%
No backup policy at all	62.8%
No policy, or a policy that's attached but blocked or over a limit	65.4%

There's no reasonable definition that gets this below 61%.

The unprotected resources are the old ones

This is the number that changed how I think about the problem. I'd assumed the unprotected resources would be new ones. Spun up last sprint, policy hasn't caught up, on someone's list.

Median age of a production resource with no recovery copy: 502 days
Median age of one with a recovery copy: 363 days
Share of unprotected resources older than 12 months: 55%

So they're about four months older on average. Not a backlog. A resident population that nobody is watching.

It's also the best argument against the obvious objection, that these resources are protected by something the assessment couldn't see. A resource that's been running 16 months with no policy, no tag, and no documentation isn't usually one somebody carefully set up a separate backup plan for.

0.9% of the gaps were on purpose

Skipping backups is sometimes correct. A box that Terraform rebuilds on every deploy, a cache that refills from upstream, a low-value system where the cost genuinely isn't worth it. Fine.

So we counted how often that decision was actually recorded. 0.9% of production resources had a documented exclusion. The other 61.2% had no policy and no note.

The distinction only matters on one day, but it matters a lot that day. "We assessed it and accepted the risk" is an answer. "We didn't know it was there" is a different conversation.

Nobody was clean

Among the 51-odd environments with 25 or more production resources:

Typical environment: 64.5% of production unprotected
More than half were above 50%
Zero reached 100%
Best in the set: 99.8%

The median tracks the aggregate almost exactly, so this isn't a few disasters dragging an average.

Why this is worse than it was two years ago

For a long time, leaving a low-tier resource unprotected was a defensible cost decision. What broke production was a human mistake or a hardware fault, both rare, both usually fixable by redeploying from code and reloading from upstream.

An AI agent with valid credentials has a different blast radius. It writes at machine speed to anything those credentials reach, and it has no idea how you tiered your estate. The cache, the queue, the read replica, and the database behind them go together.

Terraform brings back the table. Not the rows.

What to actually do about it

Count from the cloud side, not from the backup tool. Pull live inventory from every account and region via read-only APIs, then match each production resource against a recovery point. Starting from your backup tool's inventory guarantees you only count what it already knows.
Don't classify production from one tag. Look at the name, the account, what depends on it, and who gets paged when it breaks. Expect the untagged resources to be the oldest ones.
Turn every gap into a written decision. Attach a policy or write down why not, with an owner and a review date. A documented exclusion is defensible. An undecided gap isn't.
Keep recovery copies outside the blast radius. Credentials broad enough to modify production data are often broad enough to delete snapshots in the same account. Bucket versioning protects against overwrites, not against a credential that can delete versions.
Re-run the count on a schedule. New resources arrive weekly. Unless a policy attaches at creation, the coverage you proved Tuesday is stale by Friday, and in our data those gaps persist for about 16 months.

Methodology, including what we couldn't see

The numbers come from cloud data protection assessments run across 78 companies' cloud environments. Each connects through a single read-only IAM role and discovers and classifies every resource. No agents, no appliances, no manual tagging. Results aggregated and anonymized; nothing identifies a company, account, or resource.

Cohort: 78 environments, 101,340 live resources across AWS, GCP, and Azure, 53,709 production. Excluded: sandboxes, inactive environments, deleted resources, disconnected accounts, and child resources that inherit a parent's protection (an EBS volume attached to an EC2 instance counts once, through the instance). Distribution stats use environments with 25+ production resources.

How "production" was determined: from resource metadata — name, IDs, tags, the account it runs in, and how the rest of that account was already classified. Tags are one input among several, so a resource with no tags still gets classified.

The honest limitation: native snapshot detection is uneven. The assessment sees native snapshots on 98% of RDS instances, 42% of FSx file systems, 24% of EC2 instances, and almost none of the other resource types. So for S3, DynamoDB, EFS, EKS, and anything on GCP or Azure, "no recovery copy" means no policy and no recovery point in our data. A native backup, or a backup made by another tool, could exist outside that view. The age and tagging findings are why I don't think that accounts for much of the 61%, but weigh it yourself.

Also worth saying: a classifier is a model of an environment, not the environment, and it has an error rate. And counting unprotected resources says nothing about whether the protected ones would actually restore. That's a separate study, and honestly a harder one.

Full write-up with the rest of the findings is here. I'll answer what I can in the comments and pull in the engineer who ran the queries for anything I can't. And if your environment looks nothing like this, I'd like to hear what you're doing differently.

Implementing Traffic Policies in Kubernetes

Julia Salem — Wed, 30 Jun 2021 15:45:26 +0000

When setting up Kubernetes for the first time, one of the networking challenges you might face is how to safely grant outside clients access to your cluster. By default, pods within a cluster can communicate with all other pods and services. You should restrict access to anything outside of that group.

In this post, we’ll take a closer look at how to introduce a process for monitoring and observing Kubernetes traffic using Kuma, a modern distributed control plane with a bundled Envoy Proxy integration.

Setting Up a Kuma Service Mesh

Application stacks that run as individual containers need to communicate with one another and outside clients. To coordinate between all the requirements necessary to support such platforms—including security, routing and load-balancing—the concept of a service mesh emerged. The goal of a service mesh is to provide seamless management of any service on the network. Thus, while an ingress controller handles the behavior of incoming traffic, a service mesh is responsible for overseeing all aspects of the network, such as monitoring and configuration of the network.

Kuma is one example of a service mesh. It’s an open source project that works across various environments, including Kubernetes and virtual machines, and supports multi-zone deployments. Kuma is supported by the same team that built Kong, a popular API gateway that simplifies network communication. Kong has a vast plugin ecosystem that enables you to easily deploy and manage HTTP requests, responses and routes across your entire fleet. Kuma works hand-in-hand with Kong, but the two projects don’t rely on each other, as we’ll see below.

In addition to providing fine-grained traffic control capabilities, Kuma also offers rapid metrics and observability analyses. Being able to secure your networking access is only part of the solution. Since Kuma integrates with Prometheus for native data collection and Grafana for charting and viewing that data, you’ll be able to see precisely how your load balancing and client routing are behaving.

Installing Kuma is a snap. First, you can download and run the installer like so:

curl -L https://kuma.io/installer.sh | sh -

Then, switch to the installation directory:

cd kuma-1.1.2/bin

From here, you can run Kuma in multi-zone mode or standalone mode if Kuma is just in a single Kubernetes cluster. The command below will deploy Kuma in a single zone configuration, the default:

./kumactl install control-plane | kubectl apply -f -

For other environments, check out the docs on deployment.

There are several ways to interact with Kuma.

Read-only through its GUI
For write/edit access –kubectl
API (note that in a Kubernetes deployment, the API is also read-only, and interactions via kubectl are the correct process.)

To access the GUI, you’ll first need to forward the API service port:

$ kubectl port-forward svc/kuma-control-plane -n kuma-system 5681:5681

After that, you can navigate to http://127.0.0.1:5681/gui.

CNI Compatibility

Before continuing, it’s important to introduce a minor point about configuration, which has major implications.

Kubernetes uses the Container Network Interface (CNI) standard to configure networking for containers. This means that no matter how you design a CNI-compatible tool, it ought to be able to rely on the same set of protocols. Kubernetes provides an API that an ingress controller can use to set and manage the network policies. Multiple CNI-based projects have sprung up in response to enterprise-grade security and ease of use requirements. For example, one such project is Calico.

Depending on your needs, opting for a more customizable service mesh, like Kuma, can help you achieve your specific goals. For example, although Calico adheres to the Network Policies Kubernetes provides, its format for setting up traffic rules is more opaque than Kuma. Kuma provides a way of configuring network policies that run parallel to the first-class API Kubernetes provides. It should come as no surprise that Kuma is also compatible with CNI. This means you can easily swap out any network policies defined by Calico—or any project that uses a CNI-based protocol for Kuma’s traffic rules. The main differentiator between such projects comes down to features. Kuma, for example, can act as a service mesh, an observability platform and a network policy manager all in one. Other projects may have different priorities, and it is the developer’s responsibility to make sure they can all interact with one another properly.

Architecting Traffic Policies in Kubernetes with Kuma

With Kuma set up and running on Kubernetes, let’s see how to establish traffic rules to manage incoming access.

Imagine the following scenario: an eCommerce platform that relies on two microservices that communicate to meet the business’s needs—let’s call them services backend1 and backend2. A third microservice acts as a public API, and any incoming request to this service privately queries the other two. We’d like to expose the API to the public but keep the other two microservices isolated from external networks.

The pure ingress way to do this is to set up a Network Policy. However, Kuma drastically simplifies this process with an easy-to-understand YAML DSL. You can define Traffic Permission policies that explicitly identify which sources the services can communicate specific destination services.

cat <<EOF | kumactl apply -f - 
type: TrafficPermission
name: api-to-backends
mesh: default
sources:
  - match:
      service: 'publicAPI'
destinations:
  - match:
      service: 'backend1'
  - match:
      service: 'backend2'
EOF

In this manifest, the Traffic Permission policy gives the frontend permission to send traffic to the backend. The policy will reject any other source.

Traffic Permission is just one of the policies that Kuma provides. Among other features, you can also set up a Health Check policy to keep track of the health of every data plane proxy. This, too, makes use of familiar source and destination matches:

cat <<EOF | kumactl apply -f -
apiVersion: kuma.io/v1alpha1
kind: HealthCheck
mesh: default
metadata:
  name: web-to-backend-check
spec:
  sources:
  - match:
      service: 'publicAPI'
  destinations:
  - match:
      service: 'backend1'
  - match:
      service: 'backend2'
  conf:
    interval: 10s
    timeout: 2s
    unhealthyThreshold: 3
    healthyThreshold: 1
    tcp:
      send: Zm9v
      receive:
      - YmFy
      - YmF6
EOF

One Control Plane for Security, Observability and Routing

The goal of any service mesh is to provide a single location to configure how your network behaves across your entire cluster. A service mesh can simplify much of the communication across disparate services. It’s often better to opt for a more restrictive network security rather than one which is open to any connection. Implementing a zero-trust security policy with Kuma is a first-class feature, not an afterthought.

If you’d like to learn more about proper access configuration, you can check out the Kubernetes documentation on controlling access or their best practices on pod security. Kuma’s secure access patterns also provide some guidelines on how to define commonly-required networking policies.

I hope you found this information on traffic policies in Kubernetes helpful. Get in touch via the Kuma community or learn more about other ways you can leverage Kuma for your connectivity needs with these resources:

The post Implementing Traffic Policies in Kubernetes appeared first on KongHQ.

Protecting Services With API Gateway Rate Limiting

Julia Salem — Tue, 18 May 2021 13:00:19 +0000

The Kong Gateway Rate Limiting plugin is one of our most popular traffic control add-ons. You can configure the plugin with a policy for what constitutes “similar requests” (requests coming from the same IP address, for example), and you can set your limits (limit to 10 requests per minute, for example). This tutorial will walk through how simple it is to enable rate limiting in your Kong Gateway.

Rate Limiting: Protecting Your Server 101

Let’s take a step back and go over the concept of rate limiting for those who aren’t familiar.

Rate limiting is remarkably effective and ridiculously simple. It’s also regularly forgotten. Rate limiting is a defensive measure you can use to prevent your server or application from being paralyzed. By restricting the number of similar requests that can hit your server within a window of time, you ensure your server won’t be overwhelmed and debilitated.

You’re not only guarding against malicious requests. Yes, you want to shut down a bot that’s trying to discover login credentials with a brute force attack. You want to stop scrapers from slurping up your content. You want to safeguard your server from DDOS attacks.

But it’s also vital to limit non-malicious requests. Sometimes, it’s somebody else’s buggy code that hits your API endpoint 10 times a second rather than one time every 10 minutes. Or, perhaps only your premium users get unlimited API requests, while your free-tier users only get a hundred requests an hour.

Mini-Project for Kong Gateway Rate Limiting

In our mini-project for this article, we’re going to walk through a basic use case: Kong Gateway with the Rate Limiting plugin protecting a simple API server. Here are our steps:

Create Node.js Express API server with a single “hello world” endpoint.
Install and set up Kong Gateway.
Configure Kong Gateway to sit in front of our API server.
Add and configure the Rate Limiting plugin.
Test our rate limiting policies.

After walking through these steps together, you’ll have what you need to tailor the Rate Limiting plugin for your unique business needs.

Want to set up rate limiting for your API gateway with clicks instead of code? Try Konnect for free >>

Create a Node.js Express API Server

To get started, we’ll create a simple API server with a single endpoint that listens for a GET request and responds with “hello world.” At the command line, create a project folder and initialize a Node.js application:

~/$ mkdir project
~/$ cd project
~/project$ yarn init
// Accept all defaults

Then, we’ll add Express, which is the only package we’ll need:

~/project$ yarn add express

Lastly, let’s create a simple server with our “hello world” endpoint. In your project folder, create an index.js file with these contents:

/* PATH: ~/project/index.js
*/
const express = require('express')
const server = express()
const port = 3000
server.get('/', (req, res) => {
 console.log(req.headers)
 res.status(200).send('Hello world!')
})
server.listen(port, () => {
 console.log(`Server is listening on http://localhost:${port}`)
})

Now, spin up your server:

~/project$ node index.js

In your browser, you can visit http://localhost:3000. Here’s what you should see:

Let’s also use Insomnia, a desktop client for API testing. In Insomnia, we send a GET request to http://localhost:3000.

Our Node.js API server with its single endpoint is up and running. We have our 200 OK response.

Keep that terminal window with the node running. We’ll do the rest of our work in a separate terminal window.

Install and Set Up Kong Gateway

Next, we’ll install Kong Gateway to sit in front of our API server. These steps will vary depending on your local environment.

Simple usage of the Rate Limiting plugin supports configuring Kong in DB-less mode with a declarative configuration. That means, instead of sending configurations to Kong Gateway one step at a time, with those configurations stored in a database, we can use a single declarative .yml file for specifying our entire Kong configuration.

After installing Kong, we generate a starter .yml file:

~/project$ kong config init
~/project$ tree -L 1
.
├── index.js
├── kong.yml
├── node_modules
├── package.json
└── yarn.lock
1 directory, 4 files

We’ll come back to that kong.yml file in a moment.

Now, let’s tell Kong where to look for that declarative configuration file upon startup. In /etc/kong, there is a
kong.conf.default file that we’ll need to copy as kong.conf and then edit:

~/project$ cd /etc/kong
/etc/kong$ sudo su
root:/etc/kong$ cp kong.conf.default kong.conf

Next, we edit the kong.conf file. You’ll likely need root privileges to do this. There are two edits that we need to make:

# PATH: /etc/kong/kong.conf

# line ~839: Uncomment this line and set to off
database = off

# line ~1023, Uncomment this line. Set to absolute path to kong.yml
declarative_config = /PATH/TO/YOUR/project/kong.yml

Configure Kong Gateway With Our Service and Routes

Before we start up Kong, let’s edit the declarative configuration file generated in our project folder. It should look like this:

# PATH: ~/project/kong.yml

_format_version: "2.1"

services:
- name: my-api-server
  url: http://localhost:3000/
  routes:
  - name: api-routes
    paths:
    - /api

Let’s walk through what this configuration does. After setting the syntax version (2.1), we configure a new upstream service for which Kong will serve as an API gateway. Our service, which we name my-api-server, listens for requests at the URL http://localhost:3000.

We associate a route (arbitrarily named
api-routes) for our service with the path /api. Kong Gateway will listen for requests to /api, and then route those requests to our API server at http://localhost:3000.

With our declarative configuration file in place, we start Kong:

~/project$ sudo kong start

Now, in Insomnia, we send a GET request through Kong Gateway, which listens at http://localhost:8000, to the /api path:

Note that, by sending our request to port 8000, we are going through Kong Gateway to get to our API server.

Add the Kong Gateway Rate Limiting Plugin

Now that we have Kong Gateway sitting in front of our API server, we’ll add in the Rate Limiting Plugin and test it out. We will need to add a few lines to our kong.yml declarative configuration file:

# PATH: ~/project/kong.yml

_format_version: "2.1"

services:
- name: my-api-server
  url: http://localhost:3000/
  routes:
  - name: api-routes
    paths:
    - /api
  plugins:
  - name: rate-limiting
    config:
      minute: 5
      hour: 12
      policy: local

We’ve added the entire plugins section underneath our my-api-server service. We specify the name of the plugin, rate-limiting. This name is not arbitrary but refers to the actual rate-limiting plugin in the Kong package.

In this first run, we’ve configured the plugin with minute: 5, which allows for up to five requests per minute. We’ve also added hour : 12, which limits the requests per hour to 12. We’re using the local policy, which has to do with how Kong will store and increment request counts. We’ll talk about this policy configuration more below.

Let’s restart Kong:

~/project$ sudo kong restart

Now, back in Insomnia, we test out sending some requests to our API endpoint. If you send a request every few seconds, you’ll see that the first five requests received a 200 OK response. However, if you exceed five requests within a minute:

The Rate Limiting plugin detects that requests have exceeded the five-per-minute rule. It doesn’t let the subsequent request through to the API server. Instead, we get a 429 Too Many Requests response.

Our plugin works!

If you wait a minute and then try your requests again, you’ll see that you can get another five successful requests until Kong blocks you again with another 429.

At this point, we’ve sent 10 requests over several minutes. But you’ll recall that we configured our plugin with hour: 12. That means our 11th and 12th requests will be successful. However, the system will reject our 13th request even though we haven’t exceeded the five-per-minute rule . That’s because we’ll have exceeded the 12-per-hour rule.

The Rate Limiting plugin allows you to limit the number of requests per second, minute, hour, day, month and year.

Rate Limiting “Similar” Requests

We’ve configured Kong to count (and limit) requests to our server in our simple use case so far. More likely, we’ll want to apply our rate limits to similar requests. By “similar,” we might mean “coming from the same IP address” or “using the same API key.”

For example, let’s say that all users of our API server need to send requests with a unique API key set in headers as x-api-key. We can configure Kong to apply its rate limits on a per-API-key basis as follows:

# PATH: ~/project/kong.yml

_format_version: "2.1"

services:
- name: my-api-server
  url: http://localhost:3000/
  routes:
  - name: api-routes
    paths:
    - /api
  plugins:
  - name: rate-limiting
    config:
      minute: 5
      hour: 12
      policy: local
      limit_by: header
      header_name: x-api-key

The Rate Limiting plugin will apply its limits by grouping requests according to a value found in the x-api-key in the header. Requests with the same x-api-key will be considered “similar” requests.

Let’s restart Kong and see this in action:

~/project$ sudo kong restart

Back in Insomnia, we’ll adjust our GET request slightly by adding a new x-api-key header value. Let’s first set the value to user1. We send the request multiple times. Our first five requests return a 200 OK:

When we exceed the five-per-minute rule, however, this is what we see:

If we change the x-api-key to a different value (for example, user2), we immediately get 200 OK responses:

Requests from user2 are counted separately from requests from user1. Each unique x-api-key value gets (according to our rate limiting rules) up to five requests per minute and up to 12 requests per hour.

You can configure the plugin to limit_by IP address, a header value, request path or even credentials (like OAuth2 or JWT).

Counter Storage “Policy”

In our example above, we set the policy of the Rate Limiting plugin to local. This policy configuration controls how Kong will store request counters to apply its rate limits. The local policy stores in-memory counters. It’s the simplest strategy to implement (there’s nothing else you need to do!).

However, request counters with this strategy are only mostly accurate. If you want basic protection for your server, and it’s acceptable if you miscount a request here and there, then the local policy is sufficient.

The documentation for the Rate Limiting plugin instructs those with needs where “every transaction counts” to use the cluster (writes to the database) or redis (writes to Redis key-value store) policy instead.

Advanced Rate Limiting

For many common business cases, the open source Rate Limiting plugin has enough configuration options to meet your needs. For those with more complex needs (multiple limits or time windows, integration with Redis Sentinel), Kong also offers their Rate Limiting Advanced plugin for Kong Konnect.

Let’s briefly recap what we did in our mini-project. We spun up a simple API server. Then, we installed and configured Kong Gateway to sit in front of our API server. Next, we added the Rate Limiting plugin to count and limit requests to our server, showing how Kong blocks requests once we exceed certain limits within a window of time.

Lastly, we demonstrated how to group (and count) requests as “similar” with the example of a header value. Doing so enabled our plugin to differentiate counts for requests coming from two different users so that you can apply that rate limit to each user separately.

That’s all there is to it. Kong’s Rate Limiting plugin makes protecting your server ridiculously simple. As a basic traffic control defense measure, rate limiting can bring incredible power and peace of mind. You can find more details in the rate limiting plugin doc.

If you have any additional questions, post them on Kong Nation. To stay in touch, join the Kong Community.

Now that you’re successfully protecting services with the Kong Gateway Rate Limiting plugin, you may find these other tutorials helpful:

The post Protecting Services With Kong Gateway Rate Limiting appeared first on KongHQ.

Authentication with Kong's JWT Plugin

Julia Salem — Mon, 15 Mar 2021 22:38:31 +0000

As you build and maintain more applications, your authentication strategy becomes increasingly important. It may also be top of mind for your boss since technology leaders cited “improve application security” as one of their top priorities in this year’s Digital Innovation Benchmark.

The Kong Gateway JWT plugin is one strategy for API gateway authentication. JWT simplifies authentication setup, allowing you to focus more on coding and less on security.

Authentication Is Tough

You know you need a secure front door to your system. If requests don’t have the right credentials, the door should remain locked. If they do have the proper credentials, the entry should be smooth.

But how do you verify that credentials are authentic? And how do you make sure there aren’t other ways for those without the right credentials to get into your system?

Let’s walk through those scenarios as I demonstrate how to secure a service (in this case, an API server) with Kong Gateway and its JWT plugin. I’ll cover all the steps to set up, configure and test the service — giving you the foundational knowledge needed to implement these tools independently.

Core Concepts

First, let’s cover the core technologies. If you’re already familiar with these and just want to get started, feel free to skip ahead.

What Is Kong Gateway?

As more companies move from monolithic systems to microservices, a decoupled front-line API gateway to those services — providing authentication, traffic control, request and response transformation — becomes increasingly crucial. Kong Gateway, which is open source, serves as that thin layer between your users and your upstream microservices.

What Is JWT?

The JSON Web Token (JWT) format lets two parties exchange secure claims. It’s a way of saying, “I am so-and-so, which should give me access to that resource. Here is my access token to prove it.”

A JWT has a data payload signed by a trusted party to prevent spoofing. An authorizer verifies that the JWT token is authentic, allowing (or forbidding) access to that resource. Typically, a JWT payload is not encrypted; it’s open for the whole world to read. However, what’s critical is the authenticity of a token, which depends on a trusted party signing it.

What Does Kong's JWT API Gateway Plugin Do?

In this approach, the plugin serves as the JWT authorizer. It authenticates the JWT in the HTTP request by verifying that token’s claims and ensuring a trusted party signed it. Then, depending on whether these steps were successful, Kong Gateway routes the upstream service request.

Keep in mind that authentication in this context means validating the user’s credentials. That’s the job of the JWT plugin. There’s no way to know how a user got a valid JWT. The system just knows that the user has one and is presenting it for authentication. If the JWT is authentic, you can be confident that the user is who they say.

The Basic Use Case

In this basic use case, I have a login server that accepts login attempts with a user’s email and password. If the email/password checks out, the server generates and signs a JWT and hands it back to the user.

With JWT in hand, the user tries to access our microservice: a simple API server with a single endpoint. Kong Gateway sits in front of your API server, using the JWT plugin for authentication. The user presents his JWT with his request.

First, the plugin verifies the token’s authenticity. Next, it confirms the installation steps of the claims inside the payload. A common claim used is an expiration timestamp for the access token. It’s essentially saying, “This token is valid until this date and time.” So, the plugin will check the token’s expiration date.

If the JWT passes all the necessary checks, Kong Gateway grants access to the requested server endpoint. Otherwise, it responds with 401 Unauthorized.

The approach is quite simple:

Set up a basic Node.js Express server with a single endpoint.
Set up Kong Gateway as an API gateway to your server.
Enable the JWT plugin to protect your server endpoint with JWT authentication.

1. Set Up a Node.js Express Server and Endpoint

On your local machine, create a folder for your project. Then, initialize a new Node.js project. In the following examples, I’ll use yarn, but you could use npm too:


3
~$ mkdir project
~$ cd project
~/project$ yarn init  # Use all of the yarn defaults here.

Next, add Express to your project:

~/project$ yarn add express

In your project folder, create the entry point file, index.js, which will spin up an Express server with a single endpoint. Allow a GET request to /, which will respond with the string, “Hello world!”

/* PATH: ~/project/index.js
*/

const express = require('express')
const server = express()
const port = 3000

server.get('/', (req, res) => {
  console.log(req.headers)
  res.status(200).send('Hello world!')
})

server.listen(port, () => {
  console.log(`Server is listening on http://localhost:${port}`)
})

That was simple enough! Your single endpoint should log the request headers and then send “Hello world!” back to the client with a 200 status.

Start your server:

~/project$ node index.js

You can use your browser to test this new endpoint by visiting http://localhost:3000.

Your API server endpoint should be working now!

Next, use Insomnia to send the request and inspect the response. Because of its usability, you’re going to want to use Insomnia exclusively once you start sending requests with a JWT.

In Insomnia, create a GET request to http://localhost:3000.

In Insomnia, you should get a 200 OK with “Hello world!” in the response body.

It looks like the API server is up and running. Now, it’s time to put Kong Gateway in front of it.

2. Set Up Kong Gateway

I won’t cover the details here, but the Kong Gateway installation steps may look different depending on your system.

Once you’ve installed Kong, you’ll need to take a few additional steps.

DB-Less Declarative Configuration
There are two primary ways to configure Kong. Imperative configuration issues step-by-step configuration commands to Kong through its admin API. Meanwhile, declarative configuration stores the entire configuration in a single .yml file then loads it into Kong upon startup. Additionally, you can configure Kong to hook into your database, providing more control over the different nodes it manages.

For the simple setup example, I’ll use database-less declarative configuration. When you start up Kong, you’ll tell it where to find a .yml file with all of the configuration declared within.

In your project folder, run the following command, which generates an initial kong.yml declarative configuration file.


2
3
4
5
6
7
8
9
~/project$ kong config init
~/project$ tree -L 1
.
├── index.js
├── kong.yml
├── node_modules
├── package.json
└── yarn.lock
1 directory, 4 files

Next, you’ll need to configure the system’s kong.conf file before starting up Kong. If you’re working on Ubuntu, you’ll be working in /etc/kong. Here is a template to copy over and then edit.

~/project$ cd /etc/kong
/etc/kong$ sudo su

root:/etc/kong$ tree
.
├── kong.conf.default
└── kong.logrotate
0 directories, 2 files

root:/etc/kong$ cp kong.conf.default kong.conf

There are only two edits you need to make in your kong.conf file.

# PATH: /etc/kong/kong.conf

# Around line 839, uncomment and set to off
database = off

# Around line 1023, uncomment and set an absolute path to kong.yml
declarative_config = /PATH/TO/YOUR/project/kong.yml

When Kong starts up, it will be in DB-less mode, meaning it will look to your project’s kong.yml file for a configuration.

Finally, you’ll need to edit your kong.yml file to set up a gateway in front of your API server “hello world” endpoint.


2
3
4
5
6
7
8
9
10
11
12
13
/* PATH: ~/project/kong.yml
*/

_format_version: "2.1"

services:
- name: my-api-server
  url: http://localhost:3000/
routes:
- name: api-requests
  service: my-api-server
  paths:
    - /api

Let’s go over this.

The _format_version metadata specifies the version number of your declarative configuration format.

Next, you define your service, which Kong describes as “an entity representing an external upstream API or microservice.” You can name your service my-api-service and specify its URL — you’ll recall that the Express server listens for requests at http://localhost:3000.

Next, define routes, which “determine how (and if) requests are sent to their Services after they reach Kong Gateway.” The (local) URL for Kong is http://localhost:8000. You should declare your route so that Kong listens for requests at http://localhost:8000/api, then routes to your service.

Let’s see this in action. Make sure your Express server is running in a separate terminal. Then, start Kong.

~/project$ sudo kong start

In your browser, go to http://localhost:8000/api

Kong Gateway is up. Finally, add authentication.

3. Attach JWT Plugin to Kong Gateway

To add the JWT plugin, add a “plugins” definition to your kong.yml file:

/* PATH: ~/project/kong.yml
*/

_format_version: "2.1"

services:
- name: my-api-server
  url: http://localhost:3000/
routes:
- name: api-requests
  service: my-api-server
  paths:
    - /api
plugins:
- name: jwt
  service: my-api-server
  enabled: true
  config:
    key_claim_name: kid
    claims_to_verify:
    - exp

Here, you can add the plugin named jwt and attach it to your service called my-api-server. For its configuration options, tell the plugin to check the exp value to verify that the access token has not expired.

At this point, restart Kong and see what happens:

1
~/project$ sudo kong restart

The response is 401 Unauthorized. Excellent! Kong now requires a valid JWT for any requests to your API server. Next, you need to tell Kong what constitutes a valid JWT.

In kong.yml, you need to add a consumer and a credential. Kong describes consumers as being “associated with individuals using your Service, and can be used for tracking, access management, and more.” In a more elaborate setting, every one of your API users could be a consumer. That’s a use case you can read more about towards the end of this article. In this situation, your login server is your consumer. Your login server will be the entity generating JWTs and handing them out. Users who make a request to Kong will be holding a “login server” JWT.

Edit your kong.yml file by adding the “consumers” and “jwt_secrets” definitions:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
/* PATH: ~/project/kong.yml
*/

_format_version: "2.1"

services:
- name: my-api-server
  url: http://localhost:3000/
routes:
- name: api-requests
  service: my-api-server
  paths:
    - /api
plugins:
- name: jwt
  service: my-api-server
  enabled: true
  config:
    key_claim_name: kid
    claims_to_verify:
    - exp
consumers:
  - username: login_server_issuer
jwt_secrets:
  - consumer: login_server_issuer
    secret: "secret-hash-brown-bear-market-rate-limit"

You’ve added a new consumer, named login_server_issuer. Then, you added a JWT API gateway credential for that consumer, which contains the secret used to sign JWTs for this consumer. Authentication requires two parts:

The kid (key identifier) value in the JWT header, which is a unique identifier that lets the plugin determine which consumer allegedly issued this JWT
Verification of the consumer’s secret – Was this the secret used to sign this JWT API gateway? If so, then this JWT is authentic.

Before continuing, remember to restart Kong:

1
~/project$ sudo kong restart

If you want to generate a JWT for testing, you need the secret (which you have) and the key to use for the kid value. Kong gives us access to that value through its admin API at http://localhost:8001. You send a GET request to the admin API’s endpoint /consumers/CONSUMER-USERNAME/jwt. This gives us information about this consumer’s JWT credential.

As you inspect this credential’s information, you should see the JWT secret and signing algorithm. What you’re looking for, though, is the key. In the above example, that’s 1nzcMG9Xg7n1lLgmltHnkAmkt7yp4fjZ. This is what you use as the kid value in the JWT header. The Kong plugin will see this kid value, track down the associated consumer and secret, then make sure the JWT was signed with that secret.

To test this, let’s start with the happy path. You need a JWT with a header that includes the correct kid value, signed with the right secret. For simplicity, let’s do this at jwt.io. Here, you can craft your payload, set the signing secret and then copy/paste the resulting JWT.

In the payload, the kid must match the key value from above. Also, because you configured the plugin to check JWT token expiration, you should set the exp (Unix timestamp) far into the future. The name and email are inconsequential; they just demonstrate that you can put other helpful data in the JWT payload.

Lastly, include your JWT secret at the bottom for proper signing. The result is an encoded JWT.

Back in Insomnia, you have your original request that resulted in 401. You need to add “Authorization” to that request. Choose “Auth – Bearer Token,” then paste in your encoded JWT from above.

Now, with a valid JWT attached, resend the request.

Your JWT should have been validated, and Kong routed us to the API server!

If you look back at the terminal running the Express server, you’ll recall that you’re logging the request headers to the console. When the JWT plugin authenticates an access token, it writes some additional values to the upstream headers, namely the consumer id, username and credential identifier (the key value).

But what happens if your JWT is not valid? Let’s test and see.

First, sign the JWT with a different secret. Back at jwt.io, keep the payload, but change the signing secret. Copy the resulting JWT to Insomnia, and send your request again. You’ll get a 401 with “Invalid Signature.”

If your secret is correct, but the kid is incorrect, Kong won’t find an associated credential. Without that credential, there’s no way to find the secret for authenticating the JWT.

Lastly, if the exp value is in the past, then your JWT has expired. Just as you expected, you get the following response.

And that’s it! You should be up and running with Kong Gateway and the JWT Plugin acting as an authentication layer in front of an API server.

Set It and Forget It

Implementing authentication —and getting it right —is hard work. As microservices become the norm, delegating authentication makes more and more sense. “Rolling your own” implementation for JWT authentication can muddy a code base and still leave you wondering if you got it right. By using well-tested and community-adopted services that handle concerns like routing, logging or authentication, developers can shift their focus back to their projects’ unique needs.

With that, you now have a solid foundation for getting started with Kong Gateway and the JWT plugin. It’s time to get to work.

Thanks for walking through this tutorial with us. It was originally published on our blog: https://bit.ly/3eJGhCS