DEV Community: Vinod Patil

Step-Up MFA Is the Wrong Response to Credential-Theft Tools in a Privileged Session

Vinod Patil — Sun, 18 Oct 2026 12:11:41 +0000

Most session-control products respond to a risky command the same way: pause, ask for another MFA prompt, continue if the user passes.

For many commands that is reasonable. A destructive command in the wrong directory is usually a mistake by someone who is allowed to be there. A second factor confirms the person, and the person can explain.

For credential-theft tooling it is the wrong control. The question is not "is this really you?". It is "why is a credential-dumping tool being used in a production session at all?". A valid second factor does not answer that. It confirms that the right person, or the right person's stolen session, is about to dump credentials. (Disclosure: I work on zenxPAM.)

What changed in zenxPAM

Command analysis now recognises known credential-theft tooling and treats it as a hard session block. The session is disconnected. It does not get a step-up prompt, and it does not just raise an alert for someone to read later.

The order of checks is the point:

command in session
      │
      ▼
 hard-block check ── known credential-theft tooling ──► terminate session
      │ no match
      ▼
 risk scoring ──► step-up MFA / alert / allow

The hard-block check runs before the step-up MFA gate, so a credential-theft tool never reaches a prompt that a stolen session could pass. The threat response is chosen from the reason for the decision: a hard block maps to ending the session, rather than to the alert-the-security-team or step-up responses used for ordinary risk. Further commands from a session that has already been terminated are refused as well, so a blocked session cannot keep sending input in the gap before the disconnect lands.

An operator who understands what happened

When a session ends because of a policy block, the viewer shows Session Blocked, not the generic "terminated by an administrator" message. It sounds minor. An operator who sees "terminated by an administrator" phones the admin. One who sees "Session Blocked" with the policy reason knows what happened, and the audit record says the same thing.

A deliberate trade-off

Some administrative tools have legitimate uses, and a hard block will occasionally end a session that was doing honest work. That is a deliberate choice for a privileged-access product. The expectation is that the approved request scope already covers what the operator needs. If a team's work genuinely requires such a tool, the answer is a narrower, explicit approved scope for that task, not a softer default for everyone.

Why this is the right default

Every control in a PAM system is a statement about what you believe an approved session is for. If an approval says "restart the billing service", credential-theft tooling is outside that statement under any identity. Treating it as a question to be answered with another MFA tap invites the wrong conclusion, that the only problem is whether the person is who they say they are.

It also sits alongside, not instead of, the other layers: intent-scoped access, approved-application controls on the session host, and endpoint protection on the target.

If you run privileged sessions through a gateway, ask one question of your platform: what happens when a session uses credential-theft tooling? A prompt is a policy decision. A shrug is a gap.

See how zenxPAM enforces intent-scoped, just-in-time privileged sessions: https://zenxsys.com

Originally published at https://zenxsys.com/blog/block-credential-dumping-tools-privileged-sessions

Auditing Kubernetes Long-Lived Token Secrets Without Reading a Single Secret

Vinod Patil — Sun, 18 Oct 2026 11:25:41 +0000

Since Kubernetes 1.24 a service account no longer gets a token Secret automatically. That does not mean your cluster has none. Anyone who ran kubectl create secret with type: kubernetes.io/service-account-token, and every cluster upgraded from an older release, can still have tokens that never expire, sitting in a namespace nobody looks at.

They are the classic leaked credential: long-lived, bearer, and usually forgotten. Finding them sounds like a one-liner. It isn't, and the obvious way to do it is a security problem of its own. (Disclosure: I work on zenxPAM, which is where this scan lives.)

The problem with "just list the Secrets"

kubectl get secrets -A --field-selector type=kubernetes.io/service-account-token -o json

That returns each Secret with its data, meaning the token itself. Your audit tool now holds every long-lived credential in the cluster. If the tool is compromised, logs the response, or caches it, you have turned an inventory into a credential store.

Worse, there is a subtlety in the RBAC model that surprises people: the list verb on secrets is enough to read their values (a list returns full objects). Granting "just list" to an audit account is not a read-only-metadata grant. It is a grant to read every secret in scope.

Ask the API server for metadata only

The API server can return partial objects. Send this header on the list request:

Accept: application/json;as=PartialObjectMetadataList;g=meta.k8s.io;v=v1, application/json

You get names, uids, labels, annotations and timestamps back, and no data. In our lab test on a real k3s 1.30.5 cluster, the same request without that header returned data, and with it returned none. So the header is what keeps values out, not the verb.

We did not stop at trusting the header:

The scanner parses the response with a reader that extracts only allow-listed fields. It never materialises data or stringData.
If a server ignores the header and sends values anyway, the property is skipped unread, nothing is kept, logged or returned, and the scan records a warning: "The API server returned Secret values; they were discarded."
The scan only uses GET. No get secret on individual objects, no watch.

Two lists per scope: one with fieldSelector=type=kubernetes.io/service-account-token for the legacy tokens, one with a label selector for the Secrets zenxPAM itself creates as revocation markers (see the previous post in this series).

What a finding tells you

For each long-lived token Secret we record the namespace, the service account, the Secret's uid, its age, and when it was last used. The last one is the useful part, and Kubernetes gives it to you for free: the API server writes a kubernetes.io/legacy-token-last-used label on the Secret.

On Kubernetes 1.28 and later, no label means never used since tracking started. That is the "delete this" candidate.
On 1.26 and 1.27 the information is partial; below that it is unknown. The scan says so rather than guessing.
In our lab test the API server wrote the label 0.3 to 0.5 seconds after a token was used.

Four traps we hit (and fixed) building it

A page cap that retries once is a denial-of-service. Anyone who can create Secrets can attach up to 256 KiB of annotations. With a 1 MB page cap and one retry, a single planted object can stall the scan forever. The page size now keeps shrinking: 100, 25, 5, then 1, and it only fails if a single object is still over the cap.
Labels are attacker-copyable. A Secret carrying app.kubernetes.io/managed-by=zenxpam is not necessarily ours. A Secret is shown as managed only when its name is also the deterministic one the rotator uses.
A resume cursor with only a position skips namespaces. If someone edits the namespace list between pages, a position-only cursor resumes at a shifted index and the scan still ends "complete". The cursor now carries a fingerprint of the scope list, and a mismatch fails the scan instead of silently under-reporting.
The save validator and the scanner must accept the same config. If they disagree on what a valid namespace list looks like, a saved setting fails every scan.

None of these show up in a happy-path demo.

Permissions: use a separate account

The scan needs one verb, list on secrets, in the namespaces you choose (a Role per namespace), or cluster-wide with a ClusterRole. Because of the list-reads-values point above, do not reuse the rotation admin token for this. Create a separate, read-only cloud account limited to the namespaces you scan:

kubectl -n NS create role zenx-inventory --verb=list --resource=secrets
kubectl -n NS create rolebinding zenx-inventory --role=zenx-inventory --serviceaccount=NS:SERVICE_ACCOUNT

The scan itself never reads a single Secret, so get is not granted, and in our test the cluster refused get for that account while the scan still succeeded.

What happens after you find them

Findings are stale, never-used, or without an owner. From a finding you can open the rotation flow with the namespace and service account filled in. Rotation issues new, revocable tokens. It does not delete the old long-lived Secret: that stays until you remove it (kubectl -n NS delete secret NAME), because deleting a credential some forgotten consumer still uses is a decision for a person.

Honest limits

Verified on a real single-node k3s 1.30.5 cluster through the full UI and services, with a separate read-only account. Not verified on EKS, AKS or GKE, against a large number of Secrets, or against a legacy token the cluster has marked invalid.
Bound (projected) tokens are not stored anywhere, so they cannot be listed. Service accounts without a token Secret are principals, not credentials, and are not findings.
The scan is read-only. It never changes the cluster.

See how zenxPAM inventories and rotates non-human credentials: https://zenxsys.com

Originally published at https://zenxsys.com/blog/audit-kubernetes-long-lived-service-account-tokens

Kubernetes Token Rotation Doesn't Revoke Anything. Here's How to Fix That With Bound Marker Secrets

Vinod Patil — Sun, 18 Oct 2026 11:00:53 +0000

You rotate a Kubernetes service account token. The new one works. You update the consumer, close the ticket, and feel good.

Then someone asks: "Is the old token dead?"

It isn't. A token minted through the TokenRequest API stays valid until its exp claim, whether that is in one hour or in seven days. "Rotation" that only creates a new credential is key issuance. The leak window you were trying to close is still open.

This post explains the API feature that makes revocation possible, the extra machinery you need around it, and what we measured when we built it into zenxPAM, Zenxsys's agentless, just-in-time privileged access platform. (Disclosure: I work on zenxPAM.)

What we found when we ran the naive rotation

Before changing anything we ran the existing rotation path against a real single-node k3s 1.30.5 cluster. It worked, and it had gaps:

A rotation never invalidated the previous token. Depending on the requested lifetime, that is up to a 24 hour to 7 day window where a copied token still authenticates.
No preflight, so a missing RBAC verb surfaced as a failed rotation halfway through.
No check that the new token actually authenticates before the old one is dropped.
No grace window for consumers that pick up the new token a few minutes late.
No way back if the new token turns out to be wrong.

Rotation without verification, grace and rollback is how you get a 2 a.m. page.

The primitive: bound-object tokens

POST /api/v1/namespaces/{ns}/serviceaccounts/{sa}/token accepts spec.boundObjectRef. If you bind the token to an object, the API server also checks that the object still exists and has the same UID whenever the token is presented. Delete the object and the token stops working.

Bind to a Pod and the token lives and dies with the Pod. That is what projected volumes do. For a token that a CI system or an external workload holds, there is no Pod to bind to. A Secret works fine as the anchor. We call it a marker Secret: type Opaque, no data, only a name and a UID.

We proved it on the lab cluster before building on it: request a token bound to a Secret, delete the Secret, poll with the token. In our lab test the API server refused the token about 10 seconds later (9.2 to 11.0 seconds across runs). The delay is the API server's authentication cache. A design that promises instant revocation would be lying.

Two slots, A/B, per identity

Each managed identity gets two marker slots in the target namespace:

zxp-<first 12 hex of identity id>-a
zxp-<first 12 hex of identity id>-b

Both carry the labels app.kubernetes.io/managed-by=zenxpam and zenxpam.io/nhi=<id>. The current token is bound to one slot. The next rotation uses the other. That is what lets the old token keep working during a grace window while the new one is already live.

The rotation, in order

 resolve admin credentials ──► validate target ──► read current token (claims only)
            │
            ▼
 pick the free slot ──► create marker Secret ──► TokenRequest bound to it (uid from create response)
            │
            ▼
 verify: SelfSubjectReview with the NEW token must say system:serviceaccount:<ns>:<sa>
            │
            ▼
 keep previous token in the vault ──► write new token ──► grace window? ─┬─ yes: old token stays valid
                                                                         └─ no:  delete old marker Secret

Details that matter:

The new token is proven before anything switches. A SelfSubjectReview with the new token must return the expected service account. A 401 fails the rotation. A 403 or 404 (a hardened or old cluster that does not allow the check) is not a failure, but the step is marked skipped with a warning rather than silently passing.
Failure after the marker Secret is created always compensates. Delete the new marker, restore the vault, delete the retained copy. No orphaned Secrets, no half-rotated state.
The grace window is capped at the old token's remaining lifetime. An expired token needs no revoking.
Finalize deletes the old marker Secret and then polls the old token for up to 30 seconds. Because of that 10 second auth cache, "refused" is a pass and "still accepted after 30 s" is a warning with evidence, not a hidden assumption.
Roll back first proves the previous token still authenticates, then restores it and deletes the new marker. If the old token is already dead, nothing is changed.

Least privilege for the rotator itself

A rotator with cluster-admin would just move the problem. The admin service account needs, in the target namespace only:

Permission	Scope
`create` on `serviceaccounts/token`	`resourceNames`: the target service account
`create` on `secrets`	namespace wide (Kubernetes cannot restrict create by name)
`get`, `delete` on `secrets`	`resourceNames`: the two slot names only

No cluster-wide rights. The honest footnote: because create cannot be name-limited, the rotator can create Secrets in that namespace, but it can only read or delete its own two markers.

The part nobody expects: the rotator is a confused-deputy risk

A system that takes an API URL and an admin token from configuration and then calls the URL is a gadget for exfiltrating whatever token it is given. Three rules in zenxPAM close that:

The cluster address and CA come only from the cloud-account connector, never from the identity being rotated.
The admin token and CA vault keys must start with K8S_. The server refuses any other name, so a connector cannot be pointed at an unrelated vault secret and a cluster address you control.
Private or internal API hosts are refused unless an operator has put them on an allowlist.

Honest limits

Verified on a real k3s 1.30.5 cluster, not on EKS, AKS or GKE. Managed clusters may cap token lifetime or restrict SelfSubjectReview; the preflight reports the lifetime the API server grants.
If you turn binding off (KubeBindToSecret=false), the old token cannot be revoked and stays valid until expiry. The result says so, with the end time.
Revocation lags about 10 seconds. Plan the grace window with that in mind.

Why this belongs in a PAM platform and not a cron job

You can script all of this with kubectl. What you cannot easily script is the evidence: which step ran, in which order, who approved the rotation, what the verification returned, whether the old token was confirmed refused. In zenxPAM every preflight, rotation, roll back and cleanup writes an evidence record with timestamped steps, the outcome and credential fingerprints (last four characters only, never a value), and the action is audited. The answer to "is the old token dead?" is a record, not a recollection.

Next up in this series: how the same pattern (verify new, keep old, revoke on a timer, roll back on demand) works for cloud keys that have no equivalent of a bound object.

See how zenxPAM handles non-human identities: https://zenxsys.com

Originally published at https://zenxsys.com/blog/kubernetes-service-account-token-revocation

Session Recordings Are Not Evidence Until You Can Prove Nobody Touched Them

Vinod Patil — Sun, 18 Oct 2026 10:23:42 +0000

An auditor asks: "Show me what the contractor did on the production database on March 3rd."

You have a screen recording. The auditor's next question is the hard one: "How do I know this file is the one that was recorded, and that nobody trimmed it afterwards?"

If your answer is "it's in a locked folder", you have a recording but not evidence. This post covers how ZenxPAM links a recording to the request that allowed it, and how it makes later tampering detectable.

The problem with a recording on its own

A video file says what happened. It says nothing about:

who asked for the access, and who approved it;
which policy decision allowed the session;
whether the file was changed after capture;
whether it was deleted, and why.

Each gap is a place where an investigation stalls. So we treat the recording as one link in a chain, not as the artefact.

1. Capture happens at the gateway

Operators connect through a browser, and the gateway streams the session to the target through Apache Guacamole. Because all traffic passes the gateway, the recording is made there, in real time. There is no agent on the target to uninstall and no endpoint driver to disable.

Each recording gets a unique storage path of the form /{tenantId}/{sessionId}/{recording-id}, and UTC start and end timestamps are stored on the session. When the recording is uploaded, a SHA-256 hash is computed and stored with its metadata.

2. The recording is linked back to the request

Every recording points to its session, and every session points back to how it was allowed:

Access request -> Approval -> Policy evaluation -> Session -> Recording -> Audit log entries -> Immutable chain

The links are foreign keys in both directions, so an investigator can start from a recording and reach the original request, or start from a request and find every recording it produced. "Who approved this?" and "what did they do?" are answered from the same place.

3. The audit trail is a hash chain

Alongside the normal audit log, every event is anchored in an append-only chain in PostgreSQL:

each entry stores a SHA-256 hash over its id, action, entity type, timestamp, block index and the previous entry's hash;
block indexes run in sequence per tenant, so a missing entry leaves a visible gap;
the data layer rejects updates and deletes of chain entries, and the API only exposes append and read.

A verification endpoint walks the chain for a tenant and checks three things: each hash matches its recomputed value, each entry points at the real hash of the one before it, and the sequence has no gaps. A separate tamper-detection call reports any broken link.

One honest limit: this is tamper-evident, not tamper-proof. Someone with raw database access can still rewrite rows. What the chain gives you is that the rewrite is detectable the next time anyone runs verification.

4. Deleting a recording leaves a record of the deletion

Retention policies eventually remove old recordings, and that is where evidence usually disappears quietly. ZenxPAM writes audit entries for every step, and they are anchored in the same chain:

Event	What the audit trail records
Recording archived	Source path, archive target, archive path and time
Recording deleted	A tombstone with the original path and, if archived, where the copy went
Legal hold placed or released	Who did it and the stated reason
Delete attempted on a held recording	The refused attempt
Hold release attempted without authority	The attempt and the caller's roles

Legal holds also separate duties: an auditor can place a hold but cannot release it. Release needs an admin role. A held recording is skipped by the retention sweep and refused by the delete path.

Audit ingest is all-or-nothing too. If an entry cannot be anchored in the chain, the whole write is rolled back and the caller gets an error, so an unanchored audit row is never accepted silently.

5. The evidence pack

When an auditor asks for a period, the platform builds an evidence pack: counts and references for audit logs, recordings, compliance reports and policy evaluations, in a JSON manifest. A SHA-256 hash is computed over the manifest and stored with the pack, so anyone who downloads it can check it independently.

This gives your auditors something to verify, not only something to read. It is evidence for your auditors; it does not by itself make an organisation compliant with any framework.

What to take from this

If you run privileged access today, three questions are worth asking about your own setup:

Can you go from a recording to the approval that allowed it in one query?
If a recording were trimmed tomorrow, would anything notice?
If one were deleted, could you prove who deleted it and why?

If any answer is "no", the recording is a video, not evidence.

ZenxPAM docs and a demo are at zenxsys.com.

Building an Agentless, AI-Assisted Privileged Access Gateway

Vinod Patil — Sun, 18 Oct 2026 09:59:50 +0000

Traditional PAM leans on thick RDP/SSH clients, jump boxes and endpoint agents. In multi-cloud and containerised estates that means bootstrap latency on every host, exposed management ports, and audit gaps wherever an agent can't be installed. This post covers how ZenxPAM handles three of those problems: browser-native session streaming, on-prem AI session analysis, and tamper-evident audit evidence.

1. Zero-footprint interactive sessions

The operator opens a token-authenticated HTML5 client in an ordinary browser. There is no plugin, VPN client or local credential.

Browser (HTTPS / WSS)  ->  ZenxPAM gateway  ->  Apache Guacamole (guacd)  ->  RDP / SSH / VNC target

Protocol translation. Apache Guacamole 1.6.0 with custom authentication turns the browser's WebSocket stream into RDP, SSH or VNC toward the target. Targets only need to accept connections from the gateway, so management ports stay off the operator's network.
Credentials never reach the browser. Secrets are injected from the vault server-side at connect time.
Recording is a side effect of the architecture. Because every byte passes through the gateway, Guacamole records the session in real time, with no endpoint monitoring driver.
Kubernetes consoles (kubectl, k9s, Lens) are brokered through a jump host with vaulted credentials and recording.

2. Local-first AI: no data leaves the perimeter

Session telemetry is sensitive, so the AI layer calls a self-hosted Ollama instance on the same Docker network as the services. No external LLM API is involved.

Per-command risk scoring. Commands in a live session get a risk score and level plus risk indicators, and high-risk commands can be blocked with a recorded reason.
Behavioural anomaly checks run against session activity, and sessions roll up to an overall risk level.
Post-session analysis. After a session ends, the commands, deviations and MFA events are summarised by the local model, including average and peak risk scores and the riskiest commands.
Natural-language search over audit and access data runs through the same local model.

3. Tamper-evident audit evidence

Each audit event gets a linked entry in an append-only chain:

Hash is SHA-256 over the entry's id, action, entity type, timestamp, block index and previous hash.
PreviousHash links to the prior entry, with a sequential BlockIndex per tenant. A Merkle root is stored for block verification.
The chain lives in PostgreSQL. The data layer rejects any update or delete of chain entries, the API is append-only, and a foreign key prevents hard-deleting a source log that has a chain entry.
A verification endpoint walks the chain and checks hashes, links and gaps, and a tamper-detection endpoint reports broken links.
Session recordings get a SHA-256 hash at upload, and evidence packs reference recordings with their integrity hashes. Recordings can be retained in Azure Blob, S3 or GCS.

The chain is tamper-evident, not tamper-proof: someone with raw database access can still rewrite rows, and verification is what catches it.

Next steps

Docs and a deployable demo are at zenxsys.com.