A pod gets flagged mid-incident: unexpected outbound connections, a process nobody recognizes. The responder pulls the Kubernetes audit log to reconstruct what happened, and there it is, right where it should be: a create event against pods/exec on that exact pod, timestamp, source IP, the service account that made the call. Confirmation that someone got a shell into the container. Relief for about ten seconds, until the actual question comes up: what did they do once they were in there. The log has nothing. Not "nothing useful." Nothing at all.
This isn't a misconfiguration and it isn't the responder missing a setting. exec (along with attach and port-forward) isn't a normal REST call the API server handles as a single request and response. It's a streaming connection, upgraded to SPDY or WebSocket, that carries the interactive session, your keystrokes going one way, stdout and stderr coming back the other, for as long as the shell stays open. Kubernetes' audit machinery logs API requests as structured events with an optional request and response body depending on the audit policy's verbosity. But the exec session's actual traffic never travels as a request or response body in the first place, it's a raw bidirectional stream on top of the upgraded connection. Turn your audit policy up to the most verbose level available and you still get the same result: the event that says an exec happened, and silence on what happened inside it.
That gap matters most exactly when you need it least to matter, mid-incident, when "what did they run in there" is the first question anyone asks. If your response runbook says "check the audit log for exec commands," it's testing an assumption that isn't true. The audit log will tell you that access happened and roughly when. It will not tell you whether the attacker read a secret, dumped environment variables, or just checked whoami and left.
Closing that gap takes something layered in before the incident, not during it: a runtime security tool that hooks process execution at the node or kernel level (Falco is the common open-source example, watching syscalls directly rather than relying on the API server's view), or routing interactive access through a bastion or kubectl plugin that logs client-side, since the server side of a raw exec session was never going to log it for you. Neither is something you bolt on while triaging a live compromise.
This is the exact translation the Cloud Detection and Response Book is built around, not just AWS CloudTrail or Azure activity logs but the specific blind spots inside Kubernetes' own audit trail, detection-as-code and Sigma rules built with what the platform actually logs (and doesn't) in mind: https://resources.codelivly.com/product/cloud-pentesting-l2/
Top comments (0)