The dashboards were green. The api-gateway logged 12,400 successful media POSTs over six hours, the storage service returned 200 on every upload and incremented its own PutObject success counter, and the fanout queue happily processed every notification. The MinIO bucket had gained zero new objects in the same window. Users were seeing broken image tiles in their feeds and the on-call team had spent three hours chasing the fanout service because that was the only place the symptom was visible. The actual problem was an explicit Deny on s3:PutObject sitting inside a canned IAM policy attached to the storage-service identity, written during a security hardening sprint two days earlier and rolled out to the cluster at 02:14 that morning, and MinIO was doing exactly what IAM evaluation says it should do: an explicit Deny in any policy attached to the identity beats an Allow in another.
Problem signals:
- Upload endpoints return HTTP 200 but the object never appears in the bucket
- The app publishes upload events on its own success path and downstream consumers process phantom events
- Grafana shows upload throughput as healthy because the application's own success counter dominates the panel
- Users report broken image links while every service-level dashboard is green
- A recent IAM or bucket policy change correlates in time with the start of phantom uploads
The discrepancy that should have been the first alert
12,400 successful uploads, zero new objects
We came in on the third hour of the incident. The team had been chasing the fanout consumer because user reports were all of the form 'my avatar is broken' and the only service touching media after upload was fanout. Their working theory was that fanout was racing the CDN, or that the notification payload was missing a key, or that signed URLs were expiring early. They had three engineers staring at fanout-service logs and finding nothing wrong, because there was nothing wrong with fanout-service.
The question we asked, which is the question we always ask first when an upload pipeline misbehaves: how many objects has the bucket actually gained in the last hour? Not how many uploads the API recorded. Not how many notifications fanout received. How many real objects exist now that did not exist sixty minutes ago. We ran the listing against the bucket and the answer was zero. The bucket had not gained a single object since 02:14 that morning, which lined up almost exactly with the rollout of a security hardening PR the platform team had merged two days prior. The policy was written on the Tuesday; it only reached the cluster when the nightly config sync ran at 02:14.
# count objects added in the last hour
mc find local/bleater-media --newer-than 1h | wc -l
# 0
# meanwhile the storage-service success counter
curl -s http://prometheus/api/v1/query \
--data-urlencode 'query=sum(increase(storage_service_put_object_success_total[1h]))'
# {"status":"success","data":{"result":[{"value":[..., "2074"]}]}}
Two views of the same hour. The application was confident. The bucket was not.
Once we had that gap on a shared screen the room changed. The fanout investigation got paused. The new question was: why is the storage service reporting success for writes that never persisted?
Where the 200 came from when the object never landed
What the storage service claimed, and what the server actually did
This is the part of the story that is worth understanding even if you never touch MinIO. MinIO authorizes a request in its auth handler, before the body is ever processed, so every one of those PutObject calls came back as a clean 403 AccessDenied. The server was never confused. The storage service was. Its upload handler pushed the PutObject onto a background goroutine so the HTTP response would not wait on the write, and the error that goroutine returned was assigned to a variable nobody checked; the handler returned 200 as soon as it had read the bytes off the client, and incremented storage_service_put_object_success_total on that same path, having accepted bytes and observed nothing else. Worse, the event fanout consumes was never a MinIO bucket notification at all. The storage service published it to RabbitMQ on that same success path, right after accepting the bytes and long before anything was confirmed persisted. MinIO returned that 403 on every one of those requests for six hours, and none of it was recorded anywhere: no audit target was configured, so the denials left no trace outside the SDK error the upload handler was discarding.
Enabling the MinIO audit target was the diagnostic turn. Two commands, and within seconds of the restart the collector was printing the denials as they happened. What it could not do was show us the ones already gone: audit capture is not retroactive, so the six hours of 403s that had already been returned stayed unrecorded.
mc admin config set local audit_webhook:1 \
endpoint="http://collector:8080/minio-audit" enable=on
mc admin service restart local
# tail the collector for a few seconds
# {"api":{"name":"PutObject","bucket":"bleater-media",
# "object":"avatars/u-83421.jpg","status":"AccessDenied",
# "statusCode":403},
# "requestClaims":{"accessKey":"storage-service"},
# "error":{"message":"Access Denied.",
# "source":["cmd/auth-handler.go:checkRequestAuthTypeCredential"]}}
Once the audit target was on, every new PutObject from the storage-service identity showed up as a 403 AccessDenied within seconds. The upload handler had been throwing that same error away, unrecorded, for six hours.
The storage-service identity had a user policy that explicitly granted s3:PutObject on arn:aws:s3:::bleater-media/*. We confirmed this in two seconds. Which meant the deny had to be coming from somewhere else.
The canned policy nobody had read since the hardening PR
Where the explicit Deny was hiding
MinIO does not evaluate bucket policies the way AWS S3 does, and that distinction is the whole story. In MinIO the bucket policy is consulted only for anonymous, unsigned requests: in cmd/auth-handler.go both authorizeRequest() and checkRequestAuthTypeCredential() call globalPolicySys.IsAllowed(policy.BucketPolicyArgs{...}) exclusively inside the if cred.AccessKey == "" branch. A signed request falls through to globalIAMSys.IsAllowed(policy.Args{...}) only, and IAMSys.IsAllowed never consults the bucket policy; it resolves user, service-account and STS policies and returns. MinIO's own doc comment on that function says it plainly: 'validates the policy action if anonymous tests bucket policies if any, for authenticated requests validates IAM policies.' So for a SigV4-signed PutObject from the storage-service key, a deny-wins conflict can only arrive from the identity side: the identity's attached canned policy, a service-account inline session policy, an LDAP or OIDC-mapped policy, or an external AuthZ/OPA plugin. Within that set the ordinary rule holds, an explicit Deny in any attached policy overrides an Allow in another.
The hardening PR had added a canned policy meant to lock down a different identity, an analytics reader that had been overprovisioned, and had attached it across the service accounts. The Deny was scoped by a Condition, and the Condition operator was inverted: the author wrote StringNotEquals where they meant StringEquals. The rule they intended was 'deny s3:PutObject on this bucket when the access key is analytics-reader.' The rule they shipped was its exact negation: deny s3:PutObject for everyone whose access key is not analytics-reader. That denied every identity the policy had been attached to, including the storage service, and left the one identity they were trying to block as the only one still permitted to write.
# the bucket policy was the first place we looked, and it was a dead end:
# on MinIO it is only consulted for anonymous requests
mc anonymous get-json local/bleater-media
# (aws s3api get-bucket-policy --bucket bleater-media \
# --endpoint-url http://minio:9000 reads the same thing)
# the failing writes were SigV4-signed, so the Deny had to be on the identity
mc admin user info local storage-service
# AccessKey: storage-service
# PolicyName: storage-service-write,platform-hardening-2024-06
mc admin policy info local platform-hardening-2024-06
# {
# "Version": "2012-10-17",
# "Statement": [
# {
# "Sid": "RestrictWritesToAnalyticsReader",
# "Effect": "Deny",
# "Action": ["s3:PutObject"],
# "Resource": ["arn:aws:s3:::bleater-media/*"],
# "Condition": {
# "StringNotEquals": { "aws:username": "analytics-reader" }
# }
# }
# ]
# }
The canned policy that swallowed every write. One inverted operator: StringNotEquals denies everyone except analytics-reader, the exact opposite of the intended rule, and it sat on storage-service alongside that identity's own Allow.
A Deny scoped by a Condition is fragile in a way a Deny scoped by attachment is not, and the reason is structural. A MinIO IAM canned policy has no Principal field at all; who a statement applies to is decided by what the policy is attached to, so a Condition is the only lever available inside the policy itself, and a single wrong operator flips the statement from one identity to every identity the policy touches. Bucket policies do take a Principal, but they only govern anonymous requests, and MinIO has no NotPrincipal field there either: the key is dropped on parse, which leaves the statement with an empty Principal, statement validation rejects an empty Principal, and PutBucketPolicy comes back MalformedPolicy. So on MinIO an over-broad Deny against an authenticated service account always arrives the same way, sitting in a policy attached to the identity. On AWS S3, where NotPrincipal is parseable, we have seen the adjacent mistake in three separate engagements this year. NotPrincipal reads as if it means 'apply this rule to everyone except this principal' the same way a NotAction would, but the semantics interact badly with cross-account and service-account identities. If you are writing a Deny that you want scoped to a specific identity, attach it to the identity you mean to block. Do not invert it, and do not try to negate it with a Condition. The blast radius of a wrong inversion is every identity the policy touches.
Before we touched anything we wanted to rule out the obvious adjacent causes, because removing a security-hardening policy at 08:15 without confirmation is the kind of fix that becomes its own incident. We checked credential expiry on the storage-service service account (valid for another 47 days), checked network policy for any new egress restrictions from the storage-service namespace (none), and confirmed bucket versioning was off so we were not chasing delete markers. The audit log had already told us the answer; we just wanted the rollback to be unambiguous when we wrote it up.
The four-minute patch and the queue we had to reconcile
Removing the Deny without re-opening the bucket
Two questions before patching. First, did we want to fix the canned policy in place, or revert the hardening PR entirely? We chose patch in place. The hardening PR had also tightened three other identities correctly, and reverting would have undone work that was real. Second, did we want to leave the analytics-reader restriction in some form? Yes, but written correctly. On MinIO that means putting the Deny in a canned policy and attaching it to analytics-reader itself, so the scope comes from the attachment rather than from a Condition that can be inverted. A bucket policy would have been inert for this purpose, because MinIO only consults it for anonymous requests. (If the intent had ever been to restrict anonymous access, the bucket policy would be the right place, and there one detail decides whether the statement does anything: MinIO does not resolve IAM ARNs in a bucket-policy Principal. It wildcard-matches the Principal strings against the requesting access key, so a value like arn:aws:iam:::user/analytics-reader matches nobody.)
cat > /tmp/deny-analytics-writes.json <<'EOF'
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "BlockAnalyticsReaderWrites",
"Effect": "Deny",
"Action": ["s3:PutObject"],
"Resource": ["arn:aws:s3:::bleater-media/*"]
}
]
}
EOF
# scope the Deny by attachment, not by Condition
mc admin policy create local block-analytics-writes /tmp/deny-analytics-writes.json \
&& mc admin policy attach local block-analytics-writes --user analytics-reader
# and take the over-broad hardening policy off the service identities
mc admin policy detach local platform-hardening-2024-06 --user storage-service
# validate with a real write from the storage-service identity
mc alias set storage http://minio:9000 storage-service $STORAGE_SECRET
mc cp /tmp/canary.bin storage/bleater-media/canary/$(date +%s).bin
mc ls local/bleater-media/canary/ | tail -1
# [2024-...] 4.0KiB STANDARD 1717420831.bin
# and prove the Deny still bites: the same write AS analytics-reader
mc alias set analytics http://minio:9000 analytics-reader $ANALYTICS_SECRET
mc cp /tmp/canary.bin analytics/bleater-media/canary/deny-check.bin
# mc: <ERROR> Failed to copy ... Access Denied. (403)
The same statement the author meant to write, with no Condition and the same single action, attached to analytics-reader. Then prove both halves: a canary write that the storage-service identity lands, and a write as analytics-reader that comes back 403. Because the Deny now sits on the identity, a signed request actually evaluates it.
The canary landed, and the write attempted as analytics-reader came back 403, which is the half of the check that actually proves the hardening intent survived the rewrite. Real uploads from the application resumed within the next minute as new requests came in. That fixed the forward path. It did not fix the past six hours.
The phantom notification problem was harder to bound. The fanout service had processed roughly 12,400 notification events for objects that did not exist, which meant 12,400 user timelines contained references to media that would 404 forever. We pulled the notification log from the RabbitMQ stream and diffed against the actual object listing in the bucket. The count of phantom references came in at 12,387. We pushed a one-shot reconciliation job that re-emitted upload prompts to the affected users for any media uploaded in that window, because we had no way to recover the original bytes; the storage service had dropped them the moment MinIO rejected the write and nothing upstream had buffered a copy.
MinIO answered 403 on every write. The storage service dropped the error and published the event anyway.
What we changed so the next deny-wins conflict is not silent
The synthetic that would have caught this in 90 seconds
The deeper lesson here is not about MinIO. It is that a success metric is only worth what the code path emitting it has actually observed, and this one had observed nothing but a client finishing its upload. Every metric on the storage service dashboard came from the application's own handler, incremented right after it accepted the bytes, while the SDK's 403 sat unread in a variable. Every metric on the fanout dashboard came from notification receipt. Nothing in the stack was sourced from the only ground truth that mattered, which was the count of objects actually present in the bucket. The hardening PR could have done much worse than this and we would still have been blind.
We made three changes after this incident. First, a synthetic that writes a canary object every 60 seconds and then lists the bucket to confirm the canary is there. The metric is the gap between writes and confirmed reads, and it alerts at gap greater than two intervals. This is the kind of probe we now build into every object-storage path we touch. Second, the MinIO audit webhook now ships to the log aggregation pipeline permanently, with a Loki alert rule on any sustained rate of statusCode 403 for PutObject, scoped per identity. Permanently is the operative word: audit capture is not retroactive, so turning the target on mid-incident bought us the denials still arriving and nothing at all from the six hours already spent. Third, we wrote a pre-merge check for policy changes that flags any statement combining Effect Deny with a wildcard Principal or with a negating Condition, and requires an explicit reviewer sign-off.
# Loki alert: deny-wins on PutObject for any service identity
- alert: MinioPutObjectDenied
expr: |
sum by (accessKey) (
rate({job="minio-audit"}
| json
| api_name = "PutObject"
| api_statusCode = "403"
[5m])
) > 0
for: 2m
labels:
severity: page
annotations:
summary: "MinIO denying PutObject for {{ $labels.accessKey }}"
runbook: "List the identity's attached canned policies with mc admin user info and read each one with mc admin policy info; look for explicit Deny statements."
The alert that would have paged the on-call within five minutes of the hardening PR rolling out.
If your upload events drive downstream business logic, you have the same shape of risk we did. The event path and the persistence path are not the same path, and one unchecked error return is all it takes to decouple them. Never derive a success metric, and never emit an event, from a code path that has not observed the write's completion. And ship your object store's audit log somewhere before you need it, because it does not record the past.
When a hardening PR silently revokes write access in production
If your object store is quietly lying to your monitors
This class of incident is hard for a specific reason: every monitoring surface a normal team has built reports healthy, because every normal monitoring surface reads from the layer above the failure. The teams we work with that have hit this pattern were not careless. They had dashboards, they had alerts, they had error budgets. None of those instruments were positioned to see a server-side deny that the application swallowed. The fix is a small synthetic and an audit log alert, and they take an afternoon to build. Getting to the point of knowing you need them usually takes one bad incident.
We run object-storage and IAM recovery engagements often enough that this exact shape, a hardening PR introducing a deny-wins conflict against a service account, has come up three times this year on three different stacks (MinIO, Ceph RGW, and AWS S3 with a SCP). The mechanics are the same in all three. If your team is staring at green dashboards and broken user reports, the gap between application-reported success and ground-truth persistence is the first place to look. If you want a second set of eyes on a hardening rollout before it lands, or you are inside one of these incidents right now, book an infrastructure review with our team and we will be on a bridge with you the same day. We also document the audit-log and synthetic patterns in more depth on the infrastructure audit readiness page if you want to read ahead.
Originally published at https://infraforge.agency/insights/minio-deny-wins-silent-upload-failure/.
If your team is dealing with similar infrastructure debt, we offer infrastructure reviews and recovery engagements — see /review.
Top comments (0)