What Happens When "Secure" and "Available" Disagree? Building a Healthcare Access-Control System Taught Me There's No Clean Answer
Fintech's constraint was correctness, never double-charge someone. Chaos engineering's constraint was recovery, does the system actually heal, not just look healed. Healthcare access control turned out to be a different problem again: what do you do when being strict and being available point in opposite directions?
That's the question behind access-sentinel, a simulated patient-records access system I built to work through how healthcare software actually handles this tension, not in theory, but in the specific moments where the "right" answer isn't obvious.
The setup: RBAC is the easy part
Role-based access control itself isn't the hard problem. Doctors get clinical and billing data, nurses get clinical only, billing staff get financial data only, and admins (deliberately) get none of it directly; they manage the platform, not patient records. That's a straightforward policy to write and enforce at the API layer.
The hard problems start at the edges: what happens in an emergency, and what happens when part of the system is down.
Edge case one: break-glass access
Real healthcare systems have a well-known pattern called "break-glass", a clinician can override normal access restrictions in a declared emergency. The tension is obvious once you say it out loud: you can never let "emergency" become an unmonitored backdoor, but you also can never let strict access control block someone from saving a life.
I implemented this as an explicit override: a request can include a break-glass header and a required justification reason. Two things I was careful about:
-
Only clinical roles can break-glass. A billing or admin account requesting an emergency override gets a flat
403, there's no legitimate emergency-care reason for those roles to need it, so the override surface is deliberately narrow. -
A justification is mandatory, not optional. A break-glass request without a reason gets rejected with a
400before it ever touches patient data. The override isn't free, you have to state why. - Every override is logged and counted separately from normal access, specifically so a spike in break-glass usage is visible as an anomaly signal, not buried in normal traffic.
The mechanism doesn't prevent misuse, nothing purely technical can, because the whole point is that a human is allowed to override the rules. What it does is make misuse expensive to hide and cheap to detect after the fact.
Edge case two: what happens when the audit log is down
This is the design decision I spent the most time on. If every access needs to be logged, what happens to a request when the logging store itself is unreachable?
There are two clean-sounding answers, and they're both wrong on their own:
- Fail closed (reject all requests until logging is restored) is the "safe" choice, nothing goes unlogged, but it means a doctor can't pull up a patient's chart during an outage, which is a genuinely dangerous failure mode in an emergency.
- Fail open (let everything through, log later) keeps care running, but if "log later" quietly never happens, you've built an access-control system with no actual record of who saw what.
I ended up with a hybrid, split by role: clinical reads (doctor, nurse) fail open, the request succeeds, the response carries a header flagging the system as degraded, and the access event gets buffered in memory to flush to the audit store once it recovers. Non-clinical reads (billing) fail closed, a 503, because there's no emergency-care justification for blocking that traffic, so strict compliance wins by default.
I tested this by actually forcing the audit store offline under sustained load, not just a single request, and watching Grafana through the outage:
- The buffered-log counter climbed steadily during the outage, confirming events were queuing rather than silently dropping
- Billing requests sat flat at zero (correctly blocked) the whole time
- Once the store came back online, the buffered counter dropped back to zero, the queued events actually flushed, not just accumulated forever
That last part mattered most to me. "Fail open" is an easy design decision to state and a much harder one to prove, the difference between "available" and "available and honest about what happened" is exactly the buffered-and-flushed behavior, not just the initial 200 response.
The tension I didn't expect: immutability vs. the right to be forgotten
The audit log is append-only by design, every entry is chained to the previous one with a SHA-256 hash, and any attempt to edit or delete a log entry is rejected outright. That's the right call for audit integrity: if the log can be edited, it's not actually proof of anything.
But that same immutability runs directly into privacy regulations like GDPR's "right to be forgotten," which require that personal data be deletable on request. An access log that can never be edited or deleted is, by construction, incompatible with a legal right to erasure.
Real systems handle this with patterns like pseudonymized lookup tables, the immutable log stores a reference ID, not the person's actual identity, and the identity mapping (which can be deleted) lives separately. I didn't build that layer here, but naming the tension directly, rather than pretending an append-only log is a strictly better design with no trade-offs, felt more honest than leaving it out.
What this project didn't try to be
This isn't HIPAA-compliant software, and I'm not claiming it models every real constraint a production healthcare system would need — real deployments have encryption-at-rest requirements, formal compliance audits, and legal review this project never touched. What it does model honestly is the shape of the actual trade-offs: strict-but-blocking versus available-but-riskier, and the fact that a good system doesn't pick one side of that trade-off globally, it picks differently depending on what's actually at stake for a given request.
Top comments (0)