A few months ago I watched a documentary called Error by Design. Africa Uncensored and Lighthouse Reports had reconstructed the algorithm Kenya's Social Health Authority uses to decide how much you pay every month and found that it scores you partly by what your roof, walls, and floor are made of. Not your income. Your roof.
The investigation was rigorous. It reconstructed the actual model from public data and confirmed what a pre-launch report had already warned the government about: the poorest households get systematically overcharged, and it isn't a rounding error, it's the design.
I'm a computing student in Nairobi. I didn't have the standing to change SHA's algorithm. But I had a laptop and I know how to write software, so I decided to build the version I thought it should be same legal contribution rate, same statutory floor for the indigent, but scored on adjusted income instead of asset proxies and put the entire thing online, for free, for anyone to check.
This is what that actually looked like from the inside, including the parts that didn't make me look especially good.
The easy part was the algorithm
Building a means-testing formula that scores income instead of roofing material is, honestly, not the hard part. You take someone's M-Pesa activity, their KRA status, their declared assets, apply reasonable deductions for dependents and chronic illness and disability, and you have something dramatically fairer than "does this household own a radio." I had a working prototype in a few weeks.
The hard part turned out to be everything after that.
Auditing my own work turned up things I didn't expect
At some point I stopped adding features and started trying to break what I'd built, the same way I'd want someone to break the original algorithm. That turned out to be humbling.
I found a claim in my own interface that said the model was "trained with an equalized odds constraint enforced at the optimization level" sitting three lines above a metric that honestly read "Unmeasured." The model isn't trained at all. It's a deterministic rules engine. I had, without quite noticing, let a sentence drift into claiming a rigor the software didn't have. That's the exact failure mode I was trying to hold the original system accountable for.
I found a field labeled citizenIdHash that was supposed to anonymize phone numbers for an audit trail and it wasn't hashing anything. It was a template string with the word "SHA256" glued onto four real digits of the number. Cosmetically it looked like security. It wasn't.
I found a crash bug that only appeared for seasonal workers who declared a zero income in the off-season a real, common situation for agricultural and tourism workers because a variable got referenced 230 lines before it was declared. My hand-picked test cases never hit it. It took generating five hundred randomized synthetic households to find it on the first run, which taught me more about the actual limits of my own testing than anything else in the project.
I found that a single car and two cars were priced identically, because the data model had no way to represent owning more than one of anything which meant a household with one old car and one luxury SUV could report the cheap one and make the expensive one disappear from the assessment entirely.
None of these were things I'd have found by being confident. I found them by assuming I was probably wrong somewhere and going looking.
The part I'm least proud of
At one point, partway through this, I was handed what looked like a detailed intelligence briefing a leaked override dashboard, a specific court petition with a specific filing date, a World Bank funding freeze tied to the reform, all wrapped in confident, specific, citable-sounding language. It was compelling. It was also fabricated, invented by an AI system doing what those systems sometimes do when asked to speculate confidently about the future: filling gaps with plausible-sounding specifics and presenting them as fact.
I checked before using any of it. Every specific claim I could verify, I did; against primary sources, court records, the actual investigation's own published data. Most of it didn't hold up. If I hadn't checked, I'd have built part of this project's credibility on invented lawsuits and a fake leaked dashboard, and the first person who fact-checked it properly would have had every reason to stop trusting anything else I said.
I don't think that risk is unique to me. It's what happens when a confident, well-formatted answer feels like it should be true. The only fix I found was the boring one: check the specific claim, not the tone it's delivered in.
What's actually real
The things that did hold up were worth holding onto. The R² of the original reconstructed model; 0.66 for the urban submodel, 0.46 for rural is confirmed word for word in Lighthouse Reports' own published methodology. Eighty percent of the poorest households having their situation overpredicted is their number, stated directly, not something I calculated and am attributing to them. A High Court ruling did find the SHIF rollout premature and order corrective measures. A CAJ disclosure process did happen, even if the exact date I'd first been given for it was wrong by several months once I checked.
Getting the real citations right mattered more to me, by the end, than any individual bug fix because a reform proposal that plays fast and loose with its own sourcing has no standing to criticize anyone else's.
Where it stands
It's live. The full methodology and every weight is public on GitHub. There's a version anyone can run in their browser with their own numbers, entirely client-side nothing you enter is sent anywhere, which is itself a small act of the same transparency I was arguing the original system needed.
It runs a synthetic population through the exact same fairness test the original investigation used Fisher's exact test, disparate impact across county and gender instead of just asserting the word "fair" and hoping nobody checks. I'd rather it fail that test honestly than pass it by never running it.
I sent it to the two journalists who did the original investigation. I emailed KIPPRA directly, asking for a technical review, not an endorsement. I emailed SHA itself, and separately emailed the CEO. I emailed the Commission on Administrative Justice. I posted it publicly, tagged the relevant ministry, human rights and constitutional law organizations, and the outlet that broke the original story. That was two months ago. Since then, close to silence. A like here, no replies anywhere that mattered.
What I actually learned
I went into this thinking the interesting problem was the algorithm. It wasn't. The interesting problem was staying honest about my own work at the same standard I was applying to someone else's catching my own overclaims, checking my own citations, not letting a confident-sounding shortcut stand in for actually being right.
I don't know if this project changes anything about how SHA assesses four million Kenyans a month. That was never fully in my hands. What was in my hands was building something true, and being honest in public about the places it wasn't, yet, until I fixed them.
That part, at least, I got to finish.
The code, the methodology, and the full audit trail including the mistakes are public at github.com/pmkaulani/sha-pmt-reform. The live tool is at sha-pmt-reform.vercel.app.
Top comments (0)