DEV Community: Davie Kibet

The Explanation Gap: Why Explainable AI Still Struggles to Speak Human

Davie Kibet — Thu, 17 Sep 2026 18:22:59 +0000

A model predicts a 92% risk of hospital readmission. The SHAP values show age, prior admissions, and medication adherence as the top contributors. But when a nurse asks, "Why this patient, this week?" The technical answer: 'coefficients multiplied by feature values, printed to four decimal places'. Well this is correct as it is but almost useless in this conversation.

This is the unresolved core problem of explainable AI: getting a model to compute an answer was never the hard part in the long run. Getting it to explain that answer to the person who actually has to act on it, without making the explanation sound more certain than it deserves to be, is the part the field is still working through.

Why Explainability Isn't Optional

In 2015, DARPA launched its Explainable Artificial Intelligence (XAI) program with an explicit goal: to enable end users to "understand, appropriately trust, and effectively manage" AI systems. The program's retrospective makes clear that this was not an academic exercise. High-stakes domains (healthcare, criminal justice, credit, hiring) cannot tolerate decisions that are accurate on aggregate but inscrutable in individual cases.

The field has since developed a rich landscape of explanation methods. LIME (Local Interpretable Model-agnostic Explanations) fits simple surrogate models around individual predictions. Counterfactual explanations answer "what would need to change for the outcome to flip?". Saliency maps highlight influential pixels in images. Attention-based explanations point to tokens a transformer model "attended to." Rule extraction methods distill complex models into human-readable if-then statements.

None of these methods, on their own, answers the question that matters most in high-stakes decisions: Why should I, as a clinician or regulator or affected individual, believe this explanation?

Raw accuracy metrics "the model is 92% accurate" do not address this. They describe aggregate performance, not individual reasoning. A model can be highly accurate and still rely on spurious correlations, proxy variables for protected attributes, or patterns that will not hold under intervention. In healthcare, where a single false negative can cost a life, the "why" is not optional. It is the entire point.

The European Data Protection Supervisor's 2023 TechDispatch on XAI puts it plainly: "It is therefore unacceptable to have a 'black box' effect that hides the underlying logic of decisions made by AI". Opacity can hide bias, inaccuracies, and hallucinations. It prevents affected individuals from understanding, challenging, or correcting decisions that shape their lives.

What SHAP Actual Is (...and Isn't)

To understand why the communication problem is so hard, you need to understand what SHAP values actually represent.

SHAP(SHapley Additive exPlanations) is a method for attributing a model's prediction for a single instance to its input features. It is grounded in Shapley values from cooperative game theory, which answer: given a payoff produced by a coalition of players, how much credit does each player deserve?

In the ML context:

The "players" are input features.
The "payoff" is the model's prediction for a specific instance.
The Shapley value for feature i is the average marginal contribution of that feature across all possible coalitions of other features.

Formally, for a feature i and a set of features F, the Shapley value is:

ϕ_{i} = S \subseteq F ∖ i \sum \frac{∣ S ∣ ! ( ∣ F ∣ - ∣ S ∣ - 1 )!}{∣ F ∣ !} [f (S \cup i) - f (S)]

where f(S) is the model's prediction when only features in S are known.
I know, I know. The formula looks intimidating right? My bad. Lemme explain:
Imagine you're trying to figure out why your friend got 10/10 on a test.
You know they studied three things:

Studying
Getting enough sleep
Practising questions SHAP is like asking:

"How much did each of these things help my friend get that 10/10?"

But there's a catch: studying might help a lot when combined with practice, but not as much without it.

So SHAP tries different combinations:
Studying alone → score goes up a little
Studying + practice → score goes up a lot
Studying + sleep + practice → score goes up even more

It looks at all these different combinations and works out the average contribution of studying.

That's what a Shapley value is:
a way of fairly figuring out how much each feature helped produce the final prediction.

SHAP satisfies three desirable properties: local accuracy (the attributions sum to the prediction), missingness (features not present get zero attribution), and consistency (if a feature's contribution increases, its attribution does not decrease).

What SHAP is not: a causal explanation.

This distinction is easy to lose the moment attributions get turned into sentences. A SHAP value of +0.18 for "prior admissions" means: knowing this patient's prior admissions pushes the model's prediction 18 percentage points above the baseline. It does not mean that reducing prior admissions would lower risk by 18 points in the real world.

As the SHAP documentation warns: "SHAP makes transparent the correlations picked up by predictive ML models. But making correlations transparent does not make them causal!". If the model has learned that zip code is a proxy for race in a biased dataset, SHAP will dutifully explain that spurious association.

This is not a bug in SHAP. It is a feature of any method that explains model behavior rather than data-generating processes. The moment you translate SHAP values into prose; "this factor drove the prediction," "this variable caused the risk"; you risk smuggling in causal claims the method does not support.

The Communication Gap

Here is the crux of the problem.
A typical SHAP output for a single prediction looks like this (simplified):

Base value: 0.12
shap_values: [0.08, -0.03, 0.15, 0.02, -0.01]
feature_names: ['age', 'income', 'prior_admissions', 'medication_adherence', 'distance_to_clinic']

Or, in a more structured form:

Explanation(
    base_values=0.12,
    values=[0.08, -0.03, 0.15, 0.02, -0.01],
    feature_names=['age', 'income', 'prior_admissions', 'medication_adherence', 'distance_to_clinic']
)

For an ML engineer, this is informative as he/she reads Age: +0.08, Income: -0.03 fluently. For a nurse, a patient, a regulator, or a loan officer, it is nearly useless. The numbers live on a transformed scale (often log-odds). The feature names are internal identifiers, not clinical or business terms. A patient, a case worker, or a manager generally cannot, and wouldn't know how much weight to give it even if they could parse the syntax. There is no context: is 0.15 "large" for prior admissions, or typical?

Christoph Molnar's Interpretable Machine Learning emphasizes this repeatedly: interpretability is not just about producing an explanation, but producing one that the intended audience can comprehend and use. A data scientist needs different information than a patient. A regulator needs different information than a clinician.

The EDPS TechDispatch puts it sharply: explanations should be "presented in an understandable way, avoiding jargon and technical complexity". Yet most XAI tools stop at the attribution layer.

There's a second, more consequential gap hiding underneath the first one: an attribution describes how a model behaves, not how the world works. A feature with a large positive SHAP value pushed this model's prediction upward; it did not "cause" the outcome in any scientific sense. A model can pick up a spurious correlation or a genuine but non-causal association, and the attribution method has no way to distinguish these, because it's a property of the model, not of reality.

This is the trap waiting for any tool that tries to translate attributions into plain sentences: it's easy, almost automatic, for "this feature moved the prediction" to slide into "this feature causes the outcome" once it's phrased as an English sentence rather than a number.

So the real bottleneck in xAI right now isn't producing more attributions, rather, building a responsible bridge from attribution to language: one that makes the numbers legible without making them sound more certain, or more casual, than they are. This is the gap that narrashap attempts to bridge.

narrashap: A Case Study in Bridging the Gap

narrashap, a small open-source Python library, is one working example of an attempt at that bridge; not a solved version of the problem, but a useful, inspectable case study in what building toward it actually requires. Its interface is a single function:

from narrashap import narrate

narrated_shap = narrate(
    shap_values=shap_values,
    instance=patient_row,
    training_data=X_train,
    risk_percentage=65.6,
    risk_level="MODERATE RISK",
)

print(narrated_shap)

It takes SHAP output for one prediction, the specific feature values for that case, and the training data the model learned from, and produces a narrative; either from a fixed set of plain-language sentence templates, or through a language model, depending on configuration. Consider the difference directly:

Raw attribution output:

Base value: -0.114 (log-odds)
Predicted value: 0.645 (log-odds)
Race (Black): +1.58
Family history: -0.57
Vitamin D deficiency: -0.30
Hypertension: -0.13
Parity: +0.10

Generated Narrative Output:

Plain-language narrative (illustrative, not from a real tool):

This patient's predicted risk of readmission is higher than average for our population. The model's prediction is driven primarily by their history of prior hospital admissions, which is in the 92nd percentile compared to similar patients. Age also contributes positively to the predicted risk. Income and medication adherence have smaller, offsetting effects, with higher income and good adherence slightly reducing the predicted risk. Distance to the clinic has minimal influence on this prediction.

Important: This explanation describes how the model arrived at its prediction. It does not prove that these factors cause readmission risk, nor should it be used as a diagnosis or treatment recommendation.

Is causal language actively suppressed, or just discouraged in a prompt?

Asking an LLM nicely not to imply causation isn't sufficient on its own — it will occasionally slip. narrashap checks generated text against a list of banned causal phrases after generation, and rewrites if one appears. This has to be more careful than simple keyword matching: "this is not proven to cause the outcome" contains the word "proven" but is making exactly the hedged claim the system wants to allow. Getting the negation handling right took more than one attempt; an early fixed-word-lookback approach produced false positives on precisely the safe disclaimer language it needed to permit. It's a small, unglamorous detail, and exactly the kind that determines whether a safety mechanism works in practice or only in a demo.

Is the generated text checked against the numbers it claims to represent?

This is the part of the problem that gets the least attention across the broader explainability field. Most tooling evaluates whether an attribution method is faithful to the model. Very little evaluates whether a narrative built on top of that attribution is faithful to the attribution itself; whether it mentioned the features that actually mattered, got the direction right, or invented a claim the data doesn't support. narrashap's fidelity scorer is a modest, direct attempt at treating that as a measurable property rather than an assumption.

Does the explanation adapt to who's reading it?

A patient, a clinician, and a fraud analyst don't need identical phrasing, though they need the same underlying discipline about what can be claimed. Domain-specific configuration; implemented here for healthcare and, more provisionally, fraud; lets terminology and tone shift while keeping the hedging requirements fixed.

None of these are solved problems, in this project or in the field generally. But they're the right questions, and they generalize well beyond this one library.

The narrative does several things:

It frames feature values relative to the population (percentiles).
It identifies the top contributors without overstating certainty.
It explicitly disclaims causal interpretation.
It avoids medical advice. This is the kind of explanation a clinician might actually read. It is not perfect. But it is legible.

Honest Limitations

Both the broader project of AI explainability and this particular tool remain unfinished work. narrashap has one meaningfully validated real-world integration — a logistic regression health-risk model — and its design decisions have only actually been tested against that setting. Other model families should work with the underlying extraction logic in principle but haven't been confirmed. It explains individual predictions; it is not a model governance tool, a fairness audit, or a substitute for evaluating a model's behavior across a population, and no explanation-narration layer, however careful, closes that gap on its own.

Bringing a language model into the loop reintroduces familiar risks that the field as a whole hasn't fully solved: occasional hallucination, a tendency toward misplaced confidence in tone, and provider-side model updates that quietly change behavior. Guardrails like a banned-phrase check make failures more likely to be caught — they don't make an LLM path risk-free. And the negation-aware safety check here has a known edge case: a compound sentence with an unrelated negation earlier in it could, in principle, let an unrelated unsafe claim through uncaught. A narrow, acknowledged gap, not a hidden one.

The Field's Unsolved Problems

Beyond narrashap specifically, the field still struggles with:

Evaluating the narrative layer: Most XAI evaluation focuses on attribution methods, not the explanations built on top of them.
Audience-specific explanation: Patient vs. clinician vs. regulator explanations are often an afterthought, not a design norm.
Explanation drift: We monitor accuracy drift across model retraining. We rarely monitor explanation drift—whether the same input yields meaningfully different narratives over time.
Fairness at the individual level: Aggregate fairness audits are common. Surfacing fairness signals at the level of individual explanations is rarer.
Privacy-preserving generation: Local-only, no-API explanation generation is still the exception, not the rule.

Where Explainable AI Plausibly Goes Next

The future of XAI is not "more explanations." It is better explanations: faithful, appropriately uncertain, legible to their actual audience, and honest about what they are not.

Here are genuine, thoughtful directions—none of which narrashap (or most tools) claim to have solved yet:

1. Evaluating the Narrative Layer Itself

The narrative layer should be evaluated as its own research problem, separate from the attribution method. Metrics might include:

Fidelity to underlying attributions.
Comprehension by target audiences (patients, clinicians, regulators).
Resistance to misinterpretation (do users draw causal inferences where none are warranted?).

Recent work on SHAPstories and LLM-based narratives is a start. But standardized benchmarks and evaluation protocols are still emerging.

2. Audience-Specific Explanation as a Design Norm

The EDPS TechDispatch emphasizes that explanations should be tailored to their audience. A patient needs different information than a regulator. A clinician needs different information than a data scientist.

Future tools should make audience a first-class parameter, not an afterthought. This means:

Different vocabularies (clinical vs. lay terms).
Different levels of detail.
Different guardrails (e.g., stronger disclaimers for patient-facing explanations).

3. Detecting Explanation Drift

We monitor model accuracy drift. We should also monitor explanation drift: does the same input yield meaningfully different narratives after model retraining or LLM updates?

This is not just a technical problem. It is a governance problem. If explanations change, affected individuals and auditors need to know.

4. Surfacing Fairness Signals at the Individual Level

Aggregate fairness audits are important. But individuals affected by specific decisions need to know: was this decision influenced by a proxy for a protected attribute?

Future XAI tools should surface fairness-relevant signals at the individual explanation level, not just in aggregate reports.

5. Human-in-the-Loop Review as Permanent

Human review is not a transitional phase. It is a permanent design requirement for high-stakes explanations. Tools should make review easy:

Clear provenance (what model, what version, what data).
Editable narratives (so humans can correct or contextualize).
Audit trails (who reviewed, what changed).

Conclusion: One Honest Contribution

Narrashap is not a finished solution to the explanation gap. It is one honest, practical contribution to a much larger, still-unsolved problem.

It acknowledges that SHAP values alone are not enough. It bakes in guardrails against causal misinterpretation. It offers a zero-dependency fallback for sensitive environments. It treats the narrative layer as something that can (and should) be scored and audited.

The future of XAI will require many such contributions: tools that are faithful to their underlying attributions, appropriately uncertain, legible to their actual audiences, and honest about what they are not (a diagnosis, a proof, a causal claim).

The explanation gap will not close overnight. But projects like narrashap show that the field is moving in the right direction: from "we have explanations" to "we have explanations we can trust."

Below is Narrashap's github link

Further reading: Lundberg, S. & Lee, S.-I. (2017), "A Unified Approach to Interpreting Model Predictions", NeurIPS. Molnar, C., Interpretable Machine Learning.

The Stone That the 'Landlord' Rejected: African Governments vs Startups

Davie Kibet — Tue, 03 Feb 2026 21:42:29 +0000

Biblically, the stone that the builders rejected eventually becomes the cornerstone. Well, in the African tech space, the landlord would rather grind the cornerstone into asphalt to fill a fiscal pothole today than allow it to support a skyscraper tomorrow. We are witnessing a tragic architectural paradox: the very hands that should be steadying the foundation are the ones swinging the wrecking ball.

The Silence of the Flame

The wick has gone cold and the shadows have stepped forward. The silence left by KOKO Fuel’s collapse is deafening. This wasn't the quiet exit of a company that lost its way or a standard startup "pivoting" into the void; it was the sudden extinguishing of a decade-long flame.

KOKO was the rare unicorn of utility—based climate-tech that actually worked, serving 1.5 million households. Their business model was a masterclass in UN's SDG 13 (Climate Action) & SDG 7 (Affordable and Clean Energy), turning carbon credits into cheap fuel for the poor. But here is the irony: the same government that spent the last year hosting global climate summits and declaring "climate holidays" to plant trees is the one that effectively cut KOKO's oxygen. By refusing to sign a "Letter of Authorization" for their carbon credits, the state engaged in a staggering display of policy hypocrisy; publicly advocating for a green future while privately sabotaging the very pioneers making it affordable.

Credits: @davidamunga on X

But the Landlord didn't just lock the door; he burned the building down and sued the street for the smoke damage. KOKO wasn't a lightweight experiment. This was a titan of infrastructure that had invested north of $300M in clean cooking technology and logistics. They put their money where their mouth was, spending over $100M out of pocket to subsidize fuel for 1.5 million households while waiting for the government to simply fulfill its end of the bargain.

Because they understood the waywardness of the Landlord, KOKO secured a $179.6M political risk guarantee from the World Bank. This wasn't just paper; it was a shield against the exact breach of contract we are seeing today.

..the Shadows have stepped forward

Now, as the shadows lengthen, the ultimate tragedy emerges: Taxpayers are staring down a potential Ksh 23Bn bill. If the World Bank pays out that guarantee due to the government’s regulatory obstruction, they will seek to recover every cent from the state. The very people who just lost their clean fuel are now being asked to pay for the ink the government refused to use. It is a masterclass in fiscal self-sabotage.

The government’s hesitation was reportedly about "benefit sharing" they wanted a larger piece of the carbon credit pie. But by holding out for a few percentage points of revenue, they have invited a Ksh 23 billion liability. Beyond the direct Ksh 23B, there is a "ghost tax" now applied to every other Kenyan startup. International investors see the KOKO collapse and add a "Policy Risk Premium" to any future deals in Kenya.

KOKO now takes its place in a crowded graveyard, lying right next to the rusted frame of Mobius Motors (Rest Her Soul). But if you look closely at Mobius’s plot, you’ll see the flowers have long since died. The wreaths of "Buy Kenya, Build Kenya" have withered into gray stalks, and the soil is packed hard by indifference. No one is coming to visit her remains. No government delegation is laying a stone in remembrance of the thirteen years spent trying to build an African car for African roads.

Credits: @alexmwanzo on Twitter
Mobius didn't just run out of gas; it was run off the road by a Landlord that preferred the immediate hit of a tax penalty over the long-term wealth of a local automotive industry. It is a lonely corner of the cemetery. The silence there serves as a grim warning to any other builder brave enough to dream in steel: in this jurisdiction, the Landlord doesn't mourn the stones he rejects, he simply waits for the grass to cover the evidence of his own short-sightedness.

The Sky is Still Grey

As 1.5 million households reach for charcoal tonight, the "green" smoke from the government’s latest climate summit carries a bitter scent of betrayal. We were promised a transition; we were given a funeral.

African governments must decide if they want to be Property Managers or Slumlords. A Property Manager invests in the infrastructure so the tenants can thrive and pay rent for decades. A Slumlord squeezes the tenant for every cent until the building is a shell and the tenants have fled.

To the Guardians of the Paris Agreement,

The collapse of KOKO Networks in Kenya is not just a commercial failure; it is a systemic warning shot for the entire Article 6 framework. When a host government uses the Letter of Authorization (LoA) as a tool for extortion rather than an instrument of climate action, the integrity of your carbon markets is compromised. The planet cannot afford to wait for a bureaucrat to find his pen.

Drop a comment. I’m open-source :)

Beyond the Algorithm: The Tightrope Walk in Health Data Science

Davie Kibet — Fri, 25 Apr 2025 13:50:52 +0000

The application of data science methodologies holds immense promise for revolutionizing healthcare. Personalized medicine, more accurate and timely diagnostics, and the potential for vastly improved efficiency within healthcare systems are all within reach, fueled by advanced algorithms like machine learning and artificial intelligence.

However, a critical challenge lies at the heart of this transformative endeavor: the limitations inherent in the data upon which these powerful algorithms rely. The central question remains: can we truly achieve this transformative vision with the data we currently collect and can readily access?

Beyond the Hospital Walls - The Missing Pieces

Hospitals and clinics are often assumed to be data-rich environments. In reality, the datasets collected in these environments are frequently narrow, incomplete, and inconsistent. Vital signs, lab results, diagnosis codes, medications, and procedures are certainly recorded — but these are often siloed across systems, riddled with missing values, or lack the temporal resolution necessary for dynamic modeling. A significant limitation of relying solely on Electronic Health Records (EHR) is the absence of crucial contextual information that profoundly influences models to be built and their outcomes.

Comprehensive data on lifestyle factors such as diet, exercise habits, smoking and alcohol consumption, and mental well-being are often lacking or inconsistently recorded. Furthermore, EHRs typically do not capture detailed information about environmental exposures, socioeconomic determinants of health (like income, education, and housing), or patient-reported outcomes regarding their quality of life and functional status. Without this broader context, our understanding of why diseases develop and how to prevent them is inherently limited.

The increasing recognition of the importance of social determinants of health has led to efforts to incorporate this information into EHRs. However, the practicalities of consistently and accurately collecting such data within the already demanding clinical workflow present significant challenges. Clinicians have limited time, and their primary focus naturally remains on immediate patient care needs. Gathering detailed information on social circumstances might be perceived as intrusive or time-consuming, leading to incomplete or inconsistent data capture.

This lack of a holistic view can introduce biases and lead to incomplete research findings. For instance, research relying solely on hospital data might incorrectly attribute disease causes or fail to identify effective preventative measures rooted in lifestyle modifications or addressing environmental risk factors.

Locked Away: The Accessibility Obstacle

It’s truly unfortunate that valuable health data, data that could significantly improve healthcare outcomes and advance medical knowledge, frequently remains inaccessible. This data often resides outside the secure walls of established hospital systems, scattered across various platforms and databases. Alternatively, it may be present within hospitals but buried deep within lengthy and complicated reports, making it difficult to extract and utilize effectively. The process of attempting to retrieve this essential information for research purposes can often feel like navigating a complex and frustrating obstacle course, filled with numerous hurdles and challenges.

The existence of privacy laws, such as the Health Insurance Portability and Accountability Act (HIPAA) in the United States and the General Data Protection Regulation (GDPR) in Europe, is undeniably crucial and well-intentioned. These laws serve a vital purpose: to safeguard our sensitive personal health information from unauthorized access and misuse, ensuring the confidentiality and security of individuals' medical records. However, while these regulations are essential for protecting privacy, they also introduce significant complexities into the process of accessing and sharing health data for research and other legitimate purposes.

Synthetic Data: A Silver Bullet or a Faustian Bargain?

In response to the challenges of accessing real-world health data, synthetic health data has emerged as a potential alternative. This artificially generated data is designed to mimic the statistical properties of real data without containing identifiable patient information. The benefits of using synthetic data are numerous. It can overcome privacy concerns, enable broader data sharing among researchers, and facilitate the development and testing of algorithms without the need for complex data access agreements. This can significantly accelerate the pace of research in privacy-sensitive areas.

However, for all its usefulness, synthetic data is no replacement for reality. It lacks the messiness, the edge cases, the human quirks that make health data uniquely complex. Models that perform well on synthetic data often falter when exposed to real-world clinical environments.

But Why?

Synthetic Data is too 'clean'.

Real-world data is messy — full of missing values, inconsistent formats, typos, outliers, and contradictions. Synthetic data, by contrast, is usually generated using rules or models that follow distributions too neatly. So, models trained on it don’t learn how to handle chaos.

Lack of Rare Cases (Edge Cases)

In healthcare, those 1-in-1000 scenarios really matter — like a rare adverse drug reaction or an unusual combination of symptoms. Synthetic data often fails to include these rare but critical edge cases, making the model blind to them.

Missing Human Behavior and Judgment

Real medical records reflect human quirks: how doctors phrase things, how patients describe pain, or even how data entry staff input info. Synthetic data can’t replicate those subtle human factors, which actually influence outcomes a lot.

Bias in the Synthetic Generator

Synthetic data is as good as the model it is created on. Suffice to say, if that model itself was trained on biased or limited real data, the synthetic output will reflect and possibly amplify those same biases.

The Real-World Challenges: An Example in Female Sexual Health Research

The challenges faced in this research aren’t just theoretical—they’re real, tangible roadblocks that emerged when I was working on data science models in sensitive areas like female sexual health, particularly uterine fibroids. This field is critical for millions of women, yet the data barriers can feel insurmountable at times.

The Struggle to Access Reliable Data

One of my biggest frustrations has been securing comprehensive datasets. When I reached out to organizations like the WHO or healthdata.gov, I often encountered denials or bureaucratic delays. Some data exists—like the Texas Department of State Health Services’ records on fibroid diagnoses or the Global Burden of Disease Study’s prevalence statistics—but it’s scattered across different sources. Each one requires separate permissions, and the process is slow and exhausting. I’ve spent weeks just trying to get access to what should be readily available for research that could improve women’s health outcomes.

The Problem of Poorly Curated Variables

Even when I finally get my hands on a dataset, I’ve found that many aren’t well-structured for my research. One time, I opened a dataset supposedly focused on women with fibroids—only to find a redundant "gender" column. If the data is exclusively about women, why include a gender field? It’s a sign that the dataset wasn’t designed with this research in mind. I need variables like age, race, family history, and menarche age—known risk factors—but instead, I waste time cleaning irrelevant data. Furthermore, upon cleaning, I am left with meaningless columns to work with.

Fallback or perhaps fall back

When real-world data is too hard to obtain, I’ve had to rely on synthetic datasets. But this isn’t a perfect solution. If the synthetic data isn’t carefully constructed—with deep knowledge of fibroid biology and clinical realities—the models I train on it perform well in theory but fail in practice. I’ve seen models that look promising in simulations but collapse when applied to actual patient cases. It’s disheartening, knowing that these limitations could delay meaningful advancements in women’s health.

Conclusion: Care before Code
Unlocking the full potential of health data science starts with better data — not just more of it, but more meaningful, diverse, and representative variables. To get there, we need open, privacy-conscious data sharing and stronger collaboration between hospitals, researchers, policymakers, and technologists. Only through this collective effort can we move beyond the algorithm and build solutions that truly serve everyone.

This goes without saying — I was helped a great deal by Elizabeth Waithera, a data scientist, in shaping this thought, especially in highlighting the need for collective responsibility and shared access in the health data ecosystem.
Have questions? Ping me. I don’t bite (unless you're a fraudulent transaction).

Deepnote: To Use or Not To Use?

Davie Kibet — Mon, 10 Mar 2025 22:52:06 +0000

In my wandering around the various data science tools and frameworks, I discovered Deepnote, an online framework that allows you to create and run notebooks in Python.

In the rapidly evolving field of data science, tools that streamline workflows and enhance collaboration are invaluable. Deepnote stands out by combining the strengths of Power BI, Google Colab, and Jupyter Notebooks while integrating powerful AI features that simplify data science operations. This blog explores how Deepnote brings together these functionalities, making it a comprehensive tool for modern data scientists.

The Strengths of Power BI, Colab, and Jupyter

Power BI: Business Intelligence and Visualization

Power BI is renowned for its robust data visualization capabilities and business intelligence tools. It allows users to create interactive reports and dashboards, making it easier to analyze data and share insights with stakeholders.

Google Colab: Collaboration and Cloud Computing

Google Colab excels in real-time collaboration and seamless cloud computing. It allows multiple users to work on the same notebook simultaneously and provides access to powerful computing resources, including GPUs and TPUs, which are essential for training machine learning models.

Jupyter Notebooks: Flexibility and Interactivity

Jupyter Notebooks offer an interactive computing environment that supports live code, equations, visualizations, and narrative text. It’s highly flexible, supporting numerous programming languages and integrating well with various data science libraries.

How Deepnote Combines These Features

Real-Time Collaboration

Deepnote takes collaboration to the next level by enabling real-time editing and commenting, similar to Google Colab. Teams can work together seamlessly, making it easier to develop and refine models collectively. This feature enhances productivity and ensures that everyone is on the same page.

Powerful Data Visualizations

Like Power BI, Deepnote provides robust data visualization capabilities. Users can create interactive charts and graphs using built-in support for libraries like Plotly, Matplotlib, and Seaborn. These visualizations can be embedded directly into reports, making it easy to communicate findings.

Interactive and Flexible Notebooks

Deepnote’s core functionality is built around interactive notebooks, much like Jupyter. These notebooks support live code execution, making it easy to test hypotheses and iterate on models. Deepnote also supports various programming languages, ensuring flexibility in data analysis.

Integrated Data Sources and Cloud Storage

Deepnote integrates seamlessly with various data sources, including databases like PostgreSQL, MySQL, and BigQuery. It also supports cloud storage integrations with Google Drive and AWS S3, making it easy to access and store large datasets.

AI Integration: Simplifying Data Science Operations

Automated Machine Learning (AutoML)
Deepnote’s integration with AI technologies includes automated machine learning (AutoML) features. These tools automate the process of selecting, training, and tuning machine learning models, significantly reducing the time and expertise required to build high-performing models.

Natural Language Processing (NLP) Tools

Deepnote includes powerful NLP tools that enable users to analyze and process text data efficiently. These tools are integrated into the notebook environment, allowing for seamless transitions between data cleaning, analysis, and model development.

Predictive Analytics

Deepnote leverages AI to provide predictive analytics capabilities. Users can build models that predict future trends and behaviors based on historical data, enabling data-driven decision-making.

Smart Suggestions and Code Completion

Deepnote’s AI-driven smart suggestions and code completion features help users write code more efficiently. These tools reduce the learning curve for new users and speed up the development process for experienced data scientists.

Practical Use Cases

Healthcare
In healthcare, Deepnote can be used to analyze patient data, predict disease outbreaks, and optimize treatment plans. Its collaborative features enable healthcare professionals to work together, ensuring comprehensive analysis and improved patient outcomes.

Finance
Financial analysts can use Deepnote to develop predictive models for stock prices, analyze market trends, and optimize investment strategies. The integration with various data sources and powerful visualization tools makes it ideal for financial data analysis.

Retail
Retailers can leverage Deepnote to analyze customer behavior, optimize supply chains, and enhance marketing strategies. The platform’s AI integration helps in identifying patterns and making data-driven decisions to improve business performance.

Conclusion

Deepnote is a powerful tool that combines the best features of Power BI, Google Colab, and Jupyter Notebooks while integrating advanced AI capabilities. Its collaborative environment, robust visualization tools, and flexible notebook interface make it an ideal choice for data scientists. By simplifying data science operations and enhancing productivity, Deepnote is paving the way for more efficient and effective data analysis.

Cultivating Excellence: The Defining Habit of Top Data Professionals

Davie Kibet — Mon, 10 Mar 2025 22:38:52 +0000

Most people think documentation is just an afterthought something you do at the end of a project to keep things tidy.

But here’s the truth: Documentation isn’t just about writing things down; it’s a strategic tool that makes you a better data professional.

I don’t just document my work after the fact. I start before I even write a single line of code. Why? Because every decision you make, from choosing a project topic to selecting the right tools, impacts your final outcome.

📍 What Should You Be Documenting?

If you’re only documenting code, you’re missing the bigger picture.
Here’s what you need to track in every project:

✔ Project Title & Overview:
What’s the project about? What’s the goal? A clear overview helps anyone (including future you) quickly understand the project.

✔ Problem Statement
What challenge are you solving? Why does it matter? This keeps your work focused and meaningful.

✔ Tool Selection: Which tools will I use and why? Maybe Python is better for automation, but SQL is more efficient for data extraction. Documenting this helps you reflect on your choices later.

✔ Data Sources and preparation: Where is my data coming from? Is it reliable? Did I preprocess it? If you revisit the project in six months, you’ll want these answers.

✔ Decisions and Changes: Every major decision—why you used a certain algorithm, why you cleaned data a specific way—should be logged. This prevents “past you” from confusing “future you.”

✔ Key Insights & Learnings: What worked? What didn’t? What trends did I uncover? This is where the real value of your work shines. What would I do differently next time? These notes become your personal knowledge base. And it turns every project into a learning experience.

✔ Recommendations & Next Steps
What should be done based on your findings? Are there improvements or further research needed? This makes your work actionable.

📍 Where Do I Document?

•➤ For Quick Brainstorming → Paper: Nothing beats handwritten notes for capturing raw ideas and sketching out workflows.

•➤ For Ongoing Documentation → Digital Tools (GitHub, Notion, Google Docs, Jupyter Notebooks):
• Easily updated and searchable
• Allows me to add screenshots, links, and code snippets
• Keeps my workflow structured and accessible

📍 Why This Matters

Great data professionals don’t just analyze data; they think critically, track their thought process, and refine their approach over time.

If you’re not documenting your work, you’re making your job harder than it needs to be. Start early, be intentional, and turn documentation into a competitive advantage.

Your future self will thank you.

Supercharge Your Front-End Workflow in 2024: Must-Have VS Code Extensions

Davie Kibet — Thu, 08 Feb 2024 17:29:32 +0000

Let's face it, VS Code is awesome, but extensions take it to a whole new level. As a frontend dev in 2024, you need tools that boost your efficiency and stay ahead of the curve. So, ditch the vanilla setup and dive into these essential extensions:

For Code Quality & Consistency:

ESLint & Prettier: The dream team for code style. ESLint sniffs out errors, Prettier auto-magically formats for consistent beauty.
Stylelint: CSS needs love too! Stylelint enforces consistent style rules for your stylesheets.
For Productivity & Navigation:
Live Share: Real-time collaboration just got easier. Share your code editor with teammates for instant pair programming.
Bracket Pair Colorizer & Path Intellisense: Gone are the days of lost brackets and confusing file paths. These extensions add a much-needed splash of color and context.
GitLens: Dive deep into your Git history. Blame annotations, code authorship insights, and visual commit graphs - GitLens supercharges your version control workflow.
For Specific Frameworks & Languages:
React/Angular/Vue.js Snippets: Say goodbye to repetitive boilerplate. These framework-specific snippet extensions speed up development with pre-built components and code structures.
Emmet: Write HTML and CSS faster with shorthand notations that auto-expand into full code. A must-have for rapid prototyping.
Debugger for Chrome/Live Server: Debugging made easy. These extensions let you step through your code and preview changes live, directly in your browser.
Bonus Round:
Polacode: Create stunning code screenshots for presentations and docs. Share your code with flair!
CodiumAI: Generate tests automatically based on your code. This AI-powered extension helps you write better, more robust code.
Remember: This is just a starting point. Explore, experiment, and find the extensions that fit your specific workflow and preferences. Happy coding!

P.S. Did we miss your favourite extension? Please share it in the comments below!

Embracing the Future: Emerging Trends in Software Engineering

Davie Kibet — Thu, 08 Feb 2024 12:20:19 +0000

Riding the Wave: DevOps Embraces Emerging Software Engineering Trends
DevOps has revolutionized software development by fostering collaboration, automation, and agility. But the landscape is always evolving, and new trends are constantly emerging. To stay ahead of the curve, it's crucial for DevOps practitioners to embrace these innovations and integrate them into their workflows. So, let's dive into some exciting trends shaping the future of DevOps:

1. DevSecOps Takes Center Stage: Security is no longer an afterthought. DevSecOps seamlessly integrates security practices into the entire development lifecycle, from code creation to deployment. This proactive approach minimizes vulnerabilities and ensures applications are built secure from the ground up.

2. AI & Automation Unleash Efficiency: Imagine AI automating tedious tasks like configuration management, infrastructure provisioning, and performance monitoring. That's the power of AI in DevOps. It frees up valuable human resources for higher-level thinking and innovation, boosting overall team productivity.

3. Cloud-Native Reigns Supreme: Cloud adoption is exploding, and DevOps practices are adapting. We're seeing a shift towards cloud-native architectures, containerization with tools like Kubernetes, and serverless computing for more scalable and agile deployments.

4. Infrastructure as Code (IaC) Becomes Indispensable: Treating infrastructure like code allows for consistent, repeatable deployments and eliminates manual configuration errors. Tools like Terraform and Ansible are empowering DevOps teams to manage infrastructure efficiently and securely.

5. Low-Code/No-Code Democratizes Development: Low-code/no-code platforms are opening the door for non-technical individuals to contribute to the development process. This fosters collaboration and empowers citizen developers to build simple applications and automate workflows.

6. Continuous Learning, Continuous Improvement: In today's dynamic environment, continuous learning is key. DevOps teams need to embrace new technologies, methodologies, and best practices. Platforms like Gitcoin and online communities can facilitate knowledge sharing and upskilling within the DevOps community.

7. Platform Engineering Empowers Developers: Platform engineers build internal developer platforms (IDPs) that provide essential tools, resources, and services. This self-service environment empowers developers to focus on innovation and accelerate delivery.

8. Edge Computing Demands DevOps Transformation: As edge computing gains traction, managing distributed applications at the edge requires new DevOps approaches. Edge DevOps emphasizes decentralized deployments, containerization, and automation tailored for low-latency, geographically dispersed environments.

9. Sustainability in Focus: Green DevOps practices promote environmentally conscious software development and operations. This includes optimizing resource usage, reducing energy consumption, and choosing eco-friendly tools and technologies.

10. The Human Touch Remains Vital: While automation is important, it's crucial to remember that DevOps is about people. Effective communication, collaboration, and cross-functional understanding are still essential for success.

Embracing these emerging trends will position your DevOps team for continued success in the ever-evolving software development landscape. So, stay curious, experiment, and keep riding the wave of innovation!

What are your thoughts on these trends? Share your experiences and predictions for the future of DevOps in the comments below!

Let's keep the conversation going!