DEV Community: Atul Kumar

Building Responsible AI Ecosystems for Public Sector Transformation

Atul Kumar — Sun, 26 Jul 2026 06:54:22 +0000

A private company can release a flawed AI feature, roll it back, apologise, and move on. A government agency doesn't have that option. When AI is used to determine benefits eligibility, detect fraud, or prioritise citizen services, the consequences are much bigger. People affected by those decisions usually can't opt out, can't easily challenge the outcome, and can't switch to another provider. That's exactly why responsible AI in the public sector can't be treated as a compliance exercise added at the end of a project. It has to be built into the system from the very beginning.

Having worked with public sector teams adopting AI frameworks, I've seen these discussions firsthand. Many conversations start with a simple question: Should this process be automated at all? In my experience, the biggest challenge isn't a lack of good intentions. Most teams genuinely want to improve services while protecting citizens. The real problem is that the development practices, delivery timelines, and engineering patterns that work well for consumer applications often don't translate to government systems. In the public sector, the person on the other side isn't just a customer using an app. They're a citizen whose access to essential services may depend on that decision, and in most cases, there isn't an alternative provider they can turn to. That's what makes building AI for government fundamentally different.

Why Public Sector AI Is a Different Problem, Not a Harder Version of the Same One

It's easy to think of government AI as enterprise AI with a few extra approval steps and a lot more paperwork. In reality, the differences run much deeper.

In a commercial product, an inaccurate recommendation might mean a lost sale or a frustrated customer. In government, the consequences are far more significant. An incorrect decision could delay disability support, deny someone housing assistance, or wrongly flag an individual for fraud. The level of error that might be considered acceptable in an e-commerce recommendation engine simply isn't acceptable when the outcome affects a person's livelihood or access to essential services.

Another important difference is the relationship between the system and the people using it. Customers choose whether to download an app or use a company's service. Citizens rarely have that choice when interacting with a government agency. They can't simply switch to another provider if they disagree with a decision. That changes how we think about consent, transparency, and accountability. People need to understand how decisions are made, have a clear way to challenge them, and know that a human can review the outcome when necessary.

The impact of mistakes also tends to grow over time. If an AI model performs poorly in a retail application, declining sales or unhappy customers usually expose the problem quickly. In government, the warning signs are often much harder to spot. An AI system that incorrectly denies eligible citizens a benefit may continue making those mistakes for months before anyone realises there's a pattern. Many of the people affected may not have the resources, knowledge, or confidence to question an automated decision. That's why continuous monitoring, regular audits, and human oversight aren't optional features. They're essential safeguards that help ensure AI continues to serve the public fairly.

Start With the Decision, Not the Model

One of the biggest mistakes I see in public sector AI projects is that teams begin by asking, *"What can this model do?"*Instead, the first question should be, "What decision are we trying to support, and what happens if the system gets it wrong?" Those questions may sound similar, but they lead to very different design choices.

Before building anything, it's`` worth taking a step back and classifying the decision based on two factors: its impact and how easy it is to correct if a mistake happens. Some decisions are low risk and easy to reverse. For example, suggesting the next form a citizen should complete is unlikely to cause lasting harm if the recommendation isn't perfect. Those kinds of tasks are good candidates for automation with minimal oversight.

Other decisions carry much greater consequences. Determining whether someone qualifies for financial assistance or identifying a person for a compliance investigation can directly affect lives. Even if there is an appeals process, correcting a wrong decision can take weeks or months and place a significant burden on the individual. In these situations, AI should support the person making the decision, not replace them. The final judgment should remain with a human who can consider context, apply discretion, and take responsibility for the outcome.

`plaintext
Screenshot: a simple decision-classification framework

Decision: "Should this application be auto-approved?"

Impact if wrong: HIGH (loss of benefit, delayed care)
Reversibility: LOW (appeals take weeks, harm is immediate)
Affected population: Often already vulnerable

→ Classification: Human-in-the-loop required.
AI role: surface relevant facts, flag inconsistencies,
recommend a decision with reasoning — final call stays human.
`
Spending time on this classification before development begins can prevent one of the most damaging patterns I've seen in real projects. A system is introduced as a tool to help people make decisions, but over time, it quietly becomes the decision-maker. As workloads increase and case backlogs grow, staff naturally begin relying more heavily on the AI's recommendations. Eventually, approvals become routine, and human review exists more on paper than in practice.

This isn't usually the result of negligence or bad intent. It's a consequence of operational pressure. When a team is trying to work through thousands of cases, it's understandable that people look for ways to move faster. But that's exactly why these risks need to be anticipated during the design phase rather than after deployment.

If you know your organisation is likely to face staffing shortages or large case volumes, design the system with that reality in mind. Build meaningful review checkpoints, require human intervention for high-impact decisions, monitor approval patterns, and regularly check whether people are genuinely exercising judgment or simply confirming the AI's recommendation. Responsible AI isn't about assuming people will always follow the process. It's about designing systems that continue to protect fairness and accountability even when the pressure is on.

Transparency That Actually Means Something

Explainability is often reduced to technical metrics like confidence scores, SHAP values, or attention maps. Those tools can help engineers understand how a model behaves, but they rarely answer the questions that matter most to the person affected by the decision.

If a citizen is denied a benefit or service, they don't want to know that the model was 92% confident. They want clear, practical answers. Which information influenced the decision? What would need to change for the outcome to be different? If the decision is incorrect, how can it be challenged? Those are the explanations that build trust because they're useful, understandable, and actionable.

Delivering that kind of transparency isn't just a technical challenge. It's a product, policy, and operational responsibility. The system should record exactly which information contributed to each decision and preserve it in a way that can be translated into plain language for the individual involved, not just for auditors or technical teams reviewing logs months later.

Just as importantly, the appeals process has to work in practice, not only in policy documents. Giving someone a form to submit isn't enough if it disappears into an overloaded queue with no visibility or accountability. An effective appeals process is accessible, adequately staffed, and timely. People need confidence that their case will be reviewed by someone who has the authority to understand the context, correct mistakes, and make a fair decision when the AI gets it wrong.

`json
Screenshot: the difference between a technical explanation and a real one

// what the model produces internally
{
"decision": "deny",
"confidence": 0.83,
"top_features": ["income_reported", "household_size", "prior_claims"]
}

// what the citizen-facing explanation actually needs to say
"Your application was denied because reported income exceeds
the threshold for household size 3 by $412/month. If your
income has changed since you applied, you can submit updated
documentation for reassessment within 30 days."
`
This is where many public sector AI initiatives fall short. They invest heavily in building technical explainability for engineers and auditors, then assume the job is done. From a system perspective, it often is. From a citizen's perspective, it isn't.

The information needed to understand how a model reached a decision is not the same as an explanation a person can actually use. Technical insights are valuable for validating and improving models, but they don't help someone understand why they were denied a service or what they can do next.

Closing that gap requires shifting the focus from explaining the model to explaining the decision. The goal isn't just to make AI interpretable for technical teams. It's to make outcomes understandable, transparent, and actionable for the people whose lives are affected by them. That's the difference between building a system that is technically explainable and one that genuinely earns public trust.

Governance Has to Predate the Pilot, Not Follow It

A pattern I've watched repeatedly: a pilot launches with a small, well-intentioned team, gets promising results, and gets pressure to scale fast — at which point people realise there's no governance structure actually built for scale, because the pilot was small enough that informal oversight worked fine.

The fix is establishing governance as part of the pilot design, not as a phase-two concern. That means, before the first citizen is affected: a named owner accountable for the system's decisions, not just its uptime. A documented process for what happens when the system is wrong — who reviews it, how fast, and what recourse the affected person has. Clear criteria for what triggers a pause or rollback, agreed before you need them, not improvised during an incident. And a process for periodic re-evaluation, because a model that was fair and accurate at launch can drift as the population it serves changes.

`markdown
Screenshot: minimal governance record kept alongside the system

Benefits Eligibility Assistant — Governance Record

Owner: [Program Director], accountable for decision quality
Human sign-off required: all denials, all flags for investigation
Review cadence: quarterly fairness audit across demographic groups
Rollback trigger: appeal overturn rate > 15% in any 30-day window
Last audit: 2026-06-14 — no disparate impact detected
Next audit due: 2026-09-14
`
At first glance, documentation like this can feel overly formal or unnecessarily bureaucratic. That's actually the point. In the public sector, a certain level of bureaucracy is a strength, not a weakness.

Well-documented systems make it possible for people who weren't involved in the original project to understand how decisions are made, why certain design choices were made, and whether those choices still hold up over time. That becomes increasingly important because project teams change. Engineers move on, contractors finish their engagements, and new leaders inherit systems they didn't build.

Without clear documentation, much of that knowledge disappears with the people who created it. With it, future teams can review decisions, challenge assumptions, investigate problems, and make improvements with confidence. Good governance isn't about creating paperwork for its own sake. It's about ensuring that years from now, someone can still understand, evaluate, and be accountable for how the system works, even if the original builders are no longer around.

Bias and Fairness Aren't a One-Time Check
Many organisations perform a fairness audit before an AI system goes live and treat that as the finish line. In reality, it's only the starting point. A pre-deployment audit reflects the data and conditions at a single moment in time. It can't tell you what will happen months later, when the population changes, eligibility rules are updated, or a key data source quietly changes the way it collects information.

That's why fairness shouldn't be viewed as a one-time validation exercise. It needs to become part of day-to-day operations. Monitor fairness metrics continuously, review them at regular intervals, compare outcomes across different demographic groups, and define clear thresholds that trigger an investigation when something looks wrong. Waiting for an external report, a journalist, or an advocacy group to uncover a problem means the system has already failed the people it was meant to serve.

Finding and addressing issues internally is always the better outcome. It allows organisations to correct problems early, reduce harm, and strengthen public confidence before mistakes become widespread. In government, proactive oversight isn't just good engineering. It's part of maintaining public trust.

It's equally important to acknowledge that fairness isn't something a technical team can define on its own. There isn't a single mathematical formula that works for every situation. Depending on the policy objective, different interpretations of fairness can lead to different design decisions. One approach may prioritise equal outcomes across groups, while another may focus on treating similar cases consistently. Neither is universally correct.

Deciding which definition is appropriate isn't simply a modelling choice. It's a policy decision that should involve program owners, legal and ethics teams, subject matter experts, and the communities affected by the system. Data scientists can measure fairness and evaluate trade-offs, but they shouldn't be expected to decide, on their own, what fairness means for a public service. That responsibility belongs to the broader governance process.

Procurement and Vendor Dependence Are Part of the Architecture

One technical consideration that often doesn't get enough attention is what happens when the technology itself changes. AI models evolve, APIs are updated or retired, contracts expire, and vendors shift their product roadmaps. Those are normal business realities. But when an AI system is making decisions that affect citizens, it can quickly become a governance problem.

A public service shouldn't depend entirely on decisions made by a vendor that the agency can't influence or even anticipate. If a model is replaced, an API is discontinued, or a contract comes to an end, the agency still needs to deliver services reliably and consistently. That means resilience has to be part of the architecture from the beginning.

There are a few practical ways to reduce this risk. Wherever possible, keep core business rules, such as eligibility criteria and escalation policies, in systems the agency owns and understands instead of embedding them entirely within a proprietary model. That gives the organisation greater visibility into how decisions are made and makes future changes far easier to manage.

Agencies should also be able to evaluate vendor performance independently. Don't rely solely on dashboards or metrics provided by the vendor. Maintain your own monitoring, validation processes, and audit capabilities so you can verify that the system continues to behave as expected over time.

Finally, plan for migration before it becomes urgent. Vendor changes are not rare or unpredictable events. They are an expected part of the software lifecycle. If a model version you've validated is retired or a service is no longer available, you shouldn't be starting from scratch. Having a clear transition strategy in place allows the agency to adapt without disrupting services or compromising the integrity of decisions that affect the public.

What Actually Builds Trust at Scale

Looking across public sector AI projects that have earned trust, and comparing them with those that attracted criticism, a few common patterns stand out.

The successful ones don't try to solve everything at once. They begin with a single, clearly defined use case where the risks, objectives, and success criteria are well understood. Only after proving that the system works reliably in the real world do they expand its scope.

They are also transparent from the beginning. Instead of waiting for journalists or advocacy groups to ask difficult questions, they clearly explain what the AI system does, what it doesn't do, where it is used, and where human judgment still plays a role. That openness helps build confidence long before concerns arise.

Another defining characteristic is that responsibility never shifts from people to technology. For decisions with meaningful consequences, a human remains accountable for the outcome. AI provides recommendations, highlights relevant information, and improves efficiency, but it does not replace human judgment where that judgment truly matters.

The strongest implementations also give as much attention to correcting mistakes as they do to making decisions. Appeals processes, human reviews, and error correction aren't treated as administrative formalities. They are designed, tested, and maintained with the same care as the AI system itself because every automated decision should have a clear and reliable path for review.

Finally, these organisations recognise that deploying AI is the beginning of the journey, not the end. They continuously monitor performance, reassess fairness, validate models as policies and data evolve, and actively seek feedback from the communities they serve. Responsible AI isn't a milestone that's checked off after launch. It's an ongoing commitment that continues throughout the system's lifecycle.

None of these practices is particularly new or complicated. They reflect the same principles that have always defined good engineering: start small, validate before scaling, expect things to fail, design for recovery, keep people accountable, and build systems that others can understand and evaluate.

What makes public sector AI different is the impact of getting those principles wrong. When an AI system fails in government, the consequences aren't limited to lost revenue or dissatisfied customers. They affect people who often have no alternative, no meaningful choice, and sometimes very little ability to challenge the outcome. That's why responsible AI in government isn't simply about building smarter systems. It's about building systems that the public can rely on, question, and ultimately trust.

The Real Measure of Success

It's easy to judge public sector AI projects by the numbers they produce. Faster case processing, lower operational costs, and hours saved are all valuable outcomes. But efficiency alone doesn't determine whether an AI system has earned the public's trust.

A more meaningful measure is what happens when the system gets something wrong. Can a person who has been denied a benefit understand why that decision was made? Is there a straightforward way to correct mistakes? Can they expect their case to be reviewed fairly and within a reasonable timeframe? At the same time, can the agency demonstrate, with evidence, that the system continues to treat different groups fairly as policies, data, and populations change, rather than relying only on the results of an audit conducted before deployment?

Those questions are ultimately more important than how many hours the system saves. They reflect whether the technology is serving people fairly, not just operating efficiently.

Building systems that meet that standard takes more time. It requires thoughtful governance, continuous monitoring, clear documentation, and well-designed processes for review and appeal. None of those activities generates the same excitement as launching a new model or announcing impressive performance benchmarks, but they are what determine whether an AI system can be trusted over the long term.

In the public sector, people don't choose whether to interact with these systems, and they usually can't turn to another provider if something goes wrong. That reality places a higher responsibility on every AI system that supports public services. Taking a slower, more deliberate approach isn't a barrier to innovation. It's what makes innovation credible, accountable, and sustainable. Responsible AI isn't just about delivering better technology. It's about building public confidence that the technology will continue to make fair, transparent, and accountable decisions long after the initial launch.

[Boost]

Atul Kumar — Wed, 15 Jul 2026 12:42:11 +0000

Atul Kumar

Jul 15

Building AI That People Actually Use: Lessons Beyond the Hype

#ai #machinelearning #product #softwaredevelopment

3 min read

AI in 2026 Isn't About Smarter Models Anymore. It's About Smarter Decisions.

Atul Kumar — Wed, 15 Jul 2026 12:26:29 +0000

A year ago, the conversation around Artificial Intelligence sounded something like this:

"Which model is the smartest?"

Today, I think we're asking the wrong question.

The real question is:

Which AI system can consistently create value without creating new problems?

That's a much harder challenge.

And honestly, I believe that's where the industry is headed.

The Race for Bigger Models Is Slowing Down

For the past few years, every major announcement followed a familiar pattern.

A larger model.
A better benchmark.
More parameters.
Lower latency.
Higher reasoning scores.

Impressive? Absolutely.

But here's something I've noticed while talking to engineers and product teams.

Very few users wake up wondering whether today's chatbot scored two points higher on a benchmark.

They care about something far more practical.

Does it understand my request?
Can I trust the answer?
Is it fast?
Does it save me time?

If the answer is "yes," most people never ask which model is behind the scenes.

Have we become too focused on measuring AI instead of measuring its impact?

AI Has Entered Its "Reality Check" Phase

A few years ago, adding AI to a product almost guaranteed attention.

Today?

Users expect it.

Investors expect it.

Companies expect it.

Simply saying "our product uses AI" isn't enough anymore.

Now the expectations are much higher.

Can your AI reduce costs?

Can it improve productivity?

Can it automate repetitive work?

Can it generate measurable business value?

If not, people move on.

That's a healthy shift.

Technology should solve problems—not just generate headlines.

Agents Are Changing the Conversation

One of the biggest changes I've noticed is the growing focus on AI agents.

We're moving beyond chatbots that answer questions.

Instead, we're seeing systems that can:

read documents,
search databases,
plan tasks,
interact with APIs,
write code,
generate reports,
and complete multi-step workflows with minimal human intervention.

That's exciting.

But it also raises an important question.

How much autonomy should we really give AI?

Because every additional action an AI can take also increases the importance of monitoring, permissions, security, and human oversight.

Intelligence without accountability isn't enough.

Context Is Becoming More Valuable Than Intelligence

Think about two assistants.

Assistant A has the world's smartest language model.

Assistant B understands your company, your documents, your workflow, your customers, and your goals.

Which one would you choose?

I'd probably choose Assistant B.

Because context beats raw intelligence in most real-world scenarios.

This is why we're seeing so much interest in retrieval systems, enterprise knowledge bases, and AI that understands organizational context rather than simply generating fluent text.

Building Trust Is Now a Competitive Advantage

Here's something I rarely hear discussed.

Users don't expect AI to be perfect.

They expect it to be honest.

There's a huge difference.

Imagine an AI saying:

"I'm not confident enough to answer this."

That's actually better than confidently producing incorrect information.

Trust is becoming a feature.

And I think the companies that embrace transparency will build stronger products than those chasing perfect-looking demos.

The Skills Developers Need Are Changing

Not long ago, learning machine learning algorithms was enough to stand out.

Today, AI development feels much broader.

Developers are thinking about:

prompt engineering,
evaluation,
observability,
guardrails,
vector databases,
security,
governance,
cost optimisation,
and user experience.

The role has evolved from training models to designing intelligent systems.

That makes AI engineering more interesting than ever.

One Question I Keep Asking Myself

Whenever I work on an AI feature, I try to answer one question before writing a single line of code.

If AI disappeared tomorrow, would users actually miss this feature?

If the answer is "probably not," then AI is being added for the wrong reason.

Technology should make life easier—not simply make a product sound more innovative.

Final Thoughts

Artificial Intelligence is no longer in its experimental phase.

It's becoming infrastructure.

Like cloud computing or the internet itself, people will eventually stop talking about AI because it will simply become part of how software works.

The companies that succeed won't necessarily be the ones with the biggest models.

They'll be the ones that build products people trust, understand, and genuinely enjoy using.

As engineers, I think that's an exciting challenge.

Not because AI is getting smarter.

But because we're finally learning how to use it more wisely.

Let's Discuss

I'd genuinely love to hear different perspectives.

Do you think model quality still matters more than product design?
Are AI agents the future, or are they currently overhyped?
What's one AI trend in 2026 that excites—or concerns—you the most?
If you could fix one problem in today's AI ecosystem, what would it be?
Where do you see AI engineering heading over the next three years?

Looking forward to reading your thoughts in the comments.

[Boost]

Atul Kumar — Wed, 15 Jul 2026 12:12:20 +0000

Atul Kumar

Jul 15

Building AI That People Actually Use: Lessons Beyond the Hype

#ai #machinelearning #product #softwaredevelopment

3 min read

Building AI That People Actually Use: Lessons Beyond the Hype

Atul Kumar — Wed, 15 Jul 2026 12:06:44 +0000

Artificial Intelligence has become one of the most talked-about technologies of our time. Every day we hear about new models, new frameworks, and new tools that promise to change the way software is built. But after working on AI-powered applications, I have realised something interesting:

Building an AI model is often the easiest part. Building an AI product that people trust and use consistently is much harder.

Let me ask you a question:

When was the last time you chose an AI product because it used a specific model rather than because it solved your problem well?

For most people, the answer is probably "never."

That question changed the way I think about AI systems.

AI Is More Than the Model

Many developers spend weeks comparing models and benchmarks. While model selection matters, it is only one component of a complete AI solution.

A production-ready AI application also depends on:

Reliable data
Clear prompts
Strong validation
Fast response times
Security and privacy
Continuous monitoring
Feedback from real users

Even the most advanced model can produce disappointing results if these pieces are missing.

Think about your current AI project.

Is your biggest challenge really the model?
Or is it poor data quality, unclear prompts, or inconsistent outputs?

In many cases, the answer isn't what we expect.

Trust Is Harder Than Accuracy

One lesson that surprised me is that users can forgive an occasional mistake.

What they struggle to forgive is inconsistency.

Imagine this scenario:

On Monday, your AI assistant provides an excellent answer.

On Tuesday, it gives a completely incorrect response to a similar question.

Would you trust it with your work?

Probably not.

That's why consistency should be treated as an engineering goal, not just a machine learning goal.

Simple practices like validating outputs, handling edge cases gracefully, and providing transparent error messages make a significant difference.

Don't Chase Every New Model

The AI ecosystem changes incredibly fast.

Every month, there seems to be another "best" model.

Early in my journey, I felt pressure to keep replacing existing solutions with the newest release. Eventually, I realised that constant change creates unnecessary complexity.

Before switching models, I now ask myself:

Will this genuinely improve the user experience?
Can the same improvement be achieved through better prompting?
Is the additional cost and maintenance worth it?

More often than not, improving the surrounding system delivers better results than changing the model itself.

AI Should Support Human Decisions

One misconception I often encounter is the idea that AI should replace people.

In most business scenarios, I believe AI delivers the most value when it supports human decision-making rather than trying to eliminate it.

Whether it's generating content, analysing data, answering customer queries, or automating repetitive work, people still need visibility into how decisions are made.

Here's something worth considering:

If your AI system makes a critical mistake, can a human quickly understand why it happened and intervene?

If the answer is no, there's still room to improve the system.

Start Small

Many organisations want to build the next revolutionary AI platform.

My advice is much simpler.

Start with one problem.

Solve it well.

Measure the results.

Learn from user feedback.

Then expand.

Instead of asking:

"How can we use AI everywhere?"

Try asking:

"Where can AI save someone just five minutes every day?"

Small improvements that genuinely help users often create more business value than ambitious projects that never leave the prototype stage.

Final Thoughts

Artificial Intelligence is moving at an incredible pace, and there is still a lot for all of us to learn.

What excites me most isn't just the technology itself. It's the opportunity to build products that make people's work easier, faster, and more meaningful.

As developers, we shouldn't measure success by how many AI features we can add.

We should measure success by whether people continue using what we've built because it genuinely helps them.

I'd love to hear your thoughts:

What's the biggest challenge you've faced while building AI applications?
Do you prioritise model performance or user experience?
Have you ever replaced a model only to realise the real issue was elsewhere?
Where do you think AI still falls short in real-world applications?
What is one AI lesson you wish you had learned earlier?

I look forward to reading your perspectives and learning from your experiences.

The future of AI won't belong only to those with the biggest models.

It will belong to those who build solutions that users trust.