(…and not another sad statistic in a Gartner report)
Gartner just dropped another sobering forecast:
by 2027, more than 40% of agentic AI projects will be scrapped — victims of ballooning costs, intangible ROI, and governance headaches.
Here’s the thing: success in data science isn’t about dodging failure; it’s about designing your process so that success becomes the default setting. That means setting goals that actually make sense, being brutally honest about what AI can and can’t do for your business, planning like you’re building a rocket, treating your data like royalty, modeling with discipline, building apps that can take a punch, and never — ever — taking your eyes off the ball once you launch.
In this post, I’ll break down each of those moves into practical, field‑tested steps. Nail them!
1. Set Your Business Objective Like a Pro
Field Case - When “Perfect” Became the Problem
A fintech startup proudly declared their goal: “Zero fraud.” Noble? Sure. Achievable? About as likely as finding a parking spot in downtown Boston at 6 p.m. Within weeks, they were rejecting half their legitimate customers. Fraud rates dropped, but so did revenue — and customer goodwill. The pivot — “reduce fraud by 40% while keeping approval rates above 90%” - turned them from villains into heroes.
Old Fail: Setting moon‑shot goals like “100% accuracy,” “no false positives,” or “Chat itself is the product,” without defining a deliverable or business value.
Your Wins:
- Write your goal in plain business language so anyone - from your CFO to your intern - can understand it.
- Attach a number and a time frame: “increase retention by 15% in six months” beats “make customers happier.”
- Tie the metric directly to revenue, cost savings, or risk reduction so it matters to decision‑makers.
- Pressure‑test the goal with a “what if” scenario — if hitting it would break another part of the business, it’s not the right goal.
- Keep a “goal sanity” checklist and revisit it quarterly to make sure you’re still solving the right problem.
2. Be Realistic (But Still Dreamy)
Field Case - The Dashboard That Paid the Bills
A retail chain wanted “AI that predicts fashion trends” — the kind of moonshot that looks great in a pitch deck. Three months later, they realized the real money was in predicting inventory shortages. Less sexy, more profitable. Their “trend predictor” became a humble dashboard that saved millions in lost sales — and nobody cared that it didn’t make the cover of Wired.
Old Fail: Pretending your core product is AI when it’s actually a food delivery app, a laundry service, or a retail chain.
Your Moves:
- Audit your current processes and find the bottlenecks or blind spots AI could fix.
- Prioritize use cases that improve existing revenue streams before chasing “industry disruption.”
- Celebrate unglamorous wins — the boring stuff often pays the biggest bills.
- Keep the “dream” projects in a sandbox until the basics are delivering measurable ROI.
- Build a roadmap that layers quick wins first, then progressively more ambitious projects.
3. You Need a Team to Build a Rocket to Mars (Because You Kind of Are)
Field Case - A Pregnancy Problem
A healthcare AI project was meant to flag “high risk” patients. But skipping domain experts in the planning phase, the model ended up flagged “high risk” patients… who were actually just pregnant. Without someone who understands the data’s context, your “life‑saving” model can become a very expensive pregnancy test.
Old Fail: Missing diversity in the team, underestimating dataset work, rushing timelines.
Your Success:
- Ensure every project team has at least one domain expert who can sanity‑check assumptions and understand the data.
- Budget 80% of your timeline for data collection, cleaning, and labeling - it’s not glamorous, but it’s where the magic happens.
- Set delivery dates based on realistic estimates, not investor‑friendly fantasies.
- Build in checkpoints where the team can pause and reassess before committing to the next phase.
- Document every assumption so you can revisit and adjust them as you learn.
4. Treat Your Data Like a VIP Guest
Field Case - When Cats Became Guitars
An image‑classification project trained on photos where cats happened to be sitting next to guitars. The label? “Guitar.” The result? Every cat became a guitar. Technically “accurate,” but useless.
Old Fail: Too little data, dirty data, missing field data, or mislabeled examples that poison the model.
Your Take:
- Run automated checks for missing values, duplicates, and inconsistent labels.
- Have humans spot‑check random samples for labeling, sanity - machines can’t catch every nuance.
- Test your model against adversarial examples - like an apple with “iPod” taped to it - before shipping.
- Keep a “data hygiene” log so you can trace and fix issues quickly when they pop up.
- Establish a recurring “data audit day” where the team reviews and cleans the dataset.
5. Model Like You Mean It
Field Case - Lost in Translation
A social media sentiment model trained only on X (aka Twitter) slang failed miserably on LinkedIn posts. “Crushing it” meant “awesome” on X, but on LinkedIn it often meant “burnout incoming.”
Old Fail: Jumping to conclusions, skipping cross‑validation, choosing algorithms heavier than your infrastructure can handle.
Your Playbook:
- Always split your data into training, validation, and test sets - and actually use them.
- Match algorithm complexity to your deployment environment. A 200‑layer neural nets inside a mobile app — no!
- Test on data from different sources to catch context‑drift issues early.
- Monitor for model decay and retrain before performance drops below acceptable thresholds.
- Keep a “model graveyard” of past experiments so you don’t repeat mistakes.
6. Build Applications That Can Survive the Real World
Field Case - The Chatbot That Went Rogue
An AI chatbot launched without proper safeguards. Within 24 hours, it was spewing offensive content because users figured out how to “train” it in real time.
Old Fail: No safeguards, scaling issues, switching to auto‑pilot too soon, not preparing for attacks.
Your Edge:
- Simulate hostile user behavior before launch to see how your system reacts.
- Keep a human review step in place until the model has proven itself in production.
- Add anomaly detection and rate‑limiting to prevent abuse at scale.
- Maintain a rapid‑response plan for rolling back or disabling features if something goes sideways.
- Train your ops team to recognize and respond to early warning signs of failure.
7. Monitor, Measure, Optimize — Forever
Field Case - The Three‑Minute Ride That Wasn’t
A ride‑sharing app’s ETA model started showing “3 minutes” for every ride, no matter the distance. Passengers were thrilled for about 30 seconds — until they realized the number never changed. Drivers were confused, support tickets piled up, and social media had a field day. The culprit? A server clock drifted by 17 minutes, throwing off the calculations. Monitoring caught it — but only after a week of chaos.
Old Fail: Assuming it just works, missing KPIs, skipping A/B testing, ignoring real‑user feedback.
Your Levers:
- Define success metrics before launch and track them continuously.
- Run A/B tests on model updates to measure real‑world impact.
- Collect and act on feedback from actual users, not just your dev team.
- Set up alerts for anomalies so you can fix issues before they become PR disasters.
- Conduct consistent "postmortems" on both successes and setbacks to continue learning.
Final words
This post is a re‑imagining of a lecture I first gave back in 2019. Strangely enough, the challenges - and the remedies - have outlived the seismic shifts brought on by the rise of LLMs. The tech has evolved, the buzzwords have changed, but the fundamentals still decide who wins and who flames out. Get those fundamentals right, and your project won’t just survive. It’ll headline the main stage, strut in the spotlight, and have the whole boardroom singing along. 🙂
In a follow‑up post, I’ll dive deeper into my latest AI/LLM reflections - what’s changed, what hasn’t, and where I think the next big wins will come from. Stay tuned.
Originally posted here
Top comments (1)
This is pure gold for anyone trying to make AI actually deliver business value! I love how you stress realistic goals and treating data like royalty—so often, the basics are overlooked in favor of flashy moonshots. Definitely taking notes for my next project!