Sit down, let us talk about building a data science capability. Not the conference
talk version. The real one, from inside a large, heavily regulated enterprise,
over two years, with a small team, a tight budget, and no map. I inherited the
team. I did not inherit the capability around it, and that distinction is the
whole story. I am going to tell you what I was given, what I did about it, what
it cost me, and what I would tell you to do if you were starting tomorrow.
What I was given
Picture your first week. You ask for the handover and there is none. Not a thin one.
None. Models are running in production, real customers are depending on them every day,
and there is no record anywhere of how they were built, what data feeds them, who uses
their outputs, or how accurate they are. The products exist. The knowledge about them
lives in nobody's hands.
The team is small and talented, but it has been running without structure, reporting
into a business function rather than an engineering one. Everything is done on
the fly because there has never been anyone whose job it was to decide how things should
be done.
And there is infrastructure. That part matters, because the lazy version of this story
is that nothing existed. There is Kubernetes, Jenkins and CI
pipelines, Microsoft SQL Server on premises, PostgreSQL databases, Windows virtual
machines. Some work still runs off somebody's laptop. A few database environments have
unclear permissions and no obvious owner. None of that is bad technology. It was built
for applications: software that serves requests and touches a modest amount of data at
a time. Our work was going the other way, toward large volumes, reprocessed history and
model training. We had plenty of places to run software. What we did not have was a good
place to do data science at scale.
And the budget is tight. Not tight as in negotiate harder. Tight as in there
is no line item for any of what you are about to need.
Here is the first thing I want you to understand, because everything else follows from
it. In that situation, the temptation is to start fixing the loudest thing. Do not. The
loudest thing is almost never the load bearing thing. What follows is the order that
worked, and more importantly, the reasoning behind the order.
First, make the work visible
You cannot lead what you cannot see. When I arrived, work started, stalled and finished
invisibly. People were busy, genuinely busy, but the team could not answer the simplest
question about itself: what are we working on, and how is it going?
So before touching any technology, I introduced structure:
agile working through
Jira. Two week sprints. A standup every two
days rather than daily, because a small team fighting production fires does not need
another meeting, it needs a rhythm. At the end of each sprint we sat together and asked
the honest questions. What got done? What did not? Why? Who is blocked, and on what?
If you have read The Phoenix Project,
you know the idea: invisible work is unmanageable work, and the first act of any
turnaround is making the work visible. What visibility bought me was not speed. It was
knowledge. Within a few sprints I could see who knew what, which skills we had and which
we lacked, and exactly where work went to die. You are not building velocity in this
phase. You are building sight.
Write everything down, like you might leave tomorrow
Once I could see the work, the next problem introduced itself: every important thing
lived inside someone's head. In a small team that is not a culture quirk, it is an
existential risk. One resignation and a production system becomes an orphan.
So I introduced Confluence and seeded
it myself, starting with the onboarding guide I wished I had received: every tool we
use, where to find it, how to get access, which entitlements to request, which
regulations you must complete before you can do this work in a regulated environment.
That was not a filing exercise. It was aimed at the small daily friction that quietly
eats a team. Where does this thing actually run? How is it deployed? How do I get
access? Which database does it depend on? Who owns it? None of that should require
finding the one person who happens to remember. And I knew people would eventually join
us. I did not want anyone else to arrive to what I had arrived to: no handover, no map,
and a week of asking around before you can do anything useful.
Then the harder job: we went into the backyard and reverse engineered our own production
models. These were not systems we had built. They were systems we had inherited, where
the original reasoning was thin or simply gone. So we read the repositories and the code,
worked out what logic each model actually applied, what data went into it, what came out,
how often it ran, who consumed the output, and what business decision sat on the other
end of it. Nobody handed us that knowledge, so we dug it out ourselves, because the
alternative was operating systems we did not understand.
Richard Feynman, when a historian described his notebooks as a record of his thinking,
corrected him: the notes were not a record of the thinking, they were the thinking. That
is what documentation is for a team. It is not admin after the real work. It is the team
thinking in a form that survives.
And here is the part I did not fully predict. The documentation outgrew us. Teams
outside our own area use it to understand our models for their own work, and when people
later joined the team they onboarded from a documented path rather than from word of
mouth.
Then it paid off in a way I had not planned at all. Production models came under audit,
compliance review and model governance. Reviewers wanted to know what a model does, how
it works, what data it uses, how its decisions are produced, what controls exist and what
evidence backs any of it. Writing things down does not make a model compliant, and I am
not claiming it did. The claim is smaller and more useful than that: when somebody
finally asked us to explain what was running in production, we were not starting from
zero. We could walk them through it instead of spending weeks rediscovering our own
estate under pressure.
Months earlier, documentation had felt like extra work. During audit, it became
evidence.
Somewhere along the way it stopped being my project and became how the team works, which
was the real point. The principle is easy to say and hard to hold: important knowledge
belongs out of people's heads and somewhere the team can reach it. If something only
works because one particular person is available that day, you do not have a system. You
have a favour.
The cost of this phase was my own evenings, and I will not pretend otherwise. Nobody
asks for documentation. You do it because you are building something that must not
depend on you.
The long game against the short game
Now I must tell you about the hardest challenge of the whole two years, because it was
not technical.
Management, quite reasonably, wanted immediate results. Things they could count this
quarter. And almost everything I have described so far produces nothing you can count
this quarter. Sprints, documentation, governance processes: these are investments whose
returns arrive later and compound quietly. My strategy was long term, sustainability
over spectacle, and that meant living inside a permanent tension between what the
business wanted to see now and what the team needed to become.
I will be honest with you: sometimes holding that line means going against what
management is asking of you in the moment, because you can see what a scalable data
science capability requires before it is visible to anyone else. That can end well or badly, and you
should know that before you choose it. It is ownership in its purest form. If the long
game pays off, you were a builder. If it does not, you were difficult. I decided early
that I was there to plant trees,
and I accepted the cost of explaining, over and over, why I was digging when everyone
wanted fruit.
My belief, and by the end of this post I hope it becomes yours: a good data science team
is one whose systems outlive the people who built them. If I leave, if anyone leaves, if
the whole team leaves, the documentation, the pipelines, the patterns and the governance
must keep working. There is a moment in One Piece where Dr. Hiluluk asks when a man
truly dies, and answers: not when he is shot, but when he is forgotten. Systems are how
a team remembers. Build so that nothing important dies with a resignation letter,
including yours.
When the infrastructure bill arrives
Then reality did what reality does: it presented the bill for infrastructure choices
made long before I arrived.
The business migrated to a new core policy administration environment. Things broke, and
one of our major workflows began falling behind. This is a workflow handling roughly one
million trips a day. At one point, the backlog had grown to around 100 million trips
waiting to be processed.
The part people underestimate is that the new data does not stop arriving just because
you are behind. That is the whole problem, and it is simpler than it sounds:
arriving faster than we
process
-> the backlog grows
same rate in and out
-> it stops growing
-> but never shrinks
processing faster than
it arrives
-> the backlog clears
To recover we did not just need to process 100 million trips. We needed enough capacity
to work through the backlog while still handling the million or so arriving every day.
No amount of team discipline fixes that. The stack was built for hosting applications,
not for distributed data processing, and the workload and the infrastructure simply did
not share a shape.
There was a second problem, quieter and in some ways worse. The migration also introduced
inconsistencies in the data itself. So the question was never only "how do we process
more?" It was also "can we trust what we are processing?" Some of those values feed
calculations with real financial consequences, including payouts. I am not claiming wrong
payments went out. I am saying it was a risk we had to control, and quickly.
Which made the real requirement something like: work through a backlog of roughly 100
million trips, keep up with the million arriving daily, and make sure the data can be
trusted before anything downstream uses it. That is where the need came from for reliable
ingestion and ETL, reconciliation so we could prove a number, quality checks that catch
bad values before they spread, monitoring and alerts so we heard about problems from our
systems rather than from a client, lineage so we could trace where a value came from, and
the ability to safely reprocess history.
None of that was academic. Clients complained, the business pushed hard and reasonably
so, and people were under real stress for weeks.
It also changed what I thought scale meant. In a regulated environment, moving more data
is only half of it. You also have to answer who can reach this data, where a number came
from, whether the checks passed, and whether you can run the whole thing again tomorrow
and get the same answer. Scaling is not only processing more data. It is processing more
data without losing control of it.
This is also where the biggest lesson of the two years finally landed for me. I could
have spent that period improving how we built models and it would not have helped,
because the models were not the weak part. Data scientists cannot spend their time on
modelling if they are forever solving ingestion, movement, transformation, access and
quality by hand. At scale, data science is only as reliable as the data foundation
underneath it. The answer was not to buy a platform. It was to build a stronger data
engineering foundation underneath the data science.
It is fair to ask how I knew where to look. Before this role I worked at IBM and
Vodacom, in more mature data science and data engineering environments, where production
data science, scalable processing and deployment were treated as engineering disciplines
in their own right. At IBM and Vodacom I learned what a more mature capability could look
like. Here I learned what it takes to help build one.
I had seen this coming, and I had prepared for it, and this is the most transferable
lesson I own: you do not get budget for infrastructure, you get budget for pain.
But the first thing I did when the backlog arrived was not to go and design a platform.
It was to go and look at what the group already had. Large organisations are full of
capability that one team built and nobody else knows about, and rebuilding something that
already exists two floors away is an expensive way to look busy. So I went looking, and I
found another team in the group already running
Databricks. I went to them, asked how they had set it up,
and learned from what they had already worked out.
That is where the honest version of this story sits, and it is worth being precise about.
I did not introduce Databricks to the group. It was already there. What it was not was a
capability our team could use. Getting it from the first thing to the second was only
partly a technical problem.
The technical part I could do. I piloted it against our own workloads, got our data into
it, and built the first working pipelines. Part of that was a pattern for pulling data
out of our on premises databases into the cloud environment and catalog over JDBC, which
is simply the standard way applications and data systems connect to relational databases.
I built the first one, proved it on a real business problem rather than a demo, wrote it
down and taught it. The point was never for me to become the person who moves every
dataset. It was that the next person should not have to solve it again from scratch.
The rest of it was organisational, and I underestimated how much of it there would be.
Working with the team that already had the capability. Bringing in our architecture
people. Getting the commercial arrangements and the cost allocation in place. Pulling in
platform, infrastructure and networking where we needed them. Working through access and
enablement so people could actually log in and do something. That turned out to be the
real lesson of the whole exercise: getting a platform to work technically is only part of
bringing it into a team. Architecture, funding, contracts, access, support and adoption
all have to move with it. The code can be running perfectly while the capability still
does not exist.
All of this while production was still under pressure, which meant two tracks at once.
Track one was today: get enough of our data onto the platform, processing, and supporting
the business right now. Track two was the slower work of turning that into a real
capability rather than an emergency measure. You do not get to finish the second before
the first becomes urgent, and pretending otherwise is how people end up with a beautiful
architecture and an angry business.
The risk was mine and I knew it. If it had failed, there would have been no confusion
about whose idea it was. That is the ownership I keep talking about: putting your name
on an outcome before you know how it ends.
It worked. The new pattern gave us enough capacity to work through the backlog while
still handling everything arriving each day, which is the only definition of recovery
that means anything here. And something subtler happened: the team's default assumption
shifted to cloud first, and moving our data storage fully to the cloud is now one of the
team's biggest running initiatives. Nobody argued about strategy after that. Results
ended the argument, which is the only way arguments like that ever end.
But the destination was never that I got it working. It was that the team could work on
it. So the early patterns were documented, taught and handed round until they stopped
being mine, because a platform only one person understands is just a more expensive
version of the problem we started with.
Teach the team you have
Here is a constraint nobody warns you about: you will mostly not be allowed to hire your
way out. The team you have is the team you build with, and my team had been hired for a
different technical world. Models lived on laptops and personal virtual machines where
nobody else could reach them. PySpark
and Databricks were new to almost everyone. And the training budget was, you already
know, tight.
So the training programme was us. I taught, over and over: sessions on Databricks, on
PySpark, on Jira, on Confluence, always showing value rather than announcing policy,
because people adopt tools when they watch their own problem get smaller, never because
an email told them to. The goal was never for me to become the expert everyone had to
come back to. It was for anything I learned or solved once to become ordinary team
knowledge: I work it out, I prove it, I write it down, I teach it, somebody else uses it,
and eventually nobody remembers it was ever a special skill.
The best example of that is not a tool at all. One of the most valuable people in this
story already worked here. They had strong data management knowledge and, more to the
point, they knew the place: where data lived, which databases mattered, how the business
actually used them. With learning, opportunity, engineering direction and real
responsibility, they grew into the data engineering capability we badly needed. Capability
building does not always mean finding a finished specialist outside and hiring them.
Sometimes the knowledge you need is already in the building.
Then I made teaching an institution: show and tells, every Friday, with rules that are
loose on purpose. Share the model you are building. Share something you learned about
the business. Share the course you are studying, a better way to do something, a piece
of technology news. Everyone becomes able to do a bit of everyone's job, which is the
only real insurance a small team has. And this institution outgrew us exactly like the
documentation did: people from across the business now attend and give feedback, which
quietly turned a team ritual into a standing relationship with our stakeholders.
Alongside it I pushed everyone toward the studies the company sponsors, because in a
small specialised team, personal development is not a perk, it is the growth strategy.
Patterns, not heroics
By now the team could see its work, find its knowledge, and stand on real
infrastructure. The remaining gap was discipline. Deployments happened by hand, jobs run
directly on the platform by whoever built them. It worked, and in a regulated
environment it was quietly dangerous: not reproducible, not governed, not accessible to
anyone but the author.
So we built the engineering layer: this is how we deploy here. Ingestion patterns, model
development patterns, dashboarding patterns, and full
CI/CD through
GitHub workflows, with governance,
reproducibility and access control designed in rather than bolted on. A pattern is one
person's hard won competence, converted into the team's default.
And here I must tell the engine story properly, because the order matters. In my own
research I had built an open source engine that lets you describe a data or machine
learning pipeline once, as a small folder of configuration and Python, and run that same
folder anywhere. It started life serving my own experiments, and I carried on developing
it independently afterwards. That engine is
Ubunye Engine, and it is on
PyPI. Standing in front of this team's
engineering gap, I recognised that the ideas I had already worked out, about
repeatability and running the same work in different places, were the ones we needed.
They shaped how I approached the problem, and helped shape some of the patterns we
adopted as a team. Research met reality, and they strengthened each other.
Rolling patterns out is not an email. It was onboarding, practical sessions, hands on
hours every Friday, working through real pipelines together until the patterns stopped
being mine and became the team's.
Getting the foundation right also gave us room to improve everything above it. We could
start standardising how models were built, deployed, monitored and governed, with work
in source control so somebody else could pick it up and run it, instead of treating every
project as a completely different engineering problem.
Which is the distinction I would most want you to take from this. Databricks was
important. Databricks was not the capability:
PEOPLE
skills, trust, ownership
|
PRACTICES
documentation, standards
testing, CI/CD
|
DATA SCIENCE
experiments, models
deployment
|
DATA ENGINEERING
ingestion, ETL, quality
|
PLATFORM
compute, storage, catalog
A platform gives people capabilities. A pattern tells the team how work should be
structured. People and habits are what make the conventions normal. Take any layer out
and the ones above it wobble.
Learn to let go
Eventually I hit the ceiling every builder hits, and the ceiling was me.
I was running the process and leading the engineering at the same time, and doing both
badly on alternate days. Worse, every structure I personally held together was a
structure that would fail the week I went on leave, which would have betrayed the entire
philosophy of this build. So I went to management and made the case for help. We grew
the team. We brought in a project manager and scrum masters to run the process.
The budget was never going to stretch to a specialist for every gap, so a lot of what we
needed came from people who did not report to me: engineers, data engineers, platform
people, technical leads elsewhere who had already solved something like our problem. It
worked because the problems overlapped and because I had spent time on the relationships
before I needed them. You do not need everyone to report to you to build something
together.
There is a version of this where the data science team quietly becomes data science plus
data engineering plus platform enablement plus everything else nobody has picked up. That
was necessary for a while. It is a terrible permanent arrangement. So as the patterns
matured we began handing them over to the data engineering, technology and platform teams
whose job they properly are, which gives data scientists their capacity back for actual
data science. Sometimes you build what is missing, prove it, teach it, and then give it
to the team that should own it. If you want the theory underneath that move,
Team Topologies is the book: platform teams exist so
that stream teams can flow.
None of this would have worked one level up. My manager was not a micromanager, and the
room I was given is the room I was able to give the team. It is difficult to build an
autonomous team if autonomy disappears above it.
What I mean by a team that manages itself is nothing mystical. Priorities are clear. The
work is visible. People know what they own, pace themselves, ask for help when they are
stuck, and help each other without being asked. Knowledge is shared rather than hoarded.
Nobody needs chasing.
I used to think the point of leading was being the person who could always fix it. I now
think it is the opposite: the measure of the job is whether the team can say they did it
themselves and be right about it. The cost of this phase is ego, and it is the cheapest
price on this whole page once you understand what it buys.
What two years bought
So what does all of this add up to? Told plainly, the way I would tell you across a
table:
A new model used to take an unbounded amount of time to reach production. Nobody could
tell you how long, because nobody could see the path. Now it ships in days. When work
piles up, we have the capacity to work through it rather than watching it grow. New
people onboard from a documented path instead of an oral tradition.
The production models we inherited are documented and governed, and other teams use that
documentation for their own work. The research in
Accelerate found that delivery speed and
stability rise together or not at all, and that matched our experience exactly: we got
faster because we got safer.
And the change that matters more than every number: the team stopped being reactive. We
were once a team that chased whatever was on fire that day. We are now proactive for
most of what we do, which means we choose our work instead of our work choosing us.
I should be honest about where this actually sits. The capability is not finished. Some
foundational ownership is still moving toward the teams that should hold it, and there is
plenty left to improve. But the data scientists now work on the platform, the patterns
are increasingly normal rather than mine, and the knowledge is spread across people
instead of trapped in one or two. The capability is not finished. The way we work has
changed.
The systems run whether I am in the room or not. That was the goal the whole time.
The key takeaways
If you are about to build a data science capability, whether or not the team already
exists, this is what I would say to you before anything else.
- Decide what you believe, because everything else follows from it. Every move on this page is one belief wearing different clothes, and you will not hold a line for two years without one.
- Make the work visible before you touch technology. You cannot fix a team you cannot see, and no platform purchase repairs invisible work.
- Documentation is leadership, not admin. Write the guide you wish you had received. It will outgrow your team and become the organisation's memory.
- Play the long game, and know its price. Management will want countable results now, and your most important work will not be countable for months. Sometimes you will hold your line against what is being asked, because you can see what a scalable capability requires. Understand clearly: that can end as vision or as insubordination, and which one it becomes depends on what you deliver. Take that ownership with open eyes.
- Know what you would do before the crisis, and look before you build. When it arrives, bring a plan rather than a complaint. And check what your organisation already has, because the capability you are about to build from nothing may exist two floors away, waiting for somebody to do the unglamorous work of bringing it in.
- Teach the team you have. A weekly show and tell is a training programme with a budget of zero, and a small team where everyone can do a bit of everything is more resilient than a large team of silos.
- Patterns beat heroics. Convert individual competence into team defaults, and automate the important operations until nobody has to think about them.
- Let go on purpose. Hand the process to process people and the platforms to platform people. If the team only works when you are watching, you have built a performance, not a capability.
- Expect sacrifices, and choose them consciously. This build cost evenings, patience, political capital, and ego. Ownership means some of what you own may fail with your name on it. I would pay all of it again, because the alternative is a team that dies a little every time someone resigns.
One last thing, and it is the part I did not expect. The most useful skill I took from
these two years had very little to do with any tool. I learned how to build a capability
in a place where the structure I needed did not already exist. That meant working out
what was actually broken rather than what was loudest, finding the right people and
building relationships with them, asking for resources and sometimes not getting them,
managing expectations honestly, balancing a production fire against work that would only
pay off next year, and creating enough structure for the team to move without burying
them in it. I did not expect that to be the lasting lesson. It was.
Remember where this started: live systems, no handover, no useful documentation, and
everything important sitting inside somebody's head. Almost everything we built
afterwards was aimed at making sure the next person does not inherit that same darkness.
Which is the whole belief in one line. Build systems that outlive the people who built
them, including you.
None of this required genius. It required knowing why I was doing each thing, doing the
unglamorous ones in the right order, and refusing to skip the boring ones. That is what
building a capability actually is: planting trees, in the right sequence, whose shade you
may never personally sit in.
Originally published at www.tmashininisekgoto.com.
I write about machine learning engineering, MLOps and building AI systems in South Africa. More at www.tmashininisekgoto.com.
Top comments (0)