DEV Community: Marwan Mohammed

Hitting Merge: Mentally Preparing for Your First Push to Production

Marwan Mohammed — Sat, 23 May 2026 19:50:18 +0000

I pride myself on being decently good at what I do, but more importantly, at loving it. And through my journey trying to make a career out of it, I learned that toy projects are not enough, that I need to deploy something into production.

Thankfully, this is an experience I was exposed to decently early in my career. Now, after working across multiple teams to get my first live service out the door, I’m here to share the exact mental and practical guardrails I learned along the way.

Before You Dive, You Must Prepare

Your first time diving is dangerous, you cannot just dive in head first, you need to prepare your oxygen tank(s), have your diving instructor or some professional with you, and most importantly, not going into a dangerous place for your first time.

These rules apply to your first time deploying to prod as well:

you cannot dive in head first, you need to prepare your oxygen tank(s): this means ensuring your container configurations, environment variables, dependencies, and artifacts are completely decoupled from your local machine and fully portable.
Have your diving instructor or some professional with you: Always have a senior engineer, tech lead, or manager watching over your shoulder for your first roll-out. You want an experienced teammate there who can calmly take the wheel if a catastrophic edge case appears.
And most importantly, not going into a dangerous place for your first time: Depending on your scope and the company, ensure your first deployment isn't a highly sensitive, mission-critical legacy system. Start with isolated, well-bounded services where you can learn the pipeline safely.

After the dive

After the dive, you find yourself taking in the beauty of the sight you just witnessed. It feels different, new, exciting, scary, but most importantly, alive.

Your first time seeing a feature or model that you worked on working in production, processing live traffic, will feel different, it will be this weird mix of pride and fear. When you see users use that feature for the first time and it works, you'll find yourself proudly saying "I built that btw".

Reality check: Localhost vs Host

I started with a "before you dive", and an "after you dive", but what about the diving itself?

Diving in this case means accepting a harsh reality: the calculations change completely once you leave staging.

Development/Staging vs Production

The development environment and even borderline the staging environment are meant for testing if the code works and passes tests, but when it comes to production, everything changes. Suddenly you have to take into consideration query optimizations, latency reduction, scaling, etc. But in ML specifically, this paradigm shift hits even harder. I like to call this the Cost of Intelligence.

The cost of intelligence

What this refers to is simply the fact that ML models are different than traditional deterministic software, they introduce different attack vectors, somewhat more complex deployment strategies, non-trivial explainability requirements, and a ton of guardrails around model outputs. That's not even taking into consideration observability over the model, model re-training, training policies, etc.

The cost of intelligence in this case is simply the fact that you need to make sure pre and post processing are ported and scaled correctly, the model's outputs are correct and safe, and telemetry tracks data drift in real time. These are the fundamental pillars of MLOps, but they represent a constant architectural tax on modern intelligent systems.

Real World ML

In my journey moving from experimental code into production, I found my manager always asking me two things:

What is our latency, and how do we reduce it?
How is the model performing?

It feels like those are two easy questions, and at first, I thought so too! But with time, I found out that generic metrics do not cover real-life requirements.

For instance, my first production project involved orchestrating multiple interconnected models, splitting single inputs into sub-tasks, processing them asynchronously, and aggregating the final results. Standard accuracy metrics weren't enough, I had to learn how to implement strict batch bookkeeping and design or implement domain-specific metrics that reflected true business constraints and the downstream consequences of a model's prediction.

So, my most important advice here is simply know the application of your service before deploying, and understand your true KPIs

After drying up

Experimenting in notebooks is fun, building a service around a model you created is exciting, deploying them is a chore, seeing the results is like holding your firstborn. This is the cycle I felt at least, but trust me, once you have your first ever service deployed, and you clash with real world constraints, you'll learn what tens of courses and hundreds of articles won't teach you: Acceptance, rigorous preparation, and knowing when to fight the insatiable engineer's urge to over-optimize.

Final Notes

This is just the beginning. In upcoming posts, I’ll break down the specific technical steps, pipeline tools, and architectural patterns needed to make this jump safely. But before having all of your model guardrails in, you have to get your mental guardrails in place.

ML System Development and Redundancy: Stop Rebuilding the Wheel

Marwan Mohammed — Sat, 23 May 2026 17:23:17 +0000

Introduction

For the longest time I’ve found myself asking one question repeatedly:
“Do I really have to rewrite all of that every single time?”
The answer for me at the time was to create a “helper functions” repository on GH, it was a painkiller that worked, until it didn’t. Maintaining it was inefficient and exhausting, and it lacked the cohesion a real project needs.

Then, as I was hanging out with a few friends of mine who happen to be Backend Engineers (Shoutout to Youssef Tarek And Yahia Al-Touny), I saw the light. They were discussing a custom template they had built for their services. It was impressive, and it hit me: “Why don’t I create something similar for ML?”

This is when I started researching industry standards, best practices, and existing templates. Afterwards, I started working on it, and in Today’s article, you’ll get first class seats into the “How” and “Why” behind the MLOps template I created.

Templates vs Boilerplates

When I first started with this, I was confused by both terms, specifically what each one meant and how they were different from one another. So, If you’re like me, read on, if you already know, feel free to skip forward

Templates: These are architectural blueprints. They enforce a structure, ensure scalability, and allow teams to collaborate within a standardized environment.
Boilerplates: These are “copy-paste” solutions — reusable code blocks that require little to no modification to work. My goal is to build a boilerplate-flavored template, trying to get the best of both worlds!

The Graveyard: Finding Gold in Dead Code

I started by scrapping my own graveyard. I looked at every dead project, every “I’ll fix this later” comment, and every helper script I’d ever written. I asked one critical question: “Does this work outside of its intended project?” If the answer was no, I had to figure out how to make it modular.

The Pillars of the Template

1. Observability: Seeing in the Dark

Observability is arguably as important in ML as it is in standard Software Engineering. You need to be able to see how your code behaves, what succeeded and what did not, why, etc.
This makes debugging much easier, not just in development, but in production as well. And so, I started setting up my logger to track the most important information

Schemas: The underrated warrior in ML

Having a schema for everything that is not controlled by you is important, simply because validates that both inputs to the model and outputs are clean and valid..at least type safe.
So, I started setting up schemas (Pydantic) for requests, responses, app configurations, even a validator for environment variables.

Stop Switching Engines, Start Changing Oil

Previously, I created a different “engine” for every task (e.g., one for binary classification, one for multiclass). This led to massive, 1,000-line modules. Which honestly make no sense at all.
And so, I created a unified engine for every task, and then some adaptors for each task, that means that the core is the same for everything, but since each task is unique, a small adapter can be fitted to accommodate for its needs. This applies to both training, and loading/inference abstractions.

“It works on my Machine!”

An infuriating sentence, isn’t it?
Well, that’s why I decided to start working with Docker.
Because simply put, Docker makes sure that your machine is ported into whatever environment the code is shipped to, making sure that everything is consistent anywhere with minimal setup.

Developer Experience: We’re Human Too

We sometimes forget that we are human as well, and so, we need to take care of ourselves while we’re writing absolute units of computer programs.

And so, I decided to look further into this, and decided to implement the simplest form of it: Command Line Abstractions.
Initially I was thinking about Makefiles, but as I was researching more, I cam across this beautiful tool called “just”, which is basically just a wrapper around any number of commands you want, just create the configuration file (pun intended) called “justfile”, set any command you want in there with parameters if you want to and comments as “help”, and it’s running just <command>, if you need to, you could also run just list to see all commands you defined.

Breadcrumbs: Don’t lose your way in the maze

Before, I used to either manually log everything or not log anything at all, which meant that tracking my experiments, including hyper-parameters, metrics, models, artifacts, etc. was a nightmare!
Till I came across MLFlow, a piece of art disguised as a tool. It gave me the ability to build anything I want and do however many experiments I want and not lose context for how each run performs and what each run included, including data signitures, datasets, metrics, hyper-parameters, models and artifacts, etc.
This made life so much easier honestly

Notebooks: Not a liability as many think

Notebooks are notorious for making Data Scientists lazy, they just run the experiment in there, see the results, maybe log the model, and call it a day. But in reality, notebooks are much more powerful if you create them correctly!
It must give you the ability to transition from the experimentation environment into the system with the least amount of friction. Including properly logging runs and models, proper setup for pre- and post-processing, data loading, etc.
And so, I created what I call Notebook-as-a-Bridge (NAAB), not a fancy tool, but a way to write notebooks that make transitioning into a production system frictionless.

Don’t ship what you didn’t test

Writing beautiful code is meaningless if it breaks easily. In ML, this means testing:

Model loading and training

Inference logic
The API server
Proper tests don’t touch the disk or load heavy models; they use mocks to test components in isolation and in memory.

And suddenly…You’re Ready

Now, you’ll find that you actually have everything you ever need to start building any ML project, experimentation environment, testing, and even prod ready system for your use!
The more proper your template is, the faster you can go from experimentation to production, and honestly, this is the most beautiful thing ever about this.

Now what?

This template is a living organism. It will continue to iterate as I learn. Use it, fork it, and if you see an improvement, open a PR! Let’s improve the community together.

(Also, to prove this works: this template officially survived its first major project, which is now being deployed into production!)

GH Repo

You can find the repo here, if you think it is useful, do give it a star, and if you feel like you want to improve on it, feel free to fork!