Computer vision pipelines often start simple; You load an image, run a model, get some detections, and draw the results:
Image → Model → Detections → Annotation → Output
But real computer vision applications rarely stay this simple. Soon you have preprocessing, resizing, tiling, multiple inference steps, filtering, tracking, coordinate transformations, annotations, application-specific logic, and so on.
So at the end the pipeline would look more like:
┌→ Model
│
Image → Resize → Tile ─→ Model → Merge → Filter → Track → Annotate → ...
│
└→ ...
At this point, the individual steps are understandable, but the pipeline becomes difficult to understand as a whole.
The goal of our experiment is to make the data flowing through a computer vision pipeline explicit and inspectable.
The problem: the final image doesn't look right
Consider a typical detection workflow:
import supervision as sv
results = model(image)
detections = sv.Detections.from_inference(results)
detections = detections[np.isin(detections.class_id, [0])]
detections = detections[detections.confidence > 0.5]
detections = tracker.update_with_detections(detections)
...
image = box_annotator.annotate(
scene=image,
detections=detections,
)
This is perfectly reasonable code, but imagine that the final output doesn't look right, perhaps some objects are missing…
- Where did they disappear?
- Did the model fail to detect them?
- Did a confidence threshold remove them?
- Did tracking discard them?
- Did a coordinate transformation move them?
- Did an annotation step simply fail to display them?
The final image can't tell you…
You have to reconstruct the execution mentally - or start adding print statements, saving intermediate images, and inserting debugging code throughout the pipeline.
This becomes even more painful as the number of operations increases.
Making the data flow explicit
The idea behind ml-pipes is simple, the author compose the pipeline as sequence of steps, the framework run the steps.
Instead of thinking about a pipeline as a sequence of Python statements, we can represent it explicitly:
from ml_pipes.core import Pipeline
class LoadFile:
def __call__(self, image_path: str | Path) -> bytes:
path = Path(image_path)
if not path.is_file():
raise FileNotFoundError(f"Image not found: {path}")
return path.read_bytes()
...
pipeline = Pipeline([
LoadImage(),
Resize(),
Infer(),
DecodeDetections(),
FilterDetections(),
Track(),
Annotate(),
])
result = pipeline(image)
Each operation becomes a node in the data flow.
That gives us something that ordinary sequential code doesn't naturally provide:
a representation of what the pipeline actually is.
Combining ml-pipes with Supervision
This is where Supervision fits particularly well.
Supervision already provides many useful computer vision primitives:
- detections, tracking, annotators, etc.
Rather than rebuilding those operations, we can simply use them as a step in our pipeline:
from ml_pipes.core import Pipeline
import supervision as sv
model = load_my_model(...)
tracker = sv.ByteTrack()
pipeline = Pipeline([
...,
model(...),
sv.Detections.from_inference,
tracker.update_with_detections,
...,
])
While ml-pipes accepts any callable as a step, I went ahead and created my own operator package. Using that, a detection and annotation workflow can be represented as:
from ml_pipes.core import Pipeline
from ml_pipes.standard import Recall, Select, Store
import ml_pipes.supervision as sv
Pipeline([
Store("source_frame"),
sv.RoboflowInference(model_id=model_id, api_key=api_key),
Select(0),
sv.Detections.FromInference(),
Recall("source_frame"),
sv.ByteTrack(),
Recall("source_frame", prepend=True),
sv.TraceAnnotator(),
sv.BoxAnnotator(),
sv.LabelAnnotator(show_tracker_id=True, show_class=True),
])
There are two different responsibilities here.
- Supervision provides the computer-vision operations.
- ml-pipes provides the pipeline composition and execution model.
That distinction is important. The goal isn't to create yet another computer-vision library.
It's to provide a way to compose existing operations while retaining visibility into what happens between them.
What does "inspectable" actually mean?
Once the pipeline is represented explicitly, we can run inspection:
from ml_pipes.inspection import PipelineInspector
report = pipeline.inspect(image)
PipelineInspector().show(report)
Instead of only seeing the final output, now we can inspect the intermediate results:
For computer vision, this is particularly useful because the intermediate state is often something visual.
A more interesting example
A more interesting example is small-object detection, where tiling introduces additional transformations:
An Interactive Pipeline Inspection Report
This kind of pipeline is where the approach becomes particularly useful.
When an object can disappear somewhere between:
Image → Resize → Tile ─→ Model → Merge → Filter → Track → Annotate → ...
, and you want to know which step caused it.
Computer vision pipelines are data-flow problems
One thing this experiment has made increasingly clear to me is that many CV pipelines are easier to reason about as data-flow graphs than as collections of function calls.
For example:
┌──────────────┐
│ Source Image │
└──────┬───────┘
┌────▼────┐
│ Resize │
└────┬────┘
┌────▼────┐
│ Tile │
└────┬────┘
┌──────────┴──────────┐
┌────▼────┐ ┌────▼────┐
│ Model 1 │ │ Model 2 │
└────┬────┘ └────┬────┘
└──────────┬──────────┘
┌────▼────┐
│ Merge │
└────┬────┘
┌────▼────┐
│ Filter │
└────┬────┘
┌────▼────┐
│ Tracker │
└────┬────┘
┌────▼────┐
│Annotate │
└─────────┘
Once you think about the system this way, inspection becomes a natural capability of the pipeline rather than something bolted on afterwards.
This also makes debugging more interesting
Suppose the final output contains no detections.
Without pipeline inspection, you might start debugging the model, But the problem could actually be:
Image
✓
Resize
✓
Inference
✓ 42 detections
Filter
✗ 0 detections
Tracker
✗
Annotation
✗
The model was fine, the bug was in the filtering step.
This distinction matters because computer vision systems often have many places where information can be lost.
Why use Supervision?
This experiment started from a practical observation. There are already good libraries for computer vision operations.
Supervision is one example.
It provides a useful collection of reusable building blocks instead of requiring every application to implement detection visualization, tracking, filtering, and related functionality from scratch.
The relationship is roughly:
┌──────────────────────────────────────┐
│ Application │
├──────────────────────────────────────┤
│ ml-pipes │
│ composition / execution / │
│ inspection │
├──────────────────────────────────────┤
│ Supervision │
│ CV operations / tracking / │
│ annotations / utilities │
├──────────────────────────────────────┤
│ Models / OpenCV / etc. │
└──────────────────────────────────────┘
This also means that the same idea doesn't have to be tied to Supervision.
The underlying concept is broader:
make the computation graph explicit, that makes the data flowing through it inspectable.
The project is open source
The Supervision x ml-pipes integration/examples are here:
https://github.com/requiem4machines/ml-pipes-supervision
I'd be interested in hearing how other computer vision engineers currently debug complex pipelines.
Do you save intermediate images? Use notebooks? Add custom visualization code? Rely on logs and breakpoints? Or have you found a better approach?
Top comments (0)