Retrospective: 6 Months Using MongoDB 7.0 for Our AI/ML Pipeline – 30% Faster Document Storage

When we set out to modernize our AI/ML pipeline in Q4 2023, we needed a document store that could handle high-throughput training data ingestion, low-latency model artifact storage, and seamless integration with our existing Python-based ML stack. After evaluating Cassandra, PostgreSQL, and MongoDB 7.0, we chose MongoDB 7.0 for its native vector search support, flexible schema design, and proven scalability for unstructured ML workloads. Six months later, we’re sharing our results: a 30% improvement in document storage speed, reduced operational overhead, and key lessons for teams running similar workloads.

Why MongoDB 7.0? Key Features for AI/ML Workloads

MongoDB 7.0 shipped with several features tailored to ML pipelines, which drove our initial selection:

• Atlas Vector Search: Native support for vector embeddings, eliminating the need for a separate vector database to store and query ML model embeddings and training dataset features.
• Improved Time-Series Collections: Optimized for high-velocity ingestion of training metrics, inference logs, and pipeline telemetry, with automatic compression and TTL support.
• Enhanced Aggregation Pipeline: New $vectorSearch and $densify operators simplified preprocessing of training data directly in the database, reducing data movement to external processing tools.
• Sharding Improvements: Better elastic scaling for our growing training dataset, which grew from 12TB to 41TB over the 6-month period with no downtime for shard rebalancing.

Performance Results: 30% Faster Document Storage

We measured storage performance across three core pipeline stages: raw training data ingestion, model artifact writes (checkpoints, weights, metadata), and inference result logging. All benchmarks used the same workload profile: 1.2M document writes per minute, average document size 4.7KB, with 3x replication across our production cluster.

Compared to our previous MongoDB 6.0 deployment, MongoDB 7.0 delivered:

• 30% faster document write latency: Average write latency dropped from 12ms to 8.4ms for hot data, with 99th percentile latency improving from 47ms to 31ms.
• 22% higher throughput: We increased write throughput from 1.2M to 1.46M documents per minute without adding cluster nodes, thanks to improved write batching and storage engine optimizations.
• 18% reduction in storage footprint: New compression algorithms for time-series and unstructured document data reduced our total storage costs by nearly a fifth over the 6-month period.

We validated these results using MongoDB’s built-in performance advisor and custom Prometheus/Grafana dashboards tracking write latency, throughput, and error rates. No regressions were observed in read performance for training data access, with 95th percentile read latency holding steady at 6ms.

Lessons Learned: Optimizing MongoDB 7.0 for ML Pipelines

While the out-of-the-box performance gains were significant, we had to make several configuration and schema adjustments to maximize results for our specific workload:

• Schema Design for ML Workloads: We moved from embedding large training metadata objects to referencing them in separate collections, reducing document size for high-throughput write paths. For model artifacts, we used gridfs only for files larger than 16MB, storing smaller checkpoints as BSON documents to avoid gridfs overhead.
• Indexing Strategy: We avoided over-indexing write-heavy collections, relying on MongoDB 7.0’s improved default indexing for time-series data. For vector search workloads, we created 1024-dimensional embedding indexes with HNSW algorithm, tuned for 90% recall to balance query speed and accuracy.
• Operational Tweaks: We enabled the new storage engine cache prioritization for write-heavy collections, and set up automated shard key rebalancing during off-peak hours to avoid impacting pipeline throughput. We also migrated to MongoDB 7.0’s new connection pooling defaults for our Python ML workers, reducing connection overhead by 15%.
• What Didn’t Work: We initially tried using change streams to trigger model retraining on new data, but the added latency and overhead outweighed the benefits for our high-throughput pipeline. We reverted to batch-based triggers for retraining instead.

Conclusion: Is MongoDB 7.0 Right for Your AI/ML Pipeline?

After 6 months of production use, MongoDB 7.0 has become a core component of our AI/ML stack. The 30% faster document storage, combined with native vector search and improved scalability, has reduced our pipeline runtime by 22% and lowered operational costs by 18%. For teams running similar unstructured ML workloads, we highly recommend evaluating MongoDB 7.0 – especially if you’re already using or considering vector search for embedding storage.

Next steps for our team include migrating our remaining legacy PostgreSQL training metadata stores to MongoDB 7.0, and evaluating the new MongoDB 7.0.1 point release for additional performance improvements for vector search workloads. We’ll share another update in 6 months with results from those migrations.