DEV Community

Cover image for 🌐 Get started: What is MongoDB operational data layer? (Part 2) 🌐
Danny Chan for MongoDB Builders

Posted on

1 1 1 1 1

🌐 Get started: What is MongoDB operational data layer? (Part 2) 🌐

Operational Data Layer Data loading:
βœ… Data must sync with source systems
βœ… Appropriate data loading strategy
βœ… Producer systems: frequency and quantity of data changes
βœ… Consuming systems: clear requirements for data currency


Step 1: Batch extract and load:
πŸ“ Initial batch load
πŸ“ Copy application database data to Operational Data Layer
πŸ“ One-time operation to load data from source systems


*Step 2: Delta extract and load: *
πŸ”„ Starts immediately following initial batch load
πŸ”„ Real-time synchronization
πŸ”„ Incremental updates from source systems into the ODL
πŸ”„ Use Change Data Capture (CDC)
πŸ”„ Catch changes from source systems
πŸ” Matching, merging, reconciling data


Data flow and maturity model:
πŸ—οΈ Simple start
🌱 Grows in scope and strategic importance
πŸ† Delivering increased benefits to business


Phase 1: Simple ODL, offloading reads:
πŸ’» Serve only read operations
πŸ’° Cut costs
πŸ”’ High availability: takes over during source system downtime
⚑ Improve performance
πŸ“Š Handle long-running analytics queries
πŸ“ˆ Handle high read traffic peak


Phase 2: Enriched ODL for new use cases:
πŸ” Create single customer view
πŸ’³ Example: credit card transactions enriched by categorizing purchases
πŸ’° Determine their spend on each category (travel)


Phase 3: Offloading reads and writes:
πŸ“₯ Y-loading un both latency, new systems in parallel


Phase 4: ODL first:
✍️ All writes are directed to Operational Data Layer (ODL)


Phase 5: System of Record:
πŸ—ƒοΈ Operational Data Layer serve as the System of Record
🏳️ Source system can be decommissioned for cost savings
πŸ›οΈ Architectural simplicity


MongoDB for Operational Data Layer:
πŸ€— Ease: MongoDB's document model easily manage data
🧩 Flexibility: integrate multiple source systems into a single ODL, without pre-define schema
⚑ Speed: better performance when accessing data
🎨 Versatility: satisfy a range of application requirements by flexibility of document model


Example:
πŸ“ Embedding of arrays and sub-documents
πŸ“Š Modeling complex relationships and hierarchical data
πŸ—„οΈ Ability to manipulate deeply nested data without rewrite entire document
πŸ—‚οΈ Model flat, table-like structures, simple key-value pairs, text
🌍 Geospatial data
πŸ•ΈοΈ Nodes and edges used in graph processing


Processing pipelines:
πŸ” Lookups and range queries
πŸ“Š Data analytics
πŸ”¨ Transformations
πŸ” Faceted search
🌎 Geospatial processing
πŸ•΅οΈ Graph traversals



Intelligently distribute (Operational Data Layer) ODL:


Availability:
πŸ’» Multiple copies of data using replica sets
πŸ”„ Failover and recovery is fully automated


Scalability:
πŸ†™ Challenge: new source systems, adding data volume, new consuming systems, increasing workload
πŸ—„οΈ Large data sets
⚑ High throughput requirements
🧩 Solution - sharding:
πŸ€– MongoDB provides horizontal scale-out on low-cost
πŸ” Automatically partitions and distributes data across multiple physical instances


Workload isolation:
πŸ” Operational Data Layer able to safely serve disparate workloads
πŸ” Analytical queries on up-to-date data without impact on production applications


Data locality:
🌎 Allows precise control over where data is physically stored
πŸ—ΊοΈ Control geographic region for latency, governance requirements


Reference:

https://www.mongodb.com/resources/basics/implementing-an-operational-data-layer
Implementing an Operational Data Layer

https://www.mongodb.com/resources/solutions/use-cases/mainframe-modernization-reference-architecture
Mainframe Modernization Reference Architecture


Editor

Image description

Danny Chan, specialty of FSI and Serverless

Image description

Kenny Chan, specialty of FSI and Machine Learning

API Trace View

How I Cut 22.3 Seconds Off an API Call with Sentry πŸ•’

Struggling with slow API calls? Dan Mindru walks through how he used Sentry's new Trace View feature to shave off 22.3 seconds from an API call.

Get a practical walkthrough of how to identify bottlenecks, split tasks into multiple parallel tasks, identify slow AI model calls, and more.

Read more β†’

Top comments (0)

A Workflow Copilot. Tailored to You.

Pieces.app image

Our desktop app, with its intelligent copilot, streamlines coding by generating snippets, extracting code from screenshots, and accelerating problem-solving.

Read the docs