🌐 Get started: MongoDB Change streams, Concurrency, backup snapshot & checkpoint, Compound Wildcard Indexes

💫 Change streams:

• Use case: large documents, pre-images, post-images
• Access real-time data changes
• Subscribe to all data changes on database, then react to them
• Available for replica sets and sharded clusters
• Must use WiredTiger storage engine
• Can use encryption-at-rest

🕰️ Time series data:

• High-volume datasets
• Improved storage optimization and compression
• High cardinality data
• Fine-grained updates & deletes

{
    "metadata": { "sensorId": 5578, "type": "temperature" },
    "timestamp": ISODate("2021-05-18T00:00:00.000Z"),
    "temp": 12
}

Concurrency:

🔒 WiredTiger Storage Engine:

• Can't pin documents to WiredTiger cache
• Cannot reserve a portion of cache for reads & writes
• Heavy write workload affect performance
• Allocates its cache to the entire MongoDB instance

🌍 Feature: Transaction (Read & Write) Concurrency:

• Dynamically adjust maximum number of concurrent storage engine transactions (read and write)
• Optimizes database throughput during cluster overload
• Never exceeds 128 read & write tickets
• Read & write are always equal

🔍Document Level Concurrency control:

• For WiredTiger to control write operations
• Multiple clients can modify different documents of a collection at the same time
• Intent locks at the global, database and collection levels
• Detects conflicts between two operations, one will stop due to conflict operation, other one keep going
• Transparently retry conflict operation

Backup:

🛡️Feature: point-in-time snapshot
MultiVersion Concurrency Control (MVCC)

🗄️ Feature: checkpoint

• Act as recovery points
• Now-durable data in the data files
• Consistent up to and including last checkpoint
• Writing snapshot data to disk every 60 seconds
• If MongoDB encounters an error while writing a new checkpoint, MongoDB can recover from last valid checkpoint

📜 Journal:

• Ensure data durability by write-ahead log (journal) with checkpoints
• Persists all data modifications between checkpoints
• MongoDB error between checkpoints, use journal to replay all data modified

📜 Compression:

• Snappy compression for collections
• Prefix compression for indexes (deduplicate common prefixes)

📜 Memory Use:

• Utilizes both WiredTiger internal cache and filesystem cache

📜 Block compression:

• Provide significant on-disk storage savings
• Collection data in WiredTiger internal cache is uncompressed for able to be manipulated

Compound Wildcard Indexes:

🔍 Feature: Compound Wildcard Indexes

• One wildcard index replace a large number of individual indexes
• Can efficiently cover many potential queries Apply to attribute Pattern

From

{ tenantId: 1, "customFields.addr": 1 }
{ tenantId: 1, "customFields.name": 1 }
{ tenantId: 1, "customFields.blockId": 1 }

To

{ tenantId: 1, "customFields.$**": 1 }

More Features:

🔄 cluster-to-cluster sync (mongosync)

• Syncing specific data sets instead of the entire cluster

📊 approximate percentiles

• Use $percentile as an accumulator in the $group stage or as an aggregation expression Record:

{ studentId: "2345", test01: 62, test02: 81, test03: 80 }

Result:

{ studentId: '2345', testPercentiles: [ 80, 81 ] }

👤 user role

• Use user role variables within aggregation pipelines
• Example: application display different data based on user permissions

🔒 Queryable Encryption

• Security
• Fully randomized encrypted data

---

Editor

Danny Chan, specialty of FSI and Serverless

Kenny Chan, specialty of FSI and Machine Learning