The real reason for this article is I'm writing down stuff I might refer to in future because I hate the MongoDB docs.
Intro
In my on-going journey to become an evil data baron I've come to appreciate the value proposition of villainously tracking people's GPS coordinates in real time.
The problem is, if we make POST requests to update 10,000 people's coordinates every 30 seconds for 12 hours a day, we'd make already 864,000,000 calls. This is way above most rate limits. Costs would become prohibitive.
I'm not a top tier filthy rich villain machinating some big name tech mega-conglomerate. I'm just a dev, sitting at my desk, looking at Digital Ocean App Platform $5 droplets.
Fortunately, we can stream data. I've not calculated the costs in a detailed way but this seems potentially a lot cheaper.
Streaming is a very different paradigm. After a lot of research I settled on MongoDB change streams. Change streams allows us to monitor data in real time. Note: only monitoring.
It does not allow us to add or update it in real time. CRUD methods must still be done via regular HTTP calls.
That really disappointed me. I'd already spent quite some time reading its docs and then integrating it into my app. So now I'm going to rip it out and use an older strategy with a traditional self-hosted back end plus database for my current project. Sad. I really wanted to use a "serverless" DBAAS.
Nonetheless it's pretty useful. But, the documentation is byzantine. It's a kafkaesque monstrosity that gives me a literal neck ache to look at it. Documentation complexity is a kind of evil in itself.
So, the real reason for this article is I'm writing down stuff I might refer to in future.
Tech stack
- TypeScript
- React
- MongoDB Realm
Setup
I assume for the most part that you're able to do setup.
Set up cluster, database, and/or collection
Create database and/or collection
Docs.
Usually, I'd set up the database and then let my front end generate the collections. But in this case, we need the collection to show up in the Realm app dashboard. So use the MongoDB UX to create it.
In this example, imagine that I've created a database called "users" and one collection in it also called "users".
Give your front end and/or back end client access
Click on "Network Access" in the sidebar of your dashboard and add it. Docs.
Do note that if you're connecting from a service like Vercel that has dynamic IP's you'll have to set IP to 0.0.0.0/0. Docs. I don't have an opinion on the security stuff here. I probably should.
Setup Realm App
Create Realm App
Go to your MongodDB dashboard, click on the Realm tab, and click on "Create a New App". Docs.
Allow user access
It might be important to note I'm writing mainly from the perspective of a front end client here.
It was easy to miss this. In order for users to connect, we have to:
- Authenticate users, docs
- Click on "Authentication" under "Data Access" in the sidebar of Realm dashboard
- Enable the auth type you want and go from there
- Because I don't care about identifying my users, I use Anonymous Authentication
- Set up rules
- Click on "Rules" under "Data Access" in the sidebar of Realm dashboard
- Set rules for users on your collection
- owner means the user/client who inserts the record, non-owner means anyone else
Front end
Basic setup
We'll use the Realm Web SDK with React.
Run npm i realm-web.
If you're using Anonymous Authentication for users/clients then the only identifier you need to access the Realm app is the Realm app ID. It should be right at the top of your Realm dashboard.
I've placed all my Realm-related methods in the same module and instantiated a Realm app instance right at the top. All the methods below it have access to this instance:
import * as Realm from "realm-web";
const realmApp: Realm.App = new Realm.App({ id: realmAppId });
I created a method to auth users/clients. It should be nice to put user into global state but I'm not showing it here:
export const realmLogin = async () => {
const credentials = Realm.Credentials.anonymous();
try {
const user: Realm.User = await realmApp.logIn(credentials);
if (user.id !== realmApp?.currentUser?.id) {
// Handle auth fail
}
return user;
} catch (err) {
// Handle auth fail
}
};
We can have users auth/log out even though we don't really have a reason to if users are anonymous. Here's how anyway: realmApp?.currentUser?.logOut()
Set up change stream watcher
I cribbed the following code directly from the Web SDK docs.
It's important to note that we're using the Web SDK, not the Node one. They are different. It's easy to start reading the Node docs and not realise for a while that we're reading the wrong docs.
I prefer the Node change stream patterns more. But, well, we're stuck with this one for now. Anyway, here's a method for watching the stream:
const watchCollection = async () => {
// Connect to the users collection in the users database
const mongodb = realmApp?.currentUser?.mongoClient("mongodb-atlas");
const users = mongodb?.db("users").collection("users");
if (!users) {
return;
}
// This is the watcher, note the filter
const changeStream = users.watch({
filter: {
operationType: "insert",
"fullDocument.banned": false,
},
});
for await (const change of changeStream) {
switch (change.operationType) {
case "insert": {
const { documentKey, fullDocument } = change;
console.log(`new document with _id: ${documentKey}`, fullDocument);
break;
}
case "update": {
const { documentKey, fullDocument } = change;
console.log(`updated document: ${documentKey}`, fullDocument);
break;
}
case "replace": {
const { documentKey, fullDocument } = change;
console.log(`replaced document: ${documentKey}`, fullDocument);
break;
}
case "delete": {
const { documentKey } = change;
console.log(`deleted document: ${documentKey}`);
break;
}
}
}
return changeStream;
};
List of available change events.
The filter
The cool thing that attracted me to MongoDB Realm's change streams is the filter. I was so excited about it.
A lot of other databases might allow monitoring in real time, but they don't allow queries in real time on the stream. Change streams allows it.
I think Apache Kafka's interactive queries do the same thing but it's only available in Java and not in their JavaScript SDK. It's why I picked MongoDB's Realm change streams instead.
Notice that we're filtering for two criteria:
filter: {
operationType: "insert",
"fullDocument.banned": false,
}
Because of this, the Realm app will send to the client only records that have been inserted, and where the record field does not contain {"banned": true}.
The filters can probably be a lot more complex, involving mathematical operators and the like, just as we could do on regular MongoDB queries. It's just a simple example here.
Perhaps important to note is that this filter is passed on to the Realm app or kept in cookies or something (I think? Look, I didn't excavate the source code here, ok?) when our front end client connects. So if we're working in a framework that hot reloads like React, we cannot expect changes to the Realm methods to take effect just because our front end app has recompiled. We have to reload the page in the browser as well.
Get the party started
With the main methods written, all we have to do now is:
realmLogin();
watchCollection();
Functions, HTTPS endpoints, other features
Realms has some other pretty neat features, functions, triggers and HTTPS endpoints being just a few of them.
Here's one of my functions that I added to "Functions", under "Build", in the Realm sidebar. I named it "deleteAllUserData":
exports = async function(){
const userid = context.user.id;
const userColl = context.services.get("mongodb-atlas").db("users").collection("users");
const res = await userColl.deleteMany({userid});
return res;
};
Using the name, I call it in my React front end like this:
user.functions.callFunction("deleteAllUserData")
It deletes all the records created by the current connected user/client instance.
End
That's it. Now, get outta here and wreak profit.
Oldest comments (0)