I loaded over 300,000 Public Venmo Transactions into a Mongo Atlas database and use Atlas' Full Text Search to search by transaction memo.
Venmo transactions are interesting for full text search because users have to leave a public memo on every transaction. What relationships can you infer between Venmo users, based on their transaction memo? Are they friends, roommates, family, coworkers, lovers, or something else?
Choose Your Own Adventure
Uses MongoDB's Atlas Search to search hundreds of thousands of Venmo transactions. Made for dev.to's mongodb hackathon
I built this to explore using MongoDB's Atlas Search as an alternative to ElasticSearch for full text search. I used Venmo data because
- it's public
- users are forced to leave a text "comment" for every transaction
- Go to https://reidjs.github.io/venmo-search-frontend/
- Type a query
- Start backend https://github.com/reidjs/venmo-search-server
- Start frontend with
npm run dev
- Open browser to localhost:8000
Venmo dataset from https://github.com/sa7mon/venmo-data
- Out of the 7,000,000 transactions, I only used about 342,000 venmo transactions (free tier Mongo Atlas)
- To decide which transactions went into the database, I 'ranked' each transaction according to how long the comment was. Longer comment = more likely to be in database.
How to load in BSON data to atlas
mongorestore --uri mongodb+srv://[USERNAME]:[PASSWORD]@cluster0.abbj1.mongodb.net/[DATABASE] venmo.bson
How to Load in JSON data…
deploy changes to heroku:
npm run deploy
Looped through 7 million records to calculate the score on each one This nearly set my 2.6GHZ 6Core i7 32GB MBPro on fire In hindsight maybe I should have calculated a virtual field w/ mongoose, but this was the easy, dumb way of doing it. After about an hour I had at least 500,000 scores calculated, which was a big enough dataset to load into atlas