Thanks for writing this. Your article led to a breakthrough for me on the project I'm working on, though probably not in the context you imagined when writing this so thought I'd share. I have a client that uses MongoDB Atlas for their enterprise-wide system of record. But their analytics team needs the data in tabular format. Atlas has a "BI Connector" that allows querying the cluster's databases with traditional SQL - I imagine behind the scenes they are mapping any document changes out to a relational structure that mirrors the collections. However the fact that the relational structure is being maintained on a moment by moment basis impacts performance. And the BI tool itself has a technical learning curve that these non-tech folks don't want to deal with. They're familiar with SQL Server. So I proposed to write automation that can pipe the data into any system of record in their organization, using node + loopback.js. loopback provides a way to quickly generate an API (REST or GraphQL) and has connectors for almost any kind of data source. The challenge is that I didn't want to hardcode the Atlas document structures into the application, as they are in flux at the moment. I'm building an auto-mapper that can connect to the specified MongoDB database, get its collections and automatically generate SQL tables from them and populate those SQL tables from them, with in-database configuration params to control the filtering e.g. only get last 5 minutes of data from collection X using its .timestamp property. At first I tried using mongoexport.exe which lets you export to flatfiles and it takes in collections and fields as params. But the problem is it exports large numbers in scientific notation format, which loses some of the number! Like, really Mongo devs?? wtf. Just export it as a string if u have to. But nope. Brick wall. Compass has a great csv export which gets the numbers right and doesn't mess them up using scientific notation, but it's not currently automatable, i.e. you can't call its functions from the command line. I knew there had to be a way around this so I just searched for MongoDB Atlas API which dug up your article by accident (i.e. I was looking for an API I could use but no luck then I found your article. And I was like DUH, use the mongodb driver for node! No more scientific notation on bigints, and fully automatable, So thanks! I will still put the code inside loopback.js because it's a fantastically architected framework for modularizing your code into services and exposing those via API. I'm not here to sell loopback.js but I love it. And I'm always open to other ideas even if you think there's a much better way to do what I'm trying to accomplish here. Have a good one!
For further actions, you may consider blocking this person and/or reporting abuse
We're a place where coders share, stay up-to-date and grow their careers.
Thanks for writing this. Your article led to a breakthrough for me on the project I'm working on, though probably not in the context you imagined when writing this so thought I'd share. I have a client that uses MongoDB Atlas for their enterprise-wide system of record. But their analytics team needs the data in tabular format. Atlas has a "BI Connector" that allows querying the cluster's databases with traditional SQL - I imagine behind the scenes they are mapping any document changes out to a relational structure that mirrors the collections. However the fact that the relational structure is being maintained on a moment by moment basis impacts performance. And the BI tool itself has a technical learning curve that these non-tech folks don't want to deal with. They're familiar with SQL Server. So I proposed to write automation that can pipe the data into any system of record in their organization, using node + loopback.js. loopback provides a way to quickly generate an API (REST or GraphQL) and has connectors for almost any kind of data source. The challenge is that I didn't want to hardcode the Atlas document structures into the application, as they are in flux at the moment. I'm building an auto-mapper that can connect to the specified MongoDB database, get its collections and automatically generate SQL tables from them and populate those SQL tables from them, with in-database configuration params to control the filtering e.g. only get last 5 minutes of data from collection X using its .timestamp property. At first I tried using mongoexport.exe which lets you export to flatfiles and it takes in collections and fields as params. But the problem is it exports large numbers in scientific notation format, which loses some of the number! Like, really Mongo devs?? wtf. Just export it as a string if u have to. But nope. Brick wall. Compass has a great csv export which gets the numbers right and doesn't mess them up using scientific notation, but it's not currently automatable, i.e. you can't call its functions from the command line. I knew there had to be a way around this so I just searched for MongoDB Atlas API which dug up your article by accident (i.e. I was looking for an API I could use but no luck then I found your article. And I was like DUH, use the mongodb driver for node! No more scientific notation on bigints, and fully automatable, So thanks! I will still put the code inside loopback.js because it's a fantastically architected framework for modularizing your code into services and exposing those via API. I'm not here to sell loopback.js but I love it. And I'm always open to other ideas even if you think there's a much better way to do what I'm trying to accomplish here. Have a good one!