DEV Community: rachelle palmer

Supercharging Time Series Collections: Key Enhancements in MongoDB 8.0 with Block Processing

rachelle palmer — Fri, 04 Oct 2024 20:29:03 +0000

The landscape for time-series data has evolved significantly in recent years. Businesses are capturing more granular data, recognizing that the value of data rises in proportion with its precision. Storing time-stamped data and performing temporal analytics is becoming essential, even mandated across industries. As data volumes inevitably grow and precision-based analytics become increasingly crucial, tools that provide efficient ways to work with time-series data will become even more critical.

To address the rising demand for time-series analytics, we introduced Time Series Collections in MongoDB 5.0, designed to meet the expanding needs of time-stamped data. Unlike point solutions that require separate complex setups, MongoDB’s Time Series Collection enables users to simply stand up a collection and instantly leverage time-series capabilities. Over time, we’ve expanded its capabilities with features like columnar compression, enhanced temporal analytics, enriched indexing, geo-support, and seamless integration with the broader MongoDB portfolio—creating a streamlined, developer-friendly experience.

In our upcoming 8.0 release, we’re excited to introduce new features that significantly enhance scalability and query performance for managing large time-series workloads, delivering even better price-performance for our users.

Enhanced Time Series Scalability Improvements

As time-series data volumes grow, the challenge isn't just about scaling, it's about doing so efficiently, balancing resources, cost, and performance. With MongoDB 8.0, we’re introducing key optimizations in Time Series Collections to help users maximize resource value while managing increasingly complex workloads.

In previous versions, Time Series Collections inserted data in an uncompressed format, causing a larger working set, increased cache use, and write amplification, leading to high I/O as uncompressed data was written to the WiredTiger storage engine. This was especially problematic for high-cardinality workloads with millions of devices, sensors etc. With MongoDB 8.0, Time Series Collections now directly write into a column-compressed format, reducing cache usage, lowering write I/O, improving insert performance and storage efficiency.

For users, the benefits of lower cache usage directly translates into cost savings. Users will be able to extract a lot more value from their existing cluster resources resulting in better price-performance. We’ve observed throughput improvements of 2-3x, with cache usage reduced by 10-20x compared to workloads on version 7.0.
For example, workload on MongoDB 7.0 can experience performance fluctuations, leading to inconsistent write performance and a sawtooth pattern caused by I/O overload. Writing large amounts of uncompressed data to disk strained checkpoints. As shown below, a test on a 7.0 Atlas M50 (RAM - 32GB, Storage 160GB, 8 vCPUs) cluster reached peaks of 500K inserts/s while displaying this pattern:

With 8.0, the write performance is now steady eliminating I/O strain. As shown below, the same workload on Atlas M50 achieves a stable 600K inserts/s, eliminating the previous sawtooth pattern.

Block Processing with Time Series Collections

User engagement highlighted that query performance is crucial as time-series workloads scale. Traditionally, the MongoDB query engine processes data one document at a time, which can be inefficient for large-scale analytics. For Time Series Collections, this inefficiency is due to the need to unpack and reshape a large volume of compressed data. To address this, we introduced Block Processing for Time Series Collections – a new automatic query execution model that processes "blocks of data" at once, leveraging column-level summaries while avoiding the overhead of unpacking and reshaping documents. This approach significantly improves performance, particularly for aggregation that leverage stages such as $match, $sort, $group and other analytical stages like $setWindowFields.

By making each step more efficient, the overall impact becomes exponential, reducing overhead and leveraging time-series data patterns previously unavailable. Each aggregation stage processes larger data chunks, leading to a more efficient query execution model with significantly faster performance. Use cases like financial analysis and IoT, which involve intensive filtering, grouping, and sorting (i.e $match, $group, $sort), will see major performance improvements. With Block Processing, we've observed improvements ranging from 10-40x, with some large-scale aggregations reaching up to 100x.

Let’s explore how Block Processing can optimize common financial aggregations, such as generating OHLC (Open, High, Low, Close) and calculating an exponential moving average over a specified time period. In this example, we analyze an aggregation that saw a 20x improvement on an Atlas M50 replica set using MongoDB's fork of TSBS (time-series benchmarking suite). We use the TSBS finance use case that generates a workload containing 10 stock symbols, each creating an event per second over 7 days, resulting in approximately 6 million events loaded into a Time Series collection.

Here's a sample document so you can see what this looks like:

{
  "time" : ISODate("2022-01-01T00:00:00Z"),
  "tags" : {
    "symbol" : "MDB"
  },
  "_id" : ObjectId("64c4092a9451cd8064c69be1"),
  "measurement" : "price", 
  "price" : 200.13171
}

We start by creating a time-series collection market_data in MongoDB, where the timeField is set to time, and metaField to tags. The granularity is set to "seconds" to capture financial data at a 1-minute interval.

db.createCollection("market_data", {
  timeseries: {
    timeField: "time",
    metaField: "tags",
    granularity: "seconds"
  }
}

Index:
We create two compound indexes to support queries within the workload:

db.market_data.createIndex({ "tags": 1, "time": 1 }); 
db.market_data.createIndex({ "tags.symbol": 1, "time": -1 });

Query:
Next, we construct a query to generate the OHLC and exponential moving average for a group of stock symbols computed every 4 hours over a 1 day time window.

db.market_data.aggregate([
  {"$match": {"$expr": {"$gte": ["$time", {"$dateSubtract": {"startDate": new Date("2022-01-01T03:00:00Z"), "unit": "hour", "amount": 24}}]}}},
  {"$sort": {"time": 1}},
  {"$group": {
    "_id": {"symbol": "$tags.symbol", "time": {"$dateTrunc": {"date": "$time", "unit": "minute", "binSize": 240}}}, 
    "high": {"$max": "$price"},
    "low": {"$min": "$price"},
    "open": {"$first": "$price"},
    "close": {"$last": "$price"}
  }},
  {"$setWindowFields": {
    "partitionBy": "$_id.symbol",
    "sortBy": {"_id.time": 1},
    "output": {"expMovingAverage": {"$expMovingAvg": {"input": "$close", "N": 100}}}
  }},
  {"$sort": {"_id.time": -1}}
]);

Sample Output:

{
  "_id": { "symbol": "MDB", "time": ISODate("2022-01-01T02:00:00Z") },
  "high": 148.27729312597552,
  "low": 51.01901106672195,
  "open": 126.83590008130241,
  "close": 99.44471233463418,
  "expMovingAverage": 99.44471233463418
}

Compared to 7.0, this query saw its operations per second improve by an incredible 2000%, or 20x. This dramatic boost reflects our ongoing commitment to helping MongoDB users handle complex time-series data with ease, and we’re excited to see what can be achieved with these new enhancements.

Please try it out, and let us know your feedback!

contributed by Nishith Atreya and Michael Gargiulo

How to retrieve Github Repository Data using Python

rachelle palmer — Sat, 28 Sep 2024 17:21:08 +0000

Does your organization have (way) too many github repositories, and you need an easy way to summarize and keep record of what each one is for reporting, dashboard, or auditing purposes? Here's a quick script to do that very thing using the Github API.

Functions:

get_repo_info(owner, repo):
- Takes a GitHub repository owner's username (owner) and repository name (repo).
- Sends a request to GitHub's API to get repository information.
- Returns the repository's information as a JSON object if successful, or None if there is an error.
get_collaborators(collaborators_url):
- Takes the URL to the list of collaborators for a repository.
- Sends a request to fetch the list of collaborators.
- Returns a list of collaborator usernames, or an empty list if an error occurs.
get_languages(languages_url):
- Takes the URL to the repository's languages data.
- Sends a request to retrieve the programming languages used in the repository.
- Returns a list of languages, or an empty list if there is an error.
get_open_issues(owner, repo):
- Takes the repository owner's username (owner) and repository name (repo).
- Sends a request to retrieve the list of open issues in the repository.
- Returns the open issues in JSON format, or prints an error message if there's a problem.
get_repo_data(repo_url):
- Takes a repository URL, parses it to get the owner and repo values, and then calls the other functions to gather various information about the repository.
- Compiles the repository information, including its name, owner, visibility, collaborators, languages, open issues, and last activity, and returns it in a structured format (a dictionary).

import json
import requests
from pymongo import MongoClient

# MongoDB setup (replace with your actual connection details)
client = MongoClient("mongodb://localhost:27017/")
db = client["github_repos"]  # Database name
collection = db["repos"]     # Collection name

def get_repo_info(owner, repo):
    url = f"https://api.github.com/repos/{owner}/{repo}"
    headers = {"Accept": "application/vnd.github+json"}
    response = requests.get(url, headers=headers)
    if response.status_code == 200:
        return response.json()
    else:
        print(f"Error: {response.status_code}")
        return None

def get_collaborators(collaborators_url):
    response = requests.get(collaborators_url)
    if response.status_code == 200:
        return [collaborator["login"] for collaborator in response.json()]
    else:
        return []

def get_languages(languages_url):
    response = requests.get(languages_url)
    if response.status_code == 200:
        return list(response.json().keys())
    else:
        return []

def get_open_issues(owner, repo):
    url = f"https://api.github.com/repos/{owner}/{repo}/issues?state=open"
    headers = {"Accept": "application/vnd.github+json"}
    response = requests.get(url, headers=headers)
    if response.status_code == 200:
        return response.json()
    else:
        print(f"Error: {response.status_code}")
        return []

def get_repo_data(repo_url):
    owner, repo = repo_url.split("/")[-2:]
    repo_info = get_repo_info(owner, repo)

    if repo_info:
        data = {
            "Github URL": repo_url,
            "Project name": repo_info["name"],
            "Project owner": repo_info["owner"]["login"],
            "List users with access": get_collaborators(repo_info["collaborators_url"].split("{")[0]),  # remove template part of URL
            "Programming languages used": get_languages(repo_info["languages_url"]),
            "Security/visibility level": repo_info["visibility"],
            "Summary": repo_info["description"],
            "Last maintained": repo_info["pushed_at"],
            "Last release": repo_info["default_branch"],
            "Open issues": get_open_issues(owner, repo),
        }

        # Insert the data into MongoDB
        collection.insert_one(data)
        print("Data inserted into MongoDB successfully.")

        return data
    else:
        return None

# Example usage
repo_url = "https://github.com/URL"
repo_data = get_repo_data(repo_url)

if repo_data:
    print(json.dumps(repo_data, indent=4))

Using ASP.NET Core OData with MongoDB Atlas

rachelle palmer — Fri, 24 May 2024 00:42:35 +0000

Overview

This article focuses on how you can quickly get up and running using Microsoft's ASP.NET Core OData extension with MongoDB Atlas.

OData stands for Open Data Protocol which is a standard that allows developers to simplify the process of building and consuming RESTful APIs. MongoDB Atlas is MongoDB's cloud offering that allows you to leverage the full potential of MongoDB's developer data platform without any cloud vendor lock-in.

Tutorial

You need to have a MongoDB Atlas Cluster set up with sample data loaded as shown here

I'm using Visual Studio 2022 Community Edition in this tutorial. You can choose to use any IDE of your choice. Open up a new project and choose the ASP.NET Core Empty Project template.

On the following screen, name your project and click Next. Choose the latest LTS .NET Framework and make sure to disable the 'Configure for HTTPS checkbox.

Now we are going to add the MongoDB OData extension from the Nuget package manager as seen below. This uses the MongoDB .NET/C# Driver and Microsoft's AspNetCore.OData package as its dependencies.

I'm going to use the sample_restaurants database from the sample restaurants dataset loaded into my MongoDB Atlas cluster. We need some model classes as shown below in our project for this.
First we'll create the Restaurant class.


 csharp
using MongoDB.Bson;
using MongoDB.Bson.Serialization.Attributes;

namespace ODataTutorial.Models
{
    [BsonIgnoreExtraElements]
    public class Restaurant
    {
        [BsonRepresentation(BsonType.ObjectId)]
        public string Id { get; set; }

        [BsonElement("name")]
        public string Name { get; set; }

        [BsonElement("restaurant_id")]
        public string? RestaurantId { get; set; }

        [BsonElement("cuisine")]
        public string Cuisine { get; set; }

        [BsonElement("address")]
        public Address Address { get; set; }

        [BsonElement("borough")]
        public string Borough { get; set; }
    }
}

Next we’ll create the Address model class.


 csharp
using MongoDB.Bson.Serialization.Attributes;

namespace ODataTutorial.Models
{
    public class Address

  {
        [BsonElement("building")]
        public string Building { get; set; }

        [BsonElement("coord")]
        public double[] Coordinates { get; set; }

        [BsonElement("street")]
        public string Street { get; set; }

        [BsonElement("zipcode")]
        public string ZipCode { get; set; }
    }
}

The Address class does not require an _id because it is only stored as a nested subdocument.

Note that we are using the [BsonElement] attribute to tell the MongoDB C# driver to map the names in our model classes to the field names in the database. You could also choose to use a convention pack to do the same.

The [BsonIgnoreExtraElements] tells the driver to ignore other fields in the database like 'grades' so that it doesn't throw an error when deserializing such fields. You can read more about serialization using the MongoDB C# driver here.

We need the [BsonRepresentation(BsonType.ObjectId)] attribute to tell the Driver that the Id is represented as a string in our model class but as an ObjectId in the database.

Now we'll create a simple Controller that inherits from ODataController and has a Get action with the [MongoEnableQuery] attribute. This enables the use of URI query options in our API calls.


 csharp 
using Microsoft.AspNetCore.Mvc;
using Microsoft.AspNetCore.OData.Routing.Controllers;
using MongoDB.AspNetCore.OData;
using MongoDB.Driver;
using ODataTutorial.Models;

namespace ODataTutorial.Controllers
{
    public class RestaurantsController : ODataController
    {
        private readonly IMongoCollection<Restaurant> _restaurants;

        public RestaurantsController(IMongoClient mongoClient)
        {
            var database = 
mongoClient.GetDatabase("sample_restaurants");
            _restaurants = database.GetCollection<Restaurant> 
("restaurants");
        }

        [MongoEnableQuery]
        public ActionResult Get()
        {
            return Ok(_restaurants.AsQueryable());
        }
    }
}

Our Program.cs file which is the entry point of this Web Application will be quite simple and we will use the Startup.cs file to connect to our database and set up the Model Builder.


 csharp
using ODataTutorial;

public class Program
{
    public static void Main(string[] args)
    {
        var app = CreateHostBuilder(args).Build();
        app.Run();
    }

    public static IHostBuilder CreateHostBuilder(string[] args) =>
        Host.CreateDefaultBuilder(args)
            .ConfigureWebHostDefaults(webBuilder => 
            {
                webBuilder.UseStartup<Startup>();
            });
}

While setting up the modelBuilder we need to tell it to use our Id field as the primary key instead of the RestaurantID and this is done through model.HasKey(e => e.Id)


 csharp
using Microsoft.AspNetCore.OData;
using Microsoft.OData.ModelBuilder;
using MongoDB.Driver;
using ODataTutorial.Models;

namespace ODataTutorial
{
    public class Startup
    {
        public IConfiguration Configuration { get; }
        public Startup(IConfiguration configuration)
        {
            Configuration = configuration;
        }

        public void ConfigureServices(IServiceCollection services) 
        {
            var connectionString = Configuration.GetSection("MongoDB").GetValue<string>("Uri");
            if (string.IsNullOrEmpty(connectionString))
            {
                throw new InvalidOperationException("Cannot read MongoDB connection settings");
            }

            services.AddSingleton<IMongoClient>(new MongoClient(connectionString));

            var modelBuilder = new ODataConventionModelBuilder();
            var model = modelBuilder.EntitySet<Restaurant>("Restaurants").EntityType;
            model.HasKey(e => e.Id);

            services.AddControllers().AddOData(
            options =>
            {
                options.Select().Filter().OrderBy().Expand().Count().SetMaxTop(1000).AddRouteComponents(
                    "odata",
                    modelBuilder.GetEdmModel());
            });
        }

        public void Configure(IApplicationBuilder app, IWebHostEnvironment env)
        {
            app.UseRouting();
            app.UseEndpoints(endpoints =>
            {
                endpoints.MapControllers();
            });
        }
    }
}

Lastly, we need to make sure we add our MongoDB Atlas connection URL in our appsettings.json file as seen below


 json
  "MongoDB": {
    "Uri": "<Enter MongoDB Atlas Connection URL here>"
  },

Voila! That's it, we are now ready to test our OData endpoints with MongoDB Atlas.

Testing

Once you hit Build and let our application run, you should see it fire up in your chosen browser. We are using a browser in this example but we could use other tools like Postman to test the same.

The default endpoint based on the port configured (E.g http://localhost:5279/ in my project. Please note that this port will be randomly generated when creating a new project) will not show anything since we don't have it configured. However once you change it to http://localhost:5279/odata/, you should see a response like this


 json
{
  "@odata.context": "http://localhost:5279/odata/$metadata",
  "value": [
    {
      "name": "Restaurants",
      "kind": "EntitySet",
      "url": "Restaurants"
    }
  ]
}

You can also see the metadata of our data model by going to http://localhost:5279/odata/$metadata Note that our Id field is shown as the Key as intended.


 XML
<edmx:Edmx xmlns:edmx="http://docs.oasis-open.org/odata/ns/edmx" Version="4.0">
<edmx:DataServices>
  <Schema
    xmlns="http://docs.oasis-open.org/odata/ns/edm" Namespace="ODataTutorial.Models">
    <EntityType Name="Restaurant">
      <Key>
        <PropertyRef Name="Id"/>
      </Key>
      <Property Name="Id" Type="Edm.String" Nullable="false"/>
      <Property Name="Name" Type="Edm.String" Nullable="false"/>
      <Property Name="RestaurantId" Type="Edm.String"/>
      <Property Name="Cuisine" Type="Edm.String" Nullable="false"/>
      <Property Name="Address" Type="ODataTutorial.Models.Address" Nullable="false"/>
      <Property Name="Borough" Type="Edm.String" Nullable="false"/>
    </EntityType>
    <ComplexType Name="Address">
      <Property Name="Building" Type="Edm.String" Nullable="false"/>
      <Property Name="Coordinates" Type="Collection(Edm.Double)" Nullable="false"/>
      <Property Name="Street" Type="Edm.String" Nullable="false"/>
      <Property Name="ZipCode" Type="Edm.String" Nullable="false"/>
    </ComplexType>
  </Schema>
  <Schema
    xmlns="http://docs.oasis-open.org/odata/ns/edm" Namespace="Default">
    <EntityContainer Name="Container">
      <EntitySet Name="Restaurants" EntityType="ODataTutorial.Models.Restaurant"/>
    </EntityContainer>
  </Schema>
</edmx:DataServices>
</edmx:Edmx>

Now let's try to fetch data with some queries. We'll only see the first 1000 restaurants since we had SetMaxTop(1000) in our Startup.cs file. You can change this as needed. If we go to the endpoint http://localhost:5279/odata/Restaurants, we should be able to see our list of Restaurants.


 json
{
"@odata.context": "http://localhost:5279/odata/$metadata#Restaurants",
"value": [
  {
    "Id": "5eb3d668b31de5d588f42930",
    "Name": "Brunos On The Boulevard",
    "RestaurantId": "40356151",
    "Cuisine": "American",
    "Borough": "Queens",
    "Address": {
      "Building": "8825",
      "Coordinates": [ -73.8803827, 40.7643124 ],
      "Street": "Astoria Boulevard",
      "ZipCode": "11369"
    }
  },
  {
    "Id": "5eb3d668b31de5d588f42932",
    "Name": "Taste The Tropics Ice Cream",
    "RestaurantId": "40356731",
    "Cuisine": "Ice Cream, Gelato, Yogurt, Ices",
    "Borough": "Brooklyn",
    "Address": {
      "Building": "1839",
      "Coordinates": [ -73.9482609, 40.6408271 ],
      "Street": "Nostrand Avenue",
      "ZipCode": "11226"
    }
  },
  {
    "Id": "5eb3d668b31de5d588f42934",
    "Name": "C & C Catering Service",
    "RestaurantId": "40357437",
    "Cuisine": "American",
    "Borough": "Brooklyn",
    "Address": {
      "Building": "7715",
      "Coordinates": [ -73.9973325, 40.6117489 ],
      "Street": "18 Avenue",
      "ZipCode": "11214"
    }
  },
  .
  .
  .
}

Let's try to play around with some filters. This will allow us to query data without any additional logic in our application. We can search for all Italian restaurants using http://localhost:5279/odata/Restaurants?$filter=Cuisine eq 'Italian' or even better, we can search for all Italian restaurants in Queens by http://localhost:5279/odata/Restaurants?$filter=Cuisine eq 'Italian' and Borough eq 'Queens' This should give us all the restaurants satisfying that query filter.


 json
{
"@odata.context": "http://localhost:5279/odata/$metadata#Restaurants",
"value": [
  {
    "Id": "5eb3d668b31de5d588f429ed",
    "Name": "Piccola Venezia",
    "RestaurantId": "40367540",
    "Cuisine": "Italian",
    "Borough": "Queens",
    "Address": {
      "Building": "42-01",
      "Coordinates": [ -73.911784, 40.764766 ],
      "Street": "28 Avenue",
      "ZipCode": "11103"
    }
  },
  {
    "Id": "5eb3d668b31de5d588f429b8",
    "Name": "Don Peppe",
    "RestaurantId": "40366230",
    "Cuisine": "Italian",
    "Borough": "Queens",
    "Address": {
      "Building": "13558",
      "Coordinates": [ -73.8216767, 40.6689548 ],
      "Street": "Lefferts Boulevard",
      "ZipCode": "11420"
    }
  },
  {
    "Id": "5eb3d668b31de5d588f429cc",
    "Name": "Cara Mia",
    "RestaurantId": "40366812",
    "Cuisine": "Italian",
    "Borough": "Queens",
    "Address": {
      "Building": "220-20",
      "Coordinates": [ -73.7429218, 40.7305714 ],
      "Street": "Hillside Avenue",
      "ZipCode": "11427"
    }
  },
  {
    "Id": "5eb3d668b31de5d588f42a7e",
    "Name": "Aunt Bella'S Rest Of Little Neck",
    "RestaurantId": "40371807",
    "Cuisine": "Italian",
    "Borough": "Queens",
    "Address": {
      "Building": "4619",
      "Coordinates": [ -73.7363139, 40.767005 ],
      "Street": "Marathon Parkway",
      "ZipCode": "11362"
    }
  },
  .
  .
  .
}

If we only want to see selected fields, we can use the select filter as follows http://localhost:5279/odata/Restaurants?$select=Name



 json

{

"@odata.context": "http://localhost:5279/odata/$metadata#Restaurants(Name)",

"value": [

  {

    "Name": "Brunos On The Boulevard"

  },

  {

    "Name": "Taste The Tropics Ice Cream"

  },

  {

    "Name": "C & C Catering Service"

  },

  {

    "Name": "Carvel Ice Cream"

  },

  .

  .

  .

}

Conclusion

The full list of queries supported by OData can be seen here. For more query options, you can refer here.

The OData protocol makes it simple to power your REST APIs but it may have some disadvantages like lack of control over what user's are trying to access and difficulties in optimizing specific queries which should be considered before deciding whether that suits your architectural needs.

We were able to quickly get up and running with a project that uses the MongoDB extension for OData with MongoDB Atlas. You can use the OData provider too to power you REST based applications.

Are you using odata with MongoDB? We want to hear from you! please look me up on linkedin and reach out!

This article contributed by Rishit Bhatia and Rachelle Palmer from MongoDB

Fast track your FastAPI implementation and eliminate boilerplate

rachelle palmer — Wed, 14 Feb 2024 17:03:56 +0000

ICYMI: MongoDB has released a new Full Stack FastAPI, React, MongoDB (FARM) base application generator for Python webdev.

It has always been easy and quick to start building modern web applications with MongoDB and FastAPI. But we still had to make many decisions about other parts of the stack, such as authentication, testing, integration etc., and manually integrate these components.

The MongoDB FastAPI app generator simplifies this process further via:

Docker Compose Integration
Built-in Authentication System: Includes user management schemas, models, CRUD, and APIs, with OAuth2 JWT token support and magic link authentication.
MongoDB ODMantic for ODM creation
Next.js/React Frontend with middleware authorization for page access control, Form validation using React Hook Form, State management with Redux, and CSS and templates with TailwindCSS, HeroIcons, and HeadlessUI
Operational and Monitoring Tools: Includes Celery for task management, Flower for job monitoring, Traefik for seamless load balancing and HTTPS certificate automation, and Github Actions for comprehensive front-end and back-end testing.

Go from zero to fullstack FastAPI application in less than ten minutes.

Here is a link to the github repo so you can check it out!

Best Database to use with C#

rachelle palmer — Tue, 03 Jan 2023 13:03:01 +0000

One of the biggest questions for any developer is choosing what database to use. And if you google this term, you get mostly a lot of of outdated results -- posts from 2016 on stackoverflow, links to random blogs that haven't been updated in a decade, etc. And even though we might have a resource like db-engines to tell us the most popular databases, it doesn't tell me which databases C# developers actually use.

I decided to take the StackOverflow survey raw results, filter them for C# developers only, and then do some calculations on which databases are used extensively by C# developers. Voila!

Out of the 83k SO survey responses of 2021, 22,984 were developers who know C#. This is a fairly large sample, so let's proceed.

What databases are used with C#?

An astounding 55.49% of C# developers use Microsoft SQL Server. This is a gigantic number considering the number of potential choices. Microsoft SQL server is far more popular with C# developers than it is with anyone else - just 22% of all StackOverflow developers use it overall.
42.93% of C# developers use mySQL extensively, which is not that surprising
32% use sqlite.

What database do C# developers want to use?

Here's where the data gets really interesting. When you compare the number of developers using a database vs what database they say they want to use, that's where we see the difference between preference/desire ("want") vs what our jobs insist upon. This is actually true for a lot of databases, where we see the number of developers are using something is more than the number of developers who say they want to. For example:

Even though 55% of C# developers use Microsoft SQL server, just 35% say they want to use it. 35% is still a very large number, but that's a fairly significant loss.
29% of developers use PostgreSQL, and 28% want to use it (imo, a trivial difference)

We can also see there are some databases where the usage lags than the number of developers who want to use it:

MongoDB (25.34%)
Redis (20.53%)
Elastic (14.08%)

Pretty surprising results.

What databases are not used with C#?

DynamoDB, Couchbase, Cassandra, DB2 all fared very poorly with C# developers. Cassandra in particular had just 501 users, a market share of 2%. Couchbase was even worse, with 337 (1.47% of C# developers).

Building my first Go App with Github Copilot

rachelle palmer — Sat, 24 Dec 2022 03:31:48 +0000

It's been just a few months since Github made Copilot, and yet the new Github Copilot extension for VS Code already has more than 2 million downloads. This is a huge amount of people, and I wanted to see if the hype is real. I decided to try and build a few apps with a programming language that I don't know - Golang - and see how successful it might be.

My Ground Rules

Don't read a lot of Go docs. If I'm the one writing most of the code, or spending time in the docs, that would be a fail
Don't try anything TOO fancy... but don't go easy on Copilot either. This is not a marketing puff piece. I really want to know what's possible.

Experiment 1: Hello World

After installing go, the VS Code go extension, and the Github Copilot extension, I created a new file, skimmed one tutorial, and typed
package main
import
Github Copilot then suggested fmt
Github then suggested func main()
once I accepted this suggestion, GC replied with
{fmt.PrintIn("Hello World!") }
Verdict: Hell yea, this works!

Experiment 2: Build Your Own Container

For something a little more difficult, I turned to this Build Your Own Container tutorial. I typed
package main
import(
and Copilot suggested fmt.
After accepting this suggestion, I typed os, and Copilot suggested os/exec.
Then, I typed func... and Copilot veered off course with this:
main() { binary, lookErr := exec.LookPath("ls") if lookErr != nil { panic(lookErr) } args := []string{"ls", "-a", "-l", "-h"} env := os.Environ() execErr := syscall.Exec(binary, args, env) if execErr != nil { panic(execErr) }

Verdict: No.

I typed switch os.Args[1]{
and from there Copilot helpfully offers:
case "run": parent() case "child": child() default: panic("what?") }

And we're back on track!

I continue on with this tutorial and the verdict is that although Copilot sometimes offers wrong suggestions for what I want, the code itself is great. If I give it just a tiny hit, it can give me entire blocks of code that are more than 90% accurate.

Final code matches the tutorial almost exactly.

Verdict:...

Ultimately I was pretty impressed by what is possible. I still don't know golang and I'm unlikely to learn, but I did feel like I'd be capable of building an application within a matter of weeks with something like Gopilot as a sort of 'uber linter`.

Where to Learn MongoDB in 2023

rachelle palmer — Mon, 19 Dec 2022 15:13:23 +0000

If you ask OpenAI where to learn MongoDB, you get a pretty dull answer:

MongoDB offers a variety of online and offline resources to help you learn MongoDB, including official documentation, tutorials, and the MongoDB University course. Additional learning resources include the MongoDB blog and community resources like MongoDB Atlas and Stack Overflow.

It's ok but not the best response - MongoDB Atlas is a platform, not a community resource - so I'm giving you our list of where we recommend learning MongoDB.

1. YouTube

Whether you want to learn MongoDB in 30 Minutes, Learn MERN stack in an hour, or take on Full Stack Web Development with freecodecamp, or a course to learn Python in Hindi, there's pretty much something for everyone.

2. MongoDB University

Just relaunched in November, the new MongoDB University available at learn.mongodb.com completely revised its beginner level content and associate level certifications.

3. Udemy

Do you just prefer to pay for learning? Udemy has great courses for web development for every programming language. My personal favorite is the MERN stack front to back course, which is discounted right now to just $15.99 from $84.

4. MongoDB Developer Center

If you prefer to learn from step by step tutorials and code examples, the MongoDB Developer Center offers articles, quick starts, github repositories for every programming language. As an example, here's the aggregated page for Next.JS tutorials.

5. Official Documentation

And, of course, the official MongoDB documentation will always prove useful. In particular, users find the MongoDB guides helpful.

Photo by Nick Morrison on Unsplash

Best Free Courses to Learn WebDev for Beginners

rachelle palmer — Tue, 20 Sep 2022 19:57:29 +0000

If you are interested in learning web development and looking for some awesome resources online, then this post is for you.

There are so many tutorials available, it can be hard to choose which one is 'best'. I'm usually interested in doing a real project, lots of live coding, and very clear instructions. This is because when I'm trying to code, I tend to need to pause or rewind the video.

Below are some of my personal favorites:

1) Build a REST API with Node.js, Express, TypeScript, and MongoDB

2)Freecodecamp MERN stack tutorial

3) Build a Tindr Clone Full Stack Application with NodeJS and React

4) Youtube - Learn the MERN Stack in 60 Minutes

5) Build a Fullstack SvelteKit ToDo App with TailWind CSS and MongoDB

6) Serverless Vercel Svelte Project

These should be more than enough to get you started, let me know your favorite courses in the comments!

Photo by Kumiko SHIMIZU on Unsplash

Great MongoDB Resources for Python Developers

rachelle palmer — Tue, 31 May 2022 18:34:48 +0000

If you're new to MongoDB, or new to Python, it may seem overwhelming just how many resources are available to look at. Here are some of the most loved resources, tools, or tutorials that are guaranteed to get you started quickly with using MongoDB and Python.

Eve - the Python Rest API to use with MongoDB

Eve is an open source Python REST API framework powered by Flask and Cerberus. Eve offers native support for using MongoDB with python and has an easy quickstart here.

A FastAPI and MongoDB example project

If you don't want a wordy read through but really just need to look at some code, check out this github project. This is a simple CRUD application built using the MongoDB Python driver (PyMongo) and FastAPI. You can also follow the step-by-step tutorial for building this application.

pymongoarrow - a library to import/export data between MongoDB and Pandas dataframe, numpy array, parquet file, csv

pymongoarrow takes less than 15 minutes to install, understand, and use. Way better than sifting through 4 years of stack overflow posts and trying to find a correct example of how to do this.

Tech with Tim MongoDB Python Video Tutorials

This short series on youtube builds a full MongoDB Python application - video 1 is about beginner level CRUD. The second video features Schema Validation, Bulk Inserts, Data Modeling, and Advanced Queries, and the third video demonstrates full text search of MongoDB data.

The MongoDB Developer Center

Last but not least, we have our MongoDB developer center, where we feature the Python programming language and have news, tutorials, articles of interest, etc.

Prisma now offers MongoDB Support!

rachelle palmer — Mon, 25 Apr 2022 15:10:23 +0000

In very exciting news, Prisma.io recently announced the general availability of support for MongoDB. For those of you who are unfamiliar, Prisma is a next-generation objection-relational mapper (ORM) that makes development of web applications with Node.js and Typescript lightning fast. It’s worth noting that Prisma is the only fully type-safe ORM in the TypeScript ecosystem. The generated Prisma client ensures typed query results even for partial queries and relations. Tinkering with it was super easy - it’s very developer friendly and a welcome abstraction layer for error-prone querying.

How does it work?

Every project starts with a Prisma schema file. The schema file allows developers to define application models in an intuitive data modeling language. This helps prevent data inconsistencies and bloated data models. It also contains the connection to your MongoDB database and defines a generator (‘the prisma client’). The prisma client is an auto-generated and type-safe query builder for TypeScript (and Node.js). With autocompletion right in Prisma’s code editor, developers can more easily build queries of their MongoDB data without needing to look up reference documentation.

How to get started?

Despite Prisma support for MongoDB being fairly new, there are already a plethora of resources available on this topic, so I thought I would condense those into one quick guide. To get started, you’ll need to complete the following steps:

Note if you want to explore your data you can do that in the Prisma Studio visual editor, or you can use MongoDB Compass, or you can use the data explorer available in the Atlas UI, or you can use the MongoDB VS Code Extension... there are a lot of options here.

How does Prisma differ from Mongoose?

Mongoose is another ORM (or ODM, depending on who you ask…) for MongoDB. How are they different? Which one is the better fit? Here’s a simple comparison of various queries with each of them.

Most developers should find Prisma’s minimal API a welcome change from overly complex competitors in this space - certainly most of the features advertised, the average developer doesn’t really need. I’ve now built a couple of quick applications with Prisma and Node.js and I find it very intuitive and easy to get something up and running. As with all ORMs it is an abstraction layer, so you do sacrifice some flexibility in the name of usability and productivity - but I didn’t hit any of those alleged limitations and I’m not convinced most people would, even when building a more robust application.

We hope you’ll give it a try (for science!) and let us know your thoughts. We also have a whole week of activities with Prisma ahead where you can learn more about using Prisma with MongoDB - check out the launch website to learn more and join us!

You can also check out Prisma on github.

.Net Analyzer for MongoDB

rachelle palmer — Mon, 07 Feb 2022 18:42:29 +0000

C# and .NET developers know that it can sometimes be frustrating to work idiomatically with MongoDB queries and aggregations: without a way to see if your LINQ query or Builder expression corresponds to the MongoDB Query API during development, we had to wait for runtime errors in order to troubleshoot queries.

Not anymore.

With the new MongoDB Analyzer for .NET, instead of mentally mapping the idiomatic version of a query in C# to the MongoDB Query API, the MongoDB Analyzer does it for you - and even provides the generated Query API expression right in your IDE. The MongoDB Analyzer surfaces useful hints and helpful warnings on invalid expressions at compile time, bringing greater visibility to the root causes of bugs. When used together with the recently released LINQ3 provider (supported in MongoDB C#/.NET Driver 2.14.0 and higher), you can compose and understand more advanced queries easily, including aggregations. The MongoDB Analyzer can also be used from the CLI to integrate into CI/CD pipelines.

In Visual Studio, install the MongoDB.Analyzer NuGet package via package manager:

Install-Package MongoDB.Analyzer -Version 1.0.0

dotnet add package MongoDB.Analyzer --version 1.0.0

for the .net CLI.

Once installed, it will be added to your project’s Dependencies list, under Analyzers. Once the analyzer has run, you’ll find all of the diagnostic warnings output to the Error List panel. As you inspect your code, you’ll also see that any unsupported expressions will be highlighted.

MongoDB Analyzer surfaces any unsupported expressions and invalid queries at compile time. You’ll find all of these issues listed as warnings in the Visual Studio Error List. This is also quite useful as not all LINQ expressions are supported by the MongoDB C#/.NET driver right now. Similarly, supported expressions will differ slightly depending on which version of LINQ you use. I'm loving tinkering around with it.

Check out the docs and learn how to analyze your code here.

How Its Made: MongoDB Compass Export to Ruby Language

rachelle palmer — Mon, 24 Jan 2022 19:56:52 +0000

Looking for something truly useful for learning MongoDB? Imagine being able to construct a MongoDB query via the MongoDB Compass UI while exploring your data, then exporting that same query to the programming language of your choice.

In today's "How Its Made", MongoDB Engineer Neil Shweky explains how he recently built out this feature for Ruby.

Getting this functionality working required that Ruby be added as an output language to our BSON Transpilers project. As an aside, if you spent a lot of time in university (or just for fun) with compilers, parsers, and other fun functional-programming-related problems, this is a project for you.

Our BSON transpilers enable users to convert a MongoDB query from one programming language to another via a YAML template file that specifies how to generate code for the specific language in a variety of situations. For example, if the output language is Python and the input code is '3 in [1,2,3]' the following template is used to generate equivalent Ruby code:

InSyntaxTemplate:
(lhs, op, rhs) => {
  let str = '';
  if (op.includes('!') || op.includes('not')) {
    str = '!';
  }
  return${str}${rhs}.include?(${lhs})
}

This template takes in an lhs value of 3 and a rhs value of [1,2,3]. The equivalent Ruby code that is then generated to test for array membership would be:

[1, 2, 3].include?(3)

The first step in this process was to complete a series of tests which would evaluate a code statement from an input language and output it to Ruby. Then to take those learnings and complete the templates in the ruby template file.

Simple enough, right?

The most difficult part of this work was making decisions on what code is considered idiomatic to Ruby. The Ruby language offers so many ways to express the same concepts, however our goal was to choose the ideal way that would make sense to Ruby developers. As an example, assignment of a key/value pair to a Hash of { x: 1 } can be done multiple different ways:

{ x: 1 }
{ :x => 1 }
{ :'x' => 1 }
{ 'x': 1 }
{ 'x' => 1 }

Well...

Eagle-eyed Rubyists will note that the keys above vary between being Symbols and Strings so it's not technically "5 different ways of assigning exactly the same thing", but we chose this example to illustrate the flexibility the Ruby language offers. At the end of the day, we opted to use the last option, because we felt like this would "look right" to most Ruby developers.

Results in the Compass UI:

For future enhancements, we are thinking about adding Ruby as an input language. This would allow a user to convert queries written in Ruby to any of the other languages we have implemented such as Java, C#, Javascript, Python, with Golang and even Rust.

The next time you're exporting your query from within MongoDB Compass give Ruby a try and let us know what you think. If you're curious how all this works the repo for MongoDB Compass is available for exploration.

-- Rachelle, Product Lead for Developer Experience

P.S. Neil is also on twitter and devto