DEV Community: Turner Software

MiniProfiler ❤ MongoDB

James Turner — Sat, 08 Jun 2019 14:15:36 +0000

If you're familiar with .NET, you may have heard of an awesome project called MiniProfiler made by the awesome folks at Stack Overflow.

MiniProfiler / dotnet

A simple but effective mini-profiler for ASP.NET (and Core) websites

MiniProfiler for .NET (and .NET Core)

Welcome to MiniProfiler for .NET, ASP.NET, ASP.NET Core, ASP.NET MVC and generally all the combinations of those words. Documentation for MiniProfiler for .NET is in /docs, accessible via GitHub pages at: miniprofiler.com/dotnet. General information for MiniProfiler across platforms can be found at miniprofiler.com. It is part of the .NET Foundation, and operates under their code of conduct.

The current major version of MiniProfiler is v4.

Handy Links

Documentation
Samples

Building

To build the MiniProfiler solution in Visual Studio, you'll need:

Visual Studio 2019 16.3+ (or the .NET Core 3.x SDK)
The Web Compiler extension
- Note: no extension is needed if building via build.cmd or build.ps1…

View on GitHub

If you're unfamiliar, you may have gathered from the name it has something to do with profiling code - you wouldn't be wrong!

MiniProfiler isn't designed to profile every method call - it is however designed to profile the calls you specifically want to know about. These include things like database calls, your controllers or your views. You can optionally profile any other code you want through MiniProfiler's API.

MongoDB - some love it, some hate it but regardless I use it and I like it for my projects. What I don't like with MongoDB is the C# driver so as an ongoing project of mine, I have built my own wrapper for MongoDB C# driver called MongoFramework. I have written about this library previously here so I won't go into much detail besides saying it makes dealing with MongoDB similar to dealing with Entity Framework.

MongoFramework however is how MiniProfiler and MongoDB meet. You see, MiniProfiler has official packages for profiling EF6 and EF Core but doesn't actually support profiling of MongoDB queries. As I had already written a wrapper around the official MongoDB driver, I was already in a good place to extend my own integration to support profiling.

MongoFramework uses a diagnostic layer inspired by how MiniProfiler actually connects into EF Core. This diagnostic layer is per connection for MongoFramework and as an interface, can easily be swapped out for any other diagnostic tool. The diagnostic layer is invoked at every entity read/write as well as the creation of indexes.

So if you:

are building a project in .NET
are using MongoDB for persistence via MongoFramework
are using or want to use MiniProfiler

Then this might be exactly what you are looking for! Check it out on NuGet.

TurnerSoftware / MongoFramework

An "Entity Framework"-like interface for MongoDB

MongoFramework

An "Entity Framework"-like interface for MongoDB

Overview

MongoFramework tries to bring some of the nice features from Entity Framework into the world of MongoDB.

Some of the major features include:

Entity mapping for collections, IDs and properties through attributes
Indexing through attributes (including text and geospatial)
Fluent mapping builder
Entity change tracking
Changeset support (allowing for queuing multiple DB updates to run at once)
Diff-updates (only changes to an entity to be written)
Entity Buckets (clustering of small documents together, improving index performance)
Runtime type discovery (serialize and deserialize without needing to specify every "known" type)

MongoFramework is currently built on-top of the official MongoDB C# driver.

Licensing and Support

MongoFramework is licensed under the MIT license. It is free to use in personal and commercial projects.

There are support plans available that cover all active Turner Software OSS projects Support plans provide private email support, expert…

View on GitHub

Building a Polite Web Crawler

James Turner — Sat, 13 Apr 2019 13:11:04 +0000

Web crawling is the act of having a program or script accessing a website, capturing content and discovering any pages linked to from that content. On the surface it really is only performing HTTP requests and parsing HTML, both things that can be quite easily accomplished in a variety of languages and frameworks.

Web crawling is an extremely important tool for search engines or anyone wanting to perform analysis of a website. The act of crawling a site though can consume a lot of resources for the site operator depending how the site is crawled.

For example, if you crawl an 1000 page site in a few seconds, you've likely caused a not insignificant amount of server load for low-bandwidth hosting. What if you crawled a slow-loading page but your crawler didn't handle it properly, continuously re-querying the same page. What if you are just crawling pages that shouldn't be crawled. These things can lead to very upset website operators.

In a previous article, I wrote about the Robots.txt file and how that can help address these problems from the website operator's perspective. Web crawlers should (but don't have to) abide by the rules governed in that file to prevent getting blocked. In addition to the Robots.txt file, there are some other things crawlers should do to avoid being blocked.

When crawling a website on a large scale, especially for commercial purposes, it is a good idea to provide a custom User Agent, allowing website operators a chance to restrict what pages can be crawled.

Crawl frequency is another aspect you will want to to refine to allow you to crawl a site fast enough without being a performance burden. It is highly likely you will want to limit crawling to a handful of requests a second. It is also a good idea to track how long requests are taking and to start throttling the crawler to compensate for potential site load issues.

Actual footage of a server catching fire because of load, totally not from a TV Show

I spend my days programming in the world of .NET and had a need for a web crawler for a project of mine. There are some popular web crawlers already out there including Abot and DotnetSpider however for different reasons they didn't suit my needs.

I originally did have Abot setup in my project however I have been porting my project to .NET Core and it didn't support it. The library also uses a no longer support version of a library that does parsing of Robots.txt files.

With DotnetSpider, it does support .NET Core but it is designed around an entire different process of using it with message queues, model binding and built-in DB writing. These are cool features but excessive for my own needs.

I wanted a simple crawler, supporting async/await, with .NET Core support thus InfinityCrawler was born!

TurnerSoftware / InfinityCrawler

A simple but powerful web crawler library for .NET

Infinity Crawler

A simple but powerful web crawler library for .NET

Features

Obeys robots.txt (crawl delay & allow/disallow)
Obeys in-page robots rules (X-Robots-Tag header and <meta name="robots" /> tag)
Uses sitemap.xml to seed the initial crawl of the site
Built around a parallel task async/await system
Swappable request and content processors, allowing greater customisation
Auto-throttling (see below)

Licensing and Support

Infinity Crawler is licensed under the MIT license. It is free to use in personal and commercial projects.

There are support plans available that cover all active Turner Software OSS projects. Support plans provide private email support, expert usage advice for our projects, priority bug fixes and more. These support plans help fund our OSS commitments to provide better software for everyone.

Polite Crawling

The crawler is built around fast but "polite" crawling of website This is accomplished through a number of settings that allow adjustments…

View on GitHub

I'll be honest, I don't know why I called it InfinityCrawler - it sounded cool at the time so I just went with it.

This crawler is in .NET Standard and builds upon both my SitemapTools and RobotsExclusionTools libraries. It uses the Sitemap library to help seed the list of URLs it should start crawling.

It has built in support for crawl frequency including obeying frequency defined in the Robots.txt file. It can detect slow requests and auto throttle itself to avoid thrashing the website as well as detect when performance improves and return back to normal.

using InfinityCrawler;

var crawler = new Crawler();
var results = await crawler.Crawl(siteUri, new CrawlSettings
{
    UserAgent = "Your Awesome Crawler User Agent Here"
});

InfinityCrawler, while available for use in any .NET project, it still is in its early stages. I am happy with its core functionality but likely will go through a few stages of restructure as well as expanding on the testing.

I am personally pretty proud of how I implemented the async/await part but would love to talk to anyone that is an expert in this area with .NET to check my implementation and give pointers on how to improve it.

Building a flat website with ASP.NET Core

James Turner — Fri, 22 Feb 2019 01:52:49 +0000

As of last night, I launched the website for BrandVantage, my product I am building. I'm pretty proud of the site, I designed and built it myself as well as designed the logo. This all happened in just over one week since I left my job too!

I touched on a little what BrandVantage was about in a comment but this, and various other posts coming soon, are to talk about development and technologies, not me trying to sell you on it. When I do launch on Product Hunt, I will probably post about that though! 🙂

This new site, as well as my personal site and my company's website (both also launched not too long ago) are all built very similarly: a single-page, flat HTML with minified CSS.

I like having a flat site like this as it is hosting agnostic. These sites I have built, while are ASP.NET Core projects in code, they are deployed to PHP hosting. The rest of this post explains that setup a little.

The Base Project

I'm using Visual Studio with each of these projects starting out the same, an empty ASP.NET Core website project.

For getting the actual project working in development, you will need to update your Startup.cs to be something similar to below:



public class Startup
{
    public void Configure(IApplicationBuilder app, IHostingEnvironment env)
    {
        if (env.IsDevelopment())
        {
            app.UseDeveloperExceptionPage();
            app.UseBrowserLink();
        }

        app.UseDefaultFiles(new DefaultFilesOptions
        {
            DefaultFileNames = new List<string> { "index.html" }
        });

        app.UseStaticFiles();
    }
}

You want to make sure you have UseStaticFiles and set your index file to be a default otherwise it won't work as you expect.

Site Root

This setup really is taking it back to basics, I'm not using any special features of ASP.NET Core itself, there isn't some Razor template I am exporting to flat HTML. It really is just one index.html file that I've manually built.

I did look into trying to export a site from CSHTML into plain HTML but it seemed a little too complicated for a single page site. I might revisit this in the future but for now, a flat HTML page is perfect for me.

Bundling and Minification

My setup relies on the Nuget package BuildBundlerMinifier which allow the minification of CSS by just simply building the codebase as you would any other .NET codebase.

It uses a JSON file which for me looks like:



[
  {
    "outputFileName": "wwwroot/bundles/site.min.css",
    "inputFiles": [
      "wwwroot/css/normalize.css",
      "wwwroot/css/layout.css",
      "wwwroot/css/product-hunt.css",
      "wwwroot/css/responsive.css",
      "wwwroot/css/compatibility.css"
    ],
    "sourceMap": true
  },
  {
    "outputFileName": "wwwroot/fonts/font-awesome/bundles/fontawesome.min.css",
    "inputFiles": [
      "wwwroot/fonts/font-awesome/css/fontawesome.min.css",
      "wwwroot/fonts/font-awesome/css/light.min.css"
    ]
  }
]

The bundler/minifier does support JavaScript but I don't have any JS in my site besides Google Analytics and a Product Hunt script.

You can see in my site root screenshot that there is a bundles folder, that is created by this process however I have that excluded from Git.

Because my site is flat HTML, I do need to reference the bundled files directly which I do as shown here:



    <link href="/bundles/site.min.css" rel="stylesheet" />
</head>
<body>
    <link href="fonts/font-awesome/bundles/fontawesome.min.css" rel="stylesheet" />

The reason for one being in the head and one being in the body is slightly increasing the perception of a faster page load.

One downside to this entire flat HTML method is that these bundled files aren't regenerated on the fly. This means I need to build again every time I want to view a CSS change. For major works, I might manually revert back to non-minified CSS just to make development easier. Ideally though, you would have it auto-generate on modification - this is something that many other tools provide but I don't (yet) have in my setup.

If I were to use CSHTML, I could actually reference both and use tag helpers to define what environment I am running in. Again, something I might consider for the future.

Continuous Delivery

You might have noticed in that first screenshot there is a cheeky build-pipeline.yml file, this is used by Azure DevOps Pipelines to allow me to automate my build and deployment setup.

My file looks like this:



resources:
- repo: self
queue:
  name: Hosted VS2017

steps:
- task: DotNetCoreInstaller@0
  inputs:
    version: '2.2.101'

- task: DotNetCoreCLI@2
  displayName: 'dotnet build'

- task: ArchiveFiles@2
  displayName: 'Archive src/Website/wwwroot'
  inputs:
    rootFolderOrFile: src/Website/wwwroot
    includeRootFolder: false
    archiveType: 7z
    archiveFile: '$(Build.ArtifactStagingDirectory)/website.7z'

- task: PublishBuildArtifacts@1
  displayName: 'Publish Artifact: Web'
  inputs:
    ArtifactName: Web

I'm no expert on Azure DevOps Pipelines but the above is what I have working for me. First it installs a specific version of .NET Core, just so I have a stable and consistent environment (as .NET Core is already installed). I then run a build which really is just running the minification process. The next two tasks are required for getting the data to package up the build for the deployment pipeline.

At the time of making this setup, Azure didn't support a full YML config for the deployment side of the pipeline so I can't show that to you. The short version of what it does is:

Waits for a new build package coming from the master branch (allowing me to use other branches for dev without fear of deploying them)
Unpacks the artifact
Using FTPS, deploys all the changes to my hosting

Conclusion

Maybe not the greatest or most productive setup but one that works pretty well for me. What I can strongly say though is how much I love automated deployments - that stuff is amazing!

Sitemaps 101

James Turner — Sun, 27 Jan 2019 04:04:19 +0000

Our ability to search online for various things and get relevant results is quite a technical achievement, especially at the scale that search engines need to work at. They need to build large indexes of websites and content so they can process are queries and bring us to the content we are after.

I previously talked about the "Robots.txt" file and the relationship between web crawlers and the website operator. That is one piece of the puzzle where it helps both the web crawler and the website operator communicate about what shouldn't be indexed on a site.

Another piece of the puzzle are sitemaps, a file that helps you tell web crawlers and search engines about all the pages on your site, when they were last updated and the frequency of updates. This information isn't possible by just crawling pages by links in content.

A Brief History

Maybe not so surprisingly we have Google to thank about starting the concept of sitemap files back in mid 2005.

In November 2006, Yahoo and Microsoft joined Google in support of the standard with the schema "Sitemap 0.9".

Not long after that, they jointly announced support for a non-standard feature on "Robots.txt" files, allowing them to point to where sitemaps for a website can be located.

For example, here is dev.to's robots file pointing to the location of the sitemap:

# See http://www.robotstxt.org/robotstxt.html for documentation on how to use the robots.txt file
#
# To ban all spiders from the entire site uncomment the next two lines:
# User-agent: *
# Disallow: /

Sitemap: https://thepracticaldev.s3.amazonaws.com/sitemaps/sitemap.xml.gz

The Format

There are 3 flavours of sitemaps: XML, TXT and RSS

XML Sitemaps

These are most likely the only form of sitemap you will ever actually work with and what is the core format defined in the specification. That said, not all XML sitemaps are the same as there are two different types.

Normal Sitemap File

You have a number of <url> tags which all must have a <loc> tag but optionally the <lastmod>, <changefreq> and <priority> tags.

The <loc> tag is simply the absolute URL of the page on your site.

The <lastmod> tag helps indicate the "freshness" of a page. While a crawler might prioritise based on this value, I wouldn't recommend constantly updating the last modified to the current date to try and game the system.

The <changefreq> tag is only a guideline for crawlers, don't think setting it to "hourly" will make web crawlers instantly crawl your site more often.

The <priority> tag isn't for defining how important that page is compared to other websites but how important it is for the web crawler to even crawl that page. This has a default value of "0.5" when not set.

Example XML Sitemap from the specification:

<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
   <url>
      <loc>http://www.example.com/</loc>
      <lastmod>2005-01-02</lastmod>
      <changefreq>monthly</changefreq>
      <priority>0.8</priority>
   </url>
   <url>
      <loc>http://www.example.com/catalog?item=12&amp;desc=vacation_hawaii</loc>
      <changefreq>weekly</changefreq>
   </url>
   <url>
      <loc>http://www.example.com/catalog?item=73&amp;desc=vacation_new_zealand</loc>
      <lastmod>2004-12-23</lastmod>
      <changefreq>weekly</changefreq>
   </url>
   <url>
      <loc>http://www.example.com/catalog?item=74&amp;desc=vacation_newfoundland</loc>
      <lastmod>2004-12-23T18:00:15+00:00</lastmod>
      <priority>0.3</priority>
   </url>
   <url>
      <loc>http://www.example.com/catalog?item=83&amp;desc=vacation_usa</loc>
      <lastmod>2004-11-23</lastmod>
   </url>
</urlset>

Sitemap Index File

According to the standard, a normal sitemap file is limited to 50,000 URLs and a maximum size of 50MB. While I don't necessarily believe that type of limitation is still enforced, it did give to the rise of the a sitemap index file.

These files basically look a lot like a normal sitemap file but basically just point to other sitemaps. You have a number of <sitemap> tags which contain a required <loc> tag and an optional <lastmod> tag.

Example Index sitemap from the specification:

<?xml version="1.0" encoding="UTF-8"?>
<sitemapindex xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <sitemap>
    <loc>http://www.example.com/sitemap1.xml.gz</loc>
    <lastmod>2004-10-01T18:23:17+00:00</lastmod>
  </sitemap>
  <sitemap>
    <loc>http://www.example.com/sitemap2.xml.gz</loc>
    <lastmod>2004-01-01</lastmod>
  </sitemap>
</sitemapindex>

TXT Sitemaps

This type of sitemap really removes a lot of the functionality that you would find in an XML sitemap like the last modified date or how frequently a page is updated.

This format is just having each URL you want indexed on a new line with absolutely no other data.

http://www.example.com/
http://www.example.com/catalog?item=12&amp;desc=vacation_hawaii
http://www.example.com/catalog?item=73&amp;desc=vacation_new_zealand

RSS Sitemaps

While not as limited as TXT Sitemaps, RSS sitemaps have their own issues like only providing information on recent URLs.

You would use the <link> tag to define the URL you want indexed and and <pubDate> to define when it was last modified.

The future of Sitemaps

The main specification for sitemaps hasn't changed but there are some additional types of sitemaps being developed like video sitemaps, image sitemaps and special Google News sitemaps.

Google have also announced support for multilingual sitemaps where one can define the language each URL.

The support around these additional sitemap types isn't as widespread as the main XML sitemap though that may change in the future.

I wrote a thing...

My last few articles have really come around because of my work building libraries and tools that are solving problems for me and this one is no exception.

TurnerSoftware / SitemapTools

A sitemap (sitemap.xml) querying and parsing library for .NET

Sitemap Tools

A sitemap (sitemap.xml) querying and parsing library for .NET

Key features

Parses both XML sitemaps and sitemap index files
Handles GZ-compressed XML sitemaps
Supports TXT sitemaps

Licensing and Support

Sitemap Tools is licensed under the MIT license. It is free to use in personal and commercial projects.

Notes

Does not enforce sitemap standards as described at sitemaps.org
Does not validate the sitemaps
Does not support RSS sitemaps

Example

using TurnerSoftware.SitemapTools;

var sitemapQuery = new SitemapQuery();
var sitemapEntries = await sitemapQuery.GetAllSitemapsForDomainAsync("example.org");

View on GitHub

I had a need to actually parse sitemap files for a project I am working on and struggled to find any existing .NET library to do so. The latest version of my library builds upon my own "Robots.txt" parsing library (for sitemap file discovery) and supports XML sitemaps (both normal and index files) as well as TXT sitemaps.

This library and my "Robots.txt" parsing library actually build toward a third library which I will be writing an article about in the future.

More Information

sitemaps.org: The official site for the format
"Sitemaps" on Wikipedia: Covers additional details about sitemaps and any extended functionality.

No Robots Allowed

James Turner — Mon, 07 Jan 2019 01:54:36 +0000

You probably have heard about web crawlers/spiders/bots etc, generally in the context of a search engine indexing a site to appear in its search results.

This relationship between a search engine and a website operator is a delicate one. The website operator wants traffic to come to their site when people are searching for related phrases. The search engine wants to index the site so that it can get people to the most relevant content.

Website operators however do not like it when the crawler is hitting the site so hard that it is taken down nor do they like it when pages they didn't want displayed are up in search results.

A website operator has a powerful tool in their arsenal: They can just block the crawler from scanning the site at all. I mean, if their site was going down often because of being crawled too heavily or was crawling pages that they REALLY didn't want indexed, that is their only choice, right?

But what if it wasn't...

Welcome to the ring, robots.txt

The original specification for the "robots.txt" file was formed in 1994 with the aim to facilitate some level of control between website operators and the web crawlers.

It can look something like this:

# robots.txt for http://www.example.com/

User-agent: *
Disallow: /cyberworld/map/ # This is an infinite virtual URL space
Disallow: /tmp/ # these will soon disappear
Disallow: /foo.html

It is a fairly simple format, define a user-agent (or * for any) you want the following rules to apply to and add rules for disallowing particular paths. The file also supports comments after a # symbol.

This specification has been expanded on in later years like in the NoRobots RFC to include Allow rules and multiple user-agents per block.

While no official documentation on it, various web crawlers support wildcard paths, using the $ to match to the end of the path, support for Crawl-delay and support for specifying sitemaps (via Sitemap).

For example, here is dev.to's robots file:

# See http://www.robotstxt.org/robotstxt.html for documentation on how to use the robots.txt file
#
# To ban all spiders from the entire site uncomment the next two lines:
# User-agent: *
# Disallow: /

Sitemap: https://thepracticaldev.s3.amazonaws.com/sitemaps/sitemap.xml.gz

With the lack of specific disallow rules, this indicates that web crawlers can crawl any page they find.

So the next question: Is having a "robots.txt" file a guarantee that all web crawlers will behave?

No (sorry)

While true, there is no guarantee a web crawler will actually obey the robots file, it still is in their best interest otherwise they might end up being blocked.

The various big search engines will obey the rules because they need to, again their job is to get relevant content. It is the random other web crawlers people are writing in applications where you need to watch out for.

I am one of those people writing a web crawler and wanted to properly respect the websites I am crawling. While others have written libraries that can already do this, I wanted a better solution than what I found available.

My Library

TurnerSoftware / RobotsExclusionTools

A "robots.txt" parsing and querying library in C#

Robots Exclusion Tools

A "robots.txt" parsing and querying library in C#, closely following the NoRobots RFC and other details on robotstxt.org.

Features

Load Robots by string, by URI (Async) or by streams (Async)
Supports multiple user-agents and "*"
Supports Allow and Disallow
Supports Crawl-delay entries
Supports Sitemap entries
Supports wildcard paths (*) as well as must-end-with declarations ($)
Built-in "robots.txt" tokenization system (allowing extension to support other custom fields)
Built-in "robots.txt" validator (allowing to validate a tokenized file)
Dedicated parser for the data from <meta name="robots" /> tag and the X-Robots-Tag header

NoRobots RFC Compatibility

This library attempts to stick closely to the rules defined in the RFC document, including:

Global/any user-agent when none is explicitly defined (Section 3.2.1 of RFC)
Field names (eg. "User-agent") are character restricted (Section 3.3)
Allow/disallow rules are performed by order-of-occurence (Section 3.2.2)
Loading by URI applies default rules based on access to "robots.txt"…

View on GitHub

With NRobots being "an unofficial and unsupported fork" for robots file parsing, I wrote my own from scratch targeting .NET Standard 2.0. It supports all of the previously described rules while allowing flexibility to be extended later.

I wrote a custom tokenizer based on Jack Vanlightly's "Simple Tokenizer" article which is the core of my library. I wrote a validation layer on top of it to check the token patterns to make sure they adhere to the NoRobots RFC.

I do probably have a bit of the Not-Invented-Here syndrome but I think this library is a genuine step forward for anyone needing to parse robots files in .NET.

In a future post, I will go into how I use this library in two other libraries I have written.

More Information

robotstxt.org: The most official information for the file format can be found here
"Robot Exclusion standard" on Wikipedia: Covers more of the non-standard directives like addition wildcards, crawl-delay and sitemaps.

Mongo what now?

James Turner — Sat, 25 Aug 2018 15:17:42 +0000

Have you ever had your main side project spin out a side project of its own? I don't know how often this happens with others but it is seems to be a reoccurring theme with my projects.

Before I explain what this side side project is, let me explain a little how I got here.

The original side project

For the last number of years I've been working on building my own analytics tools to help manage and analyse brands. This itself deserves its own post at some point in the future when I eventually launch it but for now, the important part is to know a little about how development progressed with it.

My original concept for this was to be a PHP/MySQL site based around a little-known framework called SilverStripe. Should have been easy to build as it is one of the main technology stacks I use to build websites for clients. The further I went into development though, the further it didn't feel like the right technology for the job.

So what was PHP/MySQL changed to C#/MySQL - not exactly the most natural duo. Not long after that, for similar reasons as before, I changed again to C#/SQL.

Because I must just enjoy rewriting code, I changed again - this time, now C#/MongoDB.

These changes weren't just for kicks - it was the build up of little things along the way that made me want to do each of these changes. With the side project now using C# and MongoDB, this brings us to today.

C# + MongoDB = MongoDB C# Driver?

My experience with C# and databases have primarily been around the good ol' Linq2SQL interface which for the most part has done what I've needed to without issues. That said when I was working with C#/SQL for that original side project, I came to love Entity Framework - particularly the "Code First" experience and how you interact with collections.

An example of an EF "Code First" approach:

public class Person
{
    public int Id { get; set; }
    public string Name { get; set; }
}

public class MyContext : DbContext 
{
    public MyContext() : base() { }
    public DbSet<Person> People { get; set; }
}

...

using (var ctx = new MyContext())
{
    var entity = new Person() { Name = "James" };
    ctx.People.Add(entity);
    ctx.SaveChanges();           
}

MongoDB has a fully-featured C# driver and can probably do anything you want with the database. To me though, its downside is the API interface to it. In comparison to above, interfacing with the driver looks more like this:

public class Person
{
    public ObjectId Id { get; set; }
    public string Name { get; set; }
}

...

var client = new MongoClient("mongodb://localhost:27017");
var database = client.GetDatabase("foo");
var collection = database.GetCollection<Person>("bar");

collection.InsertOne(new Person { Name = "James" });

Now that is not an inherently bad API or anything, it just wasn't how I liked dealing with databases. What I really wanted was that Entity Framework interface but with a MongoDB backend.

If you're an Entity Framework Core fan, there does seem to be people working on proper MongoDB providers for it however back when I started working on this, that was not available. In addition to that though, my original side project is still stuck in .NET Framework, not Core.

So what did I do? I came up with an original name for my MongoDB "Entity Framework"-like library and got to work.

MongoFramework

Derivative is the new original

It may not be the first piece of code I have published for the world to see but it is probably the one that is most important to me right now.

What I wanted to build was a basic wrapper for the official MongoDB C# driver, making it feel like I was working with Entity Framework. Once I got the basics working, that is when I wanted to start being clever.

That MongoDB C# driver is good but there are some things that were cumbersome out-of-the-box. For example, performing document (aka. entity) updates required you to either use this builder/filter system and specify the keys and values that you updated OR you replace the whole document. Not liking either option, I set to build change tracking and support diff-updates in MongoFramework so you the developer doesn't need to worry.

Another example was how there is no "changeset" support in the official driver - the second you call InsertOne, it has sent that through to the DB. After a few iterations of changeset support in MongoFramework, it can now do a single BulkWrite call with all the changes in a particular MongoDbSet.

Still a Work-In-Progress

Like many side projects, MongoFramework is still a work-in-progress. There are many more features I want to add like GridFS, async reads and DB transaction support.

Even with MongoDB providers now available in EF Core, I think MongoFramework still has its place - if not in your next project, it has a place in my heart. ❤️

TurnerSoftware / MongoFramework

An "Entity Framework"-like interface for MongoDB

MongoFramework

An "Entity Framework"-like interface for MongoDB

Overview

MongoFramework tries to bring some of the nice features from Entity Framework into the world of MongoDB.

Some of the major features include:

Entity mapping for collections, IDs and properties through attributes
Indexing through attributes (including text and geospatial)
Entity change tracking
Changeset support (allowing for queuing multiple DB updates to run at once)
Diff-updates (only changes to an entity to be written)
Entity Buckets (clustering of small documents together, improving index performance)
Runtime type discovery (serialize and deserialize without needing to specify every "known" type)

MongoFramework is currently built on-top of the official MongoDB C# driver.

Extensions

These extensions are official packages that enhance the functionality of MongoFramework, integrating it with other systems and tools.

MongoFramework.Profiling.MiniProfiler

Supports profiling database reads and writes, pushing the data into MiniProfiler.

Documentation

Core Entity Mapping

The core mapping of entities and their properties…

View on GitHub