DEV Community: Nick Chapsas

CosmosDB Fluent Pagination with Cosmonaut

Nick Chapsas — Thu, 27 Sep 2018 10:41:43 +0000

Introduction

Ever wondered why the Azure Portal has scroll forward pagination and next/previous pagination but no page size + page number pagination? Go on take a wild guess. If you said "Because it runs on CosmosDB" then you're right. The Azure portal is getting it's results from CosmosDB that's why you don't see any form of server side skip and take pagination.

If you ever see a page size and page number combination in the portal then the pagination is happening on the client side. An example of that would be the "Logs" section of Application Insights, which is able to return paged results with page size and page number but those results are pre-fetched and they are just rendered on the spot. We know that because you can find all the results returned in pages of 100 in the network traffic tool of our browser of choice. The client side code will do the rest of the paging job.

CosmosDB pagination support

CosmosDB doesn't support your traditional type of skip/take pagination. Instead results are always paged with a maximum page size.

Skip take pagination is however, the most requested CosmosDB feature since August of 2014. The official ComosDB Team account replied in March of 2018 that they are planning to implement it, so we're good.

Judging based on the pace features are being delivered currently, I would not expect it to be delivered anytime soon so we need an alternative, at least for now.

So what does CosmosDB support as of today in terms of pagination?

Well, you have two values related to how results are being returned.

MaxItemCount
RequestContinuation

The MaxItemCount is the maximum nnumber of items to be returned in the enumeration operation. They can be less, but they won't exceed that number.
The RequestContinuation is the continuation token which identifies which was the last result set for this operation and points CosmosDB to the next set of data it needs to return.

This is enough functionality for someone to workaround the lack of skip take pagination. This is where Cosmonaut kicks in and does the heavy lifting for you.

Cosmonaut's implementation

Since I started working on Cosmonaut I knew that, eventually, I'd have to add fluent pagination support.

I did not want to mess with the Skip and Take LINQ methods because I know that eventually CosmosDB will support both of them (only supports Take for now).

Cosmonaut supports an IQueryable extension called WithPagination(). This method has two overloads.

WithPagination(int pageNumber, int pageSize)
WithPagination(string continuationToken, int pageSize)

As you can see, the first signature looks very much like a clear skip/take type implementation of the pagination. The second one is the direct CosmosDB approach.

How do they work?

The first signature, that has both page number and page size, can get inefficient and expensive as the page number value increases. This is because the only way this method can work is by going through all the pages from the start until it hits that requested page number and page size.

The second signature is both efficient and fast because it points straight to the next set of results. That's because you provide the continuation token.

These methods work in Cosmonaut with both LINQ and SQL queries.

How can you use them?

The first signature is as simple to use as the following.

var firstPage = await booksStore.Query().WithPagination(1, 10).OrderBy(x => x.Name).ToListAsync();
var secondPage = await booksStore.Query().WithPagination(2, 10).OrderBy(x => x.Name).ToListAsync();

As you can see, this reads very simple and I think is very straightforward. The same thing with SQL would look like this:

var firstPage = await booksStore.Query("select * from c order by c.Name asc").WithPagination(1, 10).ToListAsync();
var secondPage = await booksStore.Query("select * from c order by c.Name asc").WithPagination(2, 10).ToListAsync();

These methods work with both the CosmonautClient and the CosmosStore methods.

However, as I mentioned before, this is the method that can get inefficient as you go to higher page numbers. That's why the WithPagination signature that supports the continuation token is recommended.

"But how do I get the continuation token?"

The pagination update also introduced a new results extension method called ToPagedListAsync(). This method returns a CosmosPagedResults object. This object contains 3 very handy properties for your paged operations:

Results, which is a list of the results for your query
HasNextPage, which is a boolean indicating if there are more pages
NextPageToken, which is a string providing you the next page continuation token, if any

Let's see how this works. I'll write the same two page code that I have above but this time I'll use the continuation token approach.

var firstPage = await booksStore.Query().WithPagination(1, 10).OrderBy(x => x.Name).ToPagedListAsync();
var secondPage = await booksStore.Query().WithPagination(firstPage.NextPageToken, 10).OrderBy(x => x.Name).ToPagedListAsync();

Simple? I hope so. In a real life scenario, you would get the continuation token, store it on the client side (web app, desktop app, mobile app) and provide it for the next page.

CosmosPagedResults also support a GetNextPageAsync method which automatically gets the token and size from the last page. The same two queries would look like this:

var firstPage = await booksStore.Query().WithPagination(1, 10).OrderBy(x => x.Name).ToPagedListAsync();
var secondPage = await firstPage.GetNextPageAsync();

Note:

All of the above works with the CosmonautClient querying methods as well:

var result = await cosmonautClient.Query<Book>("localtest", "shared").WithPagination(1, 1).ToListAsync();

Pagination recommendations

Because page number and page size pagination goes though all the documents until it gets to the requested page, it can get slow and expensive. The recommended approach would be to use the page number and page size method once for the first page and get the results using the .ToPagedListAsync() method. This method will return the next continuation token and it will also tell you if there are more pages for this query. Then use the continuation token alternative of WithPagination to continue from your last query.

Keep in mind that this approach means that you have to keep state on the client for the next query, but that's what you'd do if you were using previous/next buttons anyway.

I hope this helps you make integration with CosmosDB easier. Don't forget to ask questions in the comments if something is unclear and feel free to suggest features or express ideas for Cosmonaut. Also, if you like the project then giving it a star on Github means a lot. Thank you.

Testing your CosmosDB C# code and automating it with AppVeyor

Nick Chapsas — Mon, 17 Sep 2018 13:08:29 +0000

Introduction

Don't we all love well tested code? I know I do.
It's just an awesome feeling knowing that if I change something over here, the thing over there didn't break.

However, when your code depends on a third party library, then you have to hope that the code in that library is easy to work with and easy to test.

If this isn't the first blog of mine that you read then you probably know that I'm working on an open-source ORM for CosmosDB called Cosmonaut. When I'm using a third party library and its code happens to be opensource then the first thing I do is to go to wherever it's code is hosted and check if there are tests covering it. Nobody wants to use untested code, so I knew when I started that I should cover as many scenarios and cases as possible.

Unit testing

The CosmosDB SDK is not as friendly as it could be when it comes to unit testing (and the CosmosDB SDK team knows it, so it will get better).
Normally, the way you would unit test such code would be to mock the calls that would go over the wire and just return the dataset that you want this scenario to return.

However here is a list of things that make it really hard for you to test your code.

Problem 1

The ResourceResponse class has the constructor that contains the DocumentServiceResponse parameter marked as internal. This is bad because even though you can create a ResourceReponse object from your DTO class you cannot set things like RUs consumed, response code and pretty much anything else because they are all coming from the ResourceResponseBase which also has the DocumentServiceResponse marked as internal.

Solution

To solve this problem you have to somehow set the response headers for the ResourceResponse. As you might have guessed already, then only way you can do that is via reflection. Here is an extension method that can convert your generic type object to a ResourceResponse and allows you to set the responseHeaders.

This extension method is also part of Cosmonaut

Problem 2

The SDK is using IQueryable in order to build a fluent LINQ query and use it to query your CosmosDB. This LINQ query will be then converted into SQL via an internal LINQ2CosmosDBSQL provider that the SDK comes with. The problem with that is that if you want to mock your IDocumentClient's CreateDocumentQuery to return a specific data set based on a LINQ expression then you're in for a ride.

Solution

Lets take a look at the code.

As you can see, it needs quite the setup. That's because the IQueryable needs to be setup from the ground up or else it won't be properly translated.

Problem 3

It's basically the same as Problem 2 but for SQL, which makes it slightly easier because there is no IQueryable to setup.

Solution

Almost everything in the code from "Problem 2" is the same. The only differences are between the lines 24-47. You need to replace those with the code below.

These 3 are the most common unit testing problems you will face with the CosmosDB SDK. Thankfully the other operations are easier to test, especially if you make use of the ToResourceResponse extension method.

UPDATE: As I was writing this Microsoft released the 2.0.0 version of the CosmosDB SDK packages and in them they changed the constructors of come classes.

If you are using a post 2.0.0 version of the SDK, here are the extension methods you need.

Integration testing

Unit testing is awesome, but you also want to test your system against a real database. These tests should have a wider scope than the unit-sized context of your unit tests. I won't go in depth on this one because everyone's idea of integration testing seems to be different but what you need to remember is that we are trying to test again the real CosmosDB database.

As you probably know, CosmosDB comes with it's own local emulator. That's great because we can use it to run our integration/system tests against. What's also great is that we can use the same emulator as part of our CI pipeline.

Automating unit and integration testing

I personally use AppVeyor because it is incredibly easy to setup and get started, it supports loads of stuff such as Azure services and Nuget building out of the box and it is also free for open source projects.

AppVeyor is as simple to setup as linking your Github account and selecting the repository you want it to run against. Once you set that up it will trigger a build every time you commit code. There are tons of stuff to configure if you want something more specific but in this blog I will just explain how you can setup the CosmosDB emulator in AppVeyor.

AppVeyor will look for an appveyor.yml file in your repository. If it's not there it will use the settings that can be found in the "Configuration" section. Our appveyor.yml will be pretty simple. We just want to dotnet build and dotnet test our code. Before that however we need to install the CosmosDB Emulator on the AppVeyor VM. In order to do that we will need the following powershell script:

Now the CosmosDB Emulator is installed and you can enjoy testing against it using the default CosmosDB Emulator url https://localhost:8081 and the well known key C2y6yDjf5/R+ob0N8A7Cgv30VRDJIWEHLM+4QDU5DE2nQ9nDuVTqobD4b8mGGyPMbIZnqyMsEcaGQy67XIw/Jw==.

Here is the final appveyor.yml

Once you set that up, a build will be kicked off every time you commit code in the repository and both unit and integration tests will be run against the local CosmosDB emulator instance.

Happy testing!

How to easily start using CosmosDB in your C# application in no time with Cosmonaut

Nick Chapsas — Mon, 07 May 2018 20:33:27 +0000

I've used a lot of document based NoSQL databases. CosmosDB is by far my favourite, mainly because it's much more than that. Things like it's scale-ability, the different API options, the change feed or even the emulator that Microsoft provides are amazing.

However, unless you are willing to dedicate a lot of time reading about it, the integration experience can be a nightmare for some. Things like it's pricing, it's performance, partition keys and indexing just to name a few, can be too much. On top of that you have to use the SDKs provided by Microsoft, which aren't bad at all but unless you know when to use which method, you will end up having bad performance or paying way more than what you should be.

Here's where Cosmonaut comes into play.
It is wrapper library around the SQL API of CosmosDB to allow flexible CRUD (and more) based on objects. Some might say that the SQL API SDK already does that but please keep readying and you'll see what I mean.

Installation

Cosmonaut is published on Nuget.
You can install it from the Nuget browser or the command line

Install-Package Cosmonaut
or
dotnet add package Cosmonaut

Once you add the package, integration can be as simple as adding the following line in your DI service collection.
serviceCollection.AddCosmosStore<YourObject>(cosmosSettings);

Once you do that you can get ICosmosStore<YourObject> from DI and you are ready to roll.
Alternatively, you can manually create a CosmosStore object.

CosmosStoreSettings have only 3 mandatory settings in it.

DatabaseName
AuthKey
EndpointUrl

There are more things that can be configured like the ConnectionPolicy or the IndexingPolicy but if they are not set they will default to the CosmosDB default values.

How to use

By default Cosmonaut will create/need one collection per object. However it also has logic for collection sharing between different objects. We will talk about this later.

For now all you need to know is that there is a single main restriction.

Your objects will NEED to tick one of the following checkboxes

Have a property of type string named Id
Implement the ICosmosEntity interface
Extend the CosmosEntity class
Have a property of type string with the attribute [JsonProperty("id")]

This is to ensure that your object can be stored, retrieved and updated in CosmosDB.

If you are planning to do any Select(x => x.Id) queries then you must have the [JsonProperty("id")] attribute OR extend the CosmosEntity class.

The name of the collection created by Cosmonaut (when the collection is missing) is generated in the following way. If the object has the CosmosCollection attribute then you can specify the name of the collection there. If not then a pluralized version of the object's name will be used instead. The attribute is also great if you want to add Cosmonaut in your existing CosmosDB collection.

The CosmosStore has the following methods for object manipulation:

AddAsync(TEntity entity) Adds an object in the CosmosDB collection.
UpdateAsync(TEntity entity) Updates an existing object in the CosmosDB collection.
UpsertAsync(TEntity entity) Updates an existing object in the CosmosDB collection or Adds it if it is not in the collection.
RemoveAsync(TEntity entity) Removed an object from the CosmosDB collection.

All of the above also have a Range method which allows the action to happen for a collection of items. RemoveAsync also supports expression removals based on a filter.

The operation responses also contain the ResourceResponse of the Document itself as well in order to allow for the retrieval of low level information.

When it comes to querying...

...you can simply call the .Query() method and have a IQueryable ready to use. Keep in mind that at the query level CosmosDB only supports Where, Select and SelectMany.
When you want to return the query you just built to a List or just get the first object you have two options.

You can use the LINQ method ToList() but this is a synchronous call that is not recommended. What you should do instead is to use one of the extension methods that come with Cosmonaut such as:

ToListAsync
CountAsync
FirstOrDefaultAsync
FirstAsync
SingleOrDefaultAsync
SingleAsync
MaxAsync
MinAsync

These methods will use the built int paging logic to ensure that you application doesn't get locked while Cosmonaut is retrieving documents for you.

As you can tell this gives you pretty much everything you need to get you started.

Partition Key

The partition key is one of the most important things you need to know about in CosmosDB. This blog won't explain exactly what it is and how it works but it will let you know how Cosmonaut works with it. More on partition keys here.

There are a couple of things you need to know about the partition key.

Once a collection is created without a partition key, you CANNOT add one.
Once a collection is created with a partition key, you CANNOT change it.

In non shared collections Cosmonaut will not add a partition key by default. However by using the [CosmosPartitionKey] attribute you can specify which property is your partition key property. This will be used to create the collection with the key if the collection isn't created yet.

Indexing

Indexing plays a big role when it comes to querying your document's properties.
By default a cosmosdb collection is created with the following collection rules.

{
    "indexingMode": "consistent",
    "automatic": true,
    "includedPaths": [
        {
            "path": "/*",
            "indexes": [
                {
                    "kind": "Range",
                    "dataType": "Number",
                    "precision": -1
                },
                {
                    "kind": "Hash",
                    "dataType": "String",
                    "precision": 3
                }
            ]
        }
    ],
    "excludedPaths": []
}

This is also not a blog explaining Indexing so i won't go in depth but what you need to know is that you cannot partially query strings if the kind is Hash. You can only exact match them. Cosmonaut allows you to override that at the settings level. Changing the Hash to Range would allow things like StartsWith to match the data you want.

Example: If the String datatype is Hash then exact matches like the following, cosmoStore.Query().FirstOrDefaultAsync(x => x.SomeProperty.Equals($"Nick Chapsas") will return the item if it exists in CosmosDB but cosmoStore.Query().FirstOrDefaultAsync(x => x.SomeProperty.StartsWith($"Nick Ch") will throw an error. Changing the Hash to Range will make the latter work.

However you can also override this at the query level aswell by just changing the EnableScanInQuery in FeedOptions to true.

Saving money

I get it. RU/s are scary, but don't worry, Cosmonaut is designed to take that fear away.

You see, the way CosmosDB is charging you is hourly PER collection. However if you change your RU/s in an hour for even a second then you will be charged 1 hours worth of whatever the highest RU/s for that hour was.

This can get out of hand and not every collection needs to be separated from the other. Keep in mind this is a schema-less database so why not share collections.

Well Cosmonaut has built in support for collection sharing.
All you need to do to reliably share collections without messing up your operations are two things.

Decorate your object with the SharedCosmosCollection attribute.
Implement the ISharedCosmosEntity interface.

You will also need to specify the name of the shared collection that this object will be using like that [SharedCosmosCollection("shared")].

The only compromise is that your indexing cannot be very specific because you are sharing collections.
Something that i also enabled is that if you are collection sharing then the id property will automatically become your partition key. There are two reasons that back up this choice.

You cannot add a partition key after the collection is created and it's a shame to not have at least random partition distribution.
You are not guaranteed to have any other common property between your documents.

Code

Cosmonaut is open source on Github under the MIT license.

Please consider giving it a try and reporting any issues there.
Also a star on the GitHub repo makes me feel warm inside.

Feedback is also hugely appreciated.