DEV Community: parasmm

Demystifying Inversion of Control (IoC)

parasmm — Wed, 12 Jan 2022 18:09:15 +0000

While learning about SOLID principles, specifically Dependency Inversion (D of SOLID), we come across references and confusions surrounding Inversion of control. Reading about the two did no give me straightforward answers. On top of it, looking at the sample code for the patterns referred to, for implementing Inversion of Control did not seem like inverting any control. Hence, I started checking stack overflow, blogs, et al. Writing this blog to share my understanding and collation of that information.

Before getting into details, the statements from Martin Fowler's article which gave me the right direction on differentiation on these three:

DI is about wiring,
IoC is about direction,
and DIP is about shape.

This gave a good perspective of how to look at definition and details for each of these.

Lets try to break down couple of these terms -

Dependency:

Being dependent means to require someone or something to support. Dependency in object oriented world is a relationship between objects, where, one object is dependent on other object(s) for its implementation. Think about hierarchy, inheritance, composition, etc.

Control:

Being in control means, to be able to determine the behavior or supervise the running of or maintain influence or authority over. So here we are talking about how functionality is executed or which portion of the application is taking control to execute a particular function.

Inversion:

Dictionary definition of inverted is - put upside down or in the opposite position, order, or arrangement. So in this case as well we are talking about inversion in terms of something being inverted or opposite direction. "What", we will come to in a minute.

Dependency Inversion:

The principle states:

High-level modules should not import anything from low-level modules. Both should depend on abstractions (e.g., interfaces).

Abstractions should not depend on details. Details (concrete implementations) should depend on abstractions.

Taking the above definition and dictionary references together and putting simply, we get - the direction of which object depends on which objects is inverted.

While thinking about Dependency Inversion - Think about shape of the object.

Before DIP -

You will find loads of examples online for implementing DIP. Not getting into too much details, below diagram represents before DIP state of the classes. Where high level module depends directly on low level module. Think about high level module using the function of low level module to execute its high level function successfully.

With DIP -

However, with DIP, high level module does not directly depend on the low level module. It is depending on the interface (object of the same will be provided to high level module at run time). In fact both are dependent on the interface. Thus, if we need to replace the low level module with a new one, that can be done without any impact to high level module. We can even say that High level module would not know the difference.

Inversion of Control:

Inversion of control reflects the model of relationship between a caller and callee. In classic flow of control the client has full control on environment and sequence of calls to library methods. But in case of IoC, callee takes control over some calls between callee and caller e.g. call-backs. Think about direction in flow of control. This is a common experience while extending functionality of a framework.

Quoting from Martin Fowler's blog -

The control is inverted - it calls me rather me calling the framework. This phenomenon is Inversion of Control (also known as the Hollywood Principle - "Don't call us, we'll call you").

Dependency Injection:

According to Wikipedia,dependency injection is a technique in which an object receives other objects that it depends on, called dependencies. Instead of the client specifying which service it will use, the injector tells the client what service to use.

Thus, Dependency injection is a way to implement IoC. Where the control to create the object is not with the class itself. Think about wiring up of the objects to the classes which need to use them.

Some examples of IoC:

Below are some techniques of implementing IoC. However, there were some gotchas for me which I've described here. For details of what / how of these techniques, please check those individual topics else where.

Dependency Injection: Dependency is injected at run time by DI container.The class receiving the object has not control on which actual object is created or how.

Strategy Pattern: In this pattern, the strategy / functionality / behavior can be swapped out at runtime.

As against most examples we find for implementation of strategy pattern, if we are looking at this pattern from IoC point of view then the responsibility of selection of strategy / function / behavior should be done by a different class. The client does not need to know which strategy to select. If strategy selection is done by client then it is not IoC.

Service Locator: In service locator the class requests for the object (or service) by name. If object does not exist then it would create the object and return (like factory pattern). However, if it exists then it will just return that object from cache (like registry pattern).

Template Method: It defines skeletal operations in terms of high level steps. Actual implementation of those steps is done by sub class.

Example: A abstract class called Game with below abstract methods -
Start()
Score()
End()

And a play method as -

function Play()
{
   Start();
   Score();
   End();
}

Now, sub-classes like Cricket or Soccer will implement Start(), Score() and End() functions. When client creates object of sub-class and executes play method the Start(), Score() and End() functions get called in sequence. The sequence of these calls are in turn inverted control as Cricket or Soccer classes have no control on when they get called.

Summary:

With all the people trying to simplify the understanding of Inversion of Control and Dependency Inversion Principle the only thing giving me clear direction was these words from Martin Fowler - DI is about wiring, IoC is about direction, and DIP is about shape. Additionally, patterns referred to which help in implementing IoC need to be implemented in specific way else they are true to the pattern but are not implementing IoC. Hope this article sheds some light on those items as well.

References:
https://martinfowler.com/articles/dipInTheWild.html
https://martinfowler.com/bliki/InversionOfControl.html
https://www.martinfowler.com/articles/injection.html
https://en.wikipedia.org/wiki/Inversion_of_control
https://en.wikipedia.org/wiki/Dependency_injection

Executing Prepared N1QL Queries in Couchbase using .NET Core

parasmm — Wed, 03 Nov 2021 21:27:29 +0000

While interacting with Couchbase team for some of our query tuning activities, we were suggested by them to implement Prepared N1QL Queries in our environment. While searching online, I could not find much information on how to implement and use them with .NET Core. Thought this would be a good topic to share.

What is Prepared N1QL queries in Couchbase?

Every time we issue a N1QL query in Couchbase, the query nodes need to parse the query and create a plan before the query can be executed and results returned. In an application which executes a query 100s or 1000s of times with different parameters, this plan creation for every request is an overhead. To reduce this overhead in running such frequently used queries Couchbase gives an option to use prepared queries.

One option out of the box Couchbase provides is to set a query option named adhoc to false. In this case Couchbase internally creates an identifier and saves the plan. This is in no way bad, unless we come into a scenario where we have multiple microservices or multiple pods / containers running the same query. In that scenario the same query is stored with different identifiers.

The issue here is not that the identifier is different, but there is a limit on number of prepared statements which Couchbase server can handle. Quoting from Couchbase documentation - “For Couchbase Server 6.0 and earlier, the plan is cached by the SDK (up to a limit of 5000), as well as the Query Service. On Couchbase Server 6.5 and newer, the plan is stored by the Query Service — up to an adjustable limit of 16384 plans per Query node.”

Rather than relying on the internally generated identifier, we wanted to have an option to provide the name ourselves. This way we know which queries will be run frequently in our application and we can have it parsed and ready for Couchbase to execute faster.

Comparing these to MSSQL world, a view comes to my mind.

Example using Couchabse UI:

Using the travel-sample from Couchbase installation

PREPARE AirportInfoByCity FROM 
SELECT airportname, city, country, faa  
  FROM `travel-sample` t 
WHERE type = 'airport' 
  and t.city = $City

In the query above we create a prepared query which Couchbase parses and keeps ready for subsequent execution.

To execute the same, we run EXECUTE AirportInfoByCity

The parameter will need to be set in the Runtime preferences of Couchbase UI under Named Parameters.

Looking at the execution times for these queries may not look like a lot of difference but when it comes to complex and frequent queries every millisecond counts.

Basic Stats:
I am no expert in these statistics, but on my machine running the query without prepared averaged at about 25ms and with prepared at about 12 - 13ms. Giving an average saving of about 12ms. Now, taking this a step further, if I get about 20 requests per second this is a saving of 240ms per second and 14400ms (14.4 seconds) per min. These numbers add up fast considering number of requests coming in.

Expanding this Example to .NET Core:

To start with, we’ve created an AirportController with a GetByCityPrepared function as below -

[HttpGet]
[Route("byCityPrepared/{city}")]
public async Task<IList<Airport>> GetByCityPrepared(string city)
{
    try
    {
        KeyValuePair<string, object>[] parameters = new KeyValuePair<string, object>[1];
        parameters[0] = new KeyValuePair<string, object>("$City", city);
        var queryResult = await dbHelper.ExecutePreparedQueryAsync<Airport>("AirportInfoByCity", @"select airportname, city, country, faa  
                                                                         from `travel-sample` t 
                                                                         where type = 'airport' 
                                                                         and t.city = $City",
                                                               parameters);
        return queryResult;
    }
    catch (Exception ex)
    {
        throw ex;
    }
}

Note: Airport class is scaled down version of document type of airport in travel-sample bucket. Also, dbHelper object is injected using dependency injection. Not shown for keeping things concise.

And a common DBHelper class with function ExecutePreparedQueryAsync is defined as below -

public async Task<IList<EntityType>> ExecutePreparedQueryAsync<EntityType>(string QueryName, 
                                                                string Query,
                                                                KeyValuePair<string, object>[] parameters) 
{
    var options = new QueryOptions(); 
    try 
    { 
        if(parameters !=null && parameters.Any()) 
        { 
            options.Parameter(parameters); 
        } 
        var queryResult = await _cluster.QueryAsync<EntityType>($"EXECUTE {QueryName}",  
                                        options); 

        return await queryResult.Rows.ToListAsync<EntityType>(); 
    } 
    catch (Exception ex) 
    { 
        throw ex; 
    } 
}

This example assumes that the query has already been prepared. For any reason if the prepared query gets deleted or does not exist in Couchbase the application will start failing. It is possible that the prepared got created in only one of the query nodes; and when we try to run the prepared it does not find it in that (other) query node. The error that we got was -

To overcome this scenario, we would need to have a logic in place which in case the prepared query does not exists then it would create it and then run the prepared. This would help in subsequent execution to be faster.

Let us refactor our ExecutePreparedQueryAsync by moving the execution of prepared query to a new function called TryExecutePreparedQueryAsync. Also, lets create a function BuildPreparedQuery to create prepared query in case Prepared query does not exist.

Add TryExecutePreparedQueryAsync function:

private async Task<IList<EntityType>> TryExecutePreparedQueryAsync<EntityType>(string QueryName, 
                                            QueryOptions options) 
{ 
    var queryResult = await _cluster.QueryAsync<EntityType>($"EXECUTE {QueryName}",  
                                        options); 

    if(queryResult.MetaData.Status != QueryStatus.Success) 
    { 
        HandleQueryException(queryResult.Errors); 
    } 
    return await queryResult.Rows.ToListAsync(); 
}

Add BuildPreparedQuery function:

private async Task<bool> BuildPreparedQuery(string QueryName, string Query) 
{ 
    var prepareQuery = await _cluster.QueryAsync<dynamic>($"PREPARE {QueryName} FROM {Query}"); 

    if(prepareQuery.MetaData.Status == Couchbase.Query.QueryStatus.Success) 
    { 
        return true; 
    } 
    else 
    { 
        HandleQueryException(prepareQuery.Errors); 
        return false; 
    } 
}

Note: HandleQueryException function has not been shown here for keeping it concise. Implementation for the same can be based on your requirements.

Now, in case TryExecutePreparedQueryAsync fails, call BuildPreparedQuery first, to create the prepared query and then retry execution by calling TryExecutePreparedQueryAsync. Thus we modify ExecutePreparedQueryAsync function to use above 2 functions:

public async Task<IList<EntityType>> ExecutePreparedQueryAsync<EntityType>(string QueryName, 
                                                    string Query, 
                                                    KeyValuePair<string, object>[] parameters) 
{ 
    IList<EntityType> queryResult = null; 
    var options = new QueryOptions(); 
    try 
    { 
        if(parameters !=null && parameters.Any()) 
        { 
            options.Parameter(parameters); 
        } 
        queryResult = await TryExecutePreparedQueryAsync<EntityType>(QueryName, options); 
    } 
    catch (Exception ex) 
    { 
        // Prepare the query 
        var blnPrepareQuery = await BuildPreparedQuery(QueryName, Query);  
        if(blnPrepareQuery) 
        { 
            // Execute the prepared query 
            queryResult = await TryExecutePreparedQueryAsync<EntityType>(QueryName, options); 
        } 
    }
    return queryResult;  
}

At this point, the code works all fine. Except, there is a possibility that the prepared query exists in one of the query nodes but not in other. So, if our request goes to that other query node which does not have the prepared existing, then application would throw an error. For this reason, it is suggested to first delete the existing prepared, if any, and then create a new one and execute the prepared query.

To give a pseudo code for this -

1.Try to run the Prepared query 
    a. If successful, return results 
    b. If failed, goto step 2 
2. Delete any existing prepared query 
3. Create the query as a prepared statement 
4. Execute the prepared statement and return the result

Let us add a function to delete prepared query as below -

private async Task<bool> DeletePreparedQuery(string QueryName) 
{ 
    var deletePreparedQuery = await _cluster.QueryAsync<dynamic>($"DELETE FROM system:prepareds where name = {QueryName}"); 

    if(deletePreparedQuery.MetaData.Status == Couchbase.Query.QueryStatus.Success) 
    { 
        return true; 
    } 
    else 
    { 
        HandleQueryException(deletePreparedQuery.Errors); 
        return false; 
    } 
}

Then modify the ExecutePreparedQueryAsync as below -


// if prepared does not exists then delete any existing prepared on any other query nodes 
var blnDeletePrepare = await DeletePreparedQuery(QueryName); 
if(blnDeletePrepare) 
{ 
    // Prepare the query 
    var blnPrepareQuery = await BuildPreparedQuery(QueryName, Query);  
    if(blnPrepareQuery) 
    { 
        // Execute the prepared query 
        queryResult = await TryExecutePreparedQueryAsync<EntityType>(QueryName, options); 
    } 
}

In all above functions, _cluster object has been passed through constructor using couchbase dependency injection.

Output with City as Newark:

Bonus: The error that got thrown when prepared does not exist was a bit unclear one. Digging deeper, we came to know that Couchbase 3.2 SDK by default uses a retry strategy. This retry strategy tries to run the query 10 times and in case it is still not able to process then it would throw an error. If you look at the error message, it says Too many retries: 10.

In this scenario, in retry strategy code, we can get the info that the failure is due to Query prepared statement failure. But for that we need to provide a custom implementation of retry strategy code to QueryAsync function’s options.

if(parameters !=null && parameters.Any()) 
{ 
    options.Parameter(parameters); 
} 
options.RetryStrategy(new MyRetryStrategy()); 

queryResult = await TryExecutePreparedQueryAsync<EntityType>(QueryName, options);

And add the MyRetryStrategy based on -

https://github.com/couchbase/couchbase-net-client/blob/master/src/Couchbase/Core/Retry/BestEffortRetryStrategy.cs

The link is from Couchabse .Net SDK 3.2’s master branch. The link may not work if Couchbase development team moves the file. However, the snippet I added to manage the throwing of error instead of retrying it 10 times -

public RetryAction RetryAfter(IRequest request, RetryReason reason)
{
    if (reason == RetryReason.NoRetry)
    {
        return RetryAction.Duration(null);
    }
    // added below code to modify BestEffortRetryStrategy of 
    // couchbase to stop retrying in case of QueryPreparedStatementFailure
    else if(reason == RetryReason.QueryPreparedStatementFailure)
    {
        throw new System.Exception(reason.ToString());
    }
    if (request.Idempotent || reason.AllowsNonIdempotentRetries())
    {
        var backoffDuration = _backoffCalculator.CalculateBackoff(request);
        return RetryAction.Duration(backoffDuration);
    }

    return RetryAction.Duration(null);
}

Now, in our first catch block we can check if the error message is “QueryPreparedStatementFailure”. If so, perform recreation of prepared, else handle / throw the error based on your need.

Please refer github link for complete code sample.
<13-Oct-2023> Updated github code to refer to latest nuget versions

(The code sample includes code for CRUD operation in Couchbase using .NET Core web API)

In summary, having prepared queries can save precious milli seconds while executing frequently used queries in Couchbase. And Using .Net Core 3.1 and Couchbase SDK 3.2 we can have a near foolproof prepared query execution.

Applicable to: The code has been tested with below versions of the products.

.Net Core 3.1

Couchabse SDK 3.2

Reference:

https://docs.couchbase.com/java-sdk/current/concept-docs/n1ql-query.html#prepared-statements-for-query-optimization

https://github.com/couchbase/couchbase-net-client/blob/master/src/Couchbase/Core/Retry/BestEffortRetryStrategy.cs

Couchbase CRUD with .Net Core

parasmm — Fri, 17 Sep 2021 19:26:34 +0000

Recently, I got an opportunity to work on .Net Core 9.0 with Couchbase. I wanted to share steps to create a new .Net Core Web API and implement CRUD operations with Couchbase as backend.

Pre-requisites:

Install Couchbase Community Edition 8.0.0
Install sample bucket – “travel-sample”
DotNet Core 9.0 SDK
Visual Studio Code

What is Couchbase?

Couchbase Server is an open source, distributed, JSON document database. It exposes a scale-out, key-value store with managed cache for sub-millisecond data operations, purpose-built indexers for efficient queries and a powerful query engine for executing SQL-like queries. More details can be found at their site here

Create new WebAPI project:

Execute below command to create a new .Net 9.0 project of type webapi

dotnet new webapi --name CouchbaseWebAPI

Add Package reference:

Package References and versions added for this example:

Couchbase.Extensions.DependencyInjection Version 3.8.1
CouchbaseNetClient Version 3.8.1

Run below command in the project folder -

dotnet add package Couchbase.Extensions.DependencyInjection --version 3.8.1 

dotnet add package CouchbaseNetClient --version 3.8.1 

dotnet restore

Update appsettings.json:

Update appsettings.json to include required configuration for Couchbase –

In default generated appsettings.json file you can add below configuration after the setting for AllowedHosts

  "Couchbase" : { 
    "ConnectionString": "couchbase://127.0.0.1", 
    "UseSsl": false, 
    "UserName": "<Couchbase userid>", 
    "Password": "<Couchbase password>" 
  }

Update ConfigureSevices to add Couchbase

public void ConfigureServices(IServiceCollection services) 
{ 
      services.AddControllers();
      services.AddCouchbase(Configuration.GetSection("Couchbase"));
      services.AddCouchbaseBucket<INamedBucketProvider>("travel-sample"); 
}

AddCouchbase is an extension method from “Couchbase.Extensions.DependencyInjection” which allows us to add couchbase configurations to your application. This is reading all the settings from appsettings.json which we updated in last step.

AddCouchbaseBucket is an extension method which allows us to add a particular bucket to the system. While accessing documents, application will be looking at travel-sample bucket to work with documents. This has wired up dependency injection for the application. Now, based on this in our controller class, where we want to work with travel-sample bucket we update the class. As we are about to interact with Airport documents from the travel-sample bucket, lets create a Controller class as AirportController.

private readonly IBucket _bucket;  

public AirportController(INamedBucketProvider bucketProvider) 
{
       _bucket = bucketProvider.GetBucketAsync().GetAwaiter().GetResult(); 
}

Here, in AirportController’s constructor, dependency injection is requesting for INamedBucketProvider (part of “Couchbase.Extensions.DependencyInjection”) object, which based on ConfigureServices above, gets couchbase bucket “travel-sample” assigned to _bucket which is of type IBucket (part of “Couchbase”). Now the controller is wired up to work with buckets. We can dive into individual endpoints for CRUD operations.

Let’s start with Read (CRUD): -

Read:

Create model classes named Airport and Geo:

public class Airport 
{         
    public string airportname { get; set; } 
    public string city { get; set; } 
    public string country { get; set; } 
    public string faa { get; set; } 
    public Geo geo { get; set; } 
    public string icao { get; set; } 
    public int id { get; set; } 
    public string type { get; set; } 
    public string tz { get; set; } 
} 

public class Geo 
{ 
    public double alt { get; set; } 
    public double lat { get; set; } 
    public double lon { get; set; } 
}

Add below function in AirportController for HttpGet verb

[HttpGet] 
[Route("{Id}")] 
public async Task<Airport> Get(string Id) 
{ 
      // Get default collection object 
      var collection = await _bucket.DefaultCollectionAsync(); 

      // Get single document using KV search 

      var getResult = await collection.GetAsync(Id); 
      return getResult.ContentAs<Airport>(); 
}

Run below curl command to get response from above functionality –

curl -X GET "https://localhost:5001/Airport/airport_3494" -H  "accept: text/plain"

airport_3494 is a key in the travel-sample bucket we installed. The key format in this sample bucket is type_Id, where type is the document type, here airport. And the Id is the unique identifier of the document, here 3494. The same format has been used in Create, update and delete below.

Next, let us look at Create (CRUD): -

Create:

Add an HttpPut endpoint as below

[HttpPut] 
public async Task put([FromBody]Airport airport) 
{ 
      if (airport.id != 0)
      {
           throw new Exception("Error in input data, Id should not be set!");
      } 

      // get default collection of the bucket
      var collection = await _bucket.DefaultCollectionAsync(); 

      // defaulting the id value to insert. New Id generation has different approaches which is not discussed here.
       airport.id = 1; 

       // using the collection object insert the new airport object
        await collection.InsertAsync<Airport>($"airport_{airport.id}", airport); 
}

Here, we’ve hard coded airport Id as 1 and the same value is being passed for the key name in InsertAsync call. This will throw an error (Couchbase.Core.Exceptions.KeyValue.DocumentExistsException) for subsequent calls as soon as one record is entered in the couchbase DB with Id as 1. Note: generating a new Id has couple of different approaches which is outside scope of this article.

Next up is update (CRUD): -

Update:

Add an HttpPost endpoint as below

[HttpPost] 
public async Task post([FromBody]Airport airport) 
{ 
      if (airport.id == 0) 
      {
           throw new Exception("Error in input data, Id is required!"); 
      } 

       // get default collection of the bucket
       var collection = await _bucket.DefaultCollectionAsync(); 

       // call ReplaceAsync function to save the modified version of the document 
      await collection.ReplaceAsync<Airport>($"airport_{airport.id}", airport);
}

ReplaceAsync function of collection object can be used to modify / update / replace the document. The first parameter is the KV search Key for the given document. Same as read, the key is in format type_id.

Additionally, Couchbase SDK provides us with UpsertAsync function as well, which as names suggest is for update or insert.

Lastly, let us look at Delete (CRUD): -

Delete:

Add HttpDelete endpoint to our controller

[HttpDelete] 
[Route("{Id}")] 
public async Task delete(string Id) 
{ 
    if (string.IsNullOrEmpty(Id)) 
    {     
        throw new Exception("Error in input data, Id is required!"); 
    } 

    var collection = await _bucket.DefaultCollectionAsync(); 

    // Id contains key in required k/v search. e.g. airport_1 
    await collection.RemoveAsync(Id); 
}

RemoveAsync function of collection object removes the document associated with key provided in input “Id”.

Bonus, Custom Bucket Provider:

Additionally, if we want to access multiple buckets from the system, we can use below method -

If we want to access multiple buckets, we need to create interfaces which extend from INamedBucketProvider and use that provider while adding it to controllers.

e.g.

public interface ICustomBucketProvider1 : Couchbase.Extensions.DependencyInjection.INamedBucketProvider { 

} 

public interface ICustomBucketProvider2 : Couchbase.Extensions.DependencyInjection.INamedBucketProvider { 

}

And change the Configure services function as below –

services.AddCouchbaseBucket<ICustomBucketProvider1>("travel-sample"); 

services.AddCouchbaseBucket<ICustomBucketProvider2>("gamesim-sample");

Now in controller we can add reference to both these buckets as

private readonly IBucket _bucket1; 
private readonly IBucket _bucket2; 

public AirportController(ICustomBucketProvider1 bucketProvider1, 
  ICustomBucketProvider2 bucketProvider2) 
{ 
      _bucket1 = bucketProvider1.GetBucketAsync().GetAwaiter().GetResult();
      _bucket2 = bucketProvider2.GetBucketAsync().GetAwaiter().GetResult(); 
}

The above code will wireup _bucket1 with travel-sample bucket and _bucket2 with gamesim-sample.

Please refer below link for complete code sample -
https://github.com/parasmm/CouchbaseWebAPI
<27-Jan-2026> Updated github code to refer to latest nuget versions and updated to use DotNet Core 9.0. This code base also implements Swagger UI, which can also be used to test these endpoints.

Conclusion:

So we went through wiring up .Net Core web api with Couchbase SDK and Couchbase dependency injection. Created endpoint to work with Airport documents from travel-sample bucket provided by Couchbase. Created all the usual endpoints for CRUD operations. Additionally, we saw how to connect to more than one bucket in the application.

References:

https://docs.couchbase.com/server/current/introduction/why-couchbase.html
https://blog.couchbase.com/dependency-injection-aspnet-couchbase/
https://docs.couchbase.com/dotnet-sdk/current/hello-world/start-using-sdk.html
https://docs.couchbase.com/tutorials/quick-start/quickstart-dotnet27-aspnetcore31-visualstudio-firstquery-cb65.html