DEV Community: Prasanna Sridharan

Visitor pattern

Prasanna Sridharan — Mon, 07 Nov 2022 13:10:34 +0000

Lets take a look at visitor pattern, however prior to that lets take a look at simple example and a scenario of what happens when we dont use the pattern.

Consider a shopping cart, and there are items that can be added to the cart.

    public class ShoppingCart
    {
        List<ItemBase> items1 = new List<ItemBase>
        {
            new FoodItem{Name="Icecream", Price=10, Expiry=DateTime.Now.AddDays(10)},
            new Toy{Name ="Lego", Price=100, Color="Multi"},
            new FoodItem{Name="Choclate", Price=20, Expiry=DateTime.Now.AddDays(100)},
        };
    }


    public abstract class ItemBase
    {
        public string Name { get; set; }

        public decimal Price { get; set; }
    }

    public class FoodItem : ItemBase
    {
        public DateTime Expiry { get; set; }
    }

    public class Toy : ItemBase
    {
        public string Color { get; set; }
    }

The expectation now is to have a logic to compute a discount on the shopping card during checkout.
There are atleast two ways to do this:

If the discount is flat, iterate through all the items and just apply discount on it.
However, discount may not be as simple, it could depend on the type of item and may be even some properties of the item. E.g. A food item that is going to perish sooner may have more discount; or say a toy which is of color red (most liked by kids) may have lesser discount. So the discount has additional logic dependent on the item itself in addition to percent. So, the code may look like this:

public class FoodItem : ItemBase
{
    public DateTime Expiry { get; set; }

    public override void GetDiscountedPrice()
    {
        // based on how close is the expiry give some discount
        if (Expiry > DateTime.Now.AddDays(10))
            return Price; // no discount
        else
            return Price * .5;
    }
}

public class Toy : ItemBase
{
    public string Color { get; set; }

    public override double GetDiscountedPrice()
    {
        // based on popularity of toy and how old was the launch do some logic
        return Price * .9;
    }
}
// and the processing in the shopping card would be:
public void ProcessShoppingCard()
{
    double totalDiscount = 0;
    foreach (var v in items)
    {
        totalDiscount += v.GetDiscountedPrice();
    }
}

Now for disadvantages of this approach:

The items' class files will be modified by frequent change in the logic of discounting.
Multiple class files would have to be modified to support future change
Lets say there is another request to support loyalty points calculation on the shopping cart checkout - this would need additional methods implemented in each item class.

Enter visitor pattern.

What does visitor pattern do?

It helps introduce functionality to class without modifying it.
It gives a checklist of objects for which visit methods are to be implemented and those will be present in a single class file. Making it more maintainable.

How is it different from extension methods in C# then?

Good question, extension method helps introduce functionality in a new class. However, it helps in introducing only once such functionality. What visitor does it, it gives a complete implementation set for similar (but unrelated) objects with a neat visit method, that can be injected with relevant concrete visitor implementation.

More info here: https://en.wikipedia.org/wiki/Visitor_pattern

With this pattern, there will be a IVisitor interface that would be implemented by concrete visitor classes and the items would just need to be able to "accept" a visitor. E.g.

    public interface IVisitor
    {
        void Visit(FoodItem food);

        void Visit(Toy toy);
    }

The concrete visitor will look like this

 public class DiscountCalculatorVisitor : IVisitor
    {
        public double TotalDiscount { get; private set; }

        public void Visit(FoodItem food)
        {
            if (food.Expiry > DateTime.Now.AddDays(10))
                TotalDiscount += 0; // no discount
            else
                TotalDiscount += food.Price * .5;
        }

        public void Visit(Toy toy)
        {
            if (toy.Color != "Red")
                TotalDiscount += toy.Price * 0.9;
            else
                TotalDiscount += 0;
        }
    }

The advantage comes now when we want to also compute the loyalty points. All one would need is another concrete visitor.

    public class GoldMembershipPointsCalculatorVisitor : IVisitor
    {
        public double TotalPoints { get; private set; }

        public void Visit(FoodItem food)
        {
            TotalPoints += 10;
        }

        public void Visit(Toy toy)
        {
            TotalPoints += 100;
        }
    }

The usage in main calling method would look like:

DiscountCalculatorVisitor discountCalculatorvisitor = new DiscountCalculatorVisitor();
GoldMembershipPointsCalculatorVisitor pointsVisitor = new GoldMembershipPointsCalculatorVisitor();

foreach (var v in items)
{
     v.Accept(discountCalculatorvisitor);
     v.Accept(pointsVisitor);
}

var discount = discountCalculatorvisitor.TotalDiscount;
var points = pointsVisitor.TotalPoints;

Understanding Task in .Net

Prasanna Sridharan — Wed, 16 Feb 2022 13:41:11 +0000

In .Net framework, there are couple of ways to implement asynchronous programming. We will look at the Task Parallel Library, which also happens to be the recommended way since .Net 4.0.

Using Task, with async-await keyword, the methods return a promise to the function execution.

Lets say there is some long running operation to execute. If we want to offload the long running operation to "someone" else so that the main thread can focus on doing some other work, it will be really beneficial if

The long running operation is purely I/O based operation; which means that waiting on the operation to complete, without offloading it to another thread, would waste CPU cycles.
Its a heavy CPU based application, and its a multicore machine, where the OS can optimize the CPU.

For purpose of this example, just created a P/Invoke to query all the open windows (just to simulate an example of managed-native boundary with call back to managed code).

// Import user32.dll (containing the function we need) and define
// the method corresponding to the native function.
[DllImport("user32.dll")]
private static extern int EnumWindows(EnumWC lpEnumFunc, IntPtr lParam);

private static Dictionary<Int64, int> keyValuePairs = new Dictionary<Int64, int>();

// Define the implementation of the delegate; here, we simply output the window handle.
private static bool OutputWindow(IntPtr hwnd, IntPtr lParam)
{
     var x = hwnd.ToInt64();
     if (!keyValuePairs.ContainsKey(x)) {
         keyValuePairs.Add(x, 0);
     }
     keyValuePairs[x]++;
     return true;
}

Lets take a look at the invocation as below:

private static async Task PInvokeUsingTPL()
{
    // Invoke the PInvoke call
    var handle = Task.Run(() => EnumWindows(OutputWindow, IntPtr.Zero));
    Console.WriteLine("Executed code to the PInvoke");


    // Now until the handle returns main thread of application is free to do something else
    while(handle.Status != TaskStatus.RanToCompletion)
    {
        Console.WriteLine("Doing something else");
        Thread.Sleep(500);
    }
    Console.WriteLine("handle returns and we can continue");
    await handle;
    Console.WriteLine($"Found {keyValuePairs.Count} unique windows");

}

First statement within the method actually invokes the method that makes the native external call. Given that it enumerates through the windows that are open on the machine, the main thread gets handle / promise to the execution.
While the main thread awaits, it can check the status of the handle and perform some other operation.

Note the execution:

Of course, though i used the TPL's Task.Run method to offload the execution to another .Net framework to execute as thread pool, one can still use the old construct of using threadpool manually as below:

private static async Task PInvokeUsingThreadPool()
{
    Console.WriteLine("Starting invoke");
    var response = PureThreadPoolInvocation();
    Console.WriteLine("Invoked with threadpool and now i can do more stuff");
    await response;
    Console.WriteLine("Thread pool invocation complete");
}

private static Task PureThreadPoolInvocation()
{
    TaskCompletionSource<bool> taskCompletionSource = new TaskCompletionSource<bool>();
    ThreadPool.QueueUserWorkItem(_ =>
    {
        EnumWindows(OutputWindow, IntPtr.Zero);
        taskCompletionSource.SetResult(true);
    });

    return taskCompletionSource.Task;
}

Disclaimer: there are few thread.sleep methods used just to explain the sequence of execution; one would not use them in the production grade code unless its absolutely necessary.

Read Json property values using dot notation

Prasanna Sridharan — Thu, 16 Dec 2021 11:37:41 +0000

Sometime we want to read a value of property nested deep within the json. This simple code helps get the value

Lets say the json looks like this:

{
  "level1-1": {
    "level2-1": "value2",
    "level2-2": {
      "level3-1": "value3" 
    }
  },
  "level1-2": "value1-2"
}

Then to read value3, we can use syntax like level1-1.level2-2.level3-1.

This code is PowerShell can be used.

function getValue($obj, $fullkey) {
    $ks1, $ks2 = $fullkey.Split('.', 2)
    # Technique to read a property from PSCustomObject
    $props = $obj | Get-Member | Where-Object MemberType -Like 'NoteProperty'
    foreach($p in $props)
    {
        if($p.name -eq $ks1)
        {
            # Get the value of the property name
            $val = $obj | Select-Object -ExpandProperty $p.name
            #if not reached last key yet, go recursive.
            if($ks2 -ne $null){
                return getValue -obj $val -keylist $ks2
            } 
            return $val
        }
    }

    return $null;
}

Usage:

$json = ConvertFrom-JSON '{"level1-1": { "level2-1": "value2", "level2-2": {"level3-1": "value3"}}, "level1-2":"value1-2"}'

$key = 'level1-1.level2-2.level3-1';
$res = getValue -obj $json -keylist $key
Write-Host "Key: " $key ", Value: " $res

The response would look like:

In another blog will try to show a scenario with arrays and indices.

Happy coding.

Passing Arguments to Docker windows container

Prasanna Sridharan — Thu, 28 Oct 2021 06:37:29 +0000

Docker allows passing arguments to as build arguments using following syntax, however note that this is Linux container syntax to use the variables with $ sign within the docker file.

Docker file for Linux container

FROM ubuntu:latest
ARG MYARG=defaultValue
RUN echo "The arg value is : $MYARG"

Build command looks like

docker build --build-arg MYARG=ValueFromOutside

For windows container, the arguments are to be referred like below, i.e wrapped with % symbol. So the similar code as above looks like below when executing for windows container:

Dockerfile for Windows container with default shell.

FROM mcr.microsoft.com/windows/servercore:20H2-KB4598242
ARG MYARG=PS
RUN echo "%MYARG%"

RUN powershell.exe -Command \
  $ErrorActionPreference = 'Stop'; \
  $ProgressPreference = 'SilentlyContinue'; \
  Invoke-WebRequest -Uri %MYARG% -Method Get -UseBasicParsing;

Build command looks like

docker build --build-arg MYARG=www.bing.com .

Note that it was able to print the value of variable as well as execute some PowerShell command using the argument's value.

Note that the logic depends completely on the shell that is configured to be used. By default the shell is bash for Linux and CMD for windows. Of course by using the [SHELL] statement in the Dockerfile, one can also set it to powershell in which case the argument will be accessed with following syntax: $env:VARIABLE_NAME. Refer here for more details.

FROM mcr.microsoft.com/dotnet/framework/aspnet:4.8-20210209-windowsservercore-ltsc2019
ARG SOMEURI=defaultvalue
RUN Write-Output $env:SOMEURI

RUN Invoke-WebRequest -Uri $env:SOMEURI -Method Get -UseBasicParsing;

CAP Theorem : Scenarios explained

Prasanna Sridharan — Sat, 02 Oct 2021 15:50:05 +0000

CAP theorem applies to behavior exhibited by the distributed system with respect to three attributes Consistency, Availability and Partition tolerance, for definition refer to the wiki.
The theorem defines the choice of behavior that the system can exhibit from the end user's perspective. It states that, at any given point of time, only 2 of the 3 behaviors can be guaranteed.

Lets look at what each of the 3 behavior individually mean:

Consistency: The system shall always respond to the end user's read request with consistent data. See note.
Availability: The system shall always respond to a user's request to read/write.
Partition Tolerance: The system will function even if the communication between the nodes drops i.e. partitions are created within the network. If this behavior is acceptable, then the system is tolerant to various partitions not interacting with each other and the system can continue to function with some sacrifices in quality of service.

For the below scenarios, let us consider that there are just two nodes which can hold some data. Node A is responsible for both read and write, whereas Node B is designed to be only read from.

Lets take a look at all the 3 combinations that are possible, lets start with the simplest.

Combination 1: Consistency & Availability.

In this combination, it is expected that the system is both consistent and available. It also means the system is resilient to inter node communications; and any partitions in the network is not tolerated.
When the writes happen to the Node A, then all the reads are suppose to get back the latest data consistently bounded by definition of consistency. Also the system is always available, all the read/write requests are honored without errors.

Combination 2 & 3:

These combinations are possible when the system is expected to function even when network partitions occur and system can tolerated it. I.e. even if the nodes are unable to communicate system will function.

Combination 2: Consistency and Partition tolerance:

When there are 2 or more partitions in the system, then the system can be designed to make one of the cluster/partition as the primary and have the other nodes/partitions dormant.

The primary cluster/partition is responsible and active for any data read and writes. When requests for reads are serviced by this primary node, the data is guaranteed to be latest and consistent. However if the read/write requests happen to go to nodes that are in dormant state, the system may not respond, in other words availability is not guaranteed.

When network is partitioned, then in order to keep the system consistent, partition 1 is made primary and the others are made dormant. So any request for read from system will get consistent data (if served by Node 1) or no data at all (if served by Node 2).

Combination 3: Availability and Partition tolerance:

When there are 2 or more partitions in the system, and the system can be designed to function as it was earlier. I.e. the nodes that were responsible for read and write will continue to do that; the nodes that were used for reads will continue to respond to end user's read requests. So end user will always perceive the system to be available, however data reads highly depend on the node to which the read requests goes. As there is no internode communication, the data may be out of sync, and end user may perceive the system as inconsistent.

When network is partitioned, in order to keep the system available, any read request is serviced irrespective of whether the data is stale or not. So depending on the node that services the request the data may not be perceived consistent.

Note on consistency:

The definition for consistency still depends on the implementation that was designed in the system. E.g. Eventual, Session, Strong. For purpose of above discussion, if we say system is consistent, then it is bounded by the design of consistency in the system.

Note on post communication restoration in case of partition tolerance:

This post just discusses the system's response as soon as nodes are unable to communicate with each other. Once communication between the nodes/partitions is restored, what techniques are used to synchronize the nodes is outside the purview of this article; E.g. Consensus algorithm.

Using wildcards for folder path with spark dataframe load

Prasanna Sridharan — Fri, 25 Sep 2020 20:03:11 +0000

While working with a huge volume of data, it may be required to do analysis only on certain set of data specific to say days', months' data. It is not uncommon to store data in a year/month/date or even hour/minute format.
As loading data to dataframe requires a lot of compute power and time, any optimization on data load saves a tons of resources.
So for selectively searching data in specific folder using spark dataframe load method, following wildcards can be used in the path parameter.

Environment Setup:

The files are on Azure Blob Storage with the format of yyyy/MM/dd/xyz.txt. So as to see the results, the files themselves just have one line with the date in it for easier explanation.
The examples below might show for day alone, however you can

All the files for all the days.

Format to use:
"/*/*/*/*" (One each for each hierarchy level and the last * represents the files themselves).



df = spark.read.text(mount_point + "/*/*/*/*")

Specific days/ months folder to check

Format to use:
"/*/*/1[2,9]/*" (Loads data for Day 12th and 19th of all months of all years)

"/*/*//{09,19,23/}/*" (Loads data for 9th, 19th and 23rd of all months of all years)



df = spark.read.text(mount_point + "/*/*/1[2,9]/*")

df = spark.read.text(mount_point + "/*/*//{09,19,23/}/*")

Specific series of folders

Format to use:
"/*/*/1[3-6]/*" (Loads data from Day 13th to 16th of all months of all years)



df = spark.read.text(mount_point +"/*/*/1[3-6]/*")

Combining Specific folders and some series

Format to use:
"/*/*//{09,1[8-9],2[0-1]/}/*" (Loads data for Day 9th and from 18th to 21st of all months of all years)



df = spark.read.text(mount_point +"/*/*//{09,1[8-9],2[0-1]/}/*")

Ofcourse, the other folders in the path can also use wildcards or specific values, based on need. Below is an example of 10th, 20th and 30th of Sep-2020 alone.



df = spark.read.text(mount_point +"/2020/09//{10,20,30/}/*")

Happy data mining!

Mock external dependencies in Python

Prasanna Sridharan — Mon, 27 Jul 2020 16:33:24 +0000

Testing python modules or code that have any external calls such as third-party API calls or database calls, can get difficult. This is where mocks are helpful.

Mocks are typically used when:

Calling a third-party API is redundant in testing scenarios (but required in actual production version) i.e. the data from external service doesnt often change however the http overhead skews the results.
Calling a third-party is has only premium subscription(paid) i.e. it doesn't have a test harness environment
The response from the third-party cannot be easily controlled by just changing the input i.e. simulating corner-case responses received external API.

There are several ways to implement mock, below, is a simple implementation of a http service mock using the unittest.mock library.

Mock http request

Let us assume there is a service that gives temperature back when requested with city or location:

Sample API's "get" output as seen on browser

A typical calling library within the client code would implement something like:



import requests
import json

class ExternalServiceUtility:
    def __init__(self):
        super().__init__()

    # Function to make call to external service
    def get_current_temperature_using_service(self, location):
        _temperature_service_url = "http://127.0.0.1:5001/gettemparature/{0}" #in real life would be from configuration
        formatted_url = _temperature_service_url.format(location)
        current_temperature = None

        try:
            response_obj = requests.get(formatted_url)

            # check if we got a valid response
            if response_obj.status_code != 200:
                #some logging
                return current_temperature

            # parse the response 
            response_json = response_obj.json()
            current_temperature = response_json["Temperature"]

            return current_temperature
        except Exception as ex:
            # some logging and handling
            return current_temperature

And the method to use this library would look like:



from externalserviceutility import ExternalServiceUtility
def get_actual_temperature(location):
    extutil = ExternalServiceUtility()
    temperature = extutil.get_current_temperature_using_service(location)
    print(temperature)

get_actual_temperature("Regina")

Here we may want to test the application out without calling the http endpoint. Hence the mock implementation would look as below.



from externalserviceutility import ExternalServiceUtility
import requests
from unittest.mock import patch
import json

def alternate_mock_method(args):
    data = '{"Temperature": 100, "Location": "Toronto", "Datetime": "2020-07-23", "Units": "Celsius"}'
    r = requests.Response()
    r.status_code = 200
    def json_func():
        return json.loads(data)
    r.json = json_func
    return r  

@patch('requests.get',side_effect=alternate_mock_method)
def get_temperature_with_mock(location, args):
    extutil = ExternalServiceUtility()
    temperature = extutil.get_current_temperature_using_service(location)
    print(temperature)


get_temperature_with_mock("Regina")

The same implementation can be done for other verbs within library say requests.post

Implementation details:

Include the unittest mock library and import on the top
Decorate a method using the "patch". All the calls to the method declared in the first parameter of the patch would be redirected to the function name declared as the second parameter i.e. side_effect.
Now instead of making calls to the external service, the "alternate_mock_method" implementation would be called for each requests.get call with all the child code of the method "get_temperature_with_mock" method i.e. the method decorated with the patch attribute.
Note that #get_temperature_with_mock# method requires an additional parameter in the end by syntax.

CAUTION: Note that all the calls to requests.get within the code tree structure within the mocked method would be redirected to custom mock implementation.