DEV Community: Shashank Banerjea

A MongoDB Learning journey

Shashank Banerjea — Fri, 02 Jul 2021 15:27:18 +0000

Introduction

This is my tale of fumbling and tumbling into love with MongoDB, as the data store for the product, we are building at Introvoke, to power right scale branded virtual and hybrid event experiences for organizers, OEMs and enterprises.

Most of professional life while developing applications using different technology stacks, I have primarily used RDBMS as my data store. Many times it was Microsoft SQL Server as I was primarily a .NET/C# developer working at Microsoft. I had the pleasure of working with MySQL, PostgreSQL, IBM DB2 and yes, Oracle as well, working for start ups, software vendors, educational institutions and large enterprises. Good and robust products, for the solutions they power and price points they offer.

Until I joined my new role at Introvoke in late April 2021, my primary exposure to NoSQL technology was via Azure Cosmos DB. To appeal to enterprise developers like myself, Cosmos DB offers a SQL like query language and that is what I defaulted to when I used Cosmos DB.

The choice to use MongoDB and in a hosted fashion using MongoDB Atlas was done prior to my arrival at Introvoke.

My primary responsibility at Introvoke is to build and manage analytics, integrations and API for the platform.

The first task that I was pulled into was to make calculations of consumption on our platform perform better and more precise, so I had to dive right into learning MongoDB, right away. (Ahh... I only heard of tech intensity as an industry buzzword from Satya Nadella, when I was at Microsoft and now I was feeling it).

Learning Path

This is the path I took. As anything else in life, there are always more ways than one and varies by learning style.

Pluralsight Courses

The reason I jumped in here was primarily - HABIT. As a engineer at Microsoft, this was one of many learning resources that was available to me and over the years it became the good starting point.

The courses which helped me on my path were:

Foundations of Document Databases with MongoDB by Pinal Dave: Pinal Dave has been a well known expert (See: SQL Authority) in the SQL world as well, so I trusted the content authored by Pinal would be high quality. It also seemed to me that Pinal would have insights into both SQL and NoSQL worlds, so easy choice there.
Aggregating Data across Documents in MongoDB by Axel Sirota: Aggregation is a powerful feature in MongoDB. I rave about it in the later section. This course is a good premier to what is possible with it.
Using MongoDB with Node.js by Jonathan Mills: Since I was building applications at Introvoke using Node.js, this course was good fit to combine them.

There are tons of other courses on Node.js and MongoDB on Pluralsight to explore, but I found these relevant immediately to what I aiming for and course content was up to date.

MongoDB University

What better place to learn MongoDB but from the people who made it. Here is where MongoDB University comes in.

Easy to learn courses and well defined learning path and best of all at no cost! I believe, the end goal for the courses, would be obtain MongoDB certification. I started getting links to this via e-mail after signing up for a free MongoDB Atlas account.

MongoDB documentation

Believe it or not, MongoDB documentation is very good. It is very rich in examples. I was able to follow the documentation in most cases easily and apply the examples in my queries.

Quick Start - MongoDB and Node.js article and YouTube series

I came across this four-part article and YouTube series on using Node.js and MongoDB by MongoDB's Developer Advocate Lauren Schaefer, while looking for a way on how to react to changes in the data in MongoDB collection. That would be change streams in MongoDB similar to Change Feed in Azure Cosmos DB or Change Data Capture in SQL Server.

I loved Lauren's style of delivery, her deep technical expertise and how beautifully she explains advanced topics with ease and lots of examples. The video, I started out with was last in the series, but I liked it so much, I went back read the previous three articles and watched the full video series.

Of course, Stack Overflow

This is probably a no-brainer but worth mentioning anyhow. MongoDB has a rich developer community backing it. If there is anything you are thinking about, the chances are someone has asked or has solved it for you at Stack Overflow. The only caveat is sometimes, the solutions relate to older versions of MongoDB, but I have seen the community very good about making that distinction while responding to queries, especially between MongoDB 3.x and 4.x.

My Primary Mental Blockers

As with learning new technology, I had to unlearn some old habits. Among them was how to think about storing data differently from the world of SQL JOINS.

At Introvoke, I did inherit a very de-normalized data store with various collections. But where there was gaps, I was trying to create JOINS, which was not always easy to pull off syntax wise for a newbie. Over past two months, I have gradually become better.

The same thought process also influencing me on how to store data in a normalized manner focusing on storing ids and instead of embedding documents.

One of the articles, which has help me design better and think more in terms on how data is to be accessed versus trying to store data concisely was this article - MongoDB Schema Design Best Practices by Joe Karlssson.

I think, it is a learning curve, most developers will face coming from the RDBMS world. As I discovered, getting a optimal solution does take some experimentation.

My favorite features

MongoDB Query Language

The MongoDB Query Language (MQL) is very powerful and in most instances very intuitive, though it can be verbose at times.

Aggregates

The aggregation framework, in my opinion is the most powerful feature in MongoDB and I use it on almost every functionality, I am building to return data. It can help you shape the data down to how your APIs need to return it as their data contract. Almost zero ORM. Imagine that doing it in SQL world.

The only place where I have not been able to use aggregation pipeline is data has been stored across different MongoDB clusters. That is where the Node.js application has to stitch results from different queries together, which to my amazement is real fast. That is probably a discussion for a different article.

Facets

I discovered this feature of the aggregation framework - facet, two weeks into writing queries and tying them into the API. I found it especially useful in places, where I started with the same filtered data but had to run multiple different grouping calculation on it (dimensions for data warehouse folks out there). Prior to that I was writing multiple queries for it and calling it from the application.

My Pet Peeves

BLANK results

In the SQL world, for queries with incorrect names of the tables or columns, the schema enforcement would prompt an immediate feedback. Not so in MongoDB, it returns empty result, which can become a head scratcher at times.

There have been instances where I have misspelled the collection or the field name. For example, in some collections the unique identifier for a Company entity is called - company and companyId in others.

Another one is misspelling a collection named EventsAggregates, as EventAggregates or EventsAggregate.

I do wish at least for the collection name, there was some validation, which could alert the developer of the error. The one with the document structure, the responsibility lies with the developer (or the development team) to establish naming patterns for columns across collections.

My Developer Toolkit

I found the following tools must-haves for my developer journey learning how to use MongoDB and building queries for my day to day application

MongoDB Compass

MongoDB Compass is a free cross-platform tool from MongoDB. It is great for exploring data and running queries in built in Mongo Shell. I found error reporting when developing queries so much more useful than on MongoDB extension is VS code. However for writing the MQL code, I would advice to go with an editor such as Visual Studio Code or Sublime.

MongoDB Extension for Visual Studio Code

If you are developing using Node.js or Typescript, the chances are you are using Visual Studio code. There is an excellent VS Code Extension for MongoDB which provides connection management, browsing MongoDB collections with VS code, syntax highlighting for MQL code.The only limitation I found with the plugin, is I have not been able to get the it to return more than 20 documents for a query. There is a setting when viewing document collections but it does not seem to apply to the custom queries.

Journey continues

At the time of writing this article in June 2021, it has been only two months into learning this fantastic technology. I am sure, I will have more to say in the coming days.

References

DISCLAIMER: Opinions expressed in this article are solely mine and do NOT represent those of my employer.

Azure Data Factory Automation — Adding Log Analytics for Monitoring

Shashank Banerjea — Sun, 03 Jan 2021 19:01:48 +0000

One of the challenges, that I often faced with automating deployment of Azure Data Factory has been to add monitoring to the Azure Data Factory in a scriptable way.

The monitoring is essential to broad system health. For example, to generate alerts failures or if a pipeline is running longer than expected. The preferred and documented way of generating this alert has been to push the logs to Azure Log Analytics and analyzing the logs to generate alerts or provide a broad monitoring capability integrated with other parts of the infrastructure on Azure.

The Azure Data Factory documentation does provide a method using the Azure Monitor REST API. However it is little unwieldy to use, , especially in CI/CD pipelines.

There is another method to do this is actually that is well documented but not often referred to. It is available in the documentation for Azure Monitor.

So, associating an Log Analytics as a diagnostics log and metrics sink for Azure Data Factory using Azure CLI becomes as simple as running the script below:

az monitor diagnostic-settings create \
 - name LogAnalytics02-Diagnostics \
 - resource /subscriptions/(your-subscription)/resourceGroups/(your-resource-group)/providers/Microsoft.DataFactory/factories/(data-factory-name) \
 - logs '[{"category": "PipelineRuns","enabled": true}]' \
 - metrics '[{"category": "AllMetrics","enabled": true}]' \
 - workspace /subscriptions/(your-subscription)/resourcegroups/(your-resource-group)/providers/microsoft.operationalinsights/workspaces/(your-log-analytics-workspace-name)

The pre-requisite to run this scripts are - Azure Data Factory Instance and Log Analytics workspace should already be provisioned.

Just substitute the values for subscriptions and resources with values that applies to you. The example above shows just adding the logs for PipelineRuns but you can add more values in JSON string for logs parameter.

Using Azure CLI with local storage emulator

Shashank Banerjea — Mon, 02 Dec 2019 04:31:04 +0000

The Beginnings - Unusual Circumstances

While working with a fellow engineer who uses virtualized Linux machine as their primary development environment on Windows 10 PC hosted on Virtual Box, we ran into challenges with accessing the Azure local storage emulator - Azurite, with the Azure Storage Explorer.

The root cause was the corporate proxy settings associated with the laptop that we just could not to resolve.

A mighty unusual start in a way, but that is hand we were dealt.

Finding the path forward

Our project involved developing Azure Functions using Python, which was dependent on use of Azure Storage Queues and access to local development was critical for developer isolation and productivity. But using Storage Explorer became an issue, we were left few options.

First one, drop Linux as our development environment and move to Windows or MacOS, which did not seem to have this issue. Ugly, huh? It also flies right in the face of cross-platform development.

Second one, living the hashtag - #WeAreDevelopers, write some test harness/setup code using the Azure Python SDK and keep it up to date. It was an option, not the most efficient one.

The Middle Ground - Hacked it! or really on shoulder of giants

Azure CLI is a cross platform development for developers and administrators to interact with the Azure Cloud. And until now that is how I had been using Azure CLI, to work with Azure Cloud and not for local development.

So what if, there is a way to use the Azure CLI storage command to work with local storage versus working with Cloud Storage???

IF Only, What IF??

As it turns out, THERE IS. Though not as well documented (as in here in your face... DOCUMENTED), it is possible to stitch the solution using the documentation for Azure Storage Emulator and Configuration for Azure CLI.

The solution

It turns out to be as simple as setting the value a environment variable - AZURE_STORAGE_CONNECTION_STRING, which becomes the default connection string that Azure CLI uses, if a storage account is not specified.

Here is the command to run:


$ AZURE_STORAGE_CONNECTION_STRING="UseDevelopmentStorage=true"
$ export AZURE_STORAGE_CONNECTION_STRING
$ az storage queue create -n myqueue
{
   "created" : true
}

$ az storage queue create -n anotherqueue
{
   "created" : true
}


$ az storage queue create -n newqueue
{
   "created" : true
}


$ az storage queue list --verbose

The Conclusion

Unfortunately, the firewall issues with the corporate laptop still persisted even with this. This approach did not solve the original problem.

But this unusual set of circumstances led me to discover a new way to use Azure CLI. For example - I can use it to test setup scripts, offline for Azure Storage.

So, today as I write this, I am living the motto of my group within Microsoft - "We reserve the right to become smarter".

So my fellow developers - keep hacking and be UNSTOPPABLE!

Useful Azure Data Explorer queries for Azure Data Factory pipelines

Shashank Banerjea — Fri, 04 Oct 2019 06:38:22 +0000

Monitoring the Azure Data Factory Pipelines

Monitoring Azure Data Factory is enabled through collecting diagnostic logs and posting them to Log Analytics (part of Azure Monitor). The other options include – Storage Account or Event Hub for custom processing. There are additional settings provide options to decide what information that can are to be captured. It is also possible to have multiple settings for collecting diagnostic logs, where it is possible to send different data to different collection stores.

The most readily available monitoring is available through Log Analytics Workspace and Azure Data Factory Workbook available in the Azure Marketplace. Detailed documentation to enable monitoring through is available here.

Useful Azure Data Explorer Queries

While the workbook is helpful in surfacing some of the key metrics across multiple data factory instance it is helpful to write couple of custom queries in Azure Data Explorer to review some performance statistics of a Pipeline run.

Following the instructions in the link provided in the section above, you will have created a Log Analytics Workspace to store the Azure Data Factory diagnostic data. The data for Pipeline run is available in table called – “ADFPipelineRun”.

Viewing Duration of Pipeline Runs

One of the useful queries that I find is to view how long my succeeding pipeline are taking. For this example, we assume, what we are monitoring a pipeline called - gsc2adlsgen2copy. A query for the pipeline that would look like:

    ADFPipelineRun 
    | where Status == "Succeeded"
    | where PipelineName == "gcs2adlsgen2copy"
    | project PipelineName, RunId, Start, End, (End - Start)

The query above returns all the execution time of the pipeline. This query returns all the executions of the pipeline (within the time frame set by the query explorer) and how much time they took.

Viewing number of Pipeline Runs exceeding a threshold

If we have established an SLA for the Pipeline run, we can add a threshold value into the query to view pipelines that exceed the threshold. The query would look like this with an arbitrary threshold of 15 seconds:

    ADFPipelineRun 
    | where Status == "Succeeded"
    | where PipelineName == "gcs2adlsgen2copy"
    | where (End - Start) > 15s
    | project PipelineName, RunId, Start, End, (End - Start)

Generating Alerts on missed SLA

To generate an alert from the query above, click on the “+ New Alert” button on top of the query window and select to build a new alert on the Custom query.

Shown below is the alert created for the query above, based on number of rows returned, monitored every 24 hours for past 24 hours.

Conclusion

Azure Pipelines can be monitored using Azure Monitor. Alerts can be generated using custom Azure Data Explorer queries when a pipeline takes too long or does not run over a period. More reasons to rejoice and rest easy, my fellow cloud dwellers.

Bringing R Workloads to Azure Machine Learning Service

Shashank Banerjea — Thu, 26 Sep 2019 02:51:48 +0000

Making the case

The two of the most popular platforms for data science workloads are – R and Python. Both platforms have a large ecosystem of libraries and support which enable developers and innovators to build solutions quickly and efficiently. Looking more closely however, it is quite evident that around the academic and research, R draws at least as many supporters as Python.

Azure Machine Learning (AML) Service is a hosted service which empowers organizations to streamline the building, training and deployment of machine learning models at scale.

Azure Machine Learning currently provides direct support for Python and popular ML frameworks like ONNX, PyTorch, scikit-learn, and TensorFlow. With its Studio interface and Juypter notebooks there are additional capabilities available through a Python SDK.

This article demonstrates the use of the AML Python SDK to run R Workloads in Azure Machine Learning pipelines.

This article assumes that reader has basic knowledge of the R and Python languages, familiarity with Azure Machine Learning Service, and with use of the Azure Portal. It also requires basic understanding of Docker and container registries.

Broadly Speaking

The approach to run R workloads on Azure is enabled by AML’s ability to run workloads in purpose-built Docker containers across different choices of compute.

The first step is to build a Docker container that can support R workloads. This container packages the R runtime and essential libraries required to execute the R workload.

The container is built and published to a container repository; in this example we’ll use Azure Container Registry.

The next step adds execution of Docker container to an Estimator step, which becomes part of an AML pipeline. Prior to execution, the R script will be uploaded to the AML workspace and configured as a parameter to the AML step bootstrap routine written in Python. Finally, this new AML pipeline is published to the AML Service for execution.

When the pipeline is invoked it will download the docker container, obtain the R script from workspace storage and execute it.

Configuring the Azure Machine Learning Service Workspace

Azure ML Workspace

To be able to execute R script described in this article, the Azure Machine Learning Services (AMLS) Workspace should be configured as follows:

Workspace provisioned and available.
A choice of compute successfully provisioned. This article uses the compute option of Machine Learning Compute

Creating Service Principal

The article uses the Python SDK to execute code against the AML workspace, to execute the code successfully, it also requires:

An Azure AD Service Principal created with a secret.
The service principal given access to the Azure Machine Learning Workspace as a “Contributor”.

The R Docker Container

Creating the container locally

Since we are using a Python script to bootstrap the execution of R workload, the docker container needs to support both R and Python. One popular base image for this purpose is continuumio/miniconda3:4.6.14

The Dockerfile we are using for this article is as follows:

FROM continuumio/miniconda3:4.6.14

WORKDIR /app

RUN apt-get update && yes Y | apt-get install build-essential

RUN conda install -c r r-essentials

To create a docker image, run the following command in the same path as the docker definition, stored as Dockerfile.

docker build -t aml-r .

Pulling in Additional R libraries

There may be scenarios, were additional R libraries may be required for execution of the intended workload. To enable that, the Docker image used for execution of the workload can be extended by adding line such as these to the Dockerfile.

RUN R --vanilla -e 'install.packages(c( "RPostgreSQL", \
                                        "DBI",         \
                                        "lubridate",   \
                                        "RSQLite",     \
                                        "magrittr",    \
                                        "zoo",         \
                                        "functional",  \
                                        "moments",     \
                                        "fpc",         \
                                        "RcppRoll",    \
                                        "cowplot",     \
                                        "tsne",        \
                                        "config",      \
                                        "factoextra",  \
                                        "NMF",         \
                                        "ggcorrplot",  \
                                        "umap"), repos="http://cran.r-project.org")'

The above shows the docker image being built with additional R libraries to support – PostgreSQL, SQLite and more advanced capabilities.

Publish the Docker Container to ACR

The instructions to publish a docker container image to Azure Container Registry (ACR) are provided on this link

After authenticating with the Azure Container Registry, based on the instructions provided, run the following commands (assuming myregistry is the name of the ACR created):

docker tag aml-r myregistry.azurecr.io/aml-r

This will create an alias for the image in the local repository. Next, push the image to registry with the following command:

docker push myregistry.azurecr.io/aml-r

Configuring the AML Pipeline Bootstrapper

To execute the R script, we use a initializer script written in Python. This python script (called bootstrapper.py) creates a bash shell and launches the RScript executable with the provided R script name.

Source code for the bootstrapper.py is as follows:

import os
import sys
import subprocess
import uuid


def boot(*args):
  try:
    ret = subprocess.run(args)

    if ret.returncode < 0:
        print("Pipeline step execution was terminated by signal",
             -(ret.returncode),
              file=sys.stderr)
    else:
        print("Pipeline step execution returned",
            ret.returncode,
            file=sys.stderr)

    except OSError as e:
        print("Execution failed:", e, file=sys.stderr)
        return 1

    return ret.returncode

if __name__ == "__main__":
    entry_script = sys.argv[1]
    sys.exit(boot('Rscript', '--no-site-file', '--no-environ', '--no-restore', entry_script))

Running the R code in the AML Pipeline

To enable running of the arbitrary workload in Azure Machine Learning Pipeline, the Python SDK provides us with two constructs – Estimator and EstimatorStep classes.

The Estimator class is designed for use with machine learning frameworks that do not already have an Azure Machine Learning pre-configured estimator. It wraps run configuration information to help simplify the tasks of specifying how a script is executed.

The EstimatorStep class executes the estimator as part of an Azure ML Pipeline.

The code to create an Estimator and EstimatorStep with the bootstrapper.py initializer file and hello.r R script in the custom Docker image is shown as follows:

r_script='hello.r'

aml_experiment_name='experimenthellor'
aml_compute_target='defaultcompute'
acr_details = ContainerRegistry()
acr_details.address = 'mydockerimageregistry.azurecr.io'
acr_details.username = 'mydockerimageregistry'
acr_details.password = 'mysupersecretacrpassword!'
acr_image = 'aml-r'

estimator = Estimator(source_directory='src',
                          entry_script='bootstrapper.py',
                          compute_target=aml_compute_target,
                          custom_docker_image=acr_image,
                          image_registry_details=acr_details,
                          user_managed=True)

bootstrap_args = [r_script]
inputs = []

step = EstimatorStep(
        name='execute-r',
        estimator=estimator,
        estimator_entry_script_arguments=bootstrap_args,
        inputs=inputs,
        outputs=None,
        compute_target=aml_compute_target,
        allow_reuse=False)

For this article, the initializer and R Scripts are placed in a sub-folder called src.

Putting it all together

Once, the estimator step is created, create an AML Pipeline and run it as part of the experiment. That is accomplished by the following code:

aml_pipeline = AmlPipeline(
                 workspace=aml_workspace,
                 steps=AmlStepSequence([step]),
                 description='Run R Workloads')

aml_run = published_pipeline.submit(workspace=aml_workspace,
                                       experiment_name=aml_experiment_name)
print(aml_run)

Tracking Execution of the R Workloads

The execution of the pipeline can be viewed from the Azure Portal, as part of an AML Experiment. To do this, log on to Azure Portal and navigate to AML Workspace. Click on the “Experiments” option in the left panel.

Select the experiment created for this workload. In this article, we are using “experimenthellor”.

Click on the experiment name to get the list of runs for the experiment.

Click on the latest successful run to get the list of steps in the pipeline run. During the first execution of the pipeline, the compute may require time to spin up, so the pipeline run status may show as – “NotStarted”.

Click on the pipeline step – “execute-r”, to show the obtain the details of the step.

Click on “Logs” tab to view the details of the execution of the “R” script

In the logs, you can see the output of the hello.r script is available as “[1] hello, world”. If there were errors, they would show up in the logs for troubleshooting.

Conclusion

In addition to supporting Python workload, Azure Machine Learning Service is flexible to support execution of R workloads as well. That is just one more reason to try out Azure Machine Learning Service for your next data science project. While this repository has a simple implementation intended for a quick rollout, more advanced implementations are possible with mounted volumes.

References

Credits

This solution was designed by Josh Lane (@jplane). Special shout to Josh for reviewing the contents of the repository and technical guidance.

Code Repository

The code is available on GitHub.

DEV Community: Shashank Banerjea

A MongoDB Learning journey

Introduction

Learning Path

Pluralsight Courses

MongoDB University

MongoDB documentation

Quick Start - MongoDB and Node.js article and YouTube series

Of course, Stack Overflow

My Primary Mental Blockers

My favorite features

MongoDB Query Language

Aggregates

Facets

My Pet Peeves

BLANK results

My Developer Toolkit

MongoDB Compass

MongoDB Extension for Visual Studio Code

Journey continues

References

Azure Data Factory Automation — Adding Log Analytics for Monitoring

Using Azure CLI with local storage emulator

The Beginnings - Unusual Circumstances

Finding the path forward

The Middle Ground - Hacked it! or really on shoulder of giants

The solution

The Conclusion

Useful Azure Data Explorer queries for Azure Data Factory pipelines

Monitoring the Azure Data Factory Pipelines

Useful Azure Data Explorer Queries

Viewing Duration of Pipeline Runs

Viewing number of Pipeline Runs exceeding a threshold

Generating Alerts on missed SLA

Conclusion

Bringing R Workloads to Azure Machine Learning Service

Making the case

Broadly Speaking

Configuring the Azure Machine Learning Service Workspace

Azure ML Workspace

Creating Service Principal

The R Docker Container

Creating the container locally

Pulling in Additional R libraries

Publish the Docker Container to ACR

Configuring the AML Pipeline Bootstrapper

Running the R code in the AML Pipeline

Putting it all together

Tracking Execution of the R Workloads

Conclusion

References

Credits

Code Repository

codepossible / AzureMachineLearningServiceWithR

This repository contains guidance to execute R workloads in Azure Machine Learning Service.