DEV Community: Sohaib Tariq

Show Dev: Here's how we made AI 2x faster at integrating APIs

Sohaib Tariq — Thu, 02 Apr 2026 15:36:34 +0000

We ran an experiment with our team:

Each of us asked Cursor to integrate the PayPal API into an e-commerce app multiple times.

Here are the results across all of our attempts:

13% of the attempts pulled in a deprecated PayPal SDK
87% of the attempts generated API calls based on deprecated PayPal documentation
0% of the attempts used the current, official PayPal Server SDK

Here's the interesting part, PayPal provides official API docs and SDKs for their APIs. The AI just never used them. Instead, it cobbled together code from blog posts, Stack Overflow answers, and stale training data (since PayPal is such a well-known API).

This isn't just a PayPal problem. There are millions of APIs out there. Their docs change. SDKs evolve. New API versions come out.

When an AI assistant tries to integrate an API using Web Search or model memory alone, mistakes are almost inevitable. Developers end up spending more time debugging AI output than they saved by using AI in the first place.

How we solved this problem

We built Context Plugins: given an OpenAPI spec, we generate SDKs and an MCP server that exposes structured API context to AI coding assistants.

This gives tools like Cursor access to comprehensive, up-to-date API context (including SDK documentation and API integration patterns), instead of relying on outdated training data or code scraped from GitHub.

Here's how it works:

An API provider uploads their OpenAPI spec to APIMatic
We generate high-quality SDKs in multiple programming languages
We generate an MCP server with tools and prompts that expose language-specific SDK context, optimized for LLMs
Developers install the MCP server in their IDE (Cursor, Claude Code, or GitHub Copilot)

When a developer asks to integrate an API, the coding assistant queries the MCP server, retrieves the required context (auth flows, integration patterns, latest SDK version, SDK interfaces), and generates code using the official SDK — not guesswork.

Here is a key insight we picked up along the way:

OpenAPI specs or API Reference docs alone aren't enough context for AI agents. They describe endpoints/operations and schemas, but the AI still has to infer how to handle authentication, pagination, error handling, and then translate all of that into working code. That's a long chain of inference, and every extra step is another place things could go wrong.

SDKs and SDK context cuts that chain short — much of the complexity is already wrapped in the library, so the model figures out which method to call and how to wire it up, instead of writing the entire integration from scratch.

We've just launched a pilot with PayPal, it's live on the PayPal Developer Portal. Check it out!

The benchmarks

We ran 4 controlled experiments on two real-world .NET applications, nopCommerce (mature e-commerce platform) and eShop (Microsoft's .NET reference app).

We ran the same task across the same IDE and models — with and without Context Plugins.

Aggregate results across all 4 experiments:

Metric	Without Plugin	With Plugin	Improvement
Errors	16 avg	1.5 avg	↓ 91%
Prompts needed	34 avg	15.5 avg	↓ 54%
Tokens consumed	57M avg	20M avg	↓ 65%
Manual fixes	2.75 avg	0 avg	↓ 100%

What went wrong without the plugin (real examples):

Hallucinated that SDK classes didn't exist — the agent decided OAuthAuthorizationController wasn't in the SDK and wrote a custom replacement
29+ compile errors from guessed model shapes, enums, and namespaces
Security vulnerabilities — a URL injection allowed order capture on ID mismatch; ?paid=1 query param triggered false payment approvals
Selected deprecated SDK versions despite being told to use v2.0.0
33% hallucination rate when relying on web search for API knowledge

What happened with the plugin:

Zero hallucinations across all experiments
Zero manual fixes required to the generated code
Agent verified its knowledge proactively via MCP tool calls before writing code
Clean compilation from the first attempt in 3 of 4 experiments
Eliminated 200+ lines of manual HTTP boilerplate in the eShop integration
AI completed the integration 2x faster on average

The full case study with detailed experiment logs is here: Context Plugins Case Study

Try it yourself

We've published Context Plugins for a few APIs in our Product showcase, this is the quickest way to try them out:

Try the Context Plugins Showcase

Pick an API (PayPal, Twilio, Stripe, Google Maps, Spotify, Slack, Adyen, and more), install the MCP server in your IDE, and start building.

What we learned from developers trying this

A few patterns emerged from our benchmarks and early user tests:

Trust follows grounding. Developers feel significantly better about AI output when the model is visibly pulling SDK method names and library docs, versus when it looks like it's piecing things together from web search or training data. The source of context matters as much as the correctness of the output.

The "dual responsibility" problem is real. When an AI agent has to simultaneously research an API and implement the integration, the quality of both suffers. Context Plugins separate those concerns — the MCP server brings authoritative API knowledge, the agent handles the coding. This division of labour consistently produces better results.

Developers have already hacked together solutions to this problem. Developers write their own skills and AGENT.md files to try and get their AI agents to write correct integration code for the APIs they use. Doing this involves trial and error, and they continuously need to update their local context as the API evolves.

I'd love to hear from the community:

What's the worst AI-generated API integration bug you've encountered? We're collecting failure patterns to improve our context coverage. Horror stories are especially welcome 🙃.
How are you currently handling API context for your coding assistants? AGENTS.md files? Custom MCP servers? Something else?

Have you invested in crypto yet?

Sohaib Tariq — Thu, 18 Feb 2021 08:49:10 +0000

The crypto space is buzzing right now. Bitcoin broke past $50,000 and most AltCoins are at all time high levels. Has this boom convinced you to invest yet ? Or are you waiting for the next dip to buy into crypto ? Or are you still skeptical about cryptocurrencies?
Lets get a discussion going, there's not a whole lot about crypto on DEV.

PS: I noticed that some crypto associated tags exist on DEV but I am unable to use them in my post. Does DEV discourage discussions around crypto?

Why do you use Dev.to ?

Sohaib Tariq — Wed, 27 Jan 2021 07:44:05 +0000

What do you aim to accomplish on dev? are you here for the tutorials? career advice? are you hoping to build connections with people in the tech industry or are you here to build a following? Or maybe you're just here for fun ? 😃
Let me know in the comments below.

What are the best non-programming books you have read recently ?

Sohaib Tariq — Mon, 25 Jan 2021 12:11:11 +0000

I'm looking for book recommendations to add to my reading list. I would be delighted to learn which non-technical books the dev community has been reading.
PS: No restrictions on topic/genre, anything goes😊.

What is your 1 tip for new writers on dev ?

Sohaib Tariq — Thu, 21 Jan 2021 12:10:07 +0000

Dev.to is a big community, with more people joining everyday. What tip would you like to give, that could help new users make the most of this platform ?
Your tips could be around type of content to write, how to optimize content for visibility, how to interact with people, how to engage the dev audience or anything that you have found useful in you time on dev.
THANKS!

Deploying a .Net Core Console app as Azure WebJob

Sohaib Tariq — Thu, 21 Jan 2021 11:41:44 +0000

If you are using Azure App Service, WebJobs are a great way of running background tasks without provisioning new resources or incurring additional costs.
However, Deploying a Console app as a WebJob to Azure is not the smoothest of experiences, specially if you want to setup Continuous Deployment.
In this post I'll take you through the entire process, and hopefully save you some time.

Overview

An Azure App Service instance can host multiple WebJobs. WebJobs can be deployed by placing their binaries at a predefined location, either manually, OR programmatically using a build script.

Azure supports two type of WebJobs:

Triggered
Continuous

The default location for deploying triggered WebJobs is:
d:\home\site\wwwroot\app_data\jobs\triggered\<name of job>
The default location for deploying continuous WebJobs is:
d:\home\site\wwwroot\app_data\jobs\continuous\<name of job>

Manual Deployment

If you want to get up and running quickly and want to forego CI/CD, you can manually deploy your console app as a WebJob using the Kudu Console.

Based on the type of WebJob, create a directory structure for your code according to the path shown above and upload your binaries to it.
If the entry-point for your application is an .exe file, you're good to go, however, if you are using .dll files, you may have to create a run.exe or run.cmd file to allow Azure to recognize which file to run.
Your run.cmd file may be as simple as

MyWebjob.dll

Continuous Deployment with Kudu App Service

There are multiple methods of setting up CI/CD for Azure App Service. For the purpose of this tutorial, we will use Kudu App Service. However, you will easily be able to translate the following steps to the platform of your choice.

When you set up Continuous Deployment for an App Service with Kudu, a deployment script is generated automatically. This script handles the setup and deployment of your application.
You can modify this deployment script in order to specify where your WebJob binaries ought to be published.

To access the default deployment script used by Kudu, open the Tools menu in the Kudu Debug Console and select 'Download Deployment Script'.

The downloaded folder contains 2 files, .deployment and deploy.cmd. Add both files to the root folder of your repository so that you can modify and commit them with your code.

The file we will be editing is the deploy.cmd file.

If this is your first time working with a deployment script, do not feel overwhelmed, we will only be editing a small part of it.

Deployment Script

As you will see in the Deployment section of the attached script, MSBuild is used to restore, build and publish the code for your App Service.

You will need to add a similar step for your Console App project with a slight modification:

use the OutputPath option to specify the location where the binaries for your WebJob need to go. This example is for a triggered WebJob so we will set this option to "%DEPLOYMENT_TEMP%\app_data\jobs\triggered\AnalyticsEmail"

@if "%SCM_TRACE_LEVEL%" NEQ "4" @echo off

:: ----------------------
:: KUDU Deployment Script
:: Version: 1.0.17
:: ----------------------

:: Prerequisites
:: -------------

:: Verify node.js installed
where node 2>nul >nul
IF %ERRORLEVEL% NEQ 0 (
  echo Missing node.js executable, please install node.js, if already installed make sure it can be reached from current environment.
  goto error
)

:: Setup
:: -----

setlocal enabledelayedexpansion

SET ARTIFACTS=%~dp0%..\artifacts

IF NOT DEFINED DEPLOYMENT_SOURCE (
  SET DEPLOYMENT_SOURCE=%~dp0%.
)

IF NOT DEFINED DEPLOYMENT_TARGET (
  SET DEPLOYMENT_TARGET=%ARTIFACTS%\wwwroot
)

IF NOT DEFINED NEXT_MANIFEST_PATH (
  SET NEXT_MANIFEST_PATH=%ARTIFACTS%\manifest

  IF NOT DEFINED PREVIOUS_MANIFEST_PATH (
    SET PREVIOUS_MANIFEST_PATH=%ARTIFACTS%\manifest
  )
)

IF NOT DEFINED KUDU_SYNC_CMD (
  :: Install kudu sync
  echo Installing Kudu Sync
  call npm install kudusync -g --silent
  IF !ERRORLEVEL! NEQ 0 goto error

  :: Locally just running "kuduSync" would also work
  SET KUDU_SYNC_CMD=%appdata%\npm\kuduSync.cmd
)
::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
:: Deployment
:: ----------

echo Handling ASP.NET Core Web Application deployment with MSBuild16.

:: 1. Restore, Build and publish
call :ExecuteCmd "%MSBUILD_16_DIR%\MSBuild.exe" /restore "%DEPLOYMENT_SOURCE%\AppServiceProject\AppServiceProject.csproj" /p:DeployOnBuild=true /p:configuration=Release /p:publishurl="%DEPLOYMENT_TEMP%" %SCM_BUILD_ARGS%
IF !ERRORLEVEL! NEQ 0 goto error

::--------THIS IS THE CODE WE HAVE ADDED--------------
::----------------------------------------------------

:: 1. Restore, Build and publish WebJob
call :ExecuteCmd "%MSBUILD_16_DIR%\MSBuild.exe" /restore "%DEPLOYMENT_SOURCE%\WebJobProject\WebJobProject.csproj" /p:DeployOnBuild=true /p:configuration=Release;OutputPath="%DEPLOYMENT_TEMP%\app_data\jobs\triggered\MyWebJob" /p:publishurl="%DEPLOYMENT_TEMP%" %SCM_BUILD_ARGS%

::----------------------------------------------------
IF !ERRORLEVEL! NEQ 0 goto error

:: 4. Run web job deploy script
IF DEFINED WEBJOBS_DEPLOY_CMD (
  call :ExecuteCmd "%WEBJOBS_DEPLOY_CMD%"
)

:: 2. KuduSync
call :ExecuteCmd "%KUDU_SYNC_CMD%" -v 50 -f "%DEPLOYMENT_TEMP%" -t "%DEPLOYMENT_TARGET%" -n "%NEXT_MANIFEST_PATH%" -p "%PREVIOUS_MANIFEST_PATH%" -i ".git;.hg;.deployment;deploy.cmd"
IF !ERRORLEVEL! NEQ 0 goto error

::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
goto end

:: Execute command routine that will echo out when error
:ExecuteCmd
setlocal
set _CMD_=%*
call %_CMD_%
if "%ERRORLEVEL%" NEQ "0" echo Failed exitCode=%ERRORLEVEL%, command=%_CMD_%
exit /b %ERRORLEVEL%

:error
endlocal
echo An error has occurred during web site deployment.
call :exitSetErrorLevel
call :exitFromFunction 2>nul

:exitSetErrorLevel
exit /b 1

:exitFromFunction
()

:end
endlocal
echo Finished successfully.

Making this small change in the deployment script will allow the Kudu App Service to build and deploy your WebJob to the correct location.

Schedule

We are building a triggered WebJob so we must have some sort of trigger to to execute it. For this demonstration we will use a scheduled or time-triggered WebJob. This will require a cron expression.
All we need to do is create a settings.job file inside our Console App directory which contains a cron expression against a schedule property.

{
    "schedule": "0 */15 * * * *"
}

When our WebJob is deployed, the schedule property will automatically be used to set up a schedule for our WebJob

Congratulations! you just deployed a WebJob to Azure and didnt have to pull your hair out while doing it.

If you would like more information about how Kudu handles WebJobs, you can visit the kudu project wiki on github.

Azure Datafactory pricing simplified

Sohaib Tariq — Mon, 14 Dec 2020 11:56:35 +0000

The first time I used Azure Data Factory, I found its pricing rather confusing. The pricing policy mentioned words like Read/Write Operations, Monitoring Operations and Activity Runs that did not make sense to me as a beginner.

I was simply trying to decide whether it would be more efficient and cost effective to use Data Factory rather than hand coding a data pipeline. However, I ended up spending a good few hours going through the documentation and just to get a rough cost estimate.

Below is a very basic overview of what Azure Data Factory V2 pricing looks like for data pipelines. This article should suffice for beginners who are trying out Data Factory pipelines for the first time, but please note that this is not a comprehensive overview, advanced users should refer to the official pricing documents.

Pricing

The first thing you need to know about Data Factory is that not only are you charged for executing pipelines, but also for developing/debugging and monitoring them.

If you're looking to build data pipelines in Azure Data Factory, your cost will be split into two categories:

Data Factory Operations
Pipeline Orchestration and Execution

Data Factory Operations

This is the cost associated with developing and debugging pipelines. There are 2 types of Data Factory Operations, Read/Write and Monitoring.

Read/Write:
Every time you create/edit/delete a pipeline activity or a Data Factory entity such as a dataset, linked service, integration runtime or trigger, it counts towards your Data Factory Operations cost. These are billed at $0.50 per 50,000 operations.
Monitoring:
You can monitor each pipeline run and view the status for each individual activity. For each pipeline run, you can expect to retrieve one record for the pipeline and one record for each activity or trigger.
For instance, you would be charged for 3 Monitoring activities if you debug a pipeline containing 2 activities. Monitoring activities are charged at $0.25 per 50,000 run records retrieved

As you may have concluded by now, Data Factory Operations are very inexpensive and for the most part, can be ignored for cost calculation purposes. The bulk of the cost comes from Pipeline Orchestration and Execution

Pipeline Orchestration

This refers to the cost of provisioning resources to run a pipeline and its associated activities.
Data Factory provides users with the option of running pipelines on their own servers(Self Hosted) or using a serverless integration runtime provided by Azure (Azure Integration Runtime). Pricing is slightly different for both options.

Please note that Integration runtime charges are prorated by the minute and rounded up.

Self Hosted

Every time you run a pipeline, you are charged for every activity and trigger inside that pipeline that is executed at a rate of $1.50 per 1000 Activity runs.

Azure Integration Runtime

This option is slightly cheaper at $1 per 1000 runs.

As an example, executing a pipeline with a trigger and two activities would be charged as 3 Activity runs.

Pipeline Execution

This is the cost for the compute resources required for Pipeline execution. You can expect the bulk of the cost for Azure Data Factory pipelines to fall into this category.
This is further divided into three sub-categories :

Data movement activities : This covers the cost of moving data across data stores in activities such as the copy data activity.
Pipeline activities : Pipeline activities such as Lookup, Delete and schema operations during authoring (test connection, browse folder list and table list, get schema, and preview data).
External activities : These are data transformation activities that execute in a computing environment such as Azure Databricks or Azure HDInsight. You can find a list of external activities here.

Each sub-category has separate pricing, as listed below:

Self hosted

Data movement : $0.10/hour
Pipeline activities : $0.002/hour
External activities : $0.0001/hour

Azure Integration runtime

Data movement : $0.25/DIU-hour
Pipeline activities : $0.005/hour
External activities : $0.00025/hour

You may have noticed that every sub-category is charged by the hour except Data movement which is charged in units of DIU-hour.

"A Data Integration Unit (DIU) is a measure that represents the power of a single unit in Azure Data Factory. Power is a combination of CPU, memory, and network resource allocation."
A DIU therefore, is a measure of the compute resources available to your pipeline. The more compute you allocate, the higher the cost.

The copy activity is configured to use 4 DIUs by default but you can modify this to set the value between 2 and 256 depending on your performance requirements. You can find a detailed article on DIUs and how to optimize costs and performance for the copy activity here.

Azure Also charges for inactive pipelines so its good to clean up after you're done.
A pipeline is considered inactive if it has no associated trigger or any runs within the month. An inactive pipeline is charged at $0.80 per month.

Thats all folks

The above information should allow you to make a fair cost estimate for running data pipelines in Azure Data Factory V2. However please note that you may face additional charges such as egress charges if you are copying data from an Azure database OR compute charges if you choose to create custom activities. These charges vary from case to case and are out of the scope of this blog post. The examples provided by Azure could prove useful in this regard.

2 Apps that SUPERCHARGED my productivity during lockdown

Sohaib Tariq — Wed, 28 Oct 2020 17:49:19 +0000

The Covid-19 pandemic was my first taste of working from home. The first week went by in a blur of confusion and while I didn't get much done, I still somehow ended up feeling exhausted and overworked.

It was clear that over the years, my productivity as an engineer had become so deeply entangled with my presence in a professional setting, that I would have to rethink my approach to work in order to get anything done.

The Problem

Like any good engineer, I started out by breaking my problem into manageable chunks:

1. Tracking Time: I needed a way to keep track of the number of hours of actual productive work I did, minus the intermittent distractions that working from home brings.

2. Tracking Tasks: I needed to quickly write down and prioritize all the tasks that were discussed in the multitude of zoom calls. In addition to that, I also needed to keep track of small tasks that were weighing on my mind like following up on messages/emails I had sent out or points I needed to discuss with particular co-workers when they became available.

The Solution

After some research and trying out a bunch of different apps, I settled on 2 that would solve both of the aforementioned problems:

Toggl track (formerly toggl)
OneNote

My criteria was simple, I wanted a quick and dirty solution to my problems that was going to be inexpensive or, even better, FREE.

Toggl track

What I loved about toggl is how easy it is to get up and running and the fact that it has a useable free tier, unlike many freemium apps these days.
Toggl comes with mobile, desktop and web apps with real time data syncing between them. Oh and did I mention, its free !

I set up 3 very basic projects on toggl :

work
work break (for when I took a coffee break or phone break or watching random youtube videos break; while working)
personal productivity (non-work, but productive stuff, such as writing this article)

These enabled me to track just enough of my time to make me accountable and productive without it feeling like a chore.

OneNote

Chances are, you've probably used/ heard of OneNote before. But did you know you can make kanban boards in OneNote ? ... well sort of.

That was a game changer for me.

Overnight OneNote went from being an app I occasionally used for note taking to a staple app that always remains open on my laptop.

Sorry trello, you have been replaced.

With three columns set up for "Waiting", "Todo" and "Complete" items, I was able to build a very simplistic task tracking system that works for me.

I can jot down tasks during meetings in the "Waiting" section while they come in thick and fast. There is no limit on the length and type of content I can add to a task so I can mix text, images and doodles. Then, using OneNote's convenient ability to drag and drop bullet points, I can move the ones I want to work on into the "Todo" section.

Once done, I can simply drag a task, along with all associated data over to the "Complete" section. If I want to record the time at which a particular item is completed, a timestamp is just a right-click away.

The sheer simplicity of using OneNote for task tracking is just mind blowing for me. It makes me wonder why more people don't use it, specially since its free.

tl;dr

Using OneNote and Toggl track has allowed me to reach, or dare I say, surpass my prior levels of productivity. I'm feeling a lot less stressed out because I don't have to worry about remembering small tasks. I can navigate the distractions that come with working from home, satisfied in the knowledge that I wont forget anything. And at the end of the day, I'm no longer left wondering about where all the time went.

PS

If you have any productivity tips/hacks/ideas that have worked for you, please can leave them in the comments below or tweet them to me at @thesohaibtariq. As a self professed serial procrastinator, I need all the productivity advice I can get 😀

If you found this article even remotely helpful, it would mean the world to me if you could share it with your friends. I'm trying to give this blogging thing a go, positive feedback will help me persevere through the initial lonely phase.

Barebones Authorization for Hangfire Dashboard

Sohaib Tariq — Sat, 05 Sep 2020 14:47:04 +0000

I just set up a Hangfire dashboard for our .NET Core application. As a security measure, the hangfire dashboard only allows local requests. If you wish to access the dashboard on a production server, you need to set up some form of Authorization by creating an implementation of IDashboardAuthorizationFilter. Now, there are a number of ways to do this, depending on your reuirements. All we needed was a quick solution that would allow us to see our dashboard on our development server.

If you are also looking for a quick and dirty solution,here is how I did it :

First, create an implementation of IDashboardAuthorizationFilter. We provide this implementation in the form of an Action Filter called HangfireAuthorizationFilter

You are now required to implement the Authorize method. We will authorize our user using credentials supplied in the query string. We simply store a username and password in our appsetting.json and correlate the supplied credentials with these stored credentials. If they match, we store them in a session cookie and return true, completing the Auth process.

Why do we need a cookie you say? thats because hangfire periodically makes requests to the application server in order to provide realtime data. Therefore, although the first request will contain the credentials in the query string, all subsequent calls will not, and hence, they will return a 401 status code. We fix this by including a session cookie in the reponse of our auth call. All subsequent requests made in that same session will contain this cookie and we can reauhtorize each incoming request.

public bool Authorize(DashboardContext context)
        {
            var httpContext = context.GetHttpContext();

            string userName = _settings.UserName;
            string password = _settings.Password;

            //if user has already logged in, in this session, subsequent requests will read credentials from cookie
            if (httpContext.Request.Cookies["user"] != null && httpContext.Request.Cookies["pwd"] != null
                && httpContext.Request.Cookies["user"].Equals(userName) && httpContext.Request.Cookies["pwd"].Equals(password)) {              

                    return true;
                }

            else if (context.Request.GetQuery("user") != null && context.Request.GetQuery("pwd") != null
                    && context.Request.GetQuery("user").Equals(userName) && context.Request.GetQuery("pwd").Equals(password))
            {
                //Store credentials in cookie so that subsequent requests dont require them
                httpContext.Response.Cookies.Append("user", userName);
                httpContext.Response.Cookies.Append("pwd", password);
                return true;
            }

            else return false;
        }

Next, simply include you implemenation of IDashboardAuthorizationFilter in your startup file and youre done.

 app.UseHangfireDashboard("/hangfire", new DashboardOptions
                {
                    IsReadOnlyFunc = (DashboardContext context) => _hangfireSettings.MakeDashboardReadonly,
                    Authorization = new[] { new HangFireAuthorizationFilter(_hangfireSettings)}

                });

Exponential backoff for AWS Lambda

Sohaib Tariq — Sat, 05 Sep 2020 14:43:10 +0000

I recently set up a Lambda function that reads data from an SQS Queue
and makes an API call to one of our microservices.
Naturally, this calls for an error handling mechanism, considering that the microservice
could be down or unresponsive.

AWS Lambda provides its own retry mechanism where a message is picked up from the queue by the Lambda
consumer and becomes invisible to other consumers for a specific duration called the visibility timeout.
If the consumer completes execution successfully, it automatically deletes the message from the queue.
In case of unsuccesful execution (such as a Runtime Exception), the approximate receive count
of the message is incremented and it becomes available to other consumers after the visibility timeout passes.
The number of times a message can be re-read from the queue
before it is finally sent to a Dead Letter Queue(DLQ) is configured in the Redrive policy of the
SQS Queue and is tracked via the approximate receive count.

This retry mechanism was not exactly what I had in mind for our use case. I was thinking along the
lines of a backoff strategy that keeps retrying the API call with exponentially increasing wait time;
finally sending the message to a DLQ after a set number of retries. This would give us ample time to
fix any issues with our miscroservice and prevent it from being bombarded with failing API calls.

This is what I ended up with:

First, a very basic Java function to calculate the exponential wait time, given the number of
retries recvCount:

        int randomInt = rand.nextInt(60);
        Long result = new Double(Math.pow(2, recvCount)).longValue() + 30 +randomInt;  //adding jitter to new random visibility timeout

Notice the addition of randomInt. That is 'jitter'. A bit of randomness. I read about it in some
documentation by Google Cloud and included
it as a good practice.

Next up, set the visibility timeout of the message to the value that we just calculated above. The maximum value allowed by AWS is 43200 seconds
or 12 hours.

  sqs.changeMessageVisibility(queueUrl, msg.getReceiptHandle(), newVisibilityTimeout.intValue());

Finally, we check the response to our API call. If it is a 400 or 500 series response, we throw a Runtime Exception and change the visibility timeout of the
message. This is
the easiest way I could come up with to signal unsuccessful execution of the Lambda function. Plus, we can only throw unchecked exceptions
in our handler method.

...
// api call
...  
 if (response.getStatusLine().getStatusCode() >= 400){
                            new ExponentialBackoff().setVisibilityTimeout(msg);
                            throw new RuntimeException("Request to server failed");
                        }

ExponentialBackoff is my utility class where the code that calculates and sets the visibility timeout lives. It also has some other
utility functions that are not essential for this demonstration.

There you have it; A bare bones exponential backoff implementation for AWS Lambda.

Replicating Azure SQL DB Instances on the fly for SaaS apps

Sohaib Tariq — Fri, 04 Sep 2020 16:33:31 +0000

If you are building a multi-tenant application on Azure, you may have to spin up Azure Sql instances on the fly when onboarding new tenants. I felt this would be a fairly common use case, but to my surprise, I failed to find any articles relating to this exact topic. After going through a number of different articles, I am compiling my learnings here with the hope that someone else will find this post useful.

First off, you need an Azure SDK, and not just any SDK, a ‘management’ SDK. You will find it here

The first task, as you may have guessed, is Authentication.

var authContext = new AuthenticationContext(<authority>); //"https://login.microsoftonline.com/"
var credential = new ClientCredential(<appId>, <appSecret>);
var authResult = await authContext.AcquireTokenAsync(<resource>, credential); //“https://management.azure.com/”
var token = new TokenCredentials(authResult.AccessToken, "Bearer");

Once you receive an authentication token, you can move to the next step: instantiating a SqlManagementClient.

SqlManagementClient mgmtClient = new SqlManagementClient(credentials);
mgmtClient.SubscriptionId = <subscriptionId>

I have a database already set up which contains some common data which all tenants need access to. The objective is to replicate this database every time a new tenant is added to the platform.
In order to achieve this, we will need to provide an identifier for the existing database along with the resource group, server, service tier, and Sku information for the new Database to be created.

client.Databases.CreateOrUpdate(resourceGroup, server, dbName, new Database(
                location: <location>, //centralus
                sku: new Sku(<skuName>, <tier>), //S0 , Standard
                createMode: "Copy",
                <sourceDatabaseId>)) // /subscriptions/.../resourceGroups/.../providers/Microsoft.Sql/servers/.../databases/<DB name>

The CreateOrUpdate method will return a Database from which you can retrieve the database name or resource ID as required.

And there you have it, how easy was that ? Took me a good few hours to figure it out 😅