DEV Community: Rutika Khaire

From a 13.5-Hour Mock to a 3-Hour Production Cutover

Rutika Khaire — Mon, 29 Jun 2026 12:24:19 +0000

We Planned for a Weekend Upgrade. A Mock Run Showed Us How Close We Were to a Weekend Disaster.

Every IT team has been here. A critical business system needs upgrading. The vendor says it's straightforward — run the installer, verify, go live. Leadership wants it done in one maintenance window. Your gut says it won't be that simple, but you can't explain why without sounding like you're stalling.

This is the story of how a single mock upgrade turned what could have been a stressful weekend into a calm, 3-hour production cutover — and why "just run a test first" might be the most undervalued practice in enterprise IT.

The Situation Most IT Teams Will Recognize

Our client is an association that runs their entire operation — membership, events, orders, finances, and reporting — on a single platform. The platform was two versions behind. Upgrading meant touching the system nearly every employee relied on daily.

The plan looked simple: upgrade from version A to B, then B to C, over a weekend. Users log off Friday evening, we upgrade Saturday and Sunday, everyone logs in Monday morning on the new version. The vendor's documentation suggested roughly two hours per version.

We insisted on running a mock upgrade first.

That decision saved the project.

What the Mock Revealed: The Installer Was the Easy Part

The installer finished in 40 minutes.

Everything looked perfect.

No errors.

No warnings.

If we'd been doing this in production, we probably would have celebrated and moved confidently to the next step.

Then we tried to launch the application.

It wouldn't start.

That was the moment we realized the upgrade wasn't going to be about installing software—it was going to be about understanding an entire application ecosystem.

The upgraded application expected newer runtime libraries, while several background services, supporting components, and custom modules still referenced older versions. Nothing failed during installation—the failures only appeared as each component started independently.

This wasn't one issue.

It was five separate dependency and configuration issues spread across different parts of the application stack. Each component failed differently, and the error messages often pointed us toward the wrong solution before we uncovered the real cause.

Over the next eight hours we documented every issue, refined every fix, updated our runbook, and noted exactly which configuration files needed changes across each application component.

What would have happened without the mock

We would have discovered these issues at around 2 PM on Saturday with users expecting the system back on Monday morning. The weekend would have turned into continuous troubleshooting, increasing the likelihood of extending downtime or making a difficult rollback decision under pressure.

What actually happened

When the real upgrade weekend arrived, every issue had already been solved.

The entire production cutover finished in under three hours.

The Five Things We Learned (That Apply to Any System Upgrade)

1. The Vendor's Upgrade Guide Is a Starting Point, Not a Playbook

Vendor documentation describes the happy path. It assumes clean environments, standard configurations, and minimal customization.

Real production systems rarely look like that.

Our client had of custom code, integrations, scheduled jobs, and environment-specific configurations. The vendor's documentation wasn't wrong—it simply couldn't anticipate every customer's implementation.

What to do instead

Treat the vendor guide as your baseline. Then build your own runbook that reflects your environment, your integrations, your customizations, and your validation process.

2. Mock Upgrades Aren't Rehearsals — They're Discovery

We didn't perform the mock to practice.

We performed it to discover everything we didn't know.

That distinction changes the entire mindset.

A rehearsal assumes you already understand the process.

A discovery exercise assumes there are unknown problems waiting to be uncovered.

That means budgeting more time, assigning experienced troubleshooters, documenting every surprise, and expecting to update your runbook throughout the exercise.

Our mock took approximately 13.5 hours.

Our production cutover took less than three.

The difference wasn't faster execution.

It was the absence of surprises.

3. "Successful Installation" Is a Dangerous Milestone

The installer completing successfully creates a false sense of confidence.

In our project, the installer succeeded every single time.

The failures appeared later:

when the application started
when background services initialized
when scheduled jobs executed
when integrations connected
when users exercised real business workflows

The milestones that actually matter

Can users log in?
Can they perform their daily work?
Can they create transactions?
Can reports run successfully?
Do scheduled tasks execute?
Do integrations still communicate correctly?

If your validation ends with "the application launches," you've only tested a small part of what could fail.

4. Every Component Has Its Own Configuration Universe

Modern enterprise applications are rarely a single executable.

They're collections of web applications, Windows services, scheduled jobs, APIs, reporting engines, and supporting utilities that often maintain their own configuration files.

During our upgrade, fixing the main application wasn't enough.

Next the order processing module failed.

After that, a background task failed.

Then another service exposed a slightly different variation of the same dependency issue.

The lesson was simple:

Map your application's complete topology before the upgrade.

Know every executable, every service, every scheduled process, every configuration file, and every integration point.

Then verify your fixes across all of them—not just the first application users see.

5. Your Rollback Plan Should Require Almost No Action

Many rollback plans begin with restoring backups.

Ours began with not changing production until we were confident.

The existing production server remained untouched throughout the project.

The upgrade happened on a separate server.

DNS, connection strings, and user access continued pointing to the existing environment until every validation step had passed and the client approved the cutover.

If something unexpected had happened—even after hours of work—we simply wouldn't have switched users to the new environment.

No emergency restores.

No rushed database recovery.

Just another business day on the existing system.

That design dramatically reduced both operational risk and decision pressure during the maintenance window.

The Uncomfortable Math

	Without Mock	With Mock
Planning effort	2 days	2 weeks
Production downtime	Unknown	Under 3 hours
Confidence during cutover	Low	High
Unknown issues	Many	Mostly eliminated
Monday morning	Fingers crossed	Boring (which is exactly what you want)

Yes, the mock added time to the project.

It also removed uncertainty.

Those 13.5 hours weren't extra work.

They were production troubleshooting completed safely before anyone depended on the system.

Our Final Production Runbook Looked Something Like This

Verify backups and rollback readiness.
Upgrade to Version B.
Apply documented dependency and configuration updates.
Validate application services.
Upgrade to Version C.
Reapply required configuration changes.
Execute technical smoke tests.
Execute client business test cases.
Approve cutover.
Switch users to the upgraded environment.
Monitor critical services after go-live.

By production weekend, none of these steps required improvisation.

What We'd Do Differently Next Time

No project is perfect.

Looking back, we'd improve three things.

We'd ask the client for business test cases much earlier. Our technical smoke tests were good, but only the client truly understood every business workflow that needed validation.
We'd automate the configuration updates. Applying XML and configuration changes manually worked, but scripting them would reduce both time and the possibility of human error.

The Takeaway

The least exciting part of a successful upgrade should be the production cutover.

If your maintenance window is full of surprises, those surprises usually didn't begin that weekend—they began weeks earlier when the decision was made to skip discovery.

Run the mock.

Find the ugly.

Document everything.

Refine the runbook.

Make the production upgrade boring.

Because the best infrastructure projects aren't remembered for dramatic recoveries.

They're forgotten because nothing unusual happened at all.

That's exactly how successful upgrades should feel.

My First Data Governance Project: From Confusion to Clarity (Lessons I Learned the Hard Way)

Rutika Khaire — Mon, 30 Mar 2026 15:52:14 +0000

When I was asked to work on Data Governance for the first time, I had no idea where to start.

Honestly, I was completely blank.

Like most beginners, I did what we all do:
I watched YouTube videos, read blogs, and tried to understand frameworks. Everything sounded important-and complicated.

So I decided to create a Data Governance plan.

With the help of AI tools, I quickly built something that looked impressive-structured, detailed, and “complete.”

But during the review, I faced a simple problem:

👉 I couldn’t explain what I had written.

That moment changed everything.

Lesson 1: If You Don’t Understand It, You Can’t Implement It

Using AI gave me speed-but not understanding.

I realized that Data Governance is not about creating fancy documents. It’s about clarity.

Lesson 2: Don’t Start with Complexity

My initial mistake was trying to build a “perfect” governance model from day one.

After a few reviews, I learned:

Keep things simple
Use clear language
Focus on what actually matters

Lesson 3: Start Small

Instead of complex frameworks, I shifted to:

Simple governance plans
Basic role definitions
Minimal, clear processes

And suddenly, everything became easier to explain—and implement.

Lesson 4: Always Ask ‘Why?’

One question helped me more than anything else:

👉 Why do we need Data Governance?

This helped me focus on:

Data quality
Consistency
Better decision-making
Reducing confusion

If something didn’t answer “why,” I removed it.

Lesson 5: Data Governance is About People, Not Just Documents

While preparing for stakeholder discussions, I realized:

People don’t care about frameworks-they care about impact.

So instead of explaining governance models, I focused on:

What problems it solves
How it helps the business
Why it matters to them

Lesson 6: Tell a Story, Not Just a Plan

When I prepared my demo, I didn’t just present documents.

I explained:

The current challenges
What could go wrong without governance
A simple way forward

That made all the difference.

Final Thought

My first Data Governance experience wasn’t perfect—but it taught me something important:

👉 Start simple. Understand deeply. Then build.

[Boost]

Rutika Khaire — Wed, 24 Dec 2025 06:25:02 +0000

Behind the Scenes of Data Ingestion: How Small Issues Cause Big Headaches

Rutika Khaire ・ Dec 24 '25

#dataingestion #medallion #dataengineering #datafactory

Behind the Scenes of Data Ingestion: How Small Issues Cause Big Headaches

Rutika Khaire — Wed, 24 Dec 2025 06:24:25 +0000

Data ingestion is often treated as a solved problem-until it breaks. What looks like a simple pipeline moving data from source to destination can quietly introduce inconsistencies, missing records, or silent failures that ripple across analytics and reporting systems.

In this blog, we’ll go behind the scenes of a real-world data ingestion architecture, explore common issues that arise, uncover their root causes, and share best practices to build more resilient pipelines.

Why Data Ingestion Matters

Data ingestion is the foundation of every data platform. When ingestion goes wrong:

Reports show incorrect numbers
Business decisions are based on stale or incomplete data
Engineers spend hours firefighting instead of building features

The most dangerous problems aren’t always the obvious failures—they’re the subtle ones that go unnoticed for weeks

Our Ingestion Architecture at a Glance

Azure Data Factory (ADF) for orchestration
SQL Server Views as the source layer
Medallion Architecture
Bronze: Raw ingested data
Silver: Cleaned and transformed data
Gold: Business-ready datasets

Watermark-based incremental loads to process only changed data

This architecture is scalable and efficient—but only when implemented carefully

Issue #1: Missing Deletes – “Why Is This Customer Still Active?”

Real-World Example

A customer account is deleted in the CRM system due to GDPR requirements.
However, the sales dashboard still shows the customer as active weeks later.

Impact

Compliance risk
Incorrect KPIs
Loss of trust from business users

What Actually Happened

Source System        Target Table
Customer Deleted  →  No Delete Captured

Fix (Realistic Approach)
Source (CDC Enabled) │ ├── Insert ├── Update └── Delete ──► Target Table

Enable CDC or soft-delete flags
Add delete-handling logic during MERGE
Periodically reconcile record counts

Issue #2: Race Conditions – Incremental Loads Miss Late-Arriving Updates

Real-World Example
Incremental loads often rely on a watermark column such as LastModifiedDate.

Example:

Last successful watermark = 10:00 AM
A record is updated at 9:55 AM
That update arrives late due to upstream delays

Because the timestamp is older than the watermark, the incremental query skips it entirely.

Why This Is Dangerous

No pipeline failure
No alert
Data is permanently missed unless a full reload happens

This is a silent data loss scenario.

Common Root Causes

Eventual consistency in source systems
Batch updates applied late
Reliance on timestamps without buffering

How to fix it

Use a lookback window (e.g., watermark − 5 or 10 minutes)
Prefer CDC or version-based sequencing
Reprocess recent partitions regularly

Issue #3: Over-Aggressive Filtering – “Where Did My Data Go?”

Real-World Example

A filter is added to exclude test users:

WHERE username NOT LIKE '%test%'

Suddenly, legitimate users like contest_winner disappear from reports.

Hidden Damage

No pipeline failure
No alert
Business notices weeks later

Better Filtering Strategy
Incoming Data │ ├── Valid Users ──► Continue └── Test Users ──► Logged + Reviewed
How to fix it

Use exact match lists (e.g., TestUserList)
Log filtered records
Validate filters with production samples

Best Practices

Resilient Ingestion Design
Detect Change │ Validate Data │ Apply Idempotent Load │ Monitor + Reconcile
Operational Guardrails

Row-count reconciliation
Data freshness checks
Alerting on anomalies—not just failures

Final Thoughts

Most data ingestion failures don’t crash pipelines—they quietly corrupt trust. Missed deletes, late-arriving updates, and poorly managed watermarks often go unnoticed until business users start asking uncomfortable questions.

The takeaway is simple: build ingestion pipelines for real-world behavior, not ideal scenarios. Expect delays, partial failures, and messy data. Validate early, reconcile often, and treat control logic like watermarks as first-class citizens.

In data engineering, it’s rarely the big failures that hurt the most—it’s the small ones you didn’t see coming.

My First Open Source Contribution

Rutika Khaire — Mon, 06 Oct 2025 04:55:05 +0000

For the longest time, I believed contributing to open source was something only “seasoned developers” could do. I pictured huge, complicated codebases and intimidating review processes. But recently, I made my very first open source contribution—and it turned out to be one of the most rewarding learning experiences I’ve ever had.

In this post, I’ll walk you through my journey: how I got started, what I contributed, what I learned, and why you should try it too.

How To Find the Right Project

The first challenge was figuring out where to contribute. GitHub is full of amazing repositories, but it’s easy to feel overwhelmed. I started by looking for beginner-friendly issues using labels like good first issue and help wanted.

Eventually, I came across a project called first-contributions.

Understanding the Workflow

Before writing any code, carefully read the project’s README and CONTRIBUTING.md file. These documents are gold for beginners—they explain how to set up the project, the coding style, and the process for submitting changes.

Making My First Contribution

My first contribution wasn’t something massive—it was improving a README section.

Here’s the process I followed:

Forked the repo
Cloned it to my local machine
Created a new branch for my fix
Made the changes
Committed and pushed the code
Opened a Pull Request (PR)

What I Learned

Here are my biggest takeaways from this first step into open source:

Open source isn’t as scary as it looks—maintainers usually want you to succeed.
Even small contributions make a difference. Don’t underestimate fixing a typo or improving documentation.
Reading project guidelines before jumping in saves a lot of time.
Collaboration and communication matter as much as writing code.

If you’re thinking about contributing to open source, here’s my advice:

Start small. Documentation fixes or simple bugs are great entry points.
Look for “good first issue” labels. They exist specifically to help beginners get started.
Don’t be afraid of mistakes. PR reviews are part of the learning process.

If you’ve made your first contribution, I’d love to hear your story too!

How to Scale Your Application to Handle Peak Loads and Increase Throughput

Rutika Khaire — Wed, 18 Dec 2024 12:22:05 +0000

Introduction: The Party Planner’s Dilemma

Imagine you're hosting a party. You plan for 50 guests, but suddenly 200 people show up. You need more food, more chairs, and a way to keep everyone happy without chaos. This is exactly what systems face during peak loads. How to keep everything running smoothly?

1. Scaling Up: Adding Chairs to the Party

Scaling your system during peak loads is like finding extra chairs when more guests arrive. Scaling ensures that the application has enough resources to handle increased demand.

Here’s how to do it with Azure:
Vertical Scaling (Adding More Power):
Adding resources to existing servers, like upgrading a chair to a sofa.
Example: If your Azure App Service instance needs more resources, scale up to a higher pricing tier.

How to Do It?

Navigate to your App Service.
Select Scale Up (App Service Plan) and choose a higher tier (e.g., Standard to Premium).

Horizontal Scaling (Adding More Machines):
Adding more servers, like borrowing chairs from your neighbors.
Example: During a flash sale, e-commerce platforms add more servers to handle the load.

How to Do It?

Go to your App Service.
Select Scale Out (App Service Plan) and configure autoscaling rules based on metrics like CPU usage or request count.

Azure-Specific Tip: Use Azure Monitor Autoscale to define rules like “Add one instance if CPU usage exceeds 70% for 5 minutes.” Refer this for more detailed information.

2. Thresholds: Knowing Your Party’s Limit

Every system has a breaking point—its threshold. It’s like knowing your living room can only hold 30 people before things get cramped. Thresholds are the breaking points of a system. Knowing them helps avoid disasters.

Identify Thresholds in Azure:
Example: Your SQL Database might have a DTU (Database Transaction Unit) limit of 1000.

How to Do It in Azure:

Use Azure Monitor to track resource metrics like DTU usage, CPU percentage, and memory utilization.
Set up Alerts in Azure Monitor for critical thresholds. For instance, send an email when database utilization exceeds 85%.

Set Alarms:
Example: Monitor your Azure Function’s execution time or failures.

How to Do It in Azure:

Navigate to Azure Monitor → Alerts → New Alert Rule.
Configure conditions like “Trigger an alert if CPU utilization > 80% for 5 minutes.”

3. Handling Thresholds: Avoiding the Crash

What happens when you hit a threshold? Use these strategies with Azure:

Graceful Degradation:
Example: Serve cached product pages if your backend API is overloaded.

How to Do It in Azure:

Use Azure Front Door to cache and serve static content from edge locations.
Configure Azure CDN to offload traffic from your backend during peak times.

Queueing Systems:
Example: Process orders in batches during peak traffic.

How to Do It in Azure:

Use Azure Service Bus or Azure Queue Storage to enqueue requests for asynchronous processing.
Implement a Logic App or Azure Function to process messages from the queue.

4. Throughput: Keeping the Line Moving

Throughput is the speed at which you can serve your guests. In tech terms, it’s how many requests or tasks your system can handle per second.
Here’s how to improve it with Azure:

Optimize Database Performance:
Example: If your Azure SQL Database is slow, optimize queries and scale up.

How to Do It in Azure:

Use the Query Performance Insight feature in Azure SQL to identify slow queries.
Scale your database using the DTU-based or vCore-based model.

Use Load Balancers:
Example: Distribute traffic evenly across multiple virtual machines.

How to Do It in Azure:

Set up Azure Load Balancer for Layer 4 (TCP/UDP) traffic.
Use Application Gateway for Layer 7 (HTTP/HTTPS) traffic, and enable autoscaling.

Implement Caching:
Example: Reduce repeated database calls for product details.

How to Do It in Azure:

Use Azure Cache for Redis to store frequently accessed data like product details or user sessions.

Compress Responses:
Example: Compress API responses to reduce latency.

How to Do It in Azure:

In Azure App Service, enable compression under Configuration → General Settings.

5. Put It All Together: The Ultimate Party Plan

Example Scenario:
You’re running a ticket-booking platform for a concert, and traffic surges at ticket release.

Scaling:

Use Azure App Service autoscaling to handle increased HTTP requests.
Scale your Azure SQL Database to a higher tier (e.g., Standard to Premium).

Thresholds:

Monitor DTU utilization for Azure SQL Database using Azure Monitor.
Set alerts for API response times exceeding 500ms.

Throughput:

Cache event details using Azure Cache for Redis.
Use Azure Front Door to route traffic globally and serve cached content faster.
Implement message queues with Azure Service Bus to process ticket purchases asynchronously.

Conclusion: Keep the Party Going!

Azure provides an array of tools to manage peak loads, thresholds, and throughput efficiently. With features like autoscaling, Azure Monitor, Service Bus, and Front Door, you can ensure your system remains reliable and performant—even under heavy demand.

By implementing these best practices, your applications will stay resilient, agile, and ready to handle any traffic surge.

This blog is designed for novices to scaling or those with intermediate skills who want to strengthen their understanding of handling peak loads, thresholds, and throughput. Using relatable analogies and real-world scenarios, I have tried to break down these concepts into easy-to-grasp sections.

Reference Links
Autoscaling

Boost Productivity and Data Accuracy: Essential Tactics for Software Integration

Rutika Khaire — Thu, 06 Jun 2024 13:49:22 +0000

Understanding Software Integration

Nowadays, integration has become a fundamental aspect of software development. Every organization strives to remain competitive in today's fast-paced world. Integration enables different systems to communicate seamlessly and share information efficiently.

The most effective software integration strategies involve thorough planning, clear communication, and collaborative efforts among teams.

Why do we need Software Integration

Enhanced Workflow Efficiency: Automating the flow of information between systems speeds up business processes and minimizes delays.
Unified Data Management: Integrated systems ensure that data is consistent and up-to-date across all platforms, reducing the risk of errors caused by data discrepancies.
Real-time Data Access: Changes made in one system are instantly reflected across all integrated systems, ensuring all users have access to the latest information.
Automation of Tasks: By automating routine and complex processes, integration frees up employees to focus on higher-value activities.
Reduced Operational Costs: Integration reduces the need for maintaining multiple systems and data silos, lowering IT and operational expenses.

Key Integration Tactics

What it is: Key Integration Tactics are strategies and approaches used to effectively connect different software applications and data sources. They ensure smooth information flow and collaboration between these systems.

Why they matter: Without proper tactics, integration projects can become complex, error-prone, and ultimately fail to deliver the desired benefits.

Different methods of Integration

API Integration: This allows different software systems to communicate with each other via APIs.
Middleware: There are various middleware that can be employed to facilitate communication between systems.
Webhooks: This helps in real-time data exchange.
Microservices: This breaks down applications into smaller, interconnected services.

Planning and Choosing the Right Tactics

Data mapping and standardization
This is one of the crucial tactics to ensure smooth and accurate communication between different software applications. There are several benefits that include data accuracy, efficiency, maintainability and interoperability.

Reconciliation fields
Though reconciliation fields are not a universally defined concept in software integration, but they play a crucial role in ensuring data consistency between integrated systems.

What are reconciliation fields?

Reconciliation fields are specific data points used to compare and identify discrepancies between data sets in integrated systems. They act as a common ground for both systems to verify if the information they hold aligns.

Few advantages of using reconciliation fields include pinpointing the source of the error and taking corrective actions by analyzing discrepancies in such fields. Reliable data from reconciled systems leads to better insights and informed decisions.

Error Handling and Security
Implementation of robust error handling and security measures is another crucial tact to ensure smooth integration. It is a process of anticipating, detecting, and recovering from unexpected issues that may arise during data exchange between integrated systems.

Few techniques to implement it include:

Logging errors: This is one of the very first go-to option in order to tackle issues and understand the root causes behind the problems.
Retry logics: You can implement logics that would re attempt data transfers after some delay.
Alerts: There are different ways to define alert systems to notify respective teams about issues to take necessary and appropriate actions.

Testing and Monitoring
Thorough testing and ongoing monitoring are the most essentials for ensuring the smooth operation and long-term success of your software integration project.

Testing in simple words means ensuring that the integration meets the requirements and the data exchange functions accurately. There are various types of testing like below that really help in ensuring that systems meets all expectations.

Unit testing
Integration testing
Regression testing
Smoke testing
Functional and Non-Functional testing
System testing
Load testing
Negative testing

Monitoring on the other hand involves continuously observing the performance and the health of the integrated system after it is deployed. There are different metrics available that help in tracking the exchange of data and identify failed transactions.

Future Trends in Software Integration
Software integration is constantly evolving, driven by advancements in technology and the growing need for seamless data exchange across an ever-expanding application ecosystem. Here's a glimpse into some key trends shaping the future of software integration:

The Internet of Things (IoT): Integrating with IoT devices is going to become crucial in order to manage and analyze data collected from sensors and intelligent devices.

Artificial intelligence (AI) and Machine Learning (ML): AI and ML can be used to automate tasks within the integration process, like data mapping and error handling. Additionally, AI-powered integration platforms can learn and adapt over time, optimizing performance.

Low-Code/No-Code Integration Tools: There is a rise in tools that empower business users with limited or no coding experience to build basic integrations.

These trends highlight a future where software integration becomes more accessible, automated, and secure.

Looking Ahead
By adopting the key tactics explored in this blog, you can approach software integration projects with confidence. Remember, successful integration goes beyond just connecting systems. It's about establishing a well-defined strategy, utilizing the right tools and methods, and prioritizing ongoing maintenance and monitoring.

Building Real-time Notifications with Azure Web PubSub and Azure Functions

Rutika Khaire — Fri, 01 Dec 2023 04:31:18 +0000

Hello people!!! Today I am writing this blog to share my experience of using the Azure Web Pub-Sub service within Azure Function to send real time messages to my React application.

Many a times, we have a requirement to do some processing asynchronously behind the scenes so that we don't block user's activities on the site.

I came across a similar situation where I was working on a chatbot application. This application has a feature that allows a user to upload a file and then train the file on a specific model and then once the file is trained successfully, the user should be able to query the trained data and get responses from it. Very much similar to what we do in chatgpt.

So the requirement was as below:

User uploads a file in the chat box
Clicks the send icon
File is uploaded to Azure storage
User receives a success message saying - "File uploaded successfully"

Now, while the file is getting trained asynchronously, the user should be able to continue his/her conversation with rest of the available data.

So, then when should the user query for the data that he/she just uploaded as a file??? The answer to this is whenever the file is processed successfully behind the scenes, there should be some indication to the user saying that your file is now trained successfully and is ready to serve requests.

Here comes the concept of a PubSub notification. When the file is processed or trained, the PubSub service will publish a message. That is what Pub stands for. And this published message should be subscribed by some service/resource to receive it which is termed as Sub. So PubSub stands for Publish and Subscribe.

How to Subscribe to a message

You can use WebSocket connection from your frontend application to stay connected to your publisher service so that you can receive real time messages.

In my case, the publisher is Azure function and the subscriber is React application.

Create an Azure Web PubSub service

Go to Azure portal
Type Web PubSub Service in the search bar
Click on the Create button
Select Web PubSub and then add the necessary information and create the service

Below is how the service will look like when created successfully

Go to the Keys section and see that there are connection strings available. The primary connection string is to be used while establishing connection. So keep a note of it.

Azure Function

An Azure function is an event driven, serverless compute where you can write less code to save costs and maintain the infrastructure. Read more here

So, now my azure function is responsible to read the file from Azure storage and then pass the file to an API endpoint to train it.

As, the function is event driven, I have to add two functions in my function app. One is the Blob Storage Trigger which executes whenever a new file is uploaded to the specific storage. And the other is the HTTP Trigger which helps in keeping the Websocket http connection open from the React app to the Azure function.

You have to create a function app in Azure and then add the functions to that function app.

Steps to create the function app:

Go to Azure portal
Type Function App in the search bar
Click Create
Add the necessary and required information to create the function app
After the app is created, go to Configuration and add a setting for the Web PubSub connection string as below. Here you would need the primary connection string that was created when PubSub service was created.

Below is my code structure for Azure function:

The AiTrain function is a blob storage trigger function and Negotiate is the HTTP trigger function. I am using typescript.

Negotiate function

This function is responsible to return the connection object to the subscriber. So, the index.ts file in this function has below code.

module.exports = function (context, req, connection) { context.res = { body: connection }; context.done(); };

And in the function.json file, it has below settings:

{
  "bindings": [
    {
      "authLevel": "anonymous",
      "type": "httpTrigger",
      "direction": "in",
      "name": "req",
      "methods": [
        "get",
        "post"
      ]
    },
    {
      "type": "http",
      "direction": "out",
      "name": "res"
    },
    {
      "type": "webPubSubConnection",
      "name": "connection",
      "hub": "notification",
      "direction": "in"
    }
  ],
  "scriptFile": "../dist/Negotiate/index.js"
}

AiTrain function

This function is responsible to train the uploaded file and publish a message after success.

import { AzureFunction, Context } from '@azure/functions';

const blobTrigger: AzureFunction = async function (
    context: Context,
    myBlob: string
): Promise<void> {
    try {
        context.log('TypeScript Blob trigger function');

        const blobURI = decodeURIComponent(context.bindingData.uri);

        // file processing logic


        //Azure web pubsub actions
        const actions = {
            actionName: 'sendToAll',
            data: `File processing complete!`,
            dataType: 'text',
        };

        context.bindings.actions = actions;

        // No asynchronous operations, so we can return a resolved Promise
        return Promise.resolve();
        }
    } catch (error: unknown) {
        console.error(error);
    }
};

export default blobTrigger;

The function.json file has below code:

{
  "bindings": [
    {
      "name": "myBlob",
      "type": "blobTrigger",
      "direction": "in",
      "path": "files/{name}",
      "connection": "dev_STORAGE"
    },
    {
      "type": "webPubSub",
      "name": "actions",
      "hub": "notification",
      "direction": "out"
    }
  ],
  "scriptFile": "../dist/AiTrain/index.js"
}

This completes the first part of publishing a message from Azure function.

Subscribe to published messages

In order to subscribe the messages from the React application, all you need to do is establish a Websocket connection in your code.

Below is my react code that establishes a Websocket connection to the azure function.

useEffect(() => {
        const connect = async () => {
        const res = await fetch(`${API_URI}/api/negotiate`);
        const { url, accessToken } = await res.json();
        const ws = new WebSocket(url);
        ws.onopen = () => console.log('connected');
        ws.onmessage = (event:any) => {
            console.log(JSON.stringify(event.data));
            //Code Logic
        };

        };
        connect();
    }, []);

We have to call the negotiate endpoint to keep the React app connected to the Azure function.

And that's all. You can see the console messages to see the event object that has the data sent from the Azure function. You can choose any component to display this data as a notification to the user.

Conclusion

In conclusion, Azure Web PubSub when coupled with Azure Functions and a React application opens up exciting possibilities for real-time, scalable, and interactive web experiences. We've explored how seamlessly these components integrate to create a dynamic communication channel between the server and client, enabling instant updates and collaborative features.

Keep experimenting, pushing the boundaries, and discovering new ways to leverage Azure Web PubSub. The possibilities are vast.

I hope this blog has inspired you to dive deeper into the realm of real-time communication and discover the endless possibilities that await. Happy coding!

AWS vs. Azure: Comparison

Rutika Khaire — Fri, 29 Sep 2023 14:26:03 +0000

Introduction

Amazon Web Services (AWS) and Microsoft Azure are the two giants when it comes to cloud computing. The most critical business decision is to choose the cloud platform.

Not only cloud providers, but AWS and Azure have come a long way to provide technological services to revolutionize the way organizations operate and innovate.

In this comparison, I will share my understandings of the vast ecosystems they offer, the pricing models that govern their services, the global infrastructure, and the security measures they employ to protect sensitive data.

Ecosystems

Below are the Compute Services offered

AWS comprises of AWS Elastic Beanstalk, Amazon EC2 (Elastic Compute Cloud), AWS Lambda, Amazon ECS (Elastic Container Service), and more.

Azure comprises of Azure App Service, Azure Functions, Azure Kubernetes Service (AKS), Virtual Machines (VMs), and more

Then if we talk about storage then below are the Storage Services offered

AWS offers Amazon S3 (Simple Storage Service), Amazon EBS (Elastic Block Store), Amazon RDS (Relational Database Service), Amazon DynamoDB, and Amazon Redshift

Azure offers Azure Blob Storage, Azure File Storage, Azure Table Storage, Azure Disk Storage, and Azure Data Lake Storage

Let's see the Database Services

AWS offers Amazon RDS, Amazon Aurora, Amazon DynamoDB, Amazon Redshift, and Amazon Neptune

Azure offers Azure SQL Database, Azure Database for PostgreSQL, MySQL, and MariaDB, Azure Cosmos DB, and more

Pricing

Both AWS and Azure provide a free tier that you can use to try your hands on to explore their service offerings. Both also offer pricing calculators that you can use to make cost estimations as per your needs.

There is also pay as you go pricing that charges you based on the amount of time you use the resources. It can be per minute or per hour basis.

If you want to use the resources for a longer duration like for a year or more then both AWS and Azure offer special discounted rates which are termed as Reserved Instances

Global Infrastructure

Azure global infrastructure is made up of two key components—physical infrastructure and connective network components. The physical component is comprised of 200+ physical datacenters, arranged into regions, and linked by one of the largest interconnected networks on the planet.

With the connectivity of the global Azure network, each of the Azure datacenters provides high availability, low latency, scalability, and the latest advancements in cloud infrastructure—all running on the Azure platform.

The AWS Cloud has across 102 Availability Zones within 32 geographic regions around the world, with announced plans for 12 more Availability Zones and 4 more AWS Regions.

Security Measures to protect sensitive data

To manage access to resources, AWS offers IAM roles, users and group and in Azure, you can use Azure Active Directory

If you want to secure configurable data then AWS offers Secrets Manager and Azure provides Azure Key Vault

Conclusion

This is a brief comparison that I have shared based on my experience of working on Azure and AWS. There is still lot out there in Azure and AWS that you can explore by visiting their official documentation.

Thanks for reading!

AI-Powered Coding: A Look at GitHub CoPilot's Capabilities

Rutika Khaire — Thu, 13 Apr 2023 12:05:42 +0000

Let me give you an insight of a powerful tool called the GitHub CoPilot that has revolutionized the way we write code. And congratulations to all the developers as we have an AI powered assistant that will assist us in writing fast and efficient code.

The term "CoPilot" generally refers to a person who assists the primary pilot in controlling an aircraft. Similarly, GitHub CoPilot is the term that conveys the idea of an AI powered assistant for a developer. Like a co-pilot who helps the pilot in controlling an aircraft, GitHub CoPilot helps developers in controlling the code they write.

GitHub CoPilot's Capabilities

Increased Productivity

Isn't the goal of a developer to maximize productivity? Why not utilize a potent tool to enhance your productivity?

GitHub CoPilot provides suggestions and autocompletions thus automating repetitive tasks. While it won't write the entire code according to your exact specifications, it can still be a valuable time-saver, particularly when it comes to catching unintended syntactical errors. By preventing you from spending your productive hours fixing minor mistakes that you may have inadvertently made, it allows you to focus on more meaningful tasks.

Understanding Natural Language

One of the most impressive capabilities that I found is that it is able to understand plain English language and generate code for you.

For example, if you want to write a code to filter non matching data from two JSON objects then you can simply type the text in form of a comment and it will give the code.

//Filter out non matching personIDs from teamMembers and assignmentExists const filteredTeamMembers = teamMembers.filter((e:any)=> !assignmentExists.find((n:any)=>n.dataValues.PersonID === e.dataValues.PersonID));

Helps in reducing errors

It is able to recognize errors and correct those, thus saving lot of time.

Better quality of code

As it is able to give proper code suggestions, you can focus on improving the code quality. In my case, it gave me async await code suggestions which I feel is much better than a promise based code.

Conclusion

So these are few of the capabilities that this tool offers. While there are still a few limitations as well but it is still powerful enough.

You can explore this tool by creating a free account for 2 months after which it will charge you. If you are using Visual Studio Code editor, then simply install the extension and start coding.

For more info visit link.

The easiest way to use HubSpot Search API

Rutika Khaire — Fri, 24 Mar 2023 13:47:24 +0000

Whenever working on integrations, we basically use the APIs to retrieve the information. Some APIs are pretty straightforward to understand, so just by looking at the endpoint we can make out how to use it but sometimes, you need to dig in a bit to understand the usage.

I came across this situation in my application where I wanted to retrieve all the contacts from HubSpot based on a particular property value. For eg. If there is a contact property named Company_Affiliation and I want to retrieve all the contacts with a company affiliation value of say Fictional Company then how should I use the Search API?

The Endpoint

POST
/crm/v3/objects/contacts/search

The Input

const inputHubSpotSearchObject=
    {
        "objectName":"contacts",
        "limitValue":"100",
        "afterValue":"0",
        "filters":[
            {
                "value":req.body.company_name,
                "propertyName": "company_affiliation",
                "operator": "EQ"
            }

        ]
    }

The API call

const response = await axios.post(`${process.env.API_URL}/api/hubspot/search`,inputHubSpotSearchObject,
                          {
                          headers: {
                              "Content-Type": "application/json",
                              Authorization: `Bearer ${accessToken}`,
                          }});

The Response

Conclusion

I hope you found this article helpful. For more information visit link

Happy Coding!!!

Understanding the basics of Express Middleware

Rutika Khaire — Wed, 28 Dec 2022 09:23:24 +0000

In this article, we will understand what Express middleware is and how to use it in a NodeJS application.

As the name in itself is self explanatory, Express works as a middleware in an application. Typically an application is accessed from a client system to retrieve some information from a server. So, here there is a Request that is sent by the client to the server and there is a Response sent by the server back to the requesting client.

So, Express Middleware consists of functions that are executed after a Request is sent by client and before the Response is sent by the server. Something, that we want to execute in the middle of the Request and Response.

We can have multiple middlewares that can also modify the request object and then pass it to the next function in the sequence.

The base of Express is the Application object which is typically used as app. This app object exposes different methods and here we will take a look at the listen, use, HTTP methods (get, put, post etc.).

Create a server with app object

Whenever a website is created, it has a link / URL that we use to access the website. And as the website stays on a server, there is a client that comes into picture that calls the website's URL. So, now as the website is being called, there must be someone who is listening to the calls right? In this case, the server is the listener. And that is the purpose of the listen method of app. There is a port number where these calls from a client are getting listened by the server.

The syntax for creating a server using the app object is as below:

const express = require('express')

const app = express()
const port = 3000

app.listen(port, () => {
  console.log(`Example app listening on port ${port}`)
})

Now the above code will make the server up and running at the port 3000. When we run the application we can see the message given in the console.log as below.

Now, we need to define some endpoints (path/URI) in the code that will allow the users to call different pages of the website. For, eg. a website can have a Home Page, a Contact Page, an About Page and so on. Ideally, the URL would be something like https://somedomain/Home, https://somedomain/About, https://somedomain/Contact and so on. The part of the URL like /Home, /About, /Contact etc. is what is termed as the endpoint that we should be adding in our code using the app object. Here, the HTTP methods (get,put,post etc.) come into picture. Whenever we try to access the website's page, we get some information that is displayed in the browser. So, for the above mentioned pages, we should use the get method of app.

This is termed as Routing. Below is the definition of Routing from express official documentation.

Routing refers to determining how an application responds to a client request to a particular endpoint, which is a URI (or path) and a specific HTTP request method (GET, POST, and so on).

The basic structure of a route is as following:

app.METHOD(PATH, HANDLER)

Where:

app is an instance of express.
METHOD is an HTTP request method, in lowercase.
PATH is a path on the server.
HANDLER is the function executed when the route is matched.

Let's see how we can use the HTTP methods

Following example shows the get method:

app.get('/Home', (req, res) => {
  res.send('Welcome to the Home Page!')
})

The code snippet would be as below:

It can be accessed in the browser using URL http://localhost:3000/home and we can see the following message:

An important point to note here is that if we try to access the link as http://localhost:3000, we will get an error that says Cannot GET /

What does this error mean? It means that we have not specified the base path which is "/" in our code and we are still trying to access it in the browser.

Let us add the code for base path and see how it works. Following code snippet shows a new get method for base path.

app.get('/', (req, res) => {
    res.send('This is the base call!')
  })

The complete code till now looks like below:

In the browser now if we access the URL as http://localhost:3000/, we see the below output.

The purpose of app.use method

Now, let's understand what is the app.use method used for. From the definition app.use ->

Mounts the specified middleware function or functions at the specified path: the middleware function is executed when the base of the requested path matches path.

So, in a simple language, if we want to execute some logic before the a specific path is matched and the respective handler is executed, we can configure it using the app.use method.

For eg. app.use('/apple', ...) will match “/apple”, “/apple/images”, “/apple/images/news”, and so on. So, if we call "apple/images", it will first execute handler of "/apple" and then the handler of "/apple/images"

If we configure a middleware without a path then it gets executed for every request to the app. For eg. the following function will be executed for every request.

app.use(function (req, res, next) {
  console.log('Time: %d', Date.now())
  next()
})

Middleware functions are executed sequentially, therefore the order of middleware inclusion is important.

The sequential execution of functions is achieved using the next() function. As used in the above code snippet, we are using the next() call to ensure that the next function in the sequence is executed.

Following code illustrates this:

NOTE: You have to call the URL http://localhost:3000/ in the browser to see the output

Let's see what happens if we remove the next() call from the function.

See, the log that we were seeing earlier after the Time was displayed (This is the base call!) does not appear now. This simply means that the control does not pass to the next function in sequence. So, it is important to decide the sequence of executing the functions and add those appropriately.

I hope you have enjoyed this article and hope that it helped you in understanding the basics of Express. For more information you can definitely go through the official express documentation here.