DEV Community: Vaibhav Bhutkar

Migrating from from Pre-Fabric to Microsoft Fabric – A Comparative Guide.

Vaibhav Bhutkar — Tue, 01 Jul 2025 11:28:46 +0000

Why this Matters: In today’s data-driven world, enterprises are flooded with tools: Azure Synapse, Data Lake, Power BI, Azure Data Factory, and many more. Managing them all in a cohesive, cost-effective, and high-performing architecture often feels like fitting together pieces of a complex puzzle or game.
Microsoft Fabric – a unified, SaaS-based data platform that aims to simplify the entire data lifecycle, from ingestion to insights. In this blog, we explore what it means to move from a pre-Fabric (traditional Azure) architecture to Fabric, and what benefits, challenges, and changes come with it.

Pre-Fabric Architecture
Before Fabric, a typical Microsoft data architecture looked like this:

Azure Data Factory (ADF): for data ingestion and transformation.
Azure Data Lake / Blob Storage: for storing raw data.
Power BI: for visualization.
Multiple identities, billing, and disconnected services.

Challenged with these:
Cost Optimization - Paying separately for each service, often overprovisioned.
Integration - Too many moving parts, difficult to manage end-to-end pipelines.
Learning Curve - Requires knowledge of multiple tools and their integration.
Collaboration - Teams often work in silos: engineers, analysts, and business users.

Microsoft Fabric:
Microsoft Fabric is an end-to-end analytics SaaS platform that unifies six core workloads:

Data Engineering
Data Factory
Data Science
Data Warehousing
Real-Time Analytics
Power BI

It’s built on OneLake, a single storage layer accessible by all workloads, and uses notebooks, pipelines, and datasets under one umbrella.
Core concepts Behind it -
OneLake: One storage for all data – governed, secure, and universally accessible.
Lakehouse & Warehouse Models: Unified for both engineering and business use.
Workspaces: Seamless collaboration with role-based access.

Fabric as Microsoft 365 for data. Just like Word, Excel, and Teams are tightly integrated, Fabric integrates all analytics tools in one platform.

If Microsoft provide all these features in one umbrella, then we need to consider some considerations before directly jumping to fabric.

Existing Resources: Are you heavily reliant on Synapse, ADF, etc.?
Data Volume & Type: Fabric is optimized for structured/semi-structured; advanced unstructured use cases still maturing.
Security & Compliance: Map your policies into Fabric’s workspace and OneLake controls.

Benefits if we Move to Fabric
Unified Platform: No more jumping between Azure resources or services.
Low Costing: Capacity-based pricing; easier to control.
Faster Time to Built: less code with more actions.
One environment for data engineers, analysts, and scientists.

Do We Really Need All the Data to Make Our Decisions?

Vaibhav Bhutkar — Sun, 05 Jan 2025 13:41:37 +0000

In the era of data-driven everything, we often hear the mantra, “The more data, the better the decision.” Businesses, governments, and even individuals are constantly amassing and analyzing mountains of data. But do we truly need all that data to make sound decisions? Let’s dig deeper, challenge this idea, and explore an alternative perspective.

Too Much Data, Too Many Choices

Having an abundance of data can sometimes create more confusion than clarity. Imagine a retailer examining customer purchase behavior. Does the business need every single click, scroll, and hover to optimize their strategy? Probably not. Often, the key insights lie in just a fraction of the data.

Consider a recent project I worked on involving eCommerce analytics. The client initially wanted to analyze every conceivable data point across millions of transactions. However, by focusing on just five metrics—cart abandonment rate, average order value, repeat customer rate, product views, and search-to-purchase conversion—we achieved actionable insights faster, with far less complexity.

Example: Data Ingestion in IT

Let’s pivot to an IT-specific example—data ingestion in a cloud-based architecture. Imagine an enterprise that ingests terabytes of log data from IoT devices daily. The raw data includes timestamps, device IDs, error codes, environmental metrics, and more. Initially, the IT team attempted to process all incoming data in real-time, leading to high operational costs and system slowdowns.

By reassessing their strategy, the team realized that not all data points were critical for their objective—detecting device anomalies. They identified three key metrics: error codes, device IDs, and timestamps. Filtering the raw data stream to retain only these metrics reduced ingestion volume by 70%, dramatically cutting storage costs and improving processing speeds. The refined pipeline not only met performance goals but also simplified downstream analytics.

This example highlights how focusing on the signal—the most relevant data—rather than the noise can drive efficiency and clarity in IT systems.

Signal vs. Noise

Not all data points carry equal weight. When collecting data, distinguishing between “signal” (valuable insights) and “noise” (extraneous information) is critical. A marketing campaign’s success, for example, may hinge on just a few factors: the target audience, the timing, and the message. Adding excessive layers of demographic or behavioral data can obscure rather than clarify the picture.

A memorable moment in my career involved helping a healthcare company predict patient no-shows. The initial model used over 100 variables, from weather patterns to patient health records. By eliminating irrelevant predictors, we reduced the model to 12 essential factors, improving accuracy and cutting down processing time.

The Cost of Data Overload

Collecting and storing massive amounts of data isn’t just inefficient; it’s costly. Cloud storage bills skyrocket, and processing power demands increase exponentially. Moreover, the human cost of analyzing unnecessary data—time spent by analysts, decision-makers, and developers—can be immense. A leaner data strategy often translates to more focused teams and faster results.

How to Decide What Data Matters

Define the Decision First: Begin with the decision you need to make, then identify the data required to inform it. This prevents the “data for data’s sake” trap. .
Prioritize Key Metrics: Ask, “What is the minimum amount of data needed to make this decision?” Focus on high-impact metrics.
Validate Regularly: Test whether adding more data improves your decisions. If not, it’s noise.
Leverage Sampling: Sometimes, a well-constructed sample is all you need. This is particularly true in scenarios like surveys, polling, or quality control.

Final View

The world is inundated with data, but more isn’t always better. Often, clarity comes from pruning excess information and focusing on what truly matters. By embracing a “less is more” philosophy, we not only make decisions faster but often make better ones too. So, the next time you’re faced with a mountain of data, pause and ask yourself: Do I really need all of this to decide?

Large Dataset - Pipeline, Seamless Scale-Up and Scale-Down

Vaibhav Bhutkar — Tue, 31 Dec 2024 14:49:32 +0000

In today’s data-driven world, managing and processing large datasets efficiently is more than a technical challenge—it’s a business necessity. As organizations increasingly rely on data to make decisions, the need to adapt quickly to changing data loads has become critical. But what happens when your data processing needs spike unexpectedly, or when they drop significantly during low-usage periods? This blog explores how to handle and optimize pipelines for large datasets, focusing on scaling them up and down efficiently, with a spotlight on Azure Data Factory (ADF).

The Challenge of Large Dataset Pipelines
Handling large datasets isn’t just about storage; it’s about ensuring that data can flow smoothly from one location to another without bottlenecks. The sheer volume of data can overwhelm pipelines, causing latency, failed processes, or even total system shutdowns. Traditional data pipelines often struggle to adapt to fluctuating loads, leading to inefficiencies and increased costs. Scalability isn’t just a nice-to-have feature; it’s a core requirement for any modern data processing architecture.

Building Scalable Pipelines with Azure Data Factory
Azure Data Factory (ADF) is a powerful tool designed to handle data integration and processing at scale. Its flexible architecture allows for dynamic scaling, enabling businesses to manage even the most demanding data loads efficiently. ADF’s features—such as parallelism, partitioning, and data flows—provide the foundation for building pipelines that can adjust to varying workloads. With ADF, you can create pipelines that scale up during peak data loads and scale down during quieter periods, ensuring both performance and cost-efficiency.

Scaling Up: When Data Grows Beyond Expectations
Scaling up is essential when your data volume increases rapidly, whether due to seasonal trends, marketing campaigns, or other unexpected events. Azure’s elasticity is a game-changer here. By leveraging ADF’s dynamic scaling capabilities, you can:

Increase Compute Resources: Auto-scaling integration runtimes allow your pipelines to handle larger volumes of data without manual intervention.
Optimize Parallel Processing: Splitting data into smaller partitions and processing them simultaneously boosts throughput.
Adjust Triggers Dynamically: Modify pipeline triggers to accommodate increased data ingestion rates, ensuring that no data is left unprocessed.
For instance, imagine a scenario where an eCommerce platform experiences a sudden surge in transactions during a flash sale. By dynamically scaling the ADF pipeline, you can ensure all transaction data is processed in near real-time, maintaining seamless operations and customer satisfaction.

Scaling Down: Managing Costs During Downtime
While scaling up is crucial for performance, scaling down is equally important for cost management. During periods of low data activity, unused resources can lead to unnecessary expenses. ADF provides tools to:

Deallocate Resources Automatically: By identifying low-priority operations, you can pause or terminate them during downtimes.
Minimize Redundancies: Streamline your pipelines to avoid duplicate processes and reduce resource consumption.
Monitor Usage Patterns: Use Azure Monitor to analyze pipeline activity and adjust resource allocation accordingly.
Consider a financial services firm processing customer data. During weekends or holidays, data activity might drop significantly. By scaling down resources, the firm can maintain its pipelines’ functionality while optimizing costs.

Overcoming Challenges in Scaling Pipelines
Scaling pipelines isn’t without its challenges. Common issues include latency during scale-up, resource contention, and managing pipeline dependencies. To address these:

Optimize Latency and Throughput: Use ADF’s built-in data movement features to reduce delays.
Implement Robust Monitoring: Set up alerts and dashboards to track performance and identify bottlenecks early.
Design for Resilience: Build fault-tolerant pipelines that can recover quickly from failures.

Real-Life Application: ADF Pipelines in Action
In my experience working with Azure Data Factory, I encountered a scenario where incremental database backups in .bak files needed to be migrated to an existing SQL database. The challenge lay in managing large volumes of backup data efficiently. By implementing dynamic scaling, we ensured that the pipeline adapted to varying data loads, completing the migration process seamlessly without over-provisioning resources.

Future Trends in Scalable Data Pipelines
As technology evolves, the future of scalable pipelines looks even brighter. AI and machine learning are set to play a significant role in predictive scaling, allowing pipelines to anticipate and prepare for data load changes proactively. Innovations in cloud-native tools, like enhanced ADF features, will make handling large datasets even more efficient. Businesses must also prepare for hybrid and multi-cloud environments, which add another layer of complexity to scalability.

Conclusion
By designing adaptable pipelines with tools like Azure Data Factory, organizations can optimize costs, improve reliability, and respond faster to changing data demands. Whether you’re dealing with sudden spikes in data volume or prolonged periods of low activity, a well-designed scalable pipeline is your key to success.

What strategies have you implemented to scale your data pipelines? Share your experiences or reach out to discuss specific challenges and solutions in your projects.

Importance of Design in Development !

Vaibhav Bhutkar — Mon, 30 Sep 2024 17:11:20 +0000

When we start working on any project, its tempting to dive straight in coding. The excitement of building something very quickly, especially when we are familiar with the tools, often leads up to skip essential steps like system design. However, over a time, I have learned about How crucial it is to carefully design a system before jumping in to development. I have been involved in multiple projects, analysing requirements and designing system based on these needs. Skipping this process can lead in a range of problems, which I will discuss here.
With this blog, I want to take you through the importance of system design, some of the benefits, potential problems when it's skipped, and what different prospectives - developers, clients and stakeholders - bring to the table. I will also share some real life sample design to illustrate my points.

Why System Design

System design is like the blueprint for building a house. You wouldn't built a house without a plan. so, why build an application without a solid design? The design ensures that the architecture is scalable, maintainable and meets the project requirements. Here's is the details why system design is important.

Clear Roadmap: It offers a clear path for both developers and stakeholders. It tells developer what we need to build and gives the client and stakeholders a concreate understanding of what the final product look like.
Scalability: A well designed application or system is flexible and can scale with the growth of data, users and functionalities. In the real world, application often expand beyond their initial scope, and good design can ensures system can handle that.
Risk Mitigation: When you design a system properly, you identify potential issues early on. The proactive approach reduces the risk of expensive rework later in the project.
Collaboration and Communication: Design facilitate better communication between teams. It aligns everyone on the same page - whether you'er a developer, client or project manager.

Problem when we ignore design:
Skipping or Ignoring system design can lead to numerous challenges during the project lifecycle. Here are few that I have come across.

Unclear Requirements: Without a design, developers often work based on assumptions. This leads to misunderstandings, which result in functionality that doesn't align with client expectations.
Rework and Delays: A lack of design often means going back to fix things. This not only delays the project but can result it in increasing project cost.
Technical Debt: When system aren't designed properly from the beginning, t he quick fixes used to compensate add to technical debt. Over time, this slows down the application and increase the mainteinance.

Benefits of Proper System Design
On the other hand solid design provides numerous advantages:

Efficiency: With a clear design, development become more efficient because everyone knows exactly what they need to build.
Predictable Outcomes: With a blue print in hand, you can predict more accurately what the final outcome will look like.
Easier Maintenance: A well thought - out design makes future updates and maintenance simpler because the structure of code is cleaner.

Sample System Designs
Let me share a couple of system design samples that I have been working on.
Data Ingestion Flow

This flow illustrate the design of data ingestion pipeline. We designed this flow to fetch data from FTP server, process them on VM using automated PowerShell script and push the incremental back up data to SQL Database.
Why this design: We needed an automation solution that could handle daily backups without manual intervention. The design ensures the scalability of the pipeline and handles incremental updates to avoid redundant data processing.

Perspective of System Design

Clients Perspective: Clients appreciate seeing a system design because it gives them a clear visual representation of how their application will work. Its easier for them to provide feedback before development starts, which minimizes last minutes changes.
Developer's Perspective: A developer, I can say a well-documented design simplifies the coding process. It removes ambiguity, which helps in avoiding unnecessary confusion or mistakes. With a design in hand, you know what your starting point is and what a end goal looks like.
Stakeholder's Perspective: For a stakeholders and third parties, system design ensures transparency. It provides a level of trust that the project is being developed thoughtfully and will meet their needs in terms of both functionality and scalability.

Conclusion:
System design is backbone of any successful project. It aligns everyone involved in the project, mitigates risks and ensures the application is built to scale efficiently. Whenever you're working on small internal tool or a large-scale enterprise application, never underestimate the power of good system design.

Recommended Reading:
Clean Architecture: by Robert C. Martin

Accelerated Integration: How We Met Our Deadline Ahead of Schedule.

Vaibhav Bhutkar — Fri, 28 Jun 2024 16:37:54 +0000

Introduction: This story or blog is about an Journey of an Integration project. I am sure that today everyone is aware about Integration. What it is? right? But still, let me highlight the thing what is an Integration. In simple words or for better understanding, Integration is nothing but an communication or passing data between two systems through one communication channel.

Similarly as mentioned in above definition, we had one project to integrate Salesforce with WordPress. I won't provide you the details about project itself - thats a story for another time.

In today's world of technology, things can change very quickly. Recently, we faced a big challenge: the deadline of our ongoing project was moved up. Earlier it was different and based on that deadline the progress is going on, however suddenly it gets changed from Client's end. With this change, we found that we don't have much time to complete our tasks. Our tasks was to integrate Salesforce with WordPress. In short its a two way integration, it means data will flow from Salesforce to WordPress and vice versa, both are the crucial systems for client.

The project was going smooth but suddenly we came in trouble, I am not even aware about How to handle this situation. It was very challenging and tough situation for us. I had reached out to our Manager and then after discussion, revising next flow and current state of project we got more resources for that project and target to achieve the end goal. You might think this resolve our issue, that we got resource we can complete the project and meet the deadline, but its not so easy! here! "actually problem starts"... The project was partially build and we got new resource working on that project. Our main aim to meet the deadline, but now with new resources challenge is increased "to make each team member productive from day one." Here's how we did it!

The Challenge
Our project was originally planned and to be completed in several months. This actually gives us enough time to carefully perform development and complete integration between Salesforce and WordPress. However, suddenly the deadlines was changed, we had to deliver the project much sooner than expected.

Solution
Expanding the Team
Our first step was to bring in new resources or team members. This wasn't just about having more hands on deck; it was about having right skills and ensuring everyone could hit the ground running. Everyone is much focused and with confident; as all team members are important and valuable asset for us.

How we achieved the speed, lets read below

Getteing Everyone Up to Speed
To make sure each and every team member should be productive from day one joined the team, we took several steps as below.

Clear Communication: We make sure that everyone should understand the project flow, project goals, their roles and important one that is new timeline. This actually helped to align everone's efforts.
Design Diagram: We have prepared Architectural design diagram of project, and that was iterated to small unit level as well. Once member joined we are providing that diagram and explaining project over it. It actually helps team member lot rather than reading project documentations.
Mentorship Communication: We paired new team member with experienced one, experienced one means the person working since long on that project. This not only worked with quick learning but also build a collaborative team culture.
DSM: Everyone is aware about DSM. We also held daily standup meetings to track the progress and address any issue immediately. This kept everyone one the same page and also allowed for quick adjustments. This also gives us clear picture about today's output.
Daily Check-in/Unit testing:
We also have implemented an unit test case suite in Postman, once we made check in any code over test environment we are running our test suite. It gives clear idea about everything going over production server is errorless.

- Reporting: Each day, we are reporting back to our stakeholders, detailing the tasks we had completed. This ensures transparency and kept everyone informed about our progress.

The Outcome / Result:
By focusing on above key areas, we turned a potential crises into a success story. The expanded team worked efficiently, confidently and we met the new deadline without compromising on quality. The integration of WooCommerce and Salesforce was completed smoothly.

What we learned:
Below are some key takeaway from this experience:

Become Flexible: Being able to adapt quickly to changing circumstances crucial.
Teamwork: A well coordinated team can achieve great results, even under pressure.
Preparation: Having a solid onboarding process can make a big difference in how quickly new team members become productinve.
Communication: Regular updates and clear communication are vital to keep the project on track.

Accelerated Integration is challenging, but with right approach, its possible to turn it into an opportunity for success. By expanding our team, ensuring everyone was productive from the start, and maintaining clear communication through daily reports, we were able to meet our tight deadlines and deliver high-quality integration of Salesforce and WordPress.

CopyData from REST API Application into an Azure SQL Database with extra Column; using Azure Data Factory.

Vaibhav Bhutkar — Mon, 24 Jun 2024 12:56:47 +0000

Requirement:
Our requirement is - to get data from REST API and transfer data to Azure SQL table, along with transferring data we have to transfer one extra column value to AZURE SQL instance.
To achieve this use case, we have to understand the data coming from REST Api call, so try to execute API call through Postman once if possible. In this tutorial I am going to explain about how we can call REST Api through ADF and captured/received data get stored in AZURE SQL database along with extra column.
For this tutorial we are using REST API call and AZURE SQL as target database.

Prerequisite :

Create an Azure Data Factory instance in Azure.
Create Azure SQL instance in Azure.
Get an API key for Rest Call - if authorization is there.
Postman is needed to check - Response of API call.

Set up Linked Services for REST API :
Linked service is like a connection string which define the connection information needed for service to connect other sources. To create linked service for source i.e. REST API - Go to ADF Workspace then click on "Manage" on from left side menu. Under linked services click on "New", then select REST and configure required connection details, refer below images.

Create Dataset for REST API
For creating Dataset for Rest API's, go to Author section from left panel, click on Dataset and click on "New Dataset." Select Data store as "REST" and click continue. Next is link the linked service created earlier or created in above step and then click ok.

Set up linked Services for Azure SQL DB:
To create linked service for Azure SQL DB. - Go to ADF workspace then click on "Manage" from left side menu. Under Linked service click on "New", then select Azure SQL Database and configure the connection. Add a name to lined service then select integration runtime as "AutoResolveIntegrationRuntime". Next use Azure Subscription where database is located. Then provide all details like Server Name, Username, Password, and database name to connect. Lastly click create.

*Create DataSet for Azure SQL DB: *
Similarly, Dataset created for REST API service in same way create data set for Azure SQL DB. To perform that activity; go to Author panel from left menu. Select DataSet section and click on "New Data Set". Select data store as "Azure SQL Database" and click on continue. Next link the linked service created above for Azure DataBase.

Create Pipeline and Add Copy Activity:
Now we are ready with Data Set for Azure SQL Database which is destination and Rest API service which act as source in our use case. To create activity please follow below steps.
Go to the "Author" section.
Click on "Pipelines" and then "New Pipeline".
Add copy activity to the pipeline.

Drag "Copy Data" activity from Activities pane to the pipeline canvas.
In the "Source tab", select "REST API" dataset.

In the "Sink" tab, select the Azure SQL Database dataset.

This complete activity of copy data from REST API to Azure SQL Database. Now here we need to check the response of API and based on that mapping need to consider in ADF pipeline. For that check executing API through Postman.

Add Extra column:
Now we need to add one extra column to the destination Azure SQL table. To achieve this we need to add addition column from source; put column name and value - IF it is custom then use Custom or else use as Parameter; if you are passing it from any other activity. Here we are creating Custom column. Update column Name and its value (we are passing static value here.) This completes our entire setup for process of getting data from REST API and Push it to Azure SQL Database.

Test and Run:
Run the pipeline in debug mode to test the data flow and ensure everything is working as expected.

Publish:
Once the pipeline is tested and validated, publish all the changes to make the pipeline available for scheduled runs or trigger based executions.

Conclusion:
By following all above steps, you should be able to create a robust ADF pipeline that fetches data from REST API and store it in Azure SQL Database with the necessary columns.

Generate SAS token for Azure API Management with Node Js.

Vaibhav Bhutkar — Thu, 06 Jul 2023 07:44:36 +0000

Azure API management is a platform provided by Microsoft Azure that enables organizations to publish, secure, manage, and analyze their APIs. API's allows different software application to communicate and interact with each other. Azure API management simplifies process of creating, maintaining and API deployment process.

This blog will help you out to generate access token programmatically using node js. This token is used to make direct call to Azure API management REST API. If you want sample code in c# .net then refer this link (https://learn.microsoft.com/en-us/rest/api/apimanagement/apimanagementrest/azure-api-management-rest-api-authentication)

It is possible to create SAS token manually, for that purpose use above url or navigate to Azure Management Portal and generate SAS token from there.
Login to Portal - Azure API Management Services - Deployment + Infrastructure -- Management API -- generate Token

Mention the expiry date for token in Expiry text box.
Generate Token Manually Through Code :
1. Construct a sign in string in below format -
{identifier} + "\n" + {expiry}
Here, identifier - it’s the identifier field from the API management tab from Azure API management instance.
expiry - desired expiry date of SAS token.

    const expiry = new Date();
    expiry.setDate(expiry.getDate() + 10);
    const expiryString = `${expiry.toISOString().split(".")[0]}.${formatMilliseconds(expiry.getMilliseconds())}Z`;
    const encoder = crypto.createHmac("sha512",Buffer.from(AZ_APIM_KEY, "utf8"));

(AZ_APIM_KEY - Used API Key as constant - you please use your own key from Azure)

2. Need to generate a signature by applying HMAC-SHA512 hash function to sign in string using key. Base 64 encode returned signature key.

        const dataToSign = `integration\n${expiryString}`;
        const dataToSignBytes = encoder.update(dataToSign, "utf8").digest();
        const signature = dataToSignBytes.toString("base64");

3. Finally created access token in below format.
    a. uid= {identifier}&ex={expiry}&sn={Base64 encoded signature format}

e.g. Token generated here is with above example -

Token = SharedAccessSignature uid=${AZ_APIM_IDENTIFIER}&ex=${expiryString}&sn=${signature};

Following is full code of token generation using node js -

const createToken = async () => {
  try {
    const expiry = new Date();
    expiry.setDate(expiry.getDate() + 10);

    const expiryString = `${expiry.toISOString().split(".")[0]}.${formatMilliseconds(expiry.getMilliseconds())}Z`;
    const encoder = crypto.createHmac("sha512",Buffer.from(AZ_APIM_KEY, "utf8"));

    const dataToSign = `${AZ_APIM_IDENTIFIER}\n${expiryString}`;
    const dataToSignBytes = encoder.update(dataToSign, "utf8").digest();
    const signature = dataToSignBytes.toString("base64");

    const token = `SharedAccessSignature uid=${AZ_APIM_IDENTIFIER}&ex=${expiryString}&sn=${signature}`;
    return token;

  } catch (error:any) {
    console.log(error);
    logs.insertLog(new Date(), "Error", "Crosswalk", "Expiration Utility", "1.0", "createToken", error.exceptionType, "Error observed while creating token", error.message, "", error.source, error.stackTrace);
  }
};

Use this access token as an Authorization key for further API call in Azure API management to change the subscription status/Subscriber status etc. depend on requirement. Refer below url for update user or subscription.
https://learn.microsoft.com/en-us/rest/api/apimanagement/current-ga/user/update?tabs=HTTP

const url = `${baseURL}/subscriptions/{Your details for url}`;

base url - here baseURL is constant - construct your base url based on API management url subscription or subscription id resource group etc

    const payload = {
      properties: {
        state: `expired`,
      },
    };
    const token = await createToken();
    const response = await axios.patch(url, payload, {
      headers: {
        Authorization: `${token}`,
        "content-type": "application/json",
      },
    });

Execute above call to make changes at Azure API management.

A Guide to Embedded App Development with Procore

Vaibhav Bhutkar — Mon, 03 Jul 2023 10:34:11 +0000

Now a day's, construction industry is undergoing a digital transformation, and various software platforms like Procore are leading the way in revolutionizing construction project management. With Procore's open API, developers have the opportunity to extend the platform's functionality by creating embedded apps. In this blog post we are going to discuss about "World of Embedded app development with Procore", the development points, benefits etc.

Before going in deep lets understand what is Embedded Application? -
It’s a software or application that is designed and integrated into a larger system or device to perform specific functions. The term "embedded" implies that the software is embedded within the other application or firm of a device, rather than being a standalone application running on a general-purpose computer or device.
Embedded app have limited user interfaces, as they primarily focus on performing specific tasks behind the scenes.
Now, we have some idea about Embedded application. Before starting actual development on application; Let's more understand about Procore's API. I am going to tell you what user or developer need to do if you are going to develop embedded application for Procore. We will discuss all technical points required for app development. We will not discuss about theoretical sign off document and other process.

Setup Developer Account:
As a developer you need to create signup to developer portal (https://developers.procore.com/signup) to build an application.

Once done with registration at developer portal. User or developer need to create new app at Procore development environment. Once developer creates app will get Procore Client ID, Client Secret, Sandbox environment and URL etc and access to the app page. These details are required for User/Developer for implementing authentication with Procore.

Define Your App's Purpose:
Clearly define the purpose of your embedded app. Determine the problem it aims to solve or the specific functionality it should provide within the Procore ecosystem. Consider the needs of Procore users and identify gaps or opportunities where your app can add value.
Our app purpose is simple, "Our application can be used by Procore user. To use the same; user or application user need to buy API key (or license)and utilize that key further in application". Here; we want the details of user who is going to utilize or installing our this application.

Plan your application:
Define your application architecture and workflow. Define the key functionality, features we are offering with application. Check out how our application will interact with Procore users and How app will integrate with Procore data. Draw some architectural diagrams and run all virtual use cases on it.
Review Procore's API Documentation:
To understand available end points, data model, authentication methods Thoroughly review Procore's API documentation. This documentation (https://developers.procore.com/documentation/introduction) will guide you in making API calls to fetch and manipulate data within Procore.

Build and Test app:
Start building you’re application as defined in Architectural diagram. Utilize all the end points API and stack to integrate app with Procore. Try to follow best practices provided by Procore to handle the data. Ensure the all functionality will be provided by your application as mentioned.
While building application; One point I noticed that We need to Embed this app inside Iframe. So, try to test and verify each single functionality will work under Iframe or not? User this url (https://iframetester.com/) to test your application under embedded app.

Handle Sign in With Procore:
If your application want the details of user who is trying to use our application or you are going to implement data connector app with in Iframe; then in such case - Developer need to implement Sign in With Procore functionality. To implement this functionality under I frame is quite difficult as - Under Iframe Sign in with Procore page not loads. I have implemented this functionality using window Pop up. (Means - once user click on Sign in button - one window popup loads with Sign in screen and once sign in complete, again redirected user to the main screen where he redirected.)

Client side storage not work:
Now a days most developers are using client side storage mechanism to store some key value. But this will not work here with Procore Embedded app as we are integrating our application inside Iframe. Client side storage not work in Iframe; if parent and child domain was different. It was because of cross origin policy.
I observed this issue while building our application, I tried almost all client side storage approaches () but not worked with Procore Embedded app. This is an important point each developer need to keep that in mind.

Submit Your App for Review:
Once your app is developed and tested, submit it to Procore for review. Procore has a review process to ensure that apps meet their quality and security standards. Provide all the necessary information about your app, including its purpose, functionality, and integration points with Procore.

This blog will provides a general overview of the process for developing embedded apps with Procore. It is essential to refer to Procore's official documentation and resources for detailed instructions and the latest information

Rest API vs Graph QL

Vaibhav Bhutkar — Fri, 31 Mar 2023 11:10:43 +0000

In this blog post we will discuss about what is Rest? What is GraphQL?. We will discuss about difference between Rest and GraphQL also we will check when to use Rest and GraphQL.
Now a days, REST APIs and GraphQL are two popular technologies used for building APIs. This allows developers to interact with a server and retrieve data. Both REST and GraphQL have their own unique strengths and weaknesses, and which one to use depends on your specific use case. So, we will explore the differences between REST and GraphQL and provide some overview on when to use which technology.

What is REST?

REST stands for Representational State Transfer. REST is an architectural style used for building web services. RESTful APIs use HTTP requests to GET, POST, PUT, and DELETE data. RESTful APIs mostly return data in JSON or XML format.

REST is based on a client-server architecture, which means that the client makes a request to the server, and the server responds with the requested data. RESTful APIs use URIs (Uniform Resource Identifiers) to identify resources, and HTTP methods are used to perform actions on those resources.

RESTful APIs we have been we are using since a long time and are well established, with a large community and plenty of resources available for developers. They are straightforward to build and are compatible with a wide range of programming languages.

It is an HTTP method describes the type of request that is sent to the server. They are:

GET - To reads a representation of a specified source.
POST - To creates a new specified source.
PUT - To updates/replaces every resource in a collection.
PATCH - To modifies a source.
DELETE - To deletes a source.

What is GraphQL?

GraphQL is a query language for APIs. GraphQL was developed by Facebook in 2012. GraphQL allows developers to request exactly the data what they need, and nothing more. Unlike REST, which returns fixed data structures, GraphQL lets the client specify the shape of the data it needs, and the server returns only that data.

GraphWL uses a schema to define the data that can be queried, and it uses a query language to request that data. GraphQL APIs can return data in any format, including JSON and XML.

GraphQL is become increasingly popular in recent years because it provides a more efficient and flexible way to retrieve data from an API. It eliminates the problem of over-fetching and under-fetching data, which can be a problem with RESTful APIs.

Differences between REST and GraphQL

The core difference between GraphQL and REST APIs is - GraphQL is a specification, a query language; while REST is an architectural concept for network-based software.

Another major differences between REST and GraphQL is how data is requested. In REST, the client makes a request to the server for a specific resource, and the server responds with all the data associated with that resource.
With GraphQL, the client specifies the data it needs, and the server responds with only that data.

Another difference is the way data is represented. RESTful APIs typically use JSON or XML to represent data, while GraphQL can return data in any format.

RESTful APIs are straightforward to build and are compatible with a wide range of programming languages. GraphQL is more complex to build, but it provides a more efficient and flexible way to retrieve data from an API.

When to use REST and when to use GraphQL

The choice between REST and GraphQL depends on your specific use case. Here below are some general guidelines:

Use REST if:

If your API has a fixed set of resources, and the data associated with those resources doesn't change frequently.
You need to support a wide range of clients, including web browsers, mobile devices, and IoT devices.
You need a simple API that is easy to build and maintain.

Use GraphQL if

Your API has a large number of resources, and the data associated with those resources changes frequently.
You need to support a wide range of clients, but you want to minimize the amount of data transferred over the network.
You want a flexible API that can adapt to changing client needs.

Conclusion

In conclusion, both REST and GraphQL are powerful technologies that can be used to build APIs. REST is well established and easy to build, while GraphQL provides a more efficient and flexible way to retrieve data from an API. The choice between REST and GraphQL depends on your specific use case.

Clean Code: What It Is and Why It Matters

Vaibhav Bhutkar — Fri, 31 Mar 2023 07:56:01 +0000

Introduction

I am writing this blog because, I was impressed the way he explained about Clean code and why it is necessary in developers life, when I was going through Clean code book written by Uncle Bob. Clean code is one of the most crucial and important peace concepts in software development. It refers to writing code that is easy to read, understand, and maintain. In this blog post, we will discuss what clean code is and why it matters.

What is Clean Code?

Clean code is code that is easy to read, understand, and modify. It is code that is well-organized, concise, and efficient. Clean code is essential for several reasons. First Readable - it makes it easier for other developers to understand your code. Second Maintainable - it makes it easier to maintain your code. Finally Testable - it makes it easier to identify and fix bugs.

Clean code follows several principles. One of these principles is the DRY principle, which stands for "Don't Repeat Yourself." This principle means that you should avoid duplicating code wherever is possible. Another principle is the KISS principle, which stands for "Keep It Simple, Stupid." This principle means that you should strive for simplicity in your code.

Why Does Clean Code Matter?

Clean code matters for several reasons. First, it makes it easier for other developers to understand your code. If your code is clean, readable and well-organized, other developers can quickly understand and get up to speed with your codebase. It means that they can start contributing to the project more quickly, which can help accelerate on development.

Second, clean code makes it easier to maintain your codebase. If your code is well-organized, elagant and easy to read, it will be easier to identify and fix bugs. This means that you can spend less or minimal time for debugging and more time building new features or functionalities.

Finally, clean code is essential for the long-term success of your project. As your codebase grows day to day, it will become more challenging to maintain. If your code is clean, readable and well-organized, it will be easier to scale your project and add new features over time.

How to Write Clean Code

Writing clean code takes practice and discipline. Here are some tips for writing clean code:

Use descriptive variable names - it should tell about functionality.
Write small functions - fits in screen
Avoid duplicating code
Keep functions and classes short
Write tests for your code

By following above mentioned tips, you can write code that is easy to read, understand, and maintain.

Conclusion

Clean code is essential for the success of any software project. It makes it easier for other developers to understand and maintain your codebase. It also makes it easier to identify and fix bugs. By following best practices such as the DRY and KISS principles, you can write clean code that is easy to read, understand, and maintain.

Refer Clean Code handbook written by Robert C. Martin. for more details.

Performance optimization using React Hooks! Like useCallback and useMemo.

Vaibhav Bhutkar — Tue, 04 Jan 2022 05:32:21 +0000

Performance is very important key and most common thing that each and every developer may faces at some point after building any application.

Use Effect:

The Effect Hook lets you perform side effects in function components.

Data fetching, setting up a subscription, and manually changing the DOM in React components are all examples of side effects. Whether or not you’re used to calling these operations “side effects” (or just “effects”), you’ve likely performed them in your components before.

useEffect run after every render. By default, it runs both after the first render and after every update. Instead of thinking in other terms, you might find it easier to think that effects happen “after render”. React guarantees the DOM has been updated by the time it runs the effects.

Here in above example we declare the count state variable, and we tell React we need to use an effect. We pass a function to the useEffect Hook. . Inside effect, we set message using console. When React renders this component, it will remember the effect we used, and then run our effect after updating the DOM. This happens for every render, including the first one.

Here’s where the optimization comes in. To prevent the useEffect from executing each time the function reference changes, we can use useCallback. The useCallback hook will store the reference to the function instead of the function itself. The reference of the function will only be updated when one of the dependencies of the function is updated. If you don't want the function reference to be updated ever, you can leave the dependency array empty in the same way as the dependency array of the useEffect hook. Below is the code sample of it.

When component state is changed, the component re-renders, but those re-renders can be minimized. This means faster rendering, fewer computations, minimal API calls, and etc.

Also even when we made API calls using react UseEffect at time we can use dependency object such way to stope unnecessary renders of useEffect. This can be achieved by putting some conditions inside useEffect function call.
Also we can use useMemo while exporting same as below.

While using React memo in above way, you need to check all API calls from applications and where possible use the same. It reduces unnecessary API calls.

Note: React is customizable, everyone can customize it according to his way. So this optimization can depend on the scenario.

Conclusion:
Above all are the sample examples. There are other various ways to reduce useEffects rendering calls, these depends on the requirements in application. So please explore more you will get more in react. Happy Learning !!!

Reference:

https://reactjs.org/docs/hooks-effect.html

https://blog.devgenius.io/performance-optimization-with-react-hooks-usecallback-usememo-f2e527651b79