DEV Community: Harsh Trivedi

Azure Function App Stuck on "Runtime Unreachable"? How VNet Integration and Private Endpoints Fixed It

Harsh Trivedi — Sat, 12 Sep 2026 17:43:28 +0000

A beginner-friendly, real-world walkthrough of diagnosing and fixing networking issues in a Linux Azure Function App running on Elastic Premium.

When an Azure Function App doesn't even show its runtime version, it is very easy to start looking at the application code.
But what if you haven't deployed any code yet?

That was exactly the situation I faced while setting up a new Linux Azure Function App on the Elastic Premium EP1 plan in Microsoft Azure. The Azure Portal could not properly load the runtime information, and ZIP deployment was failing before giving us any useful application-level logs.

There was no Python code to debug.

That was the first important clue.

The problem was not in the application. It was in the networking and storage connectivity required by the Function App itself.

The final solution involved connecting the Function App to the correct Virtual Network (VNet) and then creating a separate Private Endpoint and Private DNS configuration for another storage account used by the application.

This article explains what happened, why it happened, and how to troubleshoot similar problems step by step.

The Architecture We Were Working With

The environment was not a completely public Azure setup.

The Function App was running on Linux + Azure Functions Elastic Premium (EP1), and the environment already had a shared VNet containing private resources.

The storage account associated with the Function App was also protected by network restrictions and was reachable through private networking.

Few existing Function App was already working in this environment.

The new Function App, however, was not connected to the same VNet.

That small difference was enough to prevent the new Function App from reaching the storage it needed.

Azure Functions depends on storage for important runtime and application operations. For Premium plans, Azure Files is also commonly used for function app content and deployment-related functionality. Microsoft specifically notes that the storage account used by a Function App must be accessible to the app, and that Premium plans can use Azure Files for function code and configuration.

So the important question became:

If the storage account is private, how is the Function App supposed to reach it?

The answer is through the VNet.

The First Problem: "Runtime Unreachable"

Immediately after creating the Function App, we noticed a few unusual things.

The Azure Portal couldn't properly display the Functions runtime version. The host appeared unreachable, and ZIP deployment was failing without giving us the kind of application logs we would normally expect.

At first, this can be confusing.

You might think:

Is function_app.py wrong?
Is requirements.txt missing something?
Is the Python version incorrect?
Did the ZIP package have the wrong structure?

But none of those explanations made sense because the application hadn't even been deployed yet.

That changes the troubleshooting approach. If the Function App cannot initialize its host before your code is running, you should first investigate the infrastructure it depends on.

One of the most important dependencies is storage.

Azure Functions uses the storage account configured for the Function App for core operations. On Elastic Premium, Azure Files and queues can also be used to store application content and configuration. Microsoft documents AzureWebJobsStorage as the default storage connection and WEBSITE_CONTENTAZUREFILECONNECTIONSTRING as the storage connection used for Azure Files content on Elastic Premium and Consumption plans.

In our environment, this storage account was protected by network restrictions and had a private endpoint. Existing Function Apps could access it because those Function Apps had already been integrated with the correct VNet.

The new Function App wasn't.

So we had a simple mismatch:

Private storage → VNet access required → new Function App wasn't connected to the VNet.

VNet Integration: What Does It Actually Do?

This is one of the concepts that can be confusing when you are new to Azure networking.

VNet integration is primarily an outbound networking feature.

It gives the Function App a path into a VNet so that the app can reach resources inside that network, including resources exposed through private endpoints. It does not mean that your Function App suddenly becomes privately accessible from the VNet.

These are two different concepts:

VNet Integration controls how the Function App makes outbound connections.

Private Endpoint gives an Azure resource a private IP address inside a VNet.

Microsoft's documentation specifically describes regional VNet integration as an outbound capability that allows Azure Functions to access resources in the VNet, including private endpoints.

That distinction becomes extremely important when troubleshooting.

The First Fix: Connect the Function App to the VNet

The easiest way to understand the required configuration was to look at the existing Function App that was already working. We checked its networking configuration and identified the VNet and the subnet used for VNet integration.

Then we configured the new Function App using the same network architecture.

From the Azure Portal, go to:

Function App → Networking → Virtual Network Integration

Then select Add virtual network integration.

Choose the VNet used by the existing application and select the dedicated integration subnet, if in your case no such VNet exist, then create a VNet first.

For an Elastic Premium Linux Function App, the integration subnet must be delegated to:

Microsoft.Web/serverFarms

Microsoft currently documents /28 as the minimum subnet size for Elastic Premium VNet integration and recommends a larger subnet for Linux Premium workloads, because each running instance consumes an IP address and scaling can temporarily require additional addresses. You can also keep the default size when you setup the subnet.

One important rule is:

Do not use your private-endpoint subnet as the Function App integration subnet.

These subnets have different purposes.

The integration subnet is used by the Function App for VNet integration.

The private-endpoint subnet contains the private IP addresses assigned to private endpoints.

Microsoft's own Function App networking samples use separate subnets for these two purposes.

Once the integration was completed, the Function App was restarted.

After giving Azure a few minutes to complete the networking changes, the Function runtime became accessible again.

That was a strong indication that we had identified the first problem correctly.

But Then We Found Another Problem

Getting the Function App runtime working was only the first part. The application also needed to communicate with another storage account. This was not the storage account used by the Function App itself.

It was a separate business-content storage account used by the application to store processed documents.

*Example scenerio- *

You have a task that triggers the function app on a specific time to place a file from an API to your blob storage account. Now this storage account is not the same that the function app uses.
We had a simple health-check function that tested connectivity to different dependencies.

The Key Vault check was successful.

The Blob Storage check was failing.

The error looked similar to:

Failed to resolve 'contentarchive.blob.core.windows.net'
No address associated with hostname

This error was extremely useful. Why?

Because it wasn't an authentication error.

It wasn't saying:

AuthorizationPermissionMismatch

It wasn't saying:

403 Forbidden

It was saying that the hostname could not be resolved.

That pointed us toward DNS and private endpoint configuration.

Why VNet Integration Didn't Automatically Fix the Second Storage Account

This is another important concept. Connecting your Function App to a VNet does not automatically give the Function App private connectivity to every Azure resource.

The destination resource must also be configured correctly.

In our case, the second storage account did not yet have the required private endpoint.

So we effectively had this:

Function App
     |
     | VNet Integration
     ↓
   VNet
     |
     X

Private path to content storage didn't exist
The solution was to create the private networking path for that specific storage account.

Private Endpoint: What Does It Do?

A Private Endpoint creates a private network interface with a private IP address inside your VNet and connects that IP to a specific Azure service.

For example, if you create a Blob private endpoint for a storage account, the Blob service becomes reachable through a private IP.

But there is an important detail here:

Private endpoints are tied to a specific service sub-resource.

For Azure Storage, Blob, File, Queue, Table, and other services have separate endpoints and corresponding DNS zones.

For example:

Blob  → privatelink.blob.core.windows.net
File  → privatelink.file.core.windows.net
Queue → privatelink.queue.core.windows.net
Table → privatelink.table.core.windows.net

Microsoft's current private endpoint documentation confirms these service-specific DNS zones.

So creating a Blob private endpoint doesn't automatically create private connectivity for Azure Files or Queue.

This distinction matters particularly for Function Apps because the Function runtime and the application itself may use different storage services.

The Second Fix: Create a Private Endpoint for the Content Storage

We opened the business-content storage account and went to:

Networking → Private Endpoint Connections

Then we created a new private endpoint.

For the target sub-resource, we selected:

Blob

We selected the same VNet used by the Function App, but importantly, we selected the private-endpoint subnet, not the Function App's integration subnet.

During the configuration, we also enabled integration with a Private DNS Zone.

For Azure Blob Storage, the standard private DNS zone is:

privatelink.blob.core.windows.net

Microsoft recommends this naming convention for Blob private endpoints.

Once the private endpoint was provisioned and the connection was approved, the next step was verifying DNS.

Why DNS Was So Important

Creating a private endpoint gives the service a private IP. But your application doesn't normally connect to the storage account by typing its private IP.

It still uses the normal hostname:

contentarchive.blob.core.windows.net

The job of Private DNS is to make that hostname resolve to the correct private endpoint.

Conceptually, we want:

contentarchive.blob.core.windows.net
                    ↓
        Private DNS resolution
                    ↓
             Private IP
                    ↓
          Private Endpoint
                    ↓
          Storage Account

Without the correct DNS configuration, the application might continue trying to resolve the hostname through the normal public DNS path—or fail to resolve it in the private networking context.

Microsoft specifically states that applications calling private endpoints must have DNS resolution configured so the service hostname resolves to the private endpoint.

Checking the Private DNS Zone

After creating the private endpoint, we checked:

Private DNS Zones → privatelink.blob.core.windows.net

Inside the zone, we checked the record sets.

There should be an A record for the storage account that points to the private endpoint's IP address.

For example:

contentarchive
      ↓
10.x.x.x

The exact IP will obviously be different in every environment.

We then compared that address with the private IP assigned to the private endpoint.

The addresses should match.

We also checked the Virtual Network Links section.

The VNet used by the Function App must be linked to the Private DNS Zone so that workloads inside that VNet can resolve the private name.

Microsoft's DNS documentation describes the use of Private DNS Zones and VNet links for private endpoint name resolution.

One more useful detail is that when a private endpoint is created with the recommended DNS zone configuration, Azure can automatically manage the corresponding private DNS record through the private endpoint's DNS zone group.

Restart and Test Again

Once the networking and DNS configuration was complete, we restarted the Function App.

Then we ran the health-check function again.

The result changed from a DNS failure to a successful Blob connection.

A simplified version of the health-check response looked like this:

{
  "checks": {
    "key_vault": {
      "ok": true
    },
    "blob": {
      "ok": true
    }
  },
  "overall_ok": true
}

This is where health checks become extremely useful.

Instead of simply knowing that "the Function isn't working," we can test individual dependencies.

For example:

Function App
    |
    +---- Key Vault      → OK
    |
    +---- Blob Storage   → DNS failure

After fixing DNS:

Function App
    |
    +---- Key Vault      → OK
    |
    +---- Blob Storage   → OK

That tells us much more about where the problem actually exists.

An Important Troubleshooting Pattern

There is a useful pattern to remember when troubleshooting private networking.

If you see:

Failed to resolve hostname
No address associated with hostname
DNS resolution failed

start by investigating DNS and private endpoint configuration.

If DNS starts working but you then receive something like:

403 Forbidden
AuthorizationPermissionMismatch
Network access denied

that is actually progress.

It means the application can now find the destination, but something else is blocking the request.

At that point, investigate:

Storage firewall and network rules
Private endpoint approval
Identity or access permissions
Storage RBAC
Connection configuration
The specific Storage service being accessed

This is much more effective than changing application code randomly.

One Storage Account Does Not Mean One Private Endpoint

This is probably the most important lesson from the second issue.

Suppose your architecture looks like this:

Function App
      |
      ↓
     VNet
      |
      +---- Storage Account A
      |
      +---- Storage Account B
      |
      +---- Key Vault

You cannot assume that because Storage Account A has a private endpoint, Storage Account B is automatically private.

Each Azure resource needs its own private connectivity configuration.

And even within one Storage Account, different services can have different private endpoint requirements.

For example:

Storage Account
      |
      +---- Blob → Blob private endpoint
      |
      +---- File → File private endpoint
      |
      +---- Queue → Queue private endpoint

The exact configuration depends on which services your Function App actually uses.

This is particularly important for Azure Functions because the Function App's own storage requirements may include Azure Files in Elastic Premium, while your application may separately access Blob Storage for business data.

What About ZIP Deployment?

Once the networking problem was fixed, deployment could be investigated separately.

For a Python Function App, the ZIP package needs to contain the required project files at the root of the package.

For example:

function_app.py
host.json
requirements.txt

The important point is that host.json should be at the root of the extracted deployment package.

For Linux Function Apps using remote build, Microsoft currently documents these application settings:

SCM_DO_BUILD_DURING_DEPLOYMENT=true
ENABLE_ORYX_BUILD=true

Remote build allows Azure to build dependencies in the target Linux environment rather than relying on packages built on your local machine.

This is a deployment concern, though—not something to troubleshoot first when the Function host itself cannot start because it cannot reach its required storage.

That distinction saved us a lot of unnecessary debugging.

The Troubleshooting Flow I Would Follow Next Time

If I encounter a new Function App that is unreachable before any application code has been deployed, I would follow this order.

First, check the Function App itself
Confirm that the Function App is running, the hosting plan is correct, and the runtime configuration is present.

If the runtime itself cannot load, don't immediately start debugging function_app.py.

Next, check the storage dependency
Look at the Function App's storage configuration, including AzureWebJobsStorage and, for Elastic Premium, the Azure Files content configuration where applicable.

Ask:

Can this Function App actually reach the storage account it depends on?

If the storage account is publicly accessible, the problem may be elsewhere.

If it is restricted to a VNet or private endpoint, networking becomes a primary suspect.

Then, check VNet integration

Verify that the Function App is integrated with the correct VNet.

Also verify that the integration subnet is separate from the private-endpoint subnet and has the correct delegation.

For Elastic Premium Linux, Microsoft documents Microsoft.Web/serverFarms as the required delegation.

Then, check the private endpoint

For every private Azure service the application needs to access, verify that the appropriate private endpoint exists and is approved.

Don't assume that another resource's private endpoint covers it.

Then, check DNS

Make sure the correct Private DNS Zone exists and is linked to the VNet.

For Blob Storage, that means:

privatelink.blob.core.windows.net

Then verify that the storage account has the expected A record pointing to the private endpoint IP.

Finally, test the application

Once the infrastructure is confirmed, run a small health check that tests dependencies individually.

That gives you a much clearer picture than testing the entire application at once.

Quick Checklist

When troubleshooting an Azure Function App that cannot reach a private storage account, I would check these in order:

Function App storage: Is the storage account used by the Function App accessible from the Function App's network path?
VNet integration: Is the Function App connected to the correct VNet?
Integration subnet: Is the correct dedicated subnet being used and delegated to Microsoft.Web/serverFarms for Elastic Premium?
Private endpoint: Does the destination storage account have the required private endpoint for the service being accessed?
Subnet separation: Is the private endpoint using its own subnet rather than the Function App's integration subnet?
Private DNS: Does the appropriate Private DNS Zone exist and is it linked to the VNet?
DNS record: Does the storage account hostname resolve to the private endpoint's IP?
Access permissions: Once networking works, does the Function App have the required identity/RBAC or other authentication permissions?
Deployment: Only after the platform and network are healthy, investigate ZIP packaging, dependencies, and remote build configuration.

References

The following Microsoft Learn resources were used to verify the Azure networking and deployment concepts discussed in this article:

Azure Functions networking options
Storage considerations for Azure Functions
Azure Private Endpoint DNS configuration
Use private endpoints to integrate Azure Functions with a virtual network
Deployment technologies in Azure Functions
Run your Functions from a Package File in Azure

Solving the Background Verification Bottleneck for enterprises: Inside BGVerifyIQ

Harsh Trivedi — Thu, 03 Sep 2026 03:20:11 +0000

Imagine an HR manager who has just completed this month’s hiring drive. Dozens of candidates have accepted their offers, and everyone is eager to onboard them as quickly as possible. But before that can happen, every background verification request has to move through multiple checks, vendors, approvals, and status updates. What should be a straightforward operational process often turns into hours of chasing spreadsheets, checking vendor portals, and answering the same status questions over and over again. It isn’t a technology problem as much as it is a visibility problem — and that’s the challenge that inspired BGVerifyIQ.

The problem every HR team quietly struggles with

Ask any HR or Talent Acquisition team what happens after an offer letter goes out, and you’ll hear a familiar story. A candidate is selected. A background verification (BGV) request is raised with a vendor. And then… the waiting begins.

Behind that wait is a surprising amount of manual work:

HR has to log into multiple vendor portals just to check where a case stands.
Status updates live in scattered Excel sheets, emails, and vendor dashboards that rarely agree with each other.
Nobody has a single, live number for “how many candidates are actually ready to be approved today.”
SLA breaches — cases that have been open too long — get discovered late, usually when a hiring manager complains.
When a discrepancy or a “suspect” flag shows up, it can get buried in a spreadsheet row instead of being escalated immediately.
Every time leadership asks “how are our BGV vendors performing,” someone spends half a day pulling numbers together manually.
New joiners get onboarded before their verification is even reviewed properly, simply because nobody had a clear, current view of their case.

None of this is because HR teams aren’t diligent. It’s because the tools weren’t built for this specific job. Spreadsheets don’t talk to vendor systems. Vendor portals don’t talk to each other. And nobody has time to become a SQL expert just to answer a simple question like “which candidates are stuck.”

This is the exact gap BGVerifyIQ was built to close.

What BGVerifyIQ actually does

BGVerifyIQ is a single workspace that sits on top of your background verification data — requests, checks, education records, employment records — and gives HR three things they didn’t have before, all in one place:

A plain-English way to ask questions, without writing a single line of SQL.
A live, always-current dashboard of the entire BGV pipeline — no manual pulling of numbers.
A complete 360° view of any single candidate, with a clear, explainable recommendation on whether they’re ready to be approved.
Everything in the current version is intentionally read-only. It never changes your underlying data. It only helps people see it, understand it, and act on it faster. That’s an important design decision we’ll come back to later.

Why I didn’t just build “another chatbot”

There’s a lot of excitement right now about AI agents that can answer any question in a chat window. And that excitement is well placed — for the right kind of question.

But here’s the catch: a chat window is great when you have a specific question (“what’s the status of candidate Ananya?”). It’s a poor way to see the big picture (“how many candidates are pending across all vendors right now, broken down by how old each case is?”). Nobody wants to type ten different questions into a chatbot every morning just to get their daily overview.

*So instead of picking one or the other, BGVerifyIQ combines both, deliberately, as separate pages built for separate jobs:
*

Chat — for quick, specific, in-the-moment questions.
KPI Dashboard — for the daily/weekly bird’s-eye view, the numbers leadership actually cares about.
Candidate 360 — for a deep dive into one candidate before making an approval decision.
History — because past answers and past conversations shouldn’t disappear the moment you close the tab.
Settings — so each user can control their own experience and see exactly which AI models are answering their questions.

This “agent + dashboard” hybrid is, in our view, the right shape for most internal business tools going forward. The agent handles the ad hoc and the unexpected. The dashboard handles the routine and the recurring. Neither replaces the other.

A tour of the app

💬** Chat**

This is where a user can type something like “show me approval-ready candidates” or “any updates on Rao” in completely normal English. Behind the scenes, the question is turned into a safe, read-only database query, run against the data, and answered with a short summary plus the actual table of results underneath — so nothing is ever “just trust the AI,” the numbers are right there to verify.

Three quick-action buttons — Priority Queue, SLA Breaches, Vendor Performance — skip the AI translation step entirely and jump straight to pre-built, always-correct answers for the questions people ask most often. Every question and every answer is automatically saved, so nothing is lost.

📊 KPI Dashboard

This is the “morning coffee” page — a live, numbers-first view of the entire BGV pipeline: total requests, how many are cleared, how many are pending, and how many have a major discrepancy. It also shows a status breakdown, how long cases have been open (aging buckets), how each vendor is performing on turnaround time, and — critically — a live list of every case currently breaching its SLA.

At the bottom sits the Priority Queue: a ranked list of exactly which candidates need attention today, downloadable as a CSV for anyone who wants to share it further.

👤 Candidate 360

Search for any candidate by name or ID, and this page pulls together everything known about them into one screen: overall status, an approval-readiness badge, a risk score, the assigned vendor and vendor contact, a check-by-check breakdown (education, employment, address, criminal, identity), a dated timeline of everything that’s happened on the case, and — importantly — a plain-English explanation of why the system is recommending “Ready,” “Caution,” “Needs Review,” or “Escalate.” This isn’t a black-box score; every recommendation comes with the specific reasons behind it, so HR can trust it and defend the decision if asked.

From here, HR can download a one-page PDF report or email it directly to a hiring manager or stakeholder.

🕘 History

Every past conversation from the Chat page, tied to the logged-in user’s email, searchable by keyword, newest first. Click “Open” on any past conversation and it reloads right back into Chat so the person can keep going from where they left off.

⚙️ Settings

Users can update their name, role, and email, switch between light and dark theme, and — importantly for trust and transparency — see exactly which AI model is currently answering their questions, and confirm that the system is running in read-only mode.

About
This section explains what each page is about and what BGVerifyIQ stands for.

Dark Mode

What’s happening behind the scenes

For the developers and technical stakeholders reading this, here’s a quick look at how the current version is built, and why each piece exists.
db.py — handles the connection to the PostgreSQL database. It reads connection details from environment variables (so nothing sensitive is hard-coded), keeps one reusable connection pool for performance, and includes a small but important safety function: is_read_only_sql(). This function checks every single query before it's allowed to run, and blocks anything that isn't a plain SELECT. It's the guardrail that makes it safe to let an AI generate SQL on the fly — even if the AI ever tried to write something destructive, this layer would refuse it.

llm.py — the AI layer that turns a plain-English question into a safe SQL query, and turns raw query results back into a clear, professional summary. It's built so that if one AI provider is slow or down, it automatically falls back to a second one — so the app doesn't just stop working during a demo or a busy morning. It also carries strict rules about how to match candidate names and IDs (using flexible partial matching instead of exact matches, since people rarely type a name or ID perfectly), which is a big part of why the Chat page actually finds what users are looking for instead of returning "no results."

pdf_report.py — generates the one-page Candidate 360 PDF on demand, entirely in memory, with careful handling so that special characters (like smart quotes or long names) never break the PDF layout.

mailer.py — sends the PDF report directly to a stakeholder's inbox. Worth noting: this file also tells a useful real-world lesson. It was originally built using standard SMTP email sending, which worked perfectly on a local machine, but failed once deployed to a cloud platform — because most cloud hosting providers block outbound SMTP traffic to prevent spam abuse. The fix was to switch to a transactional email API (Brevo) that works over regular HTTPS, which is never blocked. It's a small example of the kind of real infrastructure problem that only shows up once you actually deploy something.

The data layer — a clean PostgreSQL schema (bgv) with tables for requests, checks, education, and employment history, plus three pre-built views (v_candidate_360, v_sla_breaches, v_vendor_performance) that do the heavy lifting of joining and aggregating data. The AI is instructed to always prefer these views over writing its own joins — which keeps its answers consistent and correct.

Try it yourself

A live version of BGVerifyIQ is deployed and ready to explore:

bgverifyiq-production.up.railway.app

Feel free to click through every page — Chat, KPI Dashboard, Candidate 360, History, and Settings — with sample data already loaded, so you can see exactly how it behaves before considering it for your own team.

From prototype to enterprise: what a real rollout looks like

What’s described above is a working, functional prototype — and a genuinely useful one. But there’s an honest and important distinction between “a tool that solves the problem well for a demo or a small team” and “a tool that’s ready to run background verification for thousands of employees across an entire organization.” Here’s what that jump actually involves.

A production-grade frontend
The current interface is built using Streamlit, which is excellent for proving out an idea fast. For an organization-wide rollout, this would typically move to a proper frontend framework — Angular, React, or similar — giving finer control over performance, accessibility, custom branding, and integration with existing internal design systems.

A real backend, with security built in from day one
An enterprise version needs a proper backend service layer, not just a script talking to a database. That means:

Authentication — logging in through the company’s existing identity provider (SSO/SAML/Azure AD), not a name-and-email form.
Authorization — making sure a regional HR analyst can only see their region’s candidates, while a compliance officer might see everything, and a vendor SPOC sees only their own cases. Not everyone should see everyone’s data. Basically Role Based Access Control, data masking policies should be enabled. [We will discuss both RBAC and Data Masking in snowflake in another article]
Data security — encryption of data at rest and in transit, strict handling of personally identifiable information (PII), and audit logs of who viewed or changed what, and when.
Proper exception handling — every failure (a database timeout, an AI provider being slow, a malformed input) should fail gracefully with a clear message, never a broken page.
Testing — automated tests for the business rules (like the risk-scoring and readiness logic), so a future code change can’t silently break how a candidate gets flagged.
A feedback mechanism — a simple way for HR users to flag when an AI-generated answer was wrong or unhelpful, so the system keeps improving based on real usage instead of guesswork.

Using AI models your organization already trusts and governs

This is one of the most important points for any organization considering this seriously.

The current prototype uses free, hosted, general-purpose AI models (Groq’s Llama model, and Google’s Gemini, as a backup). These are great for building and proving out an idea quickly — but they were not built or contracted for handling real, sensitive employee data such as names, ID numbers, addresses, and background check outcomes. An organization should not send real employee records to a public, general-purpose AI API unless that API is specifically covered by the company’s data processing agreements and security review.

The right approach for a real deployment is to use whatever AI platform your organization has already vetted and governs — most companies today are standardized on either Snowflake or Databricks for their data warehouse, and both now offer their own governed AI agents:

If your organization’s data lives in Snowflake, you can use Snowflake Cortex Agents to run directly on your existing tables and views — the exact same bgv schema and views shown above would work almost as-is. The AI never leaves Snowflake's security boundary, meaning your data doesn't get sent to an outside company's servers at all.
If your organization uses Databricks, the same idea applies with Databricks’ Mosaic AI agents running on your Lakehouse tables.

Beyond just being safer, these governed platforms typically unlock more capability too — automatically generating charts, producing longer comparative analyses (“compare this quarter’s vendor turnaround to last quarter”), and integrating with the rest of your company’s existing data and security tooling, rather than living as a separate, isolated app.

In short: build and prove the idea with free/open tools first (as I did here), then swap in your organization’s approved, governed AI platform before any real employee data touches it.

Additional functionality worth adding

Once the security and governance foundation is in place, a few natural next features come to mind:

A CRUD page to approve or update candidate status directly from the app — instead of just viewing data, an authorized HR user could mark a candidate as approved, add a remark, or update a case status, with that change written straight back to the source system, fully audit-logged.
Notifications — automatic alerts (email, Slack, Teams) the moment a case breaches SLA or a suspect flag appears, instead of HR having to go looking for it.
Integration with the HRMS/ATS — so a “Ready” status in BGVerifyIQ can automatically trigger the next onboarding step, instead of someone manually copying that status elsewhere.
Feedback and continuous improvement loop — letting users rate AI answers, which helps refine the system over time.

Why an AI agent at all — and why it matters even more on mobile

It’s worth pausing on this, because “just add an AI chatbot” can sound like a trend rather than a real solution. Here’s the actual problem an agent solves.

Traditionally, if an HR analyst wanted to answer a slightly unusual question — “how many campus hires are stuck with a discrepancy this month?” — they had one of two options: ask someone technical to write a query for them, or manually filter and cross-reference spreadsheets by hand. Both are slow, and both mean the person asking the question depends on someone else, or on their own patience with Excel.

An AI agent removes that dependency. It lets someone simply ask, in their own words, and get a trustworthy, data-backed answer in seconds — without needing to know what a database even is. That’s the real value: it turns “I need to go find someone who can pull this” into “I just asked, and I have my answer.”

This matters even more on mobile. A dashboard with charts and tables is genuinely hard to use well on a small phone screen — things get cramped, and scrolling through filters is painful. A chat interface, on the other hand, is something almost everyone is already completely comfortable with, because it looks and feels exactly like every messaging app they already use. On a phone, “just type your question” is a far better experience than “pinch and zoom into a dashboard.” This is a big part of why the agent and the dashboard are built as complementary tools rather than the dashboard being the only option — mobile users, in particular, benefit enormously from being able to just ask.

What this actually solves — the short version

For organization leaders and executives, here’s the summary:

Saves time — no more logging into multiple vendor portals or manually cross-referencing spreadsheets to answer a simple status question.
Faster approvals — a clear, explainable readiness badge tells HR exactly who’s ready to move forward, and why, without second-guessing.
Fewer surprises — SLA breaches and risk flags are surfaced automatically and immediately, instead of being discovered weeks later.
Less repetitive manual work — one-click PDF reports and direct email delivery replace manually compiling updates for hiring managers.
Better decisions, not just faster reports — the KPI dashboard turns raw data into visible bottlenecks and vendor performance trends that leadership can actually act on.
A tool people will actually use — because it meets people where they already are, whether that’s a quick chat question on a phone or a full dashboard review on a desktop.

And to be clear about where this stands today: this is proof that the idea works, built on real, functioning logic — the business rules, the read-only safety design, the recommendation logic — none of that is a mockup. What comes next, for an organization ready to roll this out at scale, is wrapping that same proven logic in enterprise-grade security, governance, and infrastructure — using AI models the organization already trusts with its data, not a public API.

The underlying idea, though, doesn’t change: give HR one place to ask, one place to see, and one place to decide — instead of five different tools that don’t talk to each other.

BGVerifyIQ assists decisions — final approval always rests with the HR team.

Disclaimer

BGVerifyIQ is a demonstration project created to showcase how AI can streamline and improve the background verification process. All data, candidate names, organizations, vendors, and background verification records shown in the application are entirely fictional and generated for demonstration purposes. No real employee, customer, or organizational data has been used at any stage of development, testing, screenshots, or deployment.

The application is built on a carefully designed sample dataset that simulates realistic background verification scenarios while ensuring privacy and confidentiality.

If you have questions about the architecture, implementation, AI workflow, or would like to discuss how a similar solution could be built for your organization, I’d be happy to connect. Feel free to reach out through LinkedIn or leave a comment on this article, and I’ll do my best to help.

Securing Enterprise AI Pipelines: Using Azure Key Vault and AES Encryption for Sensitive Documents in Snowflake/Databricks AI Workloads

Harsh Trivedi — Wed, 02 Sep 2026 17:03:05 +0000

A practical architecture for securing contracts, invoices, employee data, and business documents before they are consumed by AI models, document intelligence systems, and enterprise agents.

Organizations are rapidly building AI-powered applications on top of business documents such as contracts, invoices, employee records, financial reports, sales data, and operational data. While significant effort is spent on developing AI models and intelligent agents, security is often treated as an afterthought.

One of the biggest risks in enterprise AI is exposing sensitive information during document storage, processing, and retrieval. Simply storing files in cloud storage or data platforms is not enough. The data should also be secured during transmission.

In this article, I will walk you through a security-first architecture that combines AES encryption, Azure Key Vault, Service Principals, RBAC permissions, and controlled decryption workflows to protect sensitive documents while still enabling AI-powered document extraction, analytics, and intelligent agents.
We’ll cover:

✔ The security problem enterprises face

✔ Why encrypted storage alone is not enough

✔ Why encryption keys should never be stored in code or .env files

✔ Azure Key Vault fundamentals

✔ Service Principal authentication

✔ RBAC permissions and access control

✔ Creating and managing Key Vault secrets

✔ Python implementation examples

✔ Snowflake and Databricks integration patterns

✔ Secure document processing for AI workloads

✔ Security benefits and best practices

This architecture can be applied across Snowflake, Databricks, Microsoft Fabric, Azure AI Foundry, Synapse Analytics, and custom AI platforms.

The Growing Security Challenge in Enterprise AI

Organizations are increasingly building AI applications on top of business-critical documents.

Examples include:

Customer contracts
Legal agreements
Invoices
Financial statements
Employee records
Payroll information
Vendor documents
HR reports
Employee attrition predictions
Compliance reports
Healthcare records
Procurement documents

These documents often contain highly sensitive information.

At the same time, organizations want AI systems to:

Extract information from documents
Build search experiences
Power RAG solutions
Support AI agents
Generate business insights
Answer natural language questions
Generate KPIs, charts and visuals
This creates a difficult challenge:

How can we allow AI systems to use sensitive information without exposing the underlying documents?

The Wrong Approach

Many organizations unknowingly introduce security risks by:

Storing documents unencrypted
Storing encryption keys in source code or environment files
Keeping secrets in configuration files
Saving credentials inside notebooks
Embedding keys inside ETL pipelines

For example:

AES_KEY = "MySecretEncryptionKey"

If someone gains access to:

GitHub repositories
DevOps pipelines
Virtual machines
Databases
Storage accounts they may gain access to both the encrypted documents and the encryption key.

At that point, encryption provides little value.

The Security Principle

One of the most important security principles is:

Encrypt the data and protect the encryption key separately.

Even if attackers gain access to encrypted documents, they should not be able to access the encryption key.

This is where Azure Key Vault becomes critical.

The Solution Architecture

The architecture consists of five major layers:

Document Encryption
Secure Key Storage
Controlled Authentication
In-Memory Decryption
AI Processing

Document Source — The source of document can be anything like SAP, Cloud storage, Azure Blob Storage etc.

When we move them to our storage layer, they should be encrypted.

Step-by-Step implementation guide-

A) Why We Encrypt Documents

Consider documents such as:

Client contracts
Employee compensation data
Vendor agreements
Invoices
Strategic business reports
Attrition prediction outputs These files often contain information that should not be accessible to every user or system.

Before storing these files, we encrypt them using AES-256.

This ensures that even if storage is compromised or a developer has RBAC (Role based access control), the document remains unreadable without the encryption key.

B) Creating an AES Encryption Key

Example:

from Crypto.Random import get_random_bytes
AES-256 = 32 bytes
aes_key = get_random_bytes(32)
print(aes_key.hex()) 32 bytes = 256 bits

This creates an AES-256 encryption key.

Why AES?

AES (Advanced Encryption Standard) is one of the most widely used symmetric encryption algorithms in the world.

Advantages:

Fast
Highly secure
Industry standard
Suitable for large documents
Supported across cloud platforms AES-256 is commonly used in enterprise environments.

C) What is Azure Key Vault?

Azure Key Vault is a managed Azure service designed for securely storing and controlling access to secrets, cryptographic keys, and certificates.

Instead of storing sensitive values in application code, organizations place them inside a centralized vault protected by Azure Identity and RBAC controls.

Azure Key Vault supports three primary object types:

Secrets

Examples:

API Keys
Database Passwords
OAuth Client Secrets
AES Encryption Keys

Keys

Examples:

RSA Keys
Cryptographic Signing Keys
Customer Managed Encryption Keys

Certificates

Examples:

SSL Certificates
TLS Certificates

In our use case, the AES encryption key is stored as a Secret.

Why Azure Key Vault is required?

Centralized secret management
Fine-grained access control
Secret rotation support
Access auditing
Compliance support
Reduced insider risk

Creating Azure Key Vault

1.Navigate to: portal.azure.com

Azure Portal -> Create Resource -> Key Vault -> Create

2.Configure:

Subscription

Resource Group

Region (eg. Central India)

Vault Name (eg. projectname-aes-key-kv)

3.For Authorization/Permission Model:

Select: Azure Role Based Access Control (RBAC)

4.For Networking:

Public access: Selected networks

or

Private endpoint

5.Permissions Required to Create a Key Vault

Typically one of the following roles:

Owner
Contributor
Key Vault Contributor

Important:

Key Vault Contributor can manage the Key Vault resource itself but cannot read secrets stored within the vault.

This separation improves security.

*6.Storing the AES Key in key vault secrets and the permissions that are required to create secrets.
*
IMP: When using Azure RBAC, you will need Key Vault secret officer role to create secret.

Once the role is assigned, Create and Store the generated AES key as a secret.

Example:

Secret Name:

AES-DOCUMENT-ENCRYPTION-KEY : secret _code

The key is now centrally managed and protected.

D) Creating a Service Principal Applications, requires an identity to access Azure resources.

This is where Service Principals are used.

Think of a Service Principal as:

A non-human identity used by applications.

Example:

Snowflake/Databricks/Azure Function/Fabric Pipeline/Custom API

↓

Service Principal

↓

Azure Key Vault

↓

AES Secret

The Service Principal becomes the trusted identity that accesses the vault.

Why Service Principals Are Important

Without Service Principals:

Applications would require user credentials
Automation becomes difficult
Auditing becomes harder

Service Principals provide:

Automated authentication
Secure access
RBAC integration
Least privilege access

Required: The service principal system-managed identity must be assigned read secret role on the key vault

For Snowflake, Databricks, Fabric, or AI platforms: (Your service principal name might be: projectname_aeskey_extract_spn)

Assign Role:

Key Vault Secrets User

Can:

Read secret values
Read secret metadata

E) Python Example: Reading the AES Key

from azure.identity import ClientSecretCredential
from azure.keyvault.secrets import SecretClient
credential = ClientSecretCredential(
    tenant_id="<tenant-id>",
    client_id="<client-id>",
    client_secret="<client-secret>"
)
vault_url = "https://company-kv.vault.azure.net"
client = SecretClient(
    vault_url=vault_url,
    credential=credential
)
aes_key = client.get_secret(
    "AES-DOCUMENT-ENCRYPTION-KEY"
).value

The application retrieves the key only when needed.

F) Encrypting Sensitive Documents

from Crypto.Cipher import AES
from Crypto.Util.Padding import pad
from Crypto.Random import get_random_bytes
# AES key retrieved from Azure Key Vault
key = bytes.fromhex(aes_key_hex)
# Generate random IV (16 bytes for AES)
iv = get_random_bytes(16)
with open("contract.pdf", "rb") as file:
    plaintext = file.read()
cipher = AES.new(key, AES.MODE_CBC, iv)
ciphertext = cipher.encrypt(
    pad(plaintext, AES.block_size)
)
# Store IV along with encrypted content
encrypted_content = {
    "iv": iv,
    "ciphertext": ciphertext
}

The encrypted file can now be stored safely.

G) AI Processing Pattern

When processing begins:

Retrieve AES key from Key Vault
Decrypt document in memory
Extract structured or unstructured chunk information
Persist extracted results
Discard decrypted content

The key principle is:

Decrypt only when required and only in memory.

H) Platform Integration Examples

I) AI Agents and RAG Systems

Once document information is extracted into structured datasets, AI systems can answer questions such as:

Which contracts expire next month?
Which invoices exceed $100,000?
Which vendors require renewal?
What is the predicted attrition risk for the Sales department?
Which customers have pending obligations? The AI model operates on structured business data rather than unrestricted access to sensitive raw documents.

J) Security Benefits Achieved

This architecture provides:

Encryption at Rest — Documents remain protected in storage.
Key Separation — Encryption keys are separated from encrypted data.
Least Privilege Access — Applications receive only required permissions.
Auditing — Secret access can be monitored.
Key Rotation — Encryption keys can be replaced centrally.
Compliance Support — Supports enterprise governance and security requirements.
Reduced Attack Surface — Compromised storage does not automatically expose sensitive information.

Final Thoughts

As organizations accelerate AI adoption, securing the data feeding those AI systems becomes just as important as building the models themselves.

A secure AI platform is not only about model accuracy, vector databases, or intelligent agents. It is equally about protecting contracts, invoices, employee information, and other sensitive business assets that power those systems.

By combining AES encryption, Azure Key Vault, Service Principals, RBAC permissions, and controlled in-memory decryption, organizations can build AI platforms that are secure, scalable, compliant, and enterprise-ready.

The goal is simple:

Protect the documents.
Protect the keys.
Allow AI access only when necessary.

What next?

In my next article, I will walk you through how we implemented enterprise-grade access controls for AI-powered platforms using techniques such as:

Row-Level Security (RLS)
Row Access Policies
Dynamic Data Masking
Role-Based Access Control (RBAC)
Business Unit-based filtering
Customer-based entitlements
Financial data masking
Secure document access controls
Time-limited document download links
Agent-level authorization patterns
Governance frameworks for AI applications

We’ll explore how an AI agent can return different answers to different users for the same question, while ensuring that sensitive information remains protected and compliant with organizational policies.

Because securing the documents is important.

But securing who can access the information extracted from those documents is what truly makes an AI platform enterprise-ready.