DEV Community

Cover image for App Insights by Example — a paste-and-run playbook for your stack
Vignesh Athiappan
Vignesh Athiappan

Posted on

App Insights by Example — a paste-and-run playbook for your stack

#appinsights #azure #productivity #monitoring

No theory. Each entry = a real question → the query → what the result looks like → how to read it. Open Logs in your App Insights resource and paste.

Quick vocab you'll see in every query: requests = calls coming in (APIM operation hit, Logic App run, Function trigger). dependencies = calls going out (NetSuite, Greenhouse, SQL, Key Vault, LLM). exceptions = errors with stack traces. operation_Id = the ID that ties one end-to-end action together across all services. cloud_RoleName = which service emitted the row.

A. "Is it even working?"

1. Did my endpoint get called at all in the last hour? 

requests
| where timestamp > ago(1h)
| where name has "CreateByCustomAttributes"
| project timestamp, name, resultCode, success, duration, cloud_RoleName
| order by timestamp desc
Enter fullscreen mode Exit fullscreen mode

Sample output:
| timestamp | name | resultCode | success | duration | cloud_RoleName |
|---|---|---|---|---|---|
| 10:42:11 | POST /v1/Custom/CreateByCustomAttributes | 404 | false | 38 ms | oiboomi-prod |
| 10:41:55 | POST /v1/Custom/CreateByCustomAttributes | 404 | false | 41 ms | oiboomi-prod |

Read it: rows exist → the call is reaching APIM. Empty result → the caller never got to the gateway (DNS, network, wrong URL) — the problem is upstream of App Insights, not in your API.

2. Success vs failure count for one operation today 

requests
| where timestamp > ago(24h)
| where name has "CreateByCustomAttributes"
| summarize total=count(), failed=countif(success==false) by name
| extend failRate=round(100.0*failed/total,1)
Enter fullscreen mode Exit fullscreen mode
name total failed failRate
POST …/CreateByCustomAttributes 214 214 100

Read it: 100% fail = config/routing, not intermittent. A number like 3% = flaky (timeouts, race conditions) — different hunt entirely.

B. The 404 — full debug, the way you'd actually do it

3. Grab a failing call and get its operation_Id 

requests
| where timestamp > ago(1d)
| where name has "CreateByCustomAttributes" and success == false
| project timestamp, resultCode, operation_Id, url
| take 1
Enter fullscreen mode Exit fullscreen mode
timestamp resultCode operation_Id url
10:42:11 404 a1b2c3d4e5f6... https://…/v1/Custom/CreateByCustomAttributes

4. ⭐ The key query — is the 404 from APIM or from the backend?

Paste the operation_Id from above:

let opId = "a1b2c3d4e5f6...";
union requests, dependencies, exceptions, traces
| where operation_Id == opId
| project timestamp, itemType, name, target, resultCode, success, message=outerMessage
| order by timestamp asc
Enter fullscreen mode Exit fullscreen mode

Case 1 — no dependency row:
| timestamp | itemType | name | target | resultCode |
|---|---|---|---|---|
| 10:42:11.100 | request | POST …/CreateByCustomAttributes | | 404 |

→ APIM returned 404 without ever calling the backend. Meaning: the operation/route isn't matched in oiboomi-prod — wrong API path, operation not added to the product, or the URL template doesn't match. Fix is in APIM, not the API.

Case 2 — dependency row present, also 404:
| timestamp | itemType | name | target | resultCode |
|---|---|---|---|---|
| 10:42:11.100 | request | POST …/CreateByCustomAttributes | | 404 |
| 10:42:11.118 | dependency | POST /Custom/CreateByCustomAttributes | ConciousIntegrationAPIService | 404 |

→ APIM did forward it and the backend ConciousIntegrationAPIService returned 404. Meaning: the backend route genuinely doesn't exist at that path — check the backend base URL / rewrite policy / actual controller route.

This one query settles the argument. That "is there a dependency row or not" distinction is 90% of the value of App Insights for you.

5. See the full URL + rewritten path APIM actually sent 

let opId = "a1b2c3d4e5f6...";
dependencies
| where operation_Id == opId
| project timestamp, target, name, data, resultCode
Enter fullscreen mode Exit fullscreen mode

data shows the exact URL APIM hit downstream — compare it against what the backend expects. Nine times out of ten a /api prefix or version segment is missing here.

C. Performance / "why is it slow?"

6. Slowest operations (P50/P95/P99) 

requests
| where timestamp > ago(24h)
| summarize p50=percentile(duration,50), p95=percentile(duration,95), p99=percentile(duration,99), calls=count() by name
| order by p95 desc
Enter fullscreen mode Exit fullscreen mode
name p50 p95 p99 calls
POST /Pulse/TimeCharge/Sync 240 4100 9800 1,320
GET /Nexus/Approvals 60 180 320 8,900

Read it: P50 fine but P95/P99 huge = a slow tail (some calls hit a cold backend or a big payload), not a broadly slow endpoint. Chase the tail, not the average.

7. Which backend is dragging — NetSuite? Greenhouse? SQL? 

dependencies
| where timestamp > ago(24h)
| summarize p95=percentile(duration,95), calls=count(), fails=countif(success==false) by target, type
| order by p95 desc
Enter fullscreen mode Exit fullscreen mode
target type p95 calls fails
netsuite.suitetalk.api HTTP 6,200 540 12
yourserver.database SQL 210 40,100 0
harvest.greenhouse.io HTTP 380 210 0

Read it: NetSuite is your latency source. type tells you HTTP vs SQL vs Azure Service Bus etc. without opening any code.

8. Pull the slowest individual traces to drill into 

requests
| where timestamp > ago(6h) and name has "TimeCharge"
| top 10 by duration desc
| project timestamp, duration, resultCode, operation_Id
Enter fullscreen mode Exit fullscreen mode

Take an operation_Id, run query #4 on it → you'll see which dependency ate the time inside that specific slow run.

D. Errors / exceptions

9. Top exceptions this week (by volume) 

exceptions
| where timestamp > ago(7d)
| summarize count() by type, outerMessage
| order by count_ desc
| take 15
Enter fullscreen mode Exit fullscreen mode
type outerMessage count_
SqlException Timeout expired 91
HttpRequestException 401 Unauthorized (NetSuite) 44

Read it: ranks your real pain. The 401 pattern is exactly your OAuth-token-invalidation race — you'd catch it here without anyone reporting it.

10. Exceptions for one operation, with stack trace 

exceptions
| where timestamp > ago(1d) and operation_Name has "TimeCharge"
| project timestamp, type, outerMessage, method, details
| order by timestamp desc
Enter fullscreen mode Exit fullscreen mode

details holds the parsed stack frames — the method + line where it blew up.

11. Which request caused this exception? 

let ex = exceptions | where timestamp > ago(1d) | take 1;
ex | join kind=inner (requests) on operation_Id
| project exType=type, exMessage=outerMessage, reqName=name, reqResultCode=resultCode, url
Enter fullscreen mode Exit fullscreen mode

Read it: ties the crash back to the exact incoming call + URL that triggered it. No more guessing which input broke it.

E. Distributed tracing — the feature you're really buying

12. Follow one action across every service 

let opId = "<any operation_Id>";
union requests, dependencies, exceptions
| where operation_Id == opId
| project timestamp, cloud_RoleName, itemType, name, target, duration, resultCode
| order by timestamp asc
Enter fullscreen mode Exit fullscreen mode
timestamp cloud_RoleName itemType name target duration resultCode
.100 oiboomi-prod request POST /TimeCharge/Sync 4200 200
.140 oiboomi-prod dependency HTTP netsuite netsuite… 3900 200
.150 pulse-api request POST /billingclass 120 200

Read it: a full waterfall of one user action across APIM → NetSuite → Pulse. The .140 → 3900ms line is your bottleneck, instantly.

13. Build a mini application map with KQL (who calls whom) 

dependencies
| where timestamp > ago(1d)
| summarize calls=count(), fails=countif(success==false) by from=cloud_RoleName, to=target
| order by calls desc
Enter fullscreen mode Exit fullscreen mode

Gives you the edges of your architecture (service → backend) with failure counts. The portal's Application Map draws this visually — this is the raw version.

F. Your AI agents (cost + cache — plugs into your ₹3,096/mo model)

Assumes you emit a customEvent / customMetric per LLM call. If not yet, this is the reason to.

14. Per-call LLM latency + volume by day 

dependencies
| where timestamp > ago(7d) and target has "openai" or name has "responses"
| summarize calls=count(), p95ms=percentile(duration,95) by bin(timestamp,1d)
| render timechart
Enter fullscreen mode Exit fullscreen mode

15. Cache-hit rate (if you tag calls with a cacheHit dimension) 

customEvents
| where timestamp > ago(1d) and name == "LLMCall"
| extend hit = tostring(customDimensions.cacheHit)
| summarize total=count(), hits=countif(hit=="true")
| extend hitRatePct = round(100.0*hits/total,1)
Enter fullscreen mode Exit fullscreen mode
total hits hitRatePct
148,200 139,300 94.0

Read it: your 94% assumption becomes a live measured number. If it drifts to 80%, your monthly cost estimate is wrong and you'll see it here first.

16. Token volume trend (cost driver) if you emit tokens as a custom measurement 

customEvents
| where timestamp > ago(30d) and name == "LLMCall"
| summarize totalTokens=sum(toreal(customMeasurements.totalTokens)) by bin(timestamp,1d)
| render columnchart
Enter fullscreen mode Exit fullscreen mode

G. Logic Apps — mind the split

17. Consumption Logic App run failures (no native App Insights — via diagnostic settings → Log Analytics) 

AzureDiagnostics
| where TimeGenerated > ago(1d)
| where ResourceProvider == "MICROSOFT.LOGIC" and Category == "WorkflowRuntime"
| where status_s == "Failed"
| summarize failures=count() by resource_workflowName_s, OperationName
| order by failures desc
Enter fullscreen mode Exit fullscreen mode
resource_workflowName_s OperationName failures
GreenhouseJobSync workflowRunCompleted 7

Read it: for your 140+ Consumption apps this is how you get run-level failure counts. It's run history, not true traces — that's the limitation.

18. Standard Logic App — action-level detail (native App Insights) 

requests
| where timestamp > ago(1d) and cloud_RoleName has "your-standard-logicapp"
| summarize runs=count(), fails=countif(success==false) by name
| order by fails desc
Enter fullscreen mode Exit fullscreen mode

Read it: Standard gives you per-workflow/action telemetry as normal requests/dependencies — the richer observability you don't get on Consumption. Real bullet point for your migration break-even.

H. Alerts you'd actually set

19. Log alert: any 404 on that endpoint in the last 15 min

Save this as a scheduled log alert, fire if result count > 0:

requests
| where timestamp > ago(15m)
| where name has "CreateByCustomAttributes" and resultCode == "404"
Enter fullscreen mode Exit fullscreen mode

So the next time that route breaks after a deploy, it pages you instead of a user reporting it.

20. NetSuite dependency failure spike 

dependencies
| where timestamp > ago(15m) and target has "netsuite"
| where success == false
| summarize fails=count()
Enter fullscreen mode Exit fullscreen mode

Alert when fails > 5. Catches your OAuth token-invalidation race the moment it starts.

21. Live Metrics (not KQL) — during a deploy

Open the Live Metrics blade while you deploy Pulse Time Charge. Sub-second stream of incoming requests, failures, and CPU. If failure rate spikes the instant traffic hits the new version, you roll back before it spreads. This is your deploy safety net.

I. Cost control (ties into your optimization work)

22. What's actually costing me ingestion (last 7 days) 

union withsource=Table *
| where timestamp > ago(7d)
| summarize items=count(), gb=round(sum(_BilledSize)/1024/1024/1024, 2) by Table
| order by gb desc
Enter fullscreen mode Exit fullscreen mode
Table items gb
dependencies 12,400,000 8.9
requests 3,100,000 2.1
traces 40,000,000 14.2

Read it: traces (verbose logging) is usually the silent bill-driver. Turn down trace verbosity or sampling there first — it rarely costs you debugging power.

23. Set the levers

  • Sampling on APIM: crank to 100% while chasing a bug, drop to ~20–50% for steady state.
  • Daily cap on the workspace so a runaway loop can't bankrupt you.
  • Retention: keep 30–90 days interactive, archive the rest.

The 6 queries to memorise

  1. requests | where name has "X" | project timestamp, resultCode, success, operation_Id — is it working
  2. union requests, dependencies, exceptions | where operation_Id == "…" — the full trace (your #1 debug tool)
  3. requests | summarize percentile(duration,95) by name — what's slow
  4. dependencies | summarize percentile(duration,95), countif(success==false) by target — which backend
  5. exceptions | summarize count() by type, outerMessage — what's breaking
  6. union withsource=T * | summarize sum(_BilledSize) by T — what's costing

Everything else is a variation on these.

Top comments (0)