DEV Community: Manju Vasanth

I Taught an AI Agent to Debug Workday Integrations Like a Senior Consultant

Manju Vasanth — Fri, 24 Jul 2026 16:50:06 +0000

Built for the Agents of SigNoz hackathon: OpenTelemetry + SigNoz + MCP + a decade of enterprise integration scar tissue.

When a Workday integration fails, the error message almost never tells you the real story. A raw 403 Forbidden does not say "your Integration System User's security group is missing a domain security policy." Figuring that out takes a person who has been burned by it before. I have spent years being that person, so for this hackathon I asked a specific question: can I bottle that tribal knowledge into an agent that reads telemetry and answers like a senior consultant would?

By the end of the week, the answer was yes, and the road there included my own code crashing in ways I did not expect, twice.

What I Built

Three layers:

A mock Workday tenant. A Python/Flask simulator I built to practice integration patterns: SOAP endpoints, RaaS reports, and an Extend-style canvas that runs a config-driven, multi-step supervisory org provisioning orchestration (validate input, check existence, build a SOAP payload, call the tenant, extract results).
Full observability via SigNoz. OpenTelemetry instrumentation on every orchestration step, streaming traces, logs, and dashboards into self-hosted SigNoz, deployed with Foundry.
An AI diagnostic agent. A Python agent that talks to SigNoz's MCP server, pulls the newest failed trace, finds the failing step, and maps it to a Workday-specific root cause with fix steps.

Instrumenting a Config-Driven Pipeline

The simulator's orchestration engine executes steps from a JSON store, which made instrumentation surprisingly clean: wrap the step executor once, and every step gets a span automatically. The core of it:

def exec_steps(steps, depth=0):
    for i, s in enumerate(steps, start=1):
        typ = s.get("type", "unknown")
        ref = s.get("ref", typ)
        span_name = "step-%02d.%s.%s" % (i, typ, ref)
        with _tracer.start_as_current_span(span_name) as span:
            span.set_attribute("step.index", i)
            span.set_attribute("step.type", str(typ))
            span.set_attribute("step.ref", str(ref))
            if s.get("inject"):
                span.set_attribute("chaos.injected", str(s.get("inject")))
            try:
                exec_one(s, depth)
            except Exception as e:
                span.record_exception(e)
                span.set_status(Status(StatusCode.ERROR, str(e)))
                raise

One edit, and the SigNoz waterfall went from anonymous GET/POST spans to a readable business flow: step-02.send-http-request.CheckExists, step-05.send-http-request.CallHumanResources. I also hooked the engine's existing log function so every step message becomes a span event and a trace-correlated log line. My old "eye-catcher" debug strings suddenly carried trace IDs.

Chaos Injection: Breaking It on Purpose

To give the agent something to diagnose, I added config-driven failure injection. Adding one key to any step in the flow store simulates a realistic Workday failure:

{ "ref": "CallHumanResources", "inject": "403", ... }

Modes cover the failures I actually see in integration work: 400 (malformed WQL, usually an unescaped apostrophe in a name filter), 401 (expired ISU token), 403 (missing ISSG domain permission), 404 (bad reference), timeout, and empty result sets.

Here is the part the tutorials do not tell you: my first injected failure crashed my own application. The screen handler behind the orchestration had only ever rendered success. When the flow returned an error result, it died on a NoneType at line 196, then, after I guarded that line, at line 197. Chaos engineering found my unhandled error path before it found anything else. The traceback, fittingly, was waiting for me in SigNoz's Exceptions view, grouped and linked to the exact trace.

The Agent: Telemetry In, Workday Answers Out

SigNoz ships an MCP server (enabled through one block in Foundry's casting.yaml), and it exposes 41 tools. My agent uses three: signoz_search_traces to find the newest ERROR trace for the service, signoz_get_trace_details for the span tree, and the span attributes and status messages as evidence.

The diagnosis itself is a rule table built from experience, the mapping I would apply manually on a support ticket. The 403 entry, for example:

(re.compile(r"HTTP 403|Forbidden|lacks domain security", re.I),
 "ISSG missing domain security policy permission",
 "The ISU authenticated successfully but its security group lacks "
 "Get/Put access on the target domain.",
 ["Grant the ISU's security group the required domain security policy permission",
  "Activate pending security policy changes in the tenant",
  "Re-run the step; a 200/201 confirms the grant took effect"])

When a run fails, the agent prints the failing step, the root cause in Workday language, fix steps, and a pipeline map with the failure flagged. It also discloses when a failure was injected, if chaos.injected is present, the report says so. An agent that quietly presents staged failures as organic ones would be a demo trick, not a tool.

A confession that belongs in this section: the agent's very first run failed with a 403 of its own. I had created the SigNoz service account but skipped the role assignment step, so the MCP server rejected every query with "only viewers/editors/admins can access this resource." My permission-diagnosing agent was blocked by a missing permission. I fixed it the way its own rule table would have suggested.

The Dashboard That Called My Bluff

I built a four-panel "Workday Integration Health" dashboard with the Query Builder: p95 latency per step, failures by step, error logs, and a formula panel computing success rate as A/B*100 over root spans.

The formula panel immediately exposed a design flaw. It read 100% success while injected failures were clearly happening. The reason: my chaos hook failed the step span but let the flow complete gracefully, so the root orchestration.run span stayed clean. A run with a failed step was counting as a successful run. One line in the root wrapper fixed it, propagate ERROR to the root if any step in the trace log failed, and the panel dropped to an honest 92.5%.

I finished with an alert rule on the failure count (above 0, at least once, 5-minute rolling window). One injected run later:

What I Learned

Chaos finds your bugs first. Both crashes this week were in my code's error paths, not in the failures I injected. If your system has never seen a failure, your error handling is untested by definition.
Trace-first debugging beats log-first. This project exists partly because SigNoz found a 93x slowdown in this same simulator the week before the hackathon, a Windows IPv6 fallback on localhost that logs could never have shown me (I wrote that story up separately). The waterfall answers where, span attributes answer why.
Domain knowledge is the agent's moat. The MCP plumbing took an afternoon. The part that makes the agent useful, knowing that a 403 on a SOAP call usually means an ISSG policy gap and what to do about it, took years, and no amount of telemetry replaces it. Telemetry just delivers it faster.
Honest metrics require design. My dashboard showed 100% success while things were failing. Metrics report what you measure, not what you mean.

What I Would Do Next

A chat interface over the agent, richer WQL-level diagnosis (empty-result drift is a silent killer in reporting integrations), and pointing the same pattern at real integration platforms. Nothing here is Workday-specific in principle: named business-step spans plus a domain rule table plus an LLM to compose the narrative would work for any enterprise pipeline.

Wrap-Up

One week, one simulator, and a full observability loop: named traces, correlated logs, a live dashboard, a firing alert, and an agent that reads it all through MCP and answers like a colleague. The repo (with the Foundry casting.yaml and lock file for one-command reproduction) is here: https://github.com/ManjuVasanth/signoz-workday-agent

Built for the Agents of SigNoz hackathon by WeMakeDevs and SigNoz.

AI disclosure: I used Claude (Anthropic) as an assistant for debugging guidance, code drafting, and editing this post. The system design, Workday domain knowledge, testing, and all screenshots are from my own environment and experience.

My API Was 4 Seconds Slow. The Fix Was 9 Characters.

Manju Vasanth — Sat, 11 Jul 2026 21:20:02 +0000

I signed up for the Agents of SigNoz hackathon planning to warm up with the pre-event blog challenge: self-host SigNoz, send some data, write about a feature I liked. A quiet Saturday afternoon task.

Instead, SigNoz found a real performance bug in my own code within 30 minutes of sending it traffic. This is that story.

The App: A Mock Workday Tenant

To practice Workday integration patterns without touching a real tenant, I built a Python/Flask simulator that mimics a Workday environment: SOAP endpoints, RaaS reports, an Extend-style orchestration canvas, the works.

The flow I care about is a supervisory org provisioning orchestration. One button click runs a config-driven, multi-step pipeline: WQL lookups, widget value extraction, and SOAP calls against the simulator's own endpoints. It worked. It just always felt a little... sluggish. I assumed that was the cost of a chatty multi-step flow and moved on.

Spoiler: it was not.

Self-Hosting SigNoz (Note: Docker Compose Is Deprecated)

First surprise: if you follow older tutorials and clone the SigNoz repo looking for deploy/docker, it's gone. SigNoz recently moved installation to Foundry, a new CLI (foundryctl) that generates and deploys your whole stack from one declarative YAML file. It's new enough that barely anyone has blogged about it yet.

The entire install was three steps.

1. Install foundryctl (I'm on Windows, I grabbed the release binary and added it to PATH; on Mac/Linux there's a one-line install script).

2. Create casting.yaml:

apiVersion: v1alpha1
kind: Installation
metadata:
  name: signoz
spec:
  deployment:
    flavor: compose
    mode: docker

3. Deploy:

foundryctl cast -f casting.yaml

Foundry validated my Docker setup, generated Compose files into a pours/ directory, and started everything: SigNoz UI, OTel ingester, ClickHouse, ClickHouse Keeper, and Postgres.

docker compose -f pours/deployment/compose.yaml ps

Five services, all healthy. UI on http://localhost:8080, OTLP ingestion on 4317/4318. From zero to a running observability stack in under 15 minutes, most of which was image pulls.

Instrumenting Flask With Zero Code Changes

I wanted a baseline before writing any custom spans, so I used OpenTelemetry auto-instrumentation:

pip install opentelemetry-distro opentelemetry-exporter-otlp
opentelemetry-bootstrap -a install

Then launched my app through the OTel wrapper:

OTEL_RESOURCE_ATTRIBUTES=service.name=mock-workday-orchestrate \
OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317 \
OTEL_EXPORTER_OTLP_PROTOCOL=grpc \
opentelemetry-instrument python workday_ui.py

No code edits. The wrapper auto-instruments Flask and the requests library, so every incoming route and every outgoing HTTP call becomes a span.

I clicked through my orchestration flow 10-15 times to generate traffic, then opened SigNoz.

The First Look: Something Is Wrong

The Services page populated immediately: latency percentiles, request rate, Apdex, and a key operations table.

And it did not look good. My orchestration endpoint showed p99 latency of 4.21 seconds, and the Apdex score was sliding downhill with every run. For a local app calling itself on the same machine, that's absurd.

Time to open a trace.

The Waterfall: 99.5% of the Time Was Missing

I clicked into a 4.14 second trace for POST /extend/screen/create-sup-org. The flame graph and waterfall showed 5 spans:

POST /extend/screen/create-sup-org   4.14s   ← root
 ├─ GET  (client span)               2.05s
 │   └─ GET /tenant/soap/...         3.7ms
 └─ POST (client span)               2.08s
     └─ POST /tenant/soap/...        20.35ms

Two things jumped out.

First, the two internal calls ran sequentially and each took almost exactly 2 seconds. Suspiciously round, suspiciously identical.

Second, and this is the part that made me sit up: when I clicked the inner POST server span, the Span details panel showed it took 20.35ms, just 0.49% of total execution time. The actual handler doing the SOAP work was fast. The client span wrapping it took 2.08 seconds.

So roughly 2 seconds per call was being spent somewhere between "my code initiated the request" and "my server started handling it." On localhost. Twice per flow.

I didn't have any time.sleep() in the orchestration path (I grepped to be sure). This was real, invisible overhead.

The Smoking Gun Was a Span Attribute

The answer was sitting in the Span details panel the whole time, in the request attributes:

http.url: "http://localhost:8443/tenant/soap/human_resources"

That word, localhost, is the entire bug.

Here's what happens on Windows: when Python's requests library connects to localhost, the name resolves to both the IPv6 address (::1) and the IPv4 address (127.0.0.1), and IPv6 gets tried first. My Flask dev server was bound to IPv4 only. So every single internal call:

Attempts a connection to ::1 (IPv6)
Waits for that attempt to fail (roughly 2 seconds)
Falls back to 127.0.0.1 (IPv4)
Connects instantly, handler responds in milliseconds

A fixed 2-second tax on every internal HTTP call, completely invisible in application logs, because the application code was never slow. My "chatty flow is just sluggish" assumption had been this bug all along.

The 9-Character Fix

Replace localhost with 127.0.0.1 in the URLs my server calls at runtime.

One wrinkle worth sharing: my orchestration is config-driven, so the step URLs don't only live in Python source. They also live in a saved flow store (orchestrate_store.json). My first fix edited the Python defaults, restarted, and changed nothing, because the running flow loaded its URLs from the JSON store. Tracing the actual http.url attribute in SigNoz is what proved the old value was still live. One sed across the config and source files later:

sed -i.bak 's|http://localhost:8443|http://127.0.0.1:8443|g' orchestrate_store.json *.py

Restart, rerun the flow, open the newest trace.

Before and After

                     Before      After
Root span            4.18s   →   45ms
Internal GET         2.06s   →   11.4ms
Internal POST        2.11s   →   26.1ms

4.18 seconds down to 45 milliseconds. A 93x improvement, from changing 9 characters, found by reading one span attribute.

The new waterfall is beautiful: the client spans now hug their server spans instead of dwarfing them. And the Span details confirm the fix took: http.host: "127.0.0.1:8443".

My Favorite Feature: The Trace Waterfall + Span Attributes

The blog prompt asks for the feature I liked most, and after this experience it's not close: the trace detail view, specifically the combination of the waterfall and the Span details attribute panel.

Here's why. Logs told me nothing about this bug. My handler logs showed fast, healthy requests, because the handler was fast and healthy. Metrics alone would have told me "p99 is 4 seconds" without telling me where the time went.

The waterfall answered where: not in my handler (20ms), but in the client-side gap around it (2s). The span attributes answered why: the http.url showed exactly which hostname the live code was calling, which both exposed the IPv6 fallback and caught my incomplete first fix when the config store was still serving the old URL. Observation, localization, root cause, and verification, all in one screen, with zero custom instrumentation.

Honorable mention to Foundry. Coming from the Workday world where I think in config-driven deployments, "describe your stack in one YAML, run one command" felt immediately familiar, and it worked first try on Windows.

What's Next

This was supposed to be a warm-up, and it accidentally became a genuine debugging war story. For the main hackathon (Track 01, AI & Agent Observability), I'm building on this exact foundation: manual OpenTelemetry spans for every step of the orchestration pipeline, config-driven failure injection (401s, 403s, timeouts), and an AI diagnostic agent that reads SigNoz traces to explain why an integration step failed, not just that it did.

If a single auto-instrumented afternoon found a 93x slowdown, I'm very curious what step-level tracing will turn up.

Built for the Agents of SigNoz hackathon by WeMakeDevs and SigNoz. SigNoz is open source and OpenTelemetry-native; the self-host docs are here.

AI disclosure: I used Claude (Anthropic) as an assistant for debugging guidance and drafting this post. The setup, investigation, fix, and all screenshots are from my own environment.