DEV Community

Manju Vasanth
Manju Vasanth

Posted on

I Taught an AI Agent to Debug Workday Integrations Like a Senior Consultant

#signoz #ai #opentelemetry #observability

Built for the Agents of SigNoz hackathon: OpenTelemetry + SigNoz + MCP + a decade of enterprise integration scar tissue.

When a Workday integration fails, the error message almost never tells you the real story. A raw 403 Forbidden does not say "your Integration System User's security group is missing a domain security policy." Figuring that out takes a person who has been burned by it before. I have spent years being that person, so for this hackathon I asked a specific question: can I bottle that tribal knowledge into an agent that reads telemetry and answers like a senior consultant would?

By the end of the week, the answer was yes, and the road there included my own code crashing in ways I did not expect, twice.

What I Built

Three layers:

  1. A mock Workday tenant. A Python/Flask simulator I built to practice integration patterns: SOAP endpoints, RaaS reports, and an Extend-style canvas that runs a config-driven, multi-step supervisory org provisioning orchestration (validate input, check existence, build a SOAP payload, call the tenant, extract results).
  2. Full observability via SigNoz. OpenTelemetry instrumentation on every orchestration step, streaming traces, logs, and dashboards into self-hosted SigNoz, deployed with Foundry.
  3. An AI diagnostic agent. A Python agent that talks to SigNoz's MCP server, pulls the newest failed trace, finds the failing step, and maps it to a Workday-specific root cause with fix steps.

Instrumenting a Config-Driven Pipeline

The simulator's orchestration engine executes steps from a JSON store, which made instrumentation surprisingly clean: wrap the step executor once, and every step gets a span automatically. The core of it:

def exec_steps(steps, depth=0):
    for i, s in enumerate(steps, start=1):
        typ = s.get("type", "unknown")
        ref = s.get("ref", typ)
        span_name = "step-%02d.%s.%s" % (i, typ, ref)
        with _tracer.start_as_current_span(span_name) as span:
            span.set_attribute("step.index", i)
            span.set_attribute("step.type", str(typ))
            span.set_attribute("step.ref", str(ref))
            if s.get("inject"):
                span.set_attribute("chaos.injected", str(s.get("inject")))
            try:
                exec_one(s, depth)
            except Exception as e:
                span.record_exception(e)
                span.set_status(Status(StatusCode.ERROR, str(e)))
                raise
Enter fullscreen mode Exit fullscreen mode

One edit, and the SigNoz waterfall went from anonymous GET/POST spans to a readable business flow: step-02.send-http-request.CheckExists, step-05.send-http-request.CallHumanResources. I also hooked the engine's existing log function so every step message becomes a span event and a trace-correlated log line. My old "eye-catcher" debug strings suddenly carried trace IDs.

Chaos Injection: Breaking It on Purpose

To give the agent something to diagnose, I added config-driven failure injection. Adding one key to any step in the flow store simulates a realistic Workday failure:

{ "ref": "CallHumanResources", "inject": "403", ... }
Enter fullscreen mode Exit fullscreen mode

Modes cover the failures I actually see in integration work: 400 (malformed WQL, usually an unescaped apostrophe in a name filter), 401 (expired ISU token), 403 (missing ISSG domain permission), 404 (bad reference), timeout, and empty result sets.

Here is the part the tutorials do not tell you: my first injected failure crashed my own application. The screen handler behind the orchestration had only ever rendered success. When the flow returned an error result, it died on a NoneType at line 196, then, after I guarded that line, at line 197. Chaos engineering found my unhandled error path before it found anything else. The traceback, fittingly, was waiting for me in SigNoz's Exceptions view, grouped and linked to the exact trace.

The Agent: Telemetry In, Workday Answers Out

SigNoz ships an MCP server (enabled through one block in Foundry's casting.yaml), and it exposes 41 tools. My agent uses three: signoz_search_traces to find the newest ERROR trace for the service, signoz_get_trace_details for the span tree, and the span attributes and status messages as evidence.

The diagnosis itself is a rule table built from experience, the mapping I would apply manually on a support ticket. The 403 entry, for example:

(re.compile(r"HTTP 403|Forbidden|lacks domain security", re.I),
 "ISSG missing domain security policy permission",
 "The ISU authenticated successfully but its security group lacks "
 "Get/Put access on the target domain.",
 ["Grant the ISU's security group the required domain security policy permission",
  "Activate pending security policy changes in the tenant",
  "Re-run the step; a 200/201 confirms the grant took effect"])
Enter fullscreen mode Exit fullscreen mode

When a run fails, the agent prints the failing step, the root cause in Workday language, fix steps, and a pipeline map with the failure flagged. It also discloses when a failure was injected, if chaos.injected is present, the report says so. An agent that quietly presents staged failures as organic ones would be a demo trick, not a tool.

A confession that belongs in this section: the agent's very first run failed with a 403 of its own. I had created the SigNoz service account but skipped the role assignment step, so the MCP server rejected every query with "only viewers/editors/admins can access this resource." My permission-diagnosing agent was blocked by a missing permission. I fixed it the way its own rule table would have suggested.

The Dashboard That Called My Bluff

I built a four-panel "Workday Integration Health" dashboard with the Query Builder: p95 latency per step, failures by step, error logs, and a formula panel computing success rate as A/B*100 over root spans.

The formula panel immediately exposed a design flaw. It read 100% success while injected failures were clearly happening. The reason: my chaos hook failed the step span but let the flow complete gracefully, so the root orchestration.run span stayed clean. A run with a failed step was counting as a successful run. One line in the root wrapper fixed it, propagate ERROR to the root if any step in the trace log failed, and the panel dropped to an honest 92.5%.

I finished with an alert rule on the failure count (above 0, at least once, 5-minute rolling window). One injected run later:

What I Learned

  1. Chaos finds your bugs first. Both crashes this week were in my code's error paths, not in the failures I injected. If your system has never seen a failure, your error handling is untested by definition.
  2. Trace-first debugging beats log-first. This project exists partly because SigNoz found a 93x slowdown in this same simulator the week before the hackathon, a Windows IPv6 fallback on localhost that logs could never have shown me (I wrote that story up separately). The waterfall answers where, span attributes answer why.
  3. Domain knowledge is the agent's moat. The MCP plumbing took an afternoon. The part that makes the agent useful, knowing that a 403 on a SOAP call usually means an ISSG policy gap and what to do about it, took years, and no amount of telemetry replaces it. Telemetry just delivers it faster.
  4. Honest metrics require design. My dashboard showed 100% success while things were failing. Metrics report what you measure, not what you mean.

What I Would Do Next

A chat interface over the agent, richer WQL-level diagnosis (empty-result drift is a silent killer in reporting integrations), and pointing the same pattern at real integration platforms. Nothing here is Workday-specific in principle: named business-step spans plus a domain rule table plus an LLM to compose the narrative would work for any enterprise pipeline.

Wrap-Up

One week, one simulator, and a full observability loop: named traces, correlated logs, a live dashboard, a firing alert, and an agent that reads it all through MCP and answers like a colleague. The repo (with the Foundry casting.yaml and lock file for one-command reproduction) is here: https://github.com/ManjuVasanth/signoz-workday-agent

Built for the Agents of SigNoz hackathon by WeMakeDevs and SigNoz.

AI disclosure: I used Claude (Anthropic) as an assistant for debugging guidance, code drafting, and editing this post. The system design, Workday domain knowledge, testing, and all screenshots are from my own environment and experience.

Top comments (0)