DEV Community: DevGab

How AI Agents Verify Cross-Service Integrations: The 4-Step Verification Pattern

DevGab — Fri, 20 Mar 2026 01:07:39 +0000

The Bug Code Review Often Misses

Imagine implementing a feature perfectly — clean code, passing tests, approved PR. Now imagine that feature is completely unreachable because the service that was supposed to call it never did. No error. No failing test. Just silent, dead code sitting in production.

This is the class of bug that lives at integration boundaries — between repositories, services, teams, async producers and consumers, or config layers. It's largely invisible to traditional code review because reviewers usually operate closer to the PR boundary than the architectural boundary. The failure often isn't obvious in any one file — it's in the missing link between pieces of code that each look correct in isolation.

What I want to dig into here is a structured verification pattern I've found effective for AI-assisted audits of these gaps. You can apply the same logic manually or build tooling around it.

Why Grep Isn't Enough (But You Still Need It)

When auditing integration boundaries, one naive approach is to grep for method names across repos. Found it? Great, it exists. Didn't find it? Flag it as missing. This misses half the problem.

Existence isn't the same as reachability. A method can exist but be private. It can exist but only be called in a test context. It can exist but be conditionally gated behind a feature flag that's disabled everywhere. Verification has to go deeper than a filename match.

The 4-Step Verification Pattern

When auditing an integration boundary, a structured checklist I've found works well — whether you're prompting an AI agent or doing it manually — follows this chain:

Step 1: Confirm the Contract Exists

Start with the simplest question: does the method, endpoint, or hook actually exist in the codebase? What you're searching for depends on the boundary type:

In-process: method definition exists
HTTP/API: route and controller action exist
Async: event is emitted, consumer/worker is subscribed
Config: flag or env var is defined and wired into deployment
Data: writers and readers agree on keys and shape

# Does the method exist anywhere in this repo?
grep -r "def render_for_all_platforms" ./services/

# Does a route/action exist for it?
grep -rn "render_for_all_platforms\|render_all" ./config/routes.rb ./app/controllers/

This is table stakes. If the target doesn't exist, nothing else matters. But passing this step means almost nothing on its own.

Step 2: Check the Access Modifier

This is the step most manual reviews skip. In ordinary Ruby code, a private method can't be called with an explicit receiver — so it's a strong signal that it's not intended as a cross-boundary entry point. (Ruby being Ruby, send and metaprogramming can bypass this, but if you're relying on that for a planned integration, something has gone wrong.)

# Red flag: method is private but expected to be called from a job
class VideoRenderService
  def trigger_render(platform:)
    # public, callable — but only handles single-platform renders
  end

  private

  def render_for_all_platforms
    # intended to be called by RenderJob, but private makes it unreachable
  end
end

An agent or reviewer auditing this would flag it: the plan says render_for_all_platforms should be invoked from the job scheduler, but the access modifier prevents normal invocation. A reviewer looking only at the job scheduler PR would likely miss this.

Step 3: Find the Caller

Even if the method is public, you need to verify something actually calls it. Search across all affected repositories, not just the one where the method lives.

# Search across multiple repos (run from a parent directory)
grep -r "render_for_all_platforms" ./cms-service/
grep -r "render_for_all_platforms" ./analytics-service/
grep -r "render_for_all_platforms" ./renderer/

# If using a monorepo or shared tooling:
grep -r "render_for_all_platforms" . --include="*.rb" --include="*.ts"

Zero results across all repos when you expect a caller? That's a strong signal the feature may be unreachable. It's implemented, it works in isolation, but nothing triggers it. (Caveats: dynamic dispatch, reflection, generated clients, or external callers outside the repos you searched can produce false negatives — but zero grep hits should always trigger investigation.) This is exactly the kind of issue that slips through when PRs are reviewed in sequence by different engineers across different weeks.

Step 4: Trace the Wiring

The final step is confirming the plumbing — environment variables, feature flags, configuration values — actually connects the caller to the implementation.

# Does the calling code depend on a flag that's never set?
class RenderJob
  def perform(video_id)
    if ENV['MULTI_PLATFORM_RENDERING_ENABLED'] == 'true' # ← Is this set?
      VideoRenderService.new.render_for_all_platforms
    else
      VideoRenderService.new.render_single_platform
    end
  end
end

# Check if the env var is defined anywhere
grep -r "MULTI_PLATFORM_RENDERING_ENABLED" ./config/
grep -r "MULTI_PLATFORM_RENDERING_ENABLED" ./.env*
grep -r "MULTI_PLATFORM_RENDERING_ENABLED" ./infrastructure/

Finding the caller but not the configuration wiring means the feature is structurally present but effectively disabled. Depending on your defaults, this could mean the new code never runs — or worse, always runs when it shouldn't.

Applying This to AI Agent Design

If you're building or prompting AI agents to do this kind of audit, the pattern translates directly into a structured task definition. Here's a minimal prompt template for a phase-level audit agent:

You are auditing Phase [N]: [Phase Name].

For each integration point listed in the plan document:
1. Confirm the method/endpoint exists (exact name match)
2. Check its access modifier — is it callable by the expected caller?
3. Search all provided repositories for callers of this method
4. Verify any required configuration (env vars, flags) is wired up

Report findings as JSON with fields:
- integration_point: string
- exists: boolean
- access_modifier: "public" | "private" | "protected" | "unknown"
- callers_found: string[] (file paths)
- config_wired: boolean
- risk: "none" | "low" | "medium" | "high"
- notes: string

Asking the agent to return structured JSON means you can aggregate findings across multiple agents running in parallel on different phases, deduplicate overlapping issues, and prioritise by risk before handing off to human engineers for validation.

The Data Path Variant: JSONB and Schema Drift

The 4-step pattern works for method-level integrations, but there's a nastier variant: data contract mismatches. In systems that share a database — or where multiple code paths within one application write to the same semi-structured column — schema drift can be silent and painful.

# Service A writes metadata like this:
video.update!(metadata: { narration_url: url, timing_data: timings })

# Service B reads metadata like this:
video.metadata['tts_url'] # ← Different key, silently returns nil

For this variant, the verification pattern shifts. Instead of tracing callers, you trace writers and readers of the same data structure. Search for all write paths to the column, all read paths from the column, and map out which keys are written versus which keys are read. Mismatches are your bugs.

Running Agents in Parallel, Not Sequence

One design choice that can improve coverage: run agents for different phases simultaneously rather than one after another. Sequential auditing risks each agent's findings influencing the framing of the next, and it's just slower.

Parallel agents produce independent findings that you aggregate afterward. This catches issues that a single sequential pass might rationalise away — if one agent flags a dead method and a separate agent (with no knowledge of that finding) independently identifies that the expected caller has a bug, those two findings together paint a much clearer picture than either does alone.

Human Validation Is Non-Negotiable

It's worth being direct about this: AI agents surface candidates for issues. They're not a replacement for a human engineer confirming the finding against a live codebase.

Static search — whether by grep, an AI agent, or an AST tool — has real blind spots: dynamic dispatch, reflection, metaprogramming, framework conventions that invoke methods by naming pattern, generated clients, runtime config injection, and external callers that live outside the repos you searched. Any of these can make reachable code look unreachable, or vice versa.

Treat agent output as a prioritised investigation list, not a definitive bug report. The value is in the structured, cross-repo coverage — catching the categories of issues that would slip through — not in the infallibility of each individual finding.

Building Your Own Cross-Repo Audit

You don't need a sophisticated AI setup to apply this pattern today. Even a rough script that runs the first few verification steps across repos will surface issues. Here's an illustrative starting point (not production-grade, but enough to demonstrate the approach):

#!/usr/bin/env bash
set -euo pipefail
# Usage: ./audit_integration.sh render_for_all_platforms ./cms ./analytics ./renderer

METHOD="${1:?method name required}"
shift

printf '=== Auditing: %s ===\n' "$METHOD"

for REPO in "$@"; do
  printf '\n--- Searching in %s ---\n' "$REPO"
  # Step 1: Definitions
  printf 'Definitions:\n'
  grep -R --include='*.rb' -nE "def[[:space:]]+${METHOD}\b" "$REPO" || true
  # Step 3: References (excluding specs)
  printf 'References:\n'
  grep -R --include='*.rb' -n "${METHOD}" "$REPO" \
    | grep -vE '(_spec\.rb|/spec/)' || true
done

This is obviously a starting point — you'd want to add access modifier parsing, config checking, and structured output for aggregation. But it demonstrates the core idea: systematically checking existence and reachability across repo boundaries.

The Payoff

The methodology described here found ten real issues in one internal audit — including a fully-implemented feature that was never invoked. That's not a minor edge case catch; that's weeks of engineering work sitting invisible in production.

Integration boundary bugs are expensive because they're hard to find and easy to miss at review time. A structured, phase-level verification approach — whether run by AI agents or methodical engineers — is one of the more effective ways to catch this class of issue before it reaches users.

This article builds on a deeper exploration at devgab.com.

Have you run into integration bugs that slipped through code review because the gap lived between PRs? What's your current approach to auditing work that spans multiple repos — and would you trust an AI agent to surface those gaps?

Unpopular Opinion: Most Webhook Implementations Are Dangerously Half-Baked

DevGab — Wed, 18 Mar 2026 03:52:31 +0000

There. I said it.

After reviewing dozens of Rails codebases over the years, I've come to a uncomfortable conclusion: webhook implementations are almost universally treated as an afterthought. We copy-paste a Stripe tutorial, slap a before_action :verify_signature on a controller, and call it production-ready. It's not.

The "Just Receive It" Mentality Is Costing Us

Ask most developers what their webhook setup looks like and you'll hear some version of: "Oh, we have an endpoint that Stripe hits and we process the event." That's it. No mention of retry handling. No idempotency. No audit trail. Definitely no outbound webhooks.

This matters because real production systems are almost never one-directional. You receive events from Stripe, yes — but you're also probably sending events to your partners, your analytics pipeline, your fulfillment vendor. That outbound flow gets built as a hastily-assembled HTTParty.post buried in an ActiveRecord callback somewhere. And then it silently fails on a Tuesday at 2am.

Inbound ≠ Outbound, and Treating Them the Same Is the Root Problem

Here's the thing that took me an embarrassingly long time to internalize: receiving and sending webhooks are architecturally opposite problems, even though they look similar on the surface.

When you receive a webhook, you don't control the retry policy. You don't control when events arrive. You don't control what happens if you return a 500. The sender might retry three times, or thirty times, or never again — and you won't know until orders stop processing. Your job is to be fast, forgiving, and idempotent. Acknowledge first, process later.

# This is the correct pattern for inbound webhooks.
# Notice what it does NOT do: heavy processing inline.
def create
  return head :unauthorized unless valid_signature?

  # Acknowledge immediately — do NOT make the sender wait
  WebhookProcessorJob.perform_later(request.raw_post, provider: 'stripe')
  head :ok
end

When you send a webhook, the calculus flips entirely. Now you're responsible for the retry strategy. A network timeout is a transient failure — retry it with backoff. A 400 Bad Request means your payload is wrong — retrying it a hundred times won't fix that. These are fundamentally different failure modes that need different responses, and conflating them leads to both missed deliveries and infinite retry loops.

class OutboundWebhookJob < ApplicationJob
  # Transient errors (timeouts, 5xx): retry with backoff
  retry_on WebhookService::NetworkError, wait: :polynomially_longer, attempts: 10

  # Permanent errors (4xx, bad payload): stop immediately
  discard_on WebhookService::BadPayloadError

  def perform(endpoint_id, event_type, payload)
    WebhookService.deliver!(endpoint_id, event_type, payload)
  end
end

"But We Use Sidekiq Retries" Doesn't Cut It

I hear this one a lot. Yes, Sidekiq retries are great. No, they are not a substitute for intentional error handling in your webhook layer. Sidekiq doesn't know the difference between a 408 timeout and a 422 validation error — it just retries both. That means you're hammering a partner endpoint with a malformed payload 25 times, generating noise in their logs and potentially getting your IP rate-limited or blocked.

Smart retry logic needs to live in your application code, not be delegated entirely to your job queue's default behavior.

The Security Asymmetry Nobody Talks About

Here's another asymmetry that rarely gets called out explicitly: inbound and outbound webhooks use the same cryptographic primitive (HMAC-SHA256 is pretty much the industry standard) but with completely opposite trust models.

For inbound webhooks, you receive someone else's signature and verify it against a shared secret they gave you. For outbound webhooks, you sign your own payloads with a secret you gave to your receiver. Same algorithm, opposite direction of trust. If you build a generic "webhook signature" utility without accounting for this, you'll end up with subtle security bugs or, worse, a system where you accidentally expose your own signing key to the wrong party.

What "Production-Ready" Actually Looks Like

I'd argue that a truly production-ready webhook system needs at minimum:

Idempotency keys on inbound processing — deduplicate events before acting on them
A polymorphic audit log for both directions — you need to answer "what did we receive, and what did we send, and when?"
Differentiated retry logic on outbound — transient vs. permanent failures are not the same
Independent secret rotation per integration — one compromised partner shouldn't require you to rotate everything
Timeouts on outbound requests — seriously, don't let a slow partner endpoint block your job queue workers

Most apps I've seen have maybe two of these five. Some have none.

Why Do We Keep Getting This Wrong?

Honestly? I think it's because webhooks feel simple. It's just HTTP, right? POST to an endpoint, return 200, done. The failure modes are invisible — silent data loss doesn't throw an exception you can see in your error tracker. An outbound webhook that fails with no retry never shows up as an error unless you're explicitly monitoring for it.

We've also been conditioned by the Stripe integration tutorial, which is genuinely excellent but only covers one direction and one provider. It's a starting point, not a template for your entire webhook architecture.

The Takeaway

Stop treating webhooks as a solved problem. Start treating them as an integration boundary that deserves the same architectural rigor you'd give to any other critical data pipeline. Model the two directions separately. Make failure explicit. Build the audit trail. Your on-call rotation will thank you.

Webhooks are not hard to get right — but they require intentionality that most tutorials never bother to teach.

This article builds on a deeper exploration at devgab.com — Building Resilient Webhook Systems: A Tale of Two Directions, which includes full implementation details for signature verification, polymorphic audit trails, and retry strategies in Rails.

What does your webhook setup look like in production? Are you handling both directions explicitly, or is outbound still a Net::HTTP.post in a model somewhere? Drop your setup (or your horror stories) in the comments — genuinely curious where the community is at on this.

Rails' Four-Layer Contract: Why Every Feature Needs a Route, Policy, Controller, AND Model Method

DevGab — Tue, 17 Mar 2026 21:22:48 +0000

Here's a scenario that should terrify you: a user clicks a button, sees no error, and walks away assuming their action succeeded. Meanwhile, on the server, something quietly did nothing — or worse, did the wrong thing.

This isn't a hypothetical. It's what happens when you add a UI element to a Rails app without completing what I call the four-layer contract : Route → Policy → Controller → Model. Miss any single layer, and you get silent failures that won't show up in your logs or error trackers.

Let's break this down layer by layer, look at exactly how each one fails, and build a checklist you can use every time you add a new action to your app.

The Four Layers, Visualized

Every user-initiated action in a Rails app travels through this chain:

Browser Request
       │
       ▼
┌─────────────┐
│    Route    │ Does this URL + verb map to an action?
└──────┬──────┘
       │
       ▼
┌─────────────┐
│    Policy   │ Is this user allowed to do this?
└──────┬──────┘
       │
       ▼
┌─────────────┐
│  Controller │ Orchestrate: parse params, call model, render
└──────┬──────┘
       │
       ▼
┌─────────────┐
│    Model    │ Execute the actual business logic
└─────────────┘

The critical thing to understand: each layer fails differently. Some fail loudly (a 500 error), some fail silently (a 403 you never see), and some fail invisibly (data gets saved but wrong). Let's go through each one.

Layer 1: The Route

What happens when it's missing

If you reference a named route in a view that doesn't exist in routes.rb, Rails raises a NoMethodError on the route helper itself — but only when the view renders. If the button is conditionally rendered (e.g., only for certain roles), this can stay hidden in development and only surface in production for specific users.

<%# This line will raise NoMethodError if the route doesn't exist %>
<%= button_to "Submit for Review", transition_to_review_workflow_path(@record) %>

The sneakier failure is a copy-paste error: you meant to write transition_to_review_workflow_path but pasted transition_to_approved_workflow_path instead. Both are valid helper methods, no error is raised, but your button now triggers the wrong action entirely. No exception anywhere — just a workflow that silently does the wrong thing.

The fix

# config/routes.rb
resources :workflows do
  member do
    post :approve
    post :reject
    post :transition_to_review # ← add this explicitly
  end
end

Run rails routes | grep workflow after every new action to verify the route exists before writing any other code.

Layer 2: The Policy (Pundit)

What happens when it's missing

This one catches people off guard. If you're using Pundit and your policy class doesn't define a transition_to_review? method, you get a NoMethodError — a 500, not a 403. That's actually the good failure mode because it's loud and obvious.

The truly devious failure is when the policy method exists but has the wrong logic — typically from copying another method and forgetting to update the conditions:

# app/policies/workflow_policy.rb
class WorkflowPolicy < ApplicationPolicy
  def approve?
    user.admin?
  end

  def reject?
    user.admin? || user.reviewer?
  end

  # Copied from approve? but forgot to update the conditions
  # Authors should be able to submit for review, but this only allows admins
  def transition_to_review?
    user.admin?
  end
end

Now your authors click "Submit for Review" and get redirected with a "You're not authorised" flash message. The bug gets filed as a permissions issue. Someone spends an hour checking role assignments before realising the policy method is simply wrong. Meanwhile, a missing method would have pointed directly to the problem in seconds.

The fix

# app/policies/workflow_policy.rb
class WorkflowPolicy < ApplicationPolicy
  def transition_to_review?
    # Be explicit about who can trigger this action
    user.author? && record.draft?
  end
end

Every route you add in routes.rb should have a corresponding policy method with its own logic — never copy from another method without reviewing the conditions. Make this a code review requirement.

Layer 3: The Controller Action

What happens when it's missing (or wrong)

A missing controller action raises a AbstractController::ActionNotFound — this one is at least loud. But the silent failure mode is more interesting: the controller action exists but reads the wrong param key.

# What the form sends:
# { "notes" => "Ready for review", "record_id" => "42" }

def transition_to_review
  authorize @record

  # BUG: reads :transition_notes, but form sends :notes
  note_text = params[:transition_notes]

  @record.transition_to_review!(note: note_text)
  redirect_to @record, notice: "Submitted for review."
end

note_text is nil. The record saves. The audit log has blank notes. No exception is raised. This is the category of bug that takes 3 hours to track down because everything appears to work.

The fix: use Strong Parameters defensively

def transition_to_review
  authorize @record

  # Explicitly permit and alias if needed
  transition_params = params.require(:workflow).permit(:notes)

  @record.transition_to_review!(note: transition_params[:notes])
  redirect_to @record, notice: "Submitted for review."
end

private

def set_record
  @record = Workflow.find(params[:id])
end

Using params.require() means Rails will raise an ActionController::ParameterMissing error if the top-level key is absent — making the contract between form and controller explicit and enforced.

Layer 4: The Model Method

What happens when it's missing

This is the most visible failure: a clean NoMethodError pointing directly at the missing method. Celebrate this one — it's honest. The problem is what comes after you add the method without thinking about side effects.

# The naive implementation
def transition_to_review!(note:)
  update!(status: :pending_review, review_notes: note)
end

If you also have a before_update callback that creates audit records on status changes, you now have two code paths both writing audit records. The controller might also manually build an audit record. Result: every transition creates duplicate audit entries, and your audit trail is permanently corrupted.

The fix: make the model method the single source of truth

# app/models/workflow.rb
def transition_to_review!(note:)
  transaction do
    update!(status: :pending_review)
    audit_records.create!(
      action: :transition_to_review,
      notes: note,
      performed_by: Current.user
    )
  end
end

Then remove any duplicate audit logic from callbacks or controllers. One method, one responsibility, one writer.

The Four-Layer Checklist

Print this out and tape it to your monitor:

New Rails Action Checklist
==========================

□ Route added to routes.rb
  → Run: rails routes | grep <action_name>

□ Policy method defined
  → Matches exact action name: def <action_name>?
  → Tested with correct and incorrect user roles

□ Controller action implemented
  → Params read with correct keys (verify against form field names)
  → authorize called before any data mutation
  → Single exit path (no branching redirects without coverage)

□ Model method implemented
  → Wrapped in transaction if multiple writes
  → Audit record written in ONE place only
  → Callbacks audited for conflicts

Catching This Class of Bug Earlier

Beyond the checklist, there are two Rails-native techniques that surface layer mismatches before production:

1. Route-aware integration tests. Write a request spec for every new action that asserts a 200 (or redirect) response. A missing route causes the spec to fail immediately rather than waiting for a user report.

# spec/requests/workflows_spec.rb
RSpec.describe "Workflows", type: :request do
  describe "POST /workflows/:id/transition_to_review" do
    it "transitions the workflow and returns a redirect" do
      sign_in author_user
      post transition_to_review_workflow_path(workflow), params: {
        workflow: { notes: "Ready for review" }
      }
      expect(response).to redirect_to(workflow_path(workflow))
      expect(workflow.reload.status).to eq("pending_review")
    end
  end
end

2. Policy spec coverage for every action. Using pundit-matchers, write a spec that explicitly asserts which roles can and cannot access each action. A missing policy method causes the spec to fail loudly.

# spec/policies/workflow_policy_spec.rb
describe WorkflowPolicy do
  subject { described_class.new(user, workflow) }

  context "transition_to_review" do
    context "when user is author and workflow is draft" do
      let(:user) { build(:user, :author) }
      let(:workflow) { build(:workflow, :draft, author: user) }

      it { is_expected.to permit_action(:transition_to_review) }
    end

    context "when user is a reviewer" do
      let(:user) { build(:user, :reviewer) }
      it { is_expected.to forbid_action(:transition_to_review) }
    end
  end
end

The Mental Model Shift

The real lesson here isn't about Rails mechanics — it's about how we think about "done." When a designer updates a workflow UI and the button looks right in the browser, it feels done. But the UI is just a promise. The backend is where you keep it.

Every new button, every new form, every new link is really four tasks masquerading as one. Treat it that way, and silent failures become a lot rarer.

This article builds on a deeper exploration at The Hidden Costs of UI-First Development: A Rails Workflow Bug Postmortem.

What's your approach to preventing this kind of layer mismatch? Do you use code generation, templates, or strict PR checklists? I'd love to hear how your team handles it. 👇