Building Sub-5ms Ingestion Middleware for ServiceTitan and Jobber: Part Two - Edge Validation, Telemetry Flattening, and Automated DLQ Isolation

When you scale paid traffic across Google Local Service Ads and Meta Lead Ads into home service CRMs, basic webhook receivers quickly break down.

Standard low-code tools choke when a contact form sends unformatted phone numbers with extensions, nested form objects, or malformed JSON keys.

Building a production-grade ingestion core requires handling data normalization, parsing messy address strings, and quarantining corrupted payloads before they ever hit downstream APIs like ServiceTitan, Jobber, or GoHighLevel.

The core pipeline logic handles schema flattening, strict telephony validation, and dead-letter quarantine entirely in memory.
Universal Ingress Flattening and Schema Normalization

Lead sources never format JSON the same way. Google Lead Forms dump fields into user_column_data, Meta uses field_data, and custom website forms use nested objects or arbitrary custom keys.

The pipeline flattens these disparate schemas into a single normalized payload right at the ingress boundary:
JavaScript

const rawMerged = Object.assign(
Object.create(null),
arrayExtracted,
hsProps,
bodyData.fields || {},
bodyData.data || {},
bodyData.lead || {},
bodyData.customer || {},
bodyData.customData || {},
bodyData.custom_data || {},
bodyData.contact || {},
parsedPayloadObj || {},
bodyData.record || {},
bodyData
);

This ensures that regardless of whether the inbound webhook originates from an LSA form extension, a Facebook lead ad, or a custom landing page, the pipeline resolves all primary attributes uniformly before validation runs.
Strict Telephony and Extension Isolation

Phone numbers submitted by homeowners are notoriously messy. Users frequently include extensions, multiple numbers separated by slashes, or domestic formatting that fails strict North American Numbering Plan validation checks.

The engine extracts phone extensions, isolates primary numbers, validates NANP rules, and reformats them to strict E.164 standards:
JavaScript

let phoneBase = rawPhone.replace(/^tel:/i, '').trim();
let phoneExtension = '';
const extMatch = phoneBase.match(/(?:\s+|[,;(-]\s*)(?:ext.?|extension|x|#)\s*(\d{1,6}))?\s*$/i);

if (extMatch) {
phoneExtension = extMatch[1];
phoneBase = phoneBase.slice(0, extMatch.index).trim();
}

let digits = phoneBase.replace(/\D/g, '');
if (digits.length === 11 && digits.startsWith('1')) digits = digits.slice(1);

const isNanpValid = /^2-9\d{2}2-9\d{2}\d{4}$/.test(digits);
const cleanPhone = isNanpValid ? '+1' + digits : '';

By stripping invalid characters and handling phone extensions cleanly, outbound dispatch tools never receive malformed contact data that breaks auto-dialers or creates duplicate profiles.
Address Parsing and Dead-Letter Quarantine (DLQ)

Single-line address inputs frequently break CRM location creation because standard webhooks cannot distinguish street names from unit numbers or postal codes. The pipeline parses raw address strings, extracts postal codes and state codes, and maps structured data for downstream CRM adapters:
JavaScript

let street = rawAddress;
let unit = explicitUnit;
let city = explicitCity || '';
let state = normalizeState(explicitState);
let zip = explicitZip;

let workingAddress = rawAddress
.replace(/,\s*(?:united\s+states|u.?s.?a.?|canada)\s*.?$/i, '')
.trim();

If a payload contains junk data, spam keywords, or unrecoverable formatting errors, the pipeline routes it directly to a Dead-Letter Queue envelope with PII redaction enabled:
JavaScript

if (isDuplicateEvent) {
payload.json.status = 'QUARANTINED_DUPLICATE_EVENT';
payload.json.failure_class = 'IDEMPOTENCY_LOCK_DROP';
payload.json.quarantine_reason = 'Duplicate Fingerprint (' + payloadHash + ')';
payload.json.dlq_replay_envelope = { raw_body: redactEnvelope(bodyData), ip_origin: ipAddress };
quarantinedPayloads.push(payload);
} else if (isJunk) {
payload.json.status = 'FLAGGED_JUNK_PAYLOAD';
payload.json.failure_class = 'PERMANENT_POLICY_REJECT';
payload.json.quarantine_reason = 'Triggered Spam/Profanity Filter';
payload.json.dlq_replay_envelope = { raw_body: redactEnvelope(bodyData), ip_origin: ipAddress };
quarantinedPayloads.push(payload);
}

This architecture guarantees that corrupted webhooks, spam entries, and duplicate submissions never pollute active CRM environments, keeping dispatch queues clean and accurate for service technicians.