The worst time to discover a database connector's assumptions is when the first executive asks it a production question.
The connection succeeds. The answer sounds polished. Later, someone notices it used the wrong timezone, counted test accounts, inherited a stale metric, or exposed a field the user should not see.
A database connector should enter production through shadow mode and a controlled canary—not through a successful demo.
Build a versioned replay set that includes:
- governed business questions
- ambiguous requests that should trigger clarification
- unauthorized requests that should be refused
- stale, truncated, and partial-result cases
- database timeout and cancellation cases
Compare semantics, not prose. Check identity, tenant, metric version, source, filters, freshness, row count, truncation, policy decision, and redaction profile.
Then shadow representative live questions under a separate workload budget. Measure database impact as well as answer quality.
Only after those gates pass should a small canary group see the answer.
And define rollback criteria before launch. “It feels unreliable” is not an operational threshold.
Full rollout guide: ChatGPT database connection shadow mode
Top comments (0)