This article was created for the Google Cloud All Things Agentic Hackathon.
Inspiration
When a pharmaceutical batch has a deviation, quality teams must collect
evidence from many places. They check manufacturing records, lab results,
environmental data, supplier documents, and maintenance systems. This takes
time, and important decisions still need a clear audit trail.
AI can help with the coordination, but it should not decide whether a batch can
be released. That decision belongs to a qualified human.
We built Countersign to reduce the manual work. It investigates deviations
across departments, handles long waits, and carries out approved follow-up
actions. The Qualified Person always keeps final batch authority.
What it does
A user opens a deviation case in a simple web desk. Countersign then:
- Classifies the structured intake and securely binds the submitted evidence.
- Runs five specialist agents for manufacturing, quality control, environmental monitoring, supply, and maintenance.
- Reads evidence from Google Drive, SENAITE LIMS, BigQuery, ERPNext, and openMAINT.
- Drafts an investigation, challenges the draft, and blocks unsupported claims with a deterministic citation gate.
- Waits safely for sterility incubation when required, even if the wait lasts days or weeks.
- Builds an EU GMP Annex 16 review package and pauses for the Qualified Person.
- After the human decision, files the quality report, creates ERPNext CAPA records, opens an openMAINT corrective request, and notifies Sponsor QA.
Countersign also has a separate cross-functional investigation workflow. For a
recurring operator report with patient-safety impact, a change-controlled rule
requires the first enquiry to be an operator interview. This rule uses
structured fields, not the free-text description.
We proved this full route live: GUI intake, Google Chat question, authorized
human reply, Scheduler resume, deterministic CFI closure, Calendar review, and
Google Docs report.
The safety rule is simple: agents gather evidence and perform approved
actions, while the Qualified Person makes the batch decision. Every
machine-produced decision record carries decision_authority: none.
How we built it
Countersign uses Gemini 3.5 Flash Lite on Vertex AI with the Google Agent
Development Kit (ADK). The main case workflow and the cross-functional
investigation run as separate managed Agent Runtimes.
ADK runs the department agents in parallel. Deterministic code handles
classification, compliance checks, citations, lifecycle changes, CFI trigger
rules, idempotency, and human approval. Model-written text cannot silently
become a regulatory decision.
The platform includes:
- Agent Registry to catalog the approved fleet and tool boundaries.
- Agent Identity for workload identity without embedded credentials.
- Agent Gateway to control which identity can call each MCP tool.
- Model Armor and content controls at model and tool boundaries.
- Agent Observability, Cloud Trace, and Cloud Logging for execution evidence.
- Vertex-managed sessions for the deployed case workflow.
- Cloud SQL for sterility checkpoints and durable CFI state.
- BigQuery for tenant-scoped case memory and environmental data.
- Firestore for Vercel case-session pointers and Sponsor QA notifications.
- Cloud Scheduler to start and resume long-running work.
- Cloud Run for MCP tools, scheduler services, sponsor liaison, and the Sponsor QA receiver.
External writes are isolated and idempotent. A retry reuses the same ERPNext,
openMAINT, Sponsor QA, Calendar, and Docs records instead of creating
duplicates. Server boundaries validate tenant, case, lifecycle, and tool scope.
Models cannot choose credentials, tenants, record links, or disposition
outcomes.
The pharmaceutical demo records are synthetic and clearly labelled. The
integrations, authentication, cloud deployment, persistence, and external
writes are real.
Challenges we ran into
- Identity and database access: Agent Identity could not directly use the required Cloud SQL identity. We moved database operations behind narrow, tenant-scoped MCP tools.
- Long-running workflows: Sterility waits and Chat interviews can outlive a process. We added durable state, atomic Scheduler claims, safe resume, and idempotent actions.
- Cross-system authentication: Calendar, Docs, ERPNext, openMAINT, SENAITE, and Sponsor QA all use different access models. We added scoped identities, keyless Workspace delegation, and visible typed failures.
Accomplishments that we're proud of
- Built a multi-agent investigation workflow, not a chatbot.
- Kept the LLM outside the batch-disposition path.
- Connected five departments to real systems of record.
- Demonstrated durable pause and resume for long-running work.
- Proved the deterministic GUI-to-CFI interview route through Chat, Scheduler, Calendar, and Docs.
- Completed post-QP actions across ERPNext, openMAINT, Google Workspace, and Sponsor QA.
- Added tenant isolation, least-privilege tool policies, idempotency, typed failures, and end-to-end traces.
- Verified the current codebase with 338 passing unit tests, plus recorded live cloud and integration proofs.
What we learned
Enterprise agent safety comes from architecture, not only from prompting.
Models are useful for extraction, planning, drafting, questions, and challenge.
Deterministic code is better for authorization, thresholds, lifecycle changes,
idempotency, and final gates. Combining them makes the system useful without
giving the model decision authority.
We also learned that compliance memory needs exact provenance. Durable records
and tenant-filtered evidence stores are safer than unrestricted conversational
memory.
Observability is part of the product. Reviewers need to see which tool ran,
which identity called it, what evidence came back, and where a human acted.
The architecture is production-shaped: with SOP onboarding, validation, and
tenant review, it could be hardened into a CDMO workflow. We do not claim it is
a validated GxP product today.
What's next for Countersign
Next, we would validate the workflow with representative quality teams, onboard
more tenants, expand the controlled evidence catalogs, strengthen content
inspection, and add more approved connectors.
Countersign is not presented as a validated GxP system, a Part 11 certification
product, or an autonomous batch-release tool. It is a production-shaped
demonstration of governed agents that reduce operational work while keeping
accountable decisions with humans.
Built for the Google Cloud All Things Agentic Hackathon.
Top comments (0)