DEV Community

Google Developer Experts profile image Evan Lin
Evan Lin for Google Developer Experts

Posted on Originally published at evanlin.com on

[Agent Platform in Action (3)] Solving Stateless Container HITL Amnesia: Cross-Instance Business Card Recognition and Pause/Resu

#agents #ai #architecture #go

Phase 2: Stateless Cross-Instance HITL & GCS Artifact Architecture

Recap

In the previous two articles of this series:

Entering Phase 2 (Issue #3 & Issue #4), we face a classic and common pitfall in distributed systems when pushing to production (Cloud Run / Agent Runtime auto-scaling clusters): When containers can Scale to Zero or be load-balanced to different instances at any time, where should multimodal business card photos (Binary Images) be stored? And how can the HITL pause state, waiting for a user to click a button, be recovered across instances?

In the third installment of this series, we will deep dive into how we used ADK Go v2.6.0's artifact/gcsartifact (GCS Artifact Service) and session/vertexai (Agent Platform Sessions) in PR #11 to completely cure the "HITL amnesia" of stateless containers!

1. Why did the original HITL suffer from "Amnesia" and "State Bloat" in cloud multi-instance environments?

During local development or with a single container instance, stuffing everything into memory is fast. However, once deployed to Google Cloud Run or Vertex AI Agent Runtime, which support Horizontal Auto-scaling or Scale-to-Zero, the first version's design immediately hits two major walls:

Pain Point 1: The "State Bloat" anti-pattern of stuffing multi-MB business card []byte into Session.State

In the first version, when a LINE user sent a high-resolution business card photo, we directly put the downloaded imageBytes []byte into ADK's Session.State["image_bytes"]:

// ❌ Version 1 approach: Directly stuffing multi-MB raw image []byte into Session State
InitialState: map[string]any{
    StateKeyInputType: RouteIngest,
    StateKeyImageBytes: imageBytes, // A 3MB JPEG inflates to a ~4MB Base64 string after JSON serialization!
}
Enter fullscreen mode Exit fullscreen mode

When using the local session.InMemoryService(), this is just a Go slice pointer. But when switching to cloud-persisted Agent Platform Sessions (session/vertexai) or Cloud SQL (session/database):

  1. Serialization and network transfer for every node state change (StateDelta): As the ADK Graph Workflow moves from RouterNode to VisionOCRNode, fans out to three parallel nodes, and finally pauses at HITLConfirmNode, keeping multi-MB image binary data in Session.State significantly slows down every Cloud Session API read/write and might even trigger Cloud Session payload size limits!
  2. Difficulty in auditing and versioning history: If a user sends multiple business cards across several turns, mixing raw binary files with structured JSON state makes the conversation state extremely bloated.

Pain Point 2: Container instance switching leads to loss of HITL pendingSession memory mapping

Let's look at a real-world user scenario:

  1. Turn 1 (hits Container Instance A): A user takes a photo of a business card at a conference and sends it to the LINE Bot. Instance A completes Vision OCR and parallel background checks, then pauses at HITLConfirmNode via workflow.NewRequestInputEvent, pushing a Flex Message preview card to the user.
  2. 10-minute break: The user goes to exchange thoughts with the contact and grabs a coffee. During this time, Instance A scales to 0 due to inactivity, or Container Instance B is scaled out due to other traffic.
  3. Turn 2 (hits Container Instance B): The user clicks "✏️ Edit Fields" on LINE and enters "Change title to CTO", or clicks "✅ Confirm Save".

In the first version, our Engine used an in-memory Go map[string]pendingHITL to track which userID was waiting for which SessionID and InterruptID:

// ❌ Version 1 approach: Relying solely on a local in-memory map to determine if there's a pending HITL
type Engine struct {
    mu sync.RWMutex
    pendingSession map[string]pendingHITL // userID -> {SessionID, InterruptID}
}
Enter fullscreen mode Exit fullscreen mode

As a result, when the Turn 2 Webhook request is routed by the load balancer to the brand new Instance B, Instance B's pendingSession map is empty! HasPendingHITL(userID) returns false, and the system treats "Change title to CTO" as a brand new "Business Card Search" request, instantly breaking the HITL loop!

2. The Standard ADK 2.0 Solution: Separation of Concerns between Session State and Artifact Storage

To solve this, Google ADK 2.0's architectural design explicitly splits Agent runtime data into two independent services:

Comparison Dimension Session Service (session.Service) Artifact Service (artifact.Service)
Core Purpose Stores conversation event logs (Events), workflow breakpoints (Interrupts), and lightweight structured state (JSON State) Stores large binary files (images, PDFs, audio, CSV reports) and provides automatic version control (Versioning)
Cloud Implementation Package google.golang.org/adk/v2/session/vertexai (Agent Platform Sessions) or session/database google.golang.org/adk/v2/artifact/gcsartifact (underlying storage in Google Cloud Storage Bucket)
Local Development Implementation session.InMemoryService() artifact.InMemoryService()
Role in Business Card King Stores draft_card JSON, active_interrupt_id, and lightweight pointers (filename + version number) to the image Stores raw business card photo business_card.jpg (automatically increments Version 1, 2, 3... per upload)

With this dual-track design, the entire cross-instance multimodal business card recognition and HITL recovery process becomes crystal clear:

sequenceDiagram
    autonumber
    actor User as LINE / WebUI User
    participant InstA as Container Instance A (Turn 1: Upload Card)
    participant GCS as GCS Artifact Service<br/>(artifact/gcsartifact)
    participant Sess as Agent Platform Sessions<br/>(session/vertexai)
    participant InstB as Container Instance B (Turn 2: Edit Fields)
    participant InstC as Container Instance C (Turn 3: Confirm Save)

    User->>InstA: 1. Upload business card photo (JPEG []byte)
    InstA->>GCS: 2. Save("business_card.jpg", inlineData)<br/>Returns Version = 1
    InstA->>Sess: 3. Create Session, State only records:<br/>card_image_artifact="business_card.jpg"<br/>card_image_version=1
    InstA->>GCS: 4. VisionOCRNode calls LoadVersion("business_card.jpg", 1)
    Note over InstA,Sess: Fan-Out Parallel duplicate check / Company search / Interactive retrieval -> Generate preview card
    InstA->>Sess: 5. HITLConfirmNode triggers NewRequestInputEvent<br/>Persists active_interrupt_id and draft_card
    InstA-->>User: 6. Return LINE Flex preview card (⏸️ Workflow Paused)

    Note over InstA,InstB: ☁️ Container scales to 0 or load balancer switches to a brand new stateless Instance B

    User->>InstB: 7. Input: "Change title to CTO"
    InstB->>Sess: 8. resolvePendingHITL(userID):<br/>Query latest Session's active_interrupt_id
    Sess-->>InstB: 9. Retrieve {SessionID, InterruptID} and draft_card
    InstB->>Sess: 10. ResumeHITL -> ApplyEditNode updates title -> Re-pauses and updates active_interrupt_id
    InstB-->>User: 11. Return updated Flex preview card (Title: CTO)

    Note over InstB,InstC: ☁️ Switch again to brand new stateless Instance C

    User->>InstC: 12. Click "✅ Confirm Save" (action=confirm_save)
    InstC->>Sess: 13. resolvePendingHITL(userID) retrieves latest InterruptID
    InstC->>Sess: 14. ResumeHITL -> SaveCardNode writes to card library + clears active_interrupt_id
    InstC-->>User: 15. Return save success and AI icebreaker draft!
Enter fullscreen mode Exit fullscreen mode

3. Code Implementation Breakdown (Phase 2 — PR #11)

Now let's look at the code to see how we completed this upgrade with less than a few hundred lines of clean Go code.

1. Creating a Multi-backend Artifact Factory: sessions/artifacts.go

We added sessions/artifacts.go under the sessions package, using ADK Go v2.6.0's built-in google.golang.org/adk/v2/artifact/gcsartifact package to automatically switch between Cloud GCS Bucket and local memory based on environment variables:

package sessions

import (
    "context"
    "log/slog"
    "os"
    "strings"

    "google.golang.org/adk/v2/artifact"
    "google.golang.org/adk/v2/artifact/gcsartifact"
)

// NewArtifactService returns an ADK artifact.Service backed by Google Cloud Storage
// when ARTIFACT_BUCKET_NAME or LOGS_BUCKET_NAME is configured, and falls back to
// artifact.InMemoryService() for local development and unit tests.
func NewArtifactService(ctx context.Context) (artifact.Service, error) {
    bucketName := resolveArtifactBucketName()
    if bucketName == "" {
        slog.Info("Using in-memory artifact service (set ARTIFACT_BUCKET_NAME or LOGS_BUCKET_NAME for GCS)")
        return artifact.InMemoryService(), nil
    }

    svc, err := gcsartifact.NewService(ctx, bucketName)
    if err != nil {
        return nil, err
    }
    slog.Info("Using GCS artifact service", "bucket", bucketName)
    return svc, nil
}
Enter fullscreen mode Exit fullscreen mode

A thoughtful detail: In the agents-cli Agent Runtime deployment environment, the system usually configures LOGS_BUCKET_NAME (or our custom ARTIFACT_BUCKET_NAME). Some might accidentally fill it in with a format like gs://my-bucket/prefix. Our resolveArtifactBucketName() automatically strips the gs:// prefix and path to ensure gcsartifact.NewService connects reliably, falling back to artifact.InMemoryService() if no bucket is set.

Simultaneously, we inject ArtifactService and SessionService into launcher.Config in main.go and cmd/server/main.go:

// main.go — Injecting both SessionService and ArtifactService into ADK Universal Launcher
config := &launcher.Config{
    SessionService: sessionService,
    ArtifactService: artifactService,
    AgentLoader: NewSingleAgentLoader(rootAgent),
    A2AOptions: a2aOpts,
}
Enter fullscreen mode Exit fullscreen mode

2. Storing Business Card Images in Artifacts, Keeping only "Lightweight Pointers" in Session State

Next, we refactor internal/workflow/graph.go and internal/workflow/nodes_ingest.go.

First, we define two new lightweight State Keys and a default filename:

const (
    StateKeyImageBytes = "image_bytes" // Kept only for backward compatibility fallback
    StateKeyCardImageArtifact = "card_image_artifact" // Artifact filename
    StateKeyCardImageVersion = "card_image_version" // Artifact version number (int)

    DefaultCardImageArtifactName = "business_card.jpg"
)
Enter fullscreen mode Exit fullscreen mode

(1) Entering from LINE Webhook (Engine.StartIngest)

When a LINE user sends an image, StartIngest first calls e.artifactService.Save to write the image binary to the Artifact service and only records the filename and version number in InitialState. StateKeyImageBytes is no longer written to Session.State:

initialState := map[string]any{
    StateKeyInputType: RouteIngest,
    StateKeyRecentEvents: recentEvents,
}

if e.artifactService != nil && len(imageBytes) > 0 {
    part := genai.NewPartFromBytes(imageBytes, detectImageMIMEType(imageBytes))
    saveResp, saveErr := e.artifactService.Save(ctx, &artifact.SaveRequest{
        AppName: appName,
        UserID: userID,
        SessionID: sess.ID(),
        FileName: DefaultCardImageArtifactName,
        Part: part,
    })
    if saveErr == nil && saveResp != nil {
        initialState[StateKeyCardImageArtifact] = DefaultCardImageArtifactName
        initialState[StateKeyCardImageVersion] = saveResp.Version
    }
}
Enter fullscreen mode Exit fullscreen mode

(2) Uploading images directly from ADK WebUI / Agent Engine Playground / A2A (RouterNode)

What if the user isn't using LINE but uploads a business card photo directly via ADK WebUI (make playground) or Agent Engine Console? In RouterNode, we use ADK 2.0's ctx.Artifacts().Save(...) to transfer InlineData from ctx.UserContent() into the Artifact Service:

// internal/workflow/nodes_ingest.go — RouterNode
if getStringState(ctx, StateKeyCardImageArtifact) == "" && len(getBytesState(ctx, StateKeyImageBytes)) == 0 && ctx.UserContent() != nil {
    for _, part := range ctx.UserContent().Parts {
        if part != nil && part.InlineData != nil && len(part.InlineData.Data) > 0 {
            if ctx.Artifacts() != nil {
                mimeType := part.InlineData.MIMEType
                if mimeType == "" {
                    mimeType = detectImageMIMEType(part.InlineData.Data)
                }
                artifactPart := genai.NewPartFromBytes(part.InlineData.Data, mimeType)
                if saveResp, saveErr := ctx.Artifacts().Save(ctx, DefaultCardImageArtifactName, artifactPart); saveErr == nil {
                    setState(ctx, ev, StateKeyCardImageArtifact, DefaultCardImageArtifactName)
                    setState(ctx, ev, StateKeyCardImageVersion, saveResp.Version)
                } else {
                    setState(ctx, ev, StateKeyImageBytes, part.InlineData.Data)
                }
            }
            if getStringState(ctx, StateKeyInputType) == "" {
                setState(ctx, ev, StateKeyInputType, RouteIngest)
            }
            break
        }
    }
}
Enter fullscreen mode Exit fullscreen mode

(3) VisionOCRNode Reads specific versions of business card images on demand (loadCardImageBytes)

When the workflow enters VisionOCRNode to call Gemini Vision for multimodal recognition, it uses ctx.Artifacts().LoadVersion (or Load) to load the image into memory for the model. It's discarded after use, keeping Session.State clean:

func loadCardImageBytes(ctx agent.Context) []byte {
    if artifactName := getStringState(ctx, StateKeyCardImageArtifact); artifactName != "" && ctx.Artifacts() != nil {
        version := getIntState(ctx, StateKeyCardImageVersion)
        var (
            resp *artifact.LoadResponse
            err error
        )
        if version > 0 {
            resp, err = ctx.Artifacts().LoadVersion(ctx, artifactName, version)
        } else {
            resp, err = ctx.Artifacts().Load(ctx, artifactName)
        }
        if err == nil && resp != nil && resp.Part != nil && resp.Part.InlineData != nil {
            return resp.Part.InlineData.Data
        }
    }
    return getBytesState(ctx, StateKeyImageBytes) // Fallback
}
Enter fullscreen mode Exit fullscreen mode

3. Implementing Stateless Cross-Instance HITL State Discovery: resolvePendingHITL

With image storage solved, here's the main event: When Container Instance B receives a user's "Change title to CTO" or "✅ Confirm Save", how does it automatically retrieve the HITL workflow paused by Instance A when its local memory is empty?

The secret is: when HITLConfirmNode calls workflow.NewRequestInputEvent to pause, we've already written the current InterruptID into Session.State["active_interrupt_id"] (and SaveCardNode clears it to "" once saving is complete).

Therefore, we implemented resolvePendingHITL(ctx, userID) in internal/workflow/graph.go:

// resolvePendingHITL checks the local cache first and falls back to querying
// the shared session.Service for any session belonging to userID with a non-empty
// StateKeyActiveInterruptID. This enables stateless cross-instance HITL resume
// on Cloud Run / Vertex AI Agent Runtime.
func (e *Engine) resolvePendingHITL(ctx context.Context, userID string) (pendingHITL, bool) {
    // 1. Fast path: If a record exists in the same instance's memory, verify with sessionService if it's still paused
    e.mu.RLock()
    cached, ok := e.pendingSession[userID]
    e.mu.RUnlock()
    if ok && cached.SessionID != "" && cached.InterruptID != "" {
        if getResp, err := e.sessionService.Get(ctx, &session.GetRequest{
            AppName: appName,
            UserID: userID,
            SessionID: cached.SessionID,
        }); err == nil && getResp != nil && getResp.Session != nil {
            if activeID, err := getResp.Session.State().Get(StateKeyActiveInterruptID); err == nil {
                if s, ok := activeID.(string); ok && strings.TrimSpace(s) != "" {
                    cached.InterruptID = strings.TrimSpace(s)
                    return cached, true
                }
            }
        }
    }

    // 2. Stateless Slow Path: When hitting a brand new container instance, query the shared sessionService directly!
    listResp, err := e.sessionService.List(ctx, &session.ListRequest{
        AppName: appName,
        UserID: userID,
    })
    if err != nil || listResp == nil {
        return pendingHITL{}, false
    }

    var (
        matched pendingHITL
        latestTime time.Time
        foundActive bool
    )
    for _, sess := range listResp.Sessions {
        if sess == nil || sess.State() == nil {
            continue
        }
        val, err := sess.State().Get(StateKeyActiveInterruptID)
        if err != nil || val == nil {
            continue
        }
        interruptID, ok := val.(string)
        interruptID = strings.TrimSpace(interruptID)
        if !ok || interruptID == "" {
            continue
        }
        updatedAt := sess.LastUpdateTime()
        if !foundActive || updatedAt.After(latestTime) {
            foundActive = true
            latestTime = updatedAt
            matched = pendingHITL{
                SessionID: sess.ID(),
                InterruptID: interruptID,
            }
        }
    }

    if foundActive {
        e.mu.Lock()
        e.pendingSession[userID] = matched
        e.mu.Unlock()
        return matched, true
    }
    return pendingHITL{}, false
}
Enter fullscreen mode Exit fullscreen mode

Behold the power of this design:

  • Regardless of which stateless instance the user's request is dispatched to by the Cloud Run or Agent Runtime load balancer, HasPendingHITL(userID) and ResumeHITL(ctx, userID, userInput) can retrieve the user's latest paused {SessionID, InterruptID} directly via the shared session.Service.
  • Then, Instance B simply wraps the user's input into an ADK 2.0 standard adk_request_input (workflow.WorkflowInputFunctionCallName) FunctionResponse and calls e.runner.Run to seamlessly wake the workflow from HITLConfirmNode!

4. Writing Unit Tests to Verify "Across Three Container Instances"!

To prove this architecture works in a multi-instance environment with no shared local memory, we wrote the TestCrossInstanceHITLResumeAndArtifactStorage test in internal/workflow/graph_test.go, creating 3 independent Engine instances (instanceA, instanceB, instanceC) to simulate three stateless containers:

func TestCrossInstanceHITLResumeAndArtifactStorage(t *testing.T) {
    ctx := context.Background()
    cardStore := store.NewMemoryStore()
    mockLLM := NewMockLLM()

    // Simulating cloud-shared Agent Platform Session Service and GCS Artifact Service
    sharedSessionSvc := session.InMemoryService()
    sharedArtifactSvc := artifact.InMemoryService()

    // Creating 3 completely independent Engine instances (each with an independent and empty pendingSession map)
    instanceA, _ := NewEngine(cardStore, mockLLM,
        WithSessionService(sharedSessionSvc),
        WithArtifactService(sharedArtifactSvc),
    )
    instanceB, _ := NewEngine(cardStore, mockLLM,
        WithSessionService(sharedSessionSvc),
        WithArtifactService(sharedArtifactSvc),
    )
    instanceC, _ := NewEngine(cardStore, mockLLM,
        WithSessionService(sharedSessionSvc),
        WithArtifactService(sharedArtifactSvc),
    )

    userID := "U_cross_instance_user"
    fakeImage := []byte{0xFF, 0xD8, 0xFF, 0xE0, 0x00, 0x10, 'J', 'F', 'I', 'F'}

    // Turn 1 (hits Instance A): Upload card photo -> Write to Artifact and pause at HITLConfirmNode
    res1, _ := instanceA.StartIngest(ctx, userID, fakeImage, []string{"Met at Cloud Summit"})
    // Verify Session State has no bloated raw image_bytes, but lightweight card_image_artifact="business_card.jpg"

    // Turn 2 (hits brand new Instance B): User enters edit command 'Change title to CTO'
    // Instance B automatically detects the active_interrupt_id left by Instance A from sharedSessionSvc!
    res2, _ := instanceB.ResumeHITL(ctx, userID, "Change title to CTO")
    // Verify title updated to 'CTO' and re-entered HITL pause waiting for confirmation

    // Turn 3 (hits brand new Instance C): User clicks '✅ Confirm Save'
    res3, _ := instanceC.ResumeHITL(ctx, userID, "action=confirm_save")
    // Verify card successfully written to CardStore, AI icebreaker generated, and HasPendingHITL is zeroed for all instances!
}
Enter fullscreen mode Exit fullscreen mode

Running go test -race -v ./...:

=== RUN TestCrossInstanceHITLResumeAndArtifactStorage
--- PASS: TestCrossInstanceHITLResumeAndArtifactStorage (0.00s)
=== RUN TestRootAgentDirectImageArtifactIngestAndHITL
--- PASS: TestRootAgentDirectImageArtifactIngestAndHITL (0.00s)
PASS
ok github.com/kkdai/line-card-assistant/internal/workflow   0.642s
Enter fullscreen mode Exit fullscreen mode

Not only was the three-stage relay from instanceA $\rightarrow$ instanceB $\rightarrow$ instanceC seamless, but TestRootAgentDirectImageArtifactIngestAndHITL in app/agent_test.go also verified that when a genai.Part image is passed directly from ADK WebUI / A2A, RouterNode automatically saves it to ArtifactService and pauses at HITLConfirmNode!

4. Summary and Next Steps

Through this practical guide (PR #11, completing Issue #3 and Issue #4 in the Roadmap), we've solved the two most critical persistence challenges for moving an AI Agent from a "toy prototype" to a "cloud stateless multi-instance production environment":

  1. Separation of Large Files and State (Artifact vs. Session State): Using ADK Go v2.6.0's artifact/gcsartifact and ctx.Artifacts(), business card photos no longer bloat Session.State. Instead, they have independent GCS versioned storage, with Session State only keeping a few dozen bytes of pointers (card_image_artifact + card_image_version).
  2. Stateless Cross-Instance HITL Recovery: Combining session/vertexai (Agent Platform Sessions) and resolvePendingHITL, even if a container scales to 0 or switches instances during the minutes spent waiting for human confirmation, the system can still resume the "edit field loop" or "confirm save" perfectly.

In the next article, we will continue with cloud database persistence and a two-layer decoupled architecture, and use agents-cli eval to build an automated evaluation flywheel. Stay tuned!

Resources and Further Reading

Top comments (1)

Collapse
 
yuhaixia profile image
Yuhai Xia •

This is a really solid breakdown — the "HITL amnesia" framing lands for me. I recently wrote a postmortem about an agent that stayed green on every health check while its memory was silently gone, so I've lived a version of Pain Point 2. The resolvePendingHITL approach (local cache first, shared session state as the source of truth) is the same shape we ended up with: never trust an in-memory map across instances, always re-hydrate from the shared store. One thing worth adding from that experience: the orphan-pending case. If the human never comes back, active_interrupt_id lingers; we added a sweep so a stale interrupt can't get "resumed" into something unexpected days later. Looking forward to the persistence follow-up.