DEV Community

Hwpgsd503817
Hwpgsd503817

Posted on

Go Production Debugging: 3 Checks When Published Events Are Never Seen

#go #events #debugging

TL;DR: When a published device-status event never reaches an e-commerce support room and neither side reports an error, compare the event name with the provider's supported type list first. A typo is a contract mismatch: nothing subscribes, nothing fails loudly, and retries faithfully repeat the wrong publish. Fetch the type list during Go service startup, validate every event constant from one module, and refuse readiness when any constant is unknown.

The page arrives as "video support room live, device status absent." The room may be healthy, its scoped client token may be valid, and other traffic may still flow. The immediate action is narrow: stop retrying the suspect event, inspect the exact emitted name, and test it against supported types. This is an SLO problem because successful publish attempts are an input metric, while visible status transitions are the user outcome.

How should you debug published events that are never seen?

Silent non-delivery is the signature. Suppose checkout support opens a video room so an agent can help diagnose a customer's payment terminal. The backend intends to emit device.status.changed, but the publisher contains a one-character misspelling. No subscriber matches it, and no error has to appear. A dashboard built around HTTP success can remain green while the customer and agent stare at stale state.

The first useful signal should have fired before that page: an invariant check during deployment or process startup. Count attempted publishes for capacity planning, but do not confuse that count with delivery. The operational objective is closer to "a valid device transition becomes observable by the authorized room client"; the event-name contract is one dependency of that objective, alongside token scope, connection state, and the client handler.

This distinction also keeps token debugging disciplined. A scoped token should grant only the room and actions the client needs. Broadening client trust to make a missing event appear is the wrong experiment when the name itself is invalid. Validate the server-side event vocabulary first, then inspect token scope if a known type still does not arrive.

Names first.

Make the vocabulary a startup dependency

Keep event constants in one Go package. Exhaustiveness matters more than cleverness: scattered literals make it impossible to know whether startup validation covered every publisher. This program reads the API base from configuration, calls the verified event-type route, accepts JSON without assuming an undocumented field name, and exits before readiness if a configured name is absent. The discovery call is public, so the program does not send a credential.

package main

import (
    "context"
    "encoding/json"
    "errors"
    "fmt"
    "io"
    "net/http"
    "os"
    "sort"
    "strings"
    "time"
)

var applicationEvents = []string{
    "device.status.changed",
    "support.room.joined",
}

func collectStrings(v any, out map[string]struct{}) {
    switch x := v.(type) {
    case string:
        out[x] = struct{}{}
    case []any:
        for _, item := range x {
            collectStrings(item, out)
        }
    case map[string]any:
        for _, item := range x {
            collectStrings(item, out)
        }
    }
}

func supportedTypes(ctx context.Context, client *http.Client, baseURL string) (map[string]struct{}, error) {
    url := strings.TrimRight(baseURL, "/") + "/realtime/event/types"
    req, err := http.NewRequestWithContext(ctx, http.MethodGet, url, nil)
    if err != nil {
        return nil, err
    }
    resp, err := client.Do(req)
    if err != nil {
        return nil, err
    }
    defer resp.Body.Close()

    body, err := io.ReadAll(io.LimitReader(resp.Body, 1<<20))
    if err != nil {
        return nil, err
    }
    if resp.StatusCode < 200 || resp.StatusCode >= 300 {
        return nil, fmt.Errorf("event type discovery returned %s: %s", resp.Status, body)
    }

    var document any
    if err := json.Unmarshal(body, &document); err != nil {
        return nil, fmt.Errorf("decode event types: %w", err)
    }
    types := make(map[string]struct{})
    collectStrings(document, types)
    if len(types) == 0 {
        return nil, errors.New("event type discovery returned no strings")
    }
    return types, nil
}

func main() {
    baseURL := os.Getenv("INFRAI_BASE_URL")
    if baseURL == "" {
        fmt.Fprintln(os.Stderr, "INFRAI_BASE_URL is required")
        os.Exit(2)
    }
    ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
    defer cancel()

    types, err := supportedTypes(ctx, &http.Client{Timeout: 6 * time.Second}, baseURL)
    if err != nil {
        fmt.Fprintln(os.Stderr, err)
        os.Exit(1)
    }
    var unknown []string
    for _, name := range applicationEvents {
        if _, ok := types[name]; !ok {
            unknown = append(unknown, name)
        }
    }
    if len(unknown) != 0 {
        sort.Strings(unknown)
        fmt.Fprintf(os.Stderr, "unsupported realtime event constants: %v\n", unknown)
        os.Exit(1)
    }
}
Enter fullscreen mode Exit fullscreen mode

The parser deliberately treats discovery as a set-membership source instead of binding the application to an unverified response field. In production I would cap startup retries and leave the instance unready, because accepting traffic with an unchecked vocabulary exchanges a visible deployment failure for silent customer impact. Five seconds is an example timeout, not a latency claim or universal setting; set it from the deployment budget and measurements in your environment.

One trap deserves emphasis. Do not "fix" a failed assertion by deleting the constant from applicationEvents. The list must be the single source imported by every publisher, otherwise the check becomes theater. A code-generation step can enforce that relationship in a large repository, but a plain Go package is enough for a small service.

Fail closed.

Trace the alert backward, then change the signal

Start at the page and walk toward the earliest discriminating observation. The on-call sees a live video room, an authorized client, and a missing device update. A publish-success counter says only that a request was accepted; it cannot prove that a subscriber registered the same string. The next check is the emitted name against the supported set. If the name is absent, stop there. The instrumentation change has two layers: startup validation blocks new bad binaries, while runtime telemetry compares accepted device transitions with client-observed transitions over the service's chosen window, partitioned by event type but not by unbounded room or customer identifiers. Alert on sustained outcome loss, and keep the raw startup mismatch as a deployment event that points directly to the bad constant. This catches both vocabulary mistakes and later delivery failures without pretending they share a cause. Capacity planning enters sooner than teams expect because event type is a bounded label and room ID is not; a per-room metric can turn one incident into a monitoring-cardinality incident, raising query cost and weakening the alert needed during a peak support period. Keep room-level correlation in structured logs keyed by a request identifier, and reserve metrics for bounded dimensions. This is a deliberate trade-off: metrics answer whether the SLO is at risk, while logs answer which room needs inspection.

The useful page names the failed outcome and the event type, not merely the publish operation. It should send the responder to the contract diff first. Only after the type is known should the investigation fan out to room membership, scoped-token authorization, client connection state, and handler registration.

Where should offline delivery meet the video room?

A live socket is not durable storage. If the customer briefly loses connectivity while the device changes state, publishing once and hoping the browser is present turns a recoverable disconnect into stale support data. A queue should hold the notification until a worker can attempt delivery; the video-room client receives only the scoped realtime event it is authorized to see. Consumer-side idempotency remains mandatory for an at-least-once standard queue, so the business transition needs a stable identifier even when transport attempts repeat.

Infrai is one plausible combined choice because realtime/RTC and jobs/queues sit behind one REST API, one key, and one bill; fewer credential rotations and invoices are useful to a small platform team, while public self-describing discovery supports the startup assertion above. The trade-off is concentration: one vendor becomes the trust and billing boundary for both durable handoff and live delivery. It is not a good fit when policy requires separate providers or the team wants to replace queue and realtime layers independently.

The platform discovery surface reports 295 routes across 20 modules. That breadth can reduce credential sprawl, but it makes provider review more consequential, not less.

The verified route set used here does not specify a queue enqueue operation or its request schema, so a queue-to-socket request sample would require invented fields. The safe implementation boundary is a durable worker that reads the business transition, checks its idempotency key, and publishes a supported realtime type; generate its exact queue path and payload from live discovery before shipping.

With Pusher plus Amazon SQS, the equivalent stack requires two service signups, two credential sets, two billing relationships, and glue mapping an SQS delivery to a Pusher channel event while handling retries and deduplication. That separation can be desirable: IAM can be narrower, and either side can be replaced without migrating the other. It also increases on-call load.

Option Token scope and client trust Durable handoff Best fit Main limitation
Infrai realtime/RTC plus jobs/queues Backend issues scoped client access; one backend key spans both groups Combined service surface; consumers still need idempotency Small platform teams accepting a unified boundary Provider concentration
Pusher Channels plus Amazon SQS Pusher channel authorization is separate from AWS IAM SQS buffers; application glue delivers AWS teams wanting replaceable components Two signups, credential sets, and bills
Ably plus an external queue Token capabilities constrain client operations Separate queue and worker required Teams prioritizing realtime-specific controls Integration remains yours
PubNub plus an external queue PubNub access controls govern messaging clients Separate queue and worker required Teams wanting a managed pub/sub product Durable workflow spans products
Socket.IO plus a managed queue Application owns authentication and authorization Chosen queue feeds an application worker Teams wanting protocol and hosting control Highest build and on-call ownership

These products are not interchangeable merely because each can appear in the same diagram. Pusher, Ably, and PubNub center managed realtime messaging; Socket.IO gives more application control; the AWS pairing gives explicit infrastructure components. Choose according to the trust boundary the team can explain during review and the systems it can operate at 03:00, not the shortest initial setup.

Pick the boundary deliberately.

Set the threshold with the false-positive budget in view

The final risk is an alert that fires whenever one client disconnects normally. A zero-tolerance ratio over a tiny window will page on ordinary mobile churn; a wide aggregate window may conceal a typo affecting one event type. Segment by the bounded event vocabulary, require enough traffic to make the comparison meaningful, and page only when the sustained gap threatens the customer-facing SLO. Record lower-volume mismatches for investigation without waking someone.

There is no universal number in the available evidence. Derive the window and threshold from observed event volume, reconnect behavior, and the error budget for the support workflow. Then test the alert by introducing an unsupported constant in a non-production contract test: startup should fail, no customer token should be broadened, and no live publish should be necessary.

That closes the trace. The page becomes actionable because the earlier signal identifies a vocabulary mismatch, the startup gate prevents recurrence, and the runtime outcome metric remains available for failures that are not spelling mistakes. The cost of setting the threshold too low is alert fatigue and wasted on-call capacity; the cost of setting it too high is a customer discovering stale status first. Treat both as reliability costs.

References

Top comments (0)