DEV Community

EthanBrooks111
EthanBrooks111

Posted on

Small Business Hosted App Log Search API — Evidence Gates for Tenant Rollbacks

A checkout rollback is only safe to evaluate if the evidence separates tenant cohorts and the configuration each request actually used. TL;DR: choose a hosted application log search API for a small business when incident investigation is the requirement and the team wants to avoid operating storage and indexing; choose a deeper logging stack when rollback decisions also depend on advanced alerting, export, or governance. Self-hosting adds an infrastructure responsibility that belongs in the platform roadmap, alongside the experiment itself.

Preserve the evidence.

A successful configuration change alone cannot establish that affected tenants recovered, and a search interface cannot compensate for events that never recorded their evaluated configuration.

What would make a cohort rollback look safer than it is?

Consider a hypothetical production review for an e-commerce checkout experiment: two tenant cohorts receive different configurations, a rollback is requested, and the next investigation asks whether the treatment cohort recovered relative to the control. This is a design exercise, not a reported incident. Suppose the review receives 12 failed checkouts from 100 treatment requests and 20 from 1,000 control requests; those illustrative counts mean 12% and 2%, respectively, despite the treatment having fewer failures in absolute terms. Comparing totals would conceal the larger failure fraction.

The subtler trap comes after the change. A request that began before rollback may finish afterward, while a client that polls configuration may still evaluate an older value; grouping events solely by arrival time can therefore describe the wrong exposure. Record the experiment assignment and evaluated configuration with the application outcome, while preserving the intended rollback revision separately in the release record. These are proposed application fields, not a particular logging API's schema.

The invariant is simple: a rollback decision needs exposure, outcome, and completeness evidence for each cohort. Don't let an aggregate decline in errors stand in for those three checks.

Should a small business choose hosted app log search or self-hosted infrastructure?

Treat this buy-vs-build table as a selection rubric, rather than a performance ranking. The question is which operating boundary the team can sustain while preserving its rollback evidence.

Option Why consider it? Trade-off to evaluate
Build around self-hosted Grafana Loki Ownership of the logging stack is a deliberate platform requirement. Reserve capacity and on-call time for operating storage and the logging infrastructure.
Buy Elastic Cloud The investigation needs the depth of an Elastic-style stack. Establish which search and governance requirements justify that depth; test the actual cohort investigation.
Buy Amazon CloudWatch Logs Include this hosted alternative when comparing application log services. Its pricing documentation includes ingestion charges; estimate the event stream before treating logging as incidental.
Buy a basic hosted logs API Straightforward incident search without running storage and indexing systems. Decide separately how automated rollback signals, exports, and governance will be provided.

For teams consolidating backend administration, Infrai offers one key and one bill across backend services, plus a unified REST API with no SDK to install: plain HTTP works from any language or runtime, letting a Go release tool retrieve incident evidence without a separate vendor library. The simple API spans 295 routes across 20 modules, so adding another backend capability need not introduce another integration convention. Public, self-describing discovery requires no key and exposes request and response schemas, while runnable examples in 10 languages give the release-tool owner a concrete integration starting point before putting credentials into the workflow.

Basic incident search fits that scope, with a narrower observability boundary: select a deeper stack if the acceptance criteria require advanced alerting and notification delivery, distributed trace queries and span trees, source-map decoding, crash symbolication, session replay, heartbeat monitoring, per-user log deletion, or bulk export and subscription feeds. Trace and span identifiers in logs provide correlation; a span tree is a separate requirement. Establish a documented search contract before depending on complex cohort filters, and establish retention requirements independently rather than assuming configurable retention or cold storage.

Those boundaries matter. A search API is sufficient only if the evidence contract fits it.

For comparison, include Datadog when evaluating a broader observability product; require a trial of the same cohort reconstruction, SLO review, and operating boundary before accepting the broader scope as useful. The alternatives here are candidates, not certified configurations. Loki is the self-hosted operating commitment; Elastic Cloud is the deeper-stack candidate; CloudWatch Logs belongs in the hosted-service comparison, with ingestion volume made explicit. Vendor documentation and a representative trial must settle the detailed feature comparison. None of these product names establishes that the application recorded a complete cohort population.

Administrative consolidation also concentrates a dependency. Evaluate provider coupling against the team's operating capacity; fewer credentials alone cannot make a release safer.

Budget capacity for the experiment and its reversal

Capacity planning should begin with an application-owned estimate: checkout attempts per second multiplied by events per attempt and average encoded bytes per event. Use the peak investigation interval as well as ordinary traffic. If an illustrative design emits 3 events per checkout, 200 checkouts per second produces 600 events per second before retries or diagnostic output; that is workload arithmetic, not measured vendor throughput. A reversal may increase the demand for evidence just as the team needs to query it, so assess ingestion and investigation together rather than approving each in isolation.

Then rehearse the reversal. Decide which fields establish a cohort's denominator and how delayed delivery affects the observation window. Fewer failures may mean recovery. They may also mean missing outcomes.

Write the SLO consequence down: which checkout objective is protected, which observation window matters, and what evidence lets the release owner resume traffic. Relative improvement between cohorts can still leave both outside the checkout SLO. Sparse traffic can also leave the treatment cohort too small for a useful comparison while the aggregate looks healthy.

For sensitive checkout data, collect a pseudonymous cohort label and the minimum outcome information the investigation needs; avoid copying customer payloads merely because a log accepts strings. GDPR Article 5's data minimization principle is relevant. Minimization does not settle deletion obligations, which need their own review before selecting an evidence store.

Put the release guard before the search interface

This Go program retrieves search evidence and demonstrates an application-side evidence guard after a requested rollback. Set LOG_SEARCH_URL to the full HTTPS search endpoint ending in /v1/logs/search, and set INFRAI_API_KEY to the credential. The explicit GET sends no filter parameters and prints the response without assuming an undocumented response schema. Save it as main.go and run it with go run main.go. The guard then uses separate synthetic inputs: each cohort has 100 completed outcomes, the minimum is 100, and an absolute failure fraction above 2% causes a hold. These illustrative thresholds are neither a statistical test nor a universal SLO; retrieved search results are not automatically accepted as a complete cohort population.

package main

import (
    "fmt"
    "io"
    "net/http"
    "net/url"
    "os"
    "strconv"
    "time"
)

func search() error {
    endpoint := os.Getenv("LOG_SEARCH_URL")
    key := os.Getenv("INFRAI_API_KEY")
    u, err := url.Parse(endpoint)
    if err != nil || u.Scheme != "https" || u.Host == "" ||
        u.User != nil || u.Path != "/v1/logs/search" ||
        u.RawQuery != "" || u.Fragment != "" || key == "" {
        return fmt.Errorf("set a valid search URL and API key")
    }
    client := &http.Client{
        Timeout: 15 * time.Second,
        CheckRedirect: func(req *http.Request, via []*http.Request) error {
            return http.ErrUseLastResponse
        },
    }
    for attempt := 0; attempt < 3; attempt++ {
        req, err := http.NewRequest(http.MethodGet, endpoint, nil)
        if err != nil {
            return err
        }
        req.Header.Set("Authorization", "Bearer " + key)
        resp, err := client.Do(req)
        if err != nil {
            return err
        }
        body, readErr := io.ReadAll(io.LimitReader(resp.Body, 1048577))
        resp.Body.Close()
        if readErr != nil {
            return readErr
        }
        if len(body) > 1048576 {
            return fmt.Errorf("search response exceeds local display limit")
        }
        if resp.StatusCode == http.StatusTooManyRequests && attempt < 2 {
            delay := time.Second * time.Duration(1 << attempt)
            retry := resp.Header.Get("Retry-After")
            if retry != "" {
                if seconds, err := strconv.ParseInt(retry, 10, 64); err == nil && seconds >= 0 {
                    if seconds > 30 {
                        return fmt.Errorf("rate limited; retry no earlier than %s", retry)
                    }
                    if requested := time.Duration(seconds) * time.Second; requested > delay {
                        delay = requested
                    }
                } else if until, err := http.ParseTime(retry); err == nil {
                    if requested := time.Until(until); requested > delay {
                        delay = requested
                    }
                } else {
                    return fmt.Errorf("rate limited; inspect Retry-After before retrying")
                }
            }
            if delay > 30 * time.Second {
                return fmt.Errorf("rate limited; retry no earlier than %s", retry)
            }
            time.Sleep(delay)
            continue
        }
        if resp.StatusCode < 200 || resp.StatusCode >= 300 {
            return fmt.Errorf("search status %d: %s", resp.StatusCode, body)
        }
        fmt.Println(string(body))
        return nil
    }
    return fmt.Errorf("search retry budget exhausted")
}

type Cohort struct {
    Name              string
    Expected          int
    Completed         int
    Failed            int
    EvaluatedRevision string
}

func review(cohorts []Cohort, revision string, minimum int, limit float64) error {
    if len(cohorts) < 2 || revision == "" || minimum < 1 ||
        !(limit >= 0 && limit <= 1) {
        return fmt.Errorf("invalid review policy")
    }
    names := make(map[string]bool)
    for _, c := range cohorts {
        if c.Name == "" || names[c.Name] {
            return fmt.Errorf("missing or duplicate cohort name")
        }
        names[c.Name] = true
        if c.Expected < minimum || c.Completed != c.Expected {
            return fmt.Errorf("%s: insufficient or incomplete evidence", c.Name)
        }
        if c.EvaluatedRevision != revision {
            return fmt.Errorf("%s: rollback exposure is unconfirmed", c.Name)
        }
        if c.Failed < 0 || c.Failed > c.Completed {
            return fmt.Errorf("%s: invalid outcome counts", c.Name)
        }
        if float64(c.Failed)/float64(c.Completed) > limit {
            return fmt.Errorf("%s: failure fraction exceeds policy", c.Name)
        }
    }
    return nil
}

func main() {
    if err := search(); err != nil {
        fmt.Fprintln(os.Stderr, err)
        os.Exit(1)
    }
    cohorts := []Cohort{
        {"control", 100, 100, 1, "baseline"},
        {"treatment", 100, 100, 2, "baseline"},
    }
    if err := review(cohorts, "baseline", 100, 0.02); err != nil {
        fmt.Fprintln(os.Stderr, "HOLD:", err)
        os.Exit(1)
    }
    fmt.Println("Evidence guard passed; continue the release review.")
}
Enter fullscreen mode Exit fullscreen mode

The important input is Expected. It must come from an independently established population of admitted checkout attempts for the same observation window, deduplicated by an application-owned attempt identifier; deriving both expected and completed counts from the same incomplete search result would let missing evidence validate itself. Each input row must contain outcomes for its stated evaluated revision only. If a cohort spans revisions, split and reconcile the population before calling this guard, rather than labeling mixed exposure as baseline.

Do not make this function the rollback actuator. A failed review should hold further rollout and bring the evidence gap to the release owner; an urgent rollback can still be the correct response before complete evidence arrives. Passing this guard establishes only its explicit conditions. It does not prove statistical significance, detect every checkout regression, or establish that a configuration change has reached every client.

Where this advice stops applying

A basic hosted search API is a sensible choice when a small team needs incident reconstruction and can keep the release decision in an application-owned review. It is a poor selection criterion for a system whose acceptance test requires continuous alert delivery, independent evidence export, or detailed governance: those requirements should determine the shortlist before anyone counts dashboards or credentials.

Keep the hold.

When the experiment is too small to compare cohorts, the outcomes arrive too late for the rollback window, or the denominator cannot be independently reconciled, changing the search product does not establish recovery. Keep those cases visible in the release policy. Buy a deeper stack when its capabilities satisfy an actual requirement, and self-host only when ownership of the logging system is worth the capacity and on-call commitment. For this checkout experiment, the decisive question is whether the evidence can support a safe release decision within the required observation window.

Sources

References for the operating choices and data-minimization principle:

Top comments (0)