DEV Community

TobiasHawkins9231
TobiasHawkins9231

Posted on

Why I Compare Cloud Logging for a Startup App — Beyond Pricing

#observability #logging #sre

TL;DR: For a startup serving the EU and US, choose the least elaborate hosted logging system that can reliably reconstruct one pricing decision across a flag change, an application release, and an order. Better Stack is a good low-friction default; Grafana Cloud Logs fits teams already operating Grafana; CloudWatch Logs fits an AWS-centered system; and Datadog Logs earns its operational overhead when logs must participate in a broader, mature incident workflow. Infrai can cover basic centralized structured logs through a plain REST API, without another SDK to install, but it is a narrower fit when retention governance, alert routing, deletion, export, or trace exploration drives the decision.

Price matters after that test, not before it. Logging that cannot answer why order ord_74bf received pricing rule regional-v3 is cheap until the first disputed checkout. My decision rule is blunt: prove the reconstruction path with representative events, verify notification and retention behavior, then compare the bill using the same ingest volume and retention window.

How should a startup app compare cloud logging options?

Suppose a new pricing rule rolls out behind a flag at 10% in both the EU and US. At 14:07 UTC, support reports that a customer saw one total on the product page and another at checkout. The useful question is not, "Did the service emit an error?" It is: which rule, flag decision, release, region, and request produced each amount?

That changes the event contract. I want a stable event name, UTC timestamp, environment, region, release identifier, request and trace identifiers, order identifier, flag key, flag variant, pricing-rule version, currency, and outcome. I would also include the actor type, but avoid raw email addresses, full names, payment data, and other fields that turn an operational record into a deletion liability.

One record is rarely enough. The reconstruction normally crosses a flag evaluation event, a pricing calculation event, and the order write. Every event needs the same correlation identifiers. A trace_id field helps correlate records, but it does not create a distributed trace or a span tree; if the team needs causal trace navigation, it should select and operate a tracing product as such.

Start by proving that an operator can retrieve logs without depending on an undeclared filter. This small Go program calls the verified search route, takes the base URL and credential from the environment, honors Retry-After on a 429, and exposes the real error body instead of turning every failure into "no logs found."

package main

import (
    "fmt"
    "io"
    "net/http"
    "os"
    "strconv"
    "strings"
    "time"
)

func main() {
    baseURL := strings.TrimRight(os.Getenv("INFRAI_BASE_URL"), "/")
    apiKey := os.Getenv("INFRAI_API_KEY")
    if baseURL == "" || apiKey == "" {
        fmt.Fprintln(os.Stderr, "INFRAI_BASE_URL and INFRAI_API_KEY are required")
        os.Exit(2)
    }

    client := &http.Client{Timeout: 15 * time.Second}
    for attempt := 0; attempt < 5; attempt++ {
        req, err := http.NewRequest(http.MethodGet, baseURL+"/v1/logs/search", nil)
        if err != nil {
            fail(err)
        }
        req.Header.Set("Authorization", "Bearer "+apiKey)
        req.Header.Set("Accept", "application/json")

        resp, err := client.Do(req)
        if err != nil {
            fail(err)
        }
        if resp.StatusCode == http.StatusTooManyRequests {
            resp.Body.Close()
            delay := time.Second << attempt
            if seconds, err := strconv.Atoi(resp.Header.Get("Retry-After")); err == nil && seconds > 0 {
                delay = time.Duration(seconds) * time.Second
            }
            time.Sleep(delay)
            continue
        }
        if resp.StatusCode < 200 || resp.StatusCode >= 300 {
            body, _ := io.ReadAll(resp.Body)
            resp.Body.Close()
            fail(fmt.Errorf("search failed: %s: %s", resp.Status, strings.TrimSpace(string(body))))
        }
        _, err = io.Copy(os.Stdout, resp.Body)
        resp.Body.Close()
        if err != nil {
            fail(err)
        }
        return
    }
    fail(fmt.Errorf("search remained rate limited after 5 attempts"))
}

func fail(err error) {
    fmt.Fprintln(os.Stderr, err)
    os.Exit(1)
}
Enter fullscreen mode Exit fullscreen mode

There are no guessed query parameters in that request because the search filters are not declared in discovery metadata. Validate the supported filter contract during the trial before putting a query into a runbook. The application events themselves should remain intentionally boring: a runbook must name fields without interpreting free-form messages, and a rollback should change the flag or release without changing the vocabulary used to investigate it.

Keep it dull.

Compare the operational fit before the invoice

The products overlap at ingestion and search, then separate around integration depth and operations. A fair comparison starts from the system the team already has.

Option Strong fit Trade-off to validate in a trial
Better Stack (formerly Logtail) A small team that wants hosted logs, search, and alerting with a relatively direct onboarding path Confirm that its retention, regional storage choices, and incident workflow match the company's data policy
Amazon CloudWatch Logs Workloads already centered on AWS identity, services, and operations Dashboards, queries, subscriptions, alarms, and access policy can become several separately managed pieces
Datadog Logs A team that wants logs connected to a broad monitoring and incident-management suite Indexing, archives, rehydration, retention choices, and ingest controls deserve careful cost and governance modeling
Grafana Cloud Logs A team already using Grafana dashboards and Loki's label-oriented query model Label design is an operating decision; high-cardinality values belong in log content rather than labels
Infrai Basic EU/US centralized logs where a plain REST API, one key, and one bill reduce client-library and credential maintenance Search filters are not fully declared in discovery metadata; there is no documented alert route, per-user log deletion, bulk export or subscription stream, or configurable retention entry point

None wins in the abstract. CloudWatch's native AWS context can outweigh its setup burden. Datadog's integrated workflow can justify complexity for a staffed operations function. Grafana Cloud becomes more attractive when engineers already think in Loki queries, while Better Stack is often easier to explain to the next developer joining a small team.

The narrower REST option has a different appeal: anything able to send an authenticated HTTP request can use it, with no language-specific package version to track. Its public discovery surface exposes request schemas and runnable examples, while one API key spans 295 routes in 20 modules and those calls appear on one bill. For this rollout, a single key can reduce credential rotation and billing reconciliation when flags and logs share an operational boundary. I would keep it on the shortlist only when basic log ingestion and incident search are the real requirement, not a proxy for an eventual compliance archive or full observability suite.

Do not compare a seven-day, all-fields-indexed configuration with a thirty-day configuration that archives most data. Fix the assumptions first: daily uncompressed bytes, peak events per second, retention, indexed versus archived data, rehydration frequency, query concurrency, egress, and support tier. Then use each vendor's current calculator or pricing page. Unit prices age quickly.

Implement the rollout as an evidence chain

Before enabling the flag, deploy the event contract and test it in both regions. The flag evaluation and the pricing decision should record the evaluated variant and the immutable pricing-rule version. The order write should carry the same request and trace identifiers. If any hop creates a new identifier without preserving the old one, reconstruction stops there.

Keep flag state out of the logger's memory. Log what the application actually evaluated for that request, not what a control plane says the flag should be now. This distinction matters during propagation delays and rollback: current configuration is not historical evidence.

I use a short acceptance checklist for the first rollout:

  1. Generate one control and one treatment order in the EU, then repeat in the US.
  2. Search by order ID and recover all three event types without relying on a person's name or email.
  3. Search by rule version and compare outcomes by region and flag variant.
  4. Confirm that duplicate delivery does not create a second business action; logging pipelines and queues may retry.
  5. Record the exact rollback owner and the evidence that triggers rollback.

The fourth item is an idempotency reflex, not a logging feature. An order mutation needs its own idempotency key or uniqueness constraint. Duplicate log events should carry a stable event ID when the application's pipeline can retry, so investigators can distinguish repeated delivery from repeated execution.

There is another boundary worth making explicit. If the selected logging service does not route alerts, run threshold evaluation in an external alerting system or a scheduled query job, and send notifications through the team's established paging path. If the risk is "the evaluation job never ran," polling the logs from that same job is insufficient. Use an independent heartbeat monitor such as Healthchecks for that silent-failure case.

Verify the incident path, then rehearse rollback

Verification is a timed exercise. Give an engineer who did not build the feature an order ID and ask them to identify the region, release, flag variant, rule version, and final outcome. They should be able to do it from stored evidence alone. Record where the investigation stalls; those gaps are more informative than a polished dashboard.

Test absence too. Pause the scheduled verifier in staging and confirm that the heartbeat monitor notices the missed run. Generate a known bad pricing outcome and confirm the alert reaches the correct rotation. A query that works only when somebody remembers to open a tab is a diagnostic, not an alert.

Rollback should be deliberately small: disable the treatment flag, verify new control events in both regions, and leave the old rule version searchable for the agreed retention period. Do not erase the incident trail during rollback. Preserve the release, flag decision, and calculation outcome needed for the postmortem.

Then test the governance claims that usually hide behind a checkbox. Can an authorized operator restrict access by role? Can the company satisfy its deletion policy when an identifier appears in a log? Can records be exported in bulk for a downstream archive? Can retention differ by environment or data class? Teams that require mature retention controls, user-level deletion, subscription streams, or enterprise approval workflows should prefer a product that documents and demonstrates those paths.

No single screenshot proves this. Run the drill.

My decision boundary

For a small startup with conventional application logs, I would begin the proof with Better Stack and Grafana Cloud, add CloudWatch when the workload is predominantly AWS, and include Datadog when the desired end state joins logs to a wider operations platform. I would include the plain REST option when minimizing client integration is important and scheduled external alert evaluation is acceptable.

I would reject any candidate that cannot reconstruct the pricing decision across region, release, flag, rule, and order. I would also reject a narrow logging service when legal deletion, long-term export, configurable retention, distributed tracing, crash symbolication, session replay, or synthetic monitoring is already a committed requirement. Those are product boundaries, not backlog details to wave away during procurement.

The final choice should come from two artifacts: a successful reconstruction drill and a cost model built from the same workload assumptions. That is enough structure for a startup decision, while leaving room to change tools when the operating model changes.

References

Top comments (0)