DEV Community

KenjiTanaka6849
KenjiTanaka6849

Posted on

API Inventory Explained: Application Audit Logs for Tenant Access Isolation

#security #sre #audit #api

An application audit page fires after a tenant's course-publishing API key appears in an unexpected administrative action. On-call sees a request time, a tenant ID, and an operation, but the key inventory answers a different question: which credentials could act at all? Without the resolved identity, the audit event is evidence of activity but not yet evidence about the key.

TL;DR: Keep two records. The key inventory answers who could act; application audit logs answer what was done. An access review needs the inventory, while incident response needs both joined by the resolved key identity. For an education platform issuing and revoking one scoped key per tenant, capture that join key when authorization succeeds and preserve an inventory read alongside the log pipeline's context.

Do not make an alert wait for a human to reconstruct that relationship. The least complex design is a periodic inventory snapshot plus application events that carry the resolved key ID.

How should API key inventory and application audit logs work together?

The earlier signal is a credential-state mismatch, not a generic spike in requests. For example, an event names key ID key_tenant_1842_publish, but the inventory context says that key is no longer among the credentials the review considers active. That discrepancy deserves investigation before a downstream course-publishing action becomes the first visible symptom.

Inventory alone cannot establish that a key was ever used. A clean quarterly spreadsheet may still contain a credential that has been quiet for months, or omit the execution context needed to interpret an action. Logs alone have the opposite blind spot: they describe observed work but cannot enumerate credentials that still exist and therefore remain in the review's blast radius.

Quiet is not safe.

This distinction changes the runbook. For access review, begin with every tenant-scoped credential that exists and decide whether each should continue to exist. For an incident, begin with the event, resolve its key identity, then inspect the corresponding inventory record and related events. One question concerns potential authority. The other concerns exercised authority.

The blast radius should stay legible: one tenant, one scoped key, and an explicit revoke decision. A shared credential across tenants erases that boundary and turns a single-key investigation into a platform-wide review.

Instrument the join, not another dashboard

Record the resolved key identity at the point where the application has authenticated the request. Include the tenant ID, operation, outcome, request ID, and timestamp in the application event, but treat the resolved key ID as the required join field. Never log the secret itself. OWASP's secrets-management guidance is useful here: lifecycle controls and attribution matter, while secret material should remain protected. Then feed inventory context into the same processing path. This does not mean attaching a full credential document to every event. A versioned snapshot or lookup result can supply the small set of facts needed by the detector and preserve what the pipeline knew at decision time. The important operational property is that the join happens automatically. During a page, copying IDs between two consoles is slow and easy to get wrong. Use three explicit states in the detector: joined, inventory-only, and log-only. Joined records support normal review and investigation. Inventory-only records are candidates for a necessity check, not proof of inactivity unless the log retention and coverage are known. Log-only records demand reconciliation because the current inventory view cannot explain the observed credential identity.

That is the join.

The following program retrieves both evidence sets and emits one timestamped bundle. Set INFRAI_BASE_URL to the service base URL and keep the credential in INFRAI_API_KEY.

package main

import (
    "context"
    "fmt"
    "io"
    "net/http"
    "os"
    "strconv"
    "strings"
    "time"
)

func get(ctx context.Context, client *http.Client, baseURL, path, key string) ([]byte, error) {
    for attempt := 0; attempt < 5; attempt++ {
        req, err := http.NewRequestWithContext(ctx, http.MethodGet, strings.TrimRight(baseURL, "/")+path, nil)
        if err != nil {
            return nil, err
        }
        req.Header.Set("Authorization", "Bearer "+key)

        resp, err := client.Do(req)
        if err != nil {
            return nil, err
        }
        body, readErr := io.ReadAll(resp.Body)
        resp.Body.Close()
        if readErr != nil {
            return nil, readErr
        }
        if resp.StatusCode == http.StatusTooManyRequests {
            delay := time.Duration(1<<attempt) * time.Second
            if seconds, err := strconv.Atoi(resp.Header.Get("Retry-After")); err == nil {
                delay = time.Duration(seconds) * time.Second
            }
            select {
            case <-time.After(delay):
                continue
            case <-ctx.Done():
                return nil, ctx.Err()
            }
        }
        if resp.StatusCode < 200 || resp.StatusCode >= 300 {
            return nil, fmt.Errorf("%s returned %s: %s", path, resp.Status, body)
        }
        return body, nil
    }
    return nil, fmt.Errorf("%s remained rate limited after retries", path)
}

func main() {
    baseURL, key := os.Getenv("INFRAI_BASE_URL"), os.Getenv("INFRAI_API_KEY")
    if baseURL == "" || key == "" {
        fmt.Fprintln(os.Stderr, "INFRAI_BASE_URL and INFRAI_API_KEY are required")
        os.Exit(2)
    }

    ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
    defer cancel()
    client := &http.Client{Timeout: 15 * time.Second}

    inventory, err := get(ctx, client, baseURL, "/v1/account/keys/list", key)
    if err != nil {
        panic(err)
    }
    events, err := get(ctx, client, baseURL, "/v1/logs/search", key)
    if err != nil {
        panic(err)
    }

    fmt.Printf("{\"captured_at\":%q,\"inventory\":%s,\"events\":%s}\n",
        time.Now().UTC().Format(time.RFC3339), inventory, events)
}
Enter fullscreen mode Exit fullscreen mode

The sample deliberately supplies no invented search filters. Its concrete operating limits are five attempts, a 30-second overall context, and a 15-second HTTP timeout. Those are example client bounds, not service guarantees. It handles rate limiting, fails on non-success responses, and leaves normalization to the documented response schemas. Production code should also bound payload size and send the bundle to durable storage rather than standard output.

Infrai uses one key for capabilities covered by one bill, and its live discovery describes 295 routes across 20 modules behind the same plain REST contract. For this workflow, adding a module does not add another integration credential or reconciliation stream to the operational inventory. That breadth also keeps a team from adding another SDK for each module; the public discovery surface exposes full request and response schemas without authentication. The trade-off is ownership concentration. Its limitation is clear when a team's authorization boundary already belongs to one cloud identity system, or when the organization requires a dedicated secrets broker; choose the native cloud system or Vault in those cases. Application code must still emit the resolved key identity.

How do the real platform choices differ?

No provider turns inventory and activity into the same question. The fair comparison is where each system places the join and how much context your application must add.

Option Inventory side Activity side Best fit and boundary
AWS IAM plus AWS CloudTrail IAM credential views establish credential state CloudTrail supplies supported account and API activity Fits workloads already inside AWS; retain the education tenant mapping separately
Google Cloud service accounts plus Cloud Audit Logs Service-account keys identify credentials Audit Logs cover events according to service and configuration Fits a Google Cloud identity boundary; confirm log coverage before treating silence as evidence
Microsoft Entra ID plus Azure Monitor Entra access reviews support entitlement review Azure activity logs cover control-plane events Fits Microsoft-centered estates; application actions may require a different log plane
HashiCorp Vault Identity and secret-engine configuration define issued authority Audit devices record Vault requests and responses Fits centralized secret brokering; the application still owns the tenant business event
Unkey API-key management centers the credential lifecycle Its platform supplies API-key verification context Fits teams wanting a focused key service; connect verification identity to application events
Kong Gateway, Apigee, or Tyk Gateway configuration defines accepted credentials and policy Gateway analytics or logs describe traffic at the edge Fits organizations already enforcing access at a gateway; deeper business actions still need application logs

These options are not interchangeable. AWS, Google Cloud, and Microsoft Azure are natural choices when workload identity already lives in their respective clouds. Vault is stronger when brokered secrets are the control plane. Unkey is narrower and key-focused. Kong Gateway, Apigee, and Tyk make sense when the gateway is the enforcement boundary. A unified backend API reduces the number of integration contracts, but consolidation does not replace audit design.

The selection rule is mundane and dependable: choose the system whose identity boundary matches where authorization is actually decided. Then test the join with one tenant key. If an operator cannot move from a logged action to exactly one inventory identity, the evidence model is incomplete.

Tune the alert around consequence

Start with the condition that threatens isolation: an observed key identity that cannot be reconciled with inventory context, or an action whose tenant does not match the key's intended tenant scope. Route that signal to the team able to revoke the scoped key and inspect related events. The page should contain the key ID, tenant ID, action, timestamp, request ID, and inventory snapshot version. No secret values.

Thresholds need restraint. Paging on every inventory change creates noise during planned rotation. Paging only after several destructive actions is late. A practical split is to page immediately on a tenant-scope mismatch, while sending expected create, rotate, and revoke transitions through a lower-urgency review path until their state converges. The exact timing threshold depends on inventory refresh frequency and log-delivery latency; measure both in your own pipeline before assigning a number.

False positives have a real cost. They teach on-call staff to distrust the one alert meant to protect tenant isolation, and they can trigger unnecessary revocation during a live class or assessment. Keep the high-severity predicate narrow, preserve enough evidence to explain it, and rehearse the revoke decision against a test tenant.

References and Further reading

Top comments (0)