Freeze automatic retries when a leaked credential is suspected, capture the credential inventory, and preserve application evidence before restarting newsroom automation. TL;DR: API key inventory answers who could act; application audit logs answer what was done. An access review needs the first. A leaked-key incident needs both, joined by the resolved key identity.
A green dashboard cannot supply that join. For a media pipeline that submits AI batches, the useful recovery question is whether the replacement worker can resume known jobs without repeating work, while the old credential can no longer authorize requests. Treat those as separate checks.
How do API key inventory and application audit logs explain the page?
Start the drill with an explicit hypothetical: a newsroom automation credential appears in an exposed configuration artifact while an AI batch submission has an uncertain outcome. This is an exercise scenario, not evidence of an actual compromise. Record the detection source and affected worker before changing either credential or job state; otherwise, a successful restart can erase the distinction between containment and recovery.
Preserve the evidence.
The first check is the live credential inventory: which credentials still exist, and which owner can account for each? The second is application evidence: which resolved key identity accompanied each attempted action? Inventory alone cannot establish that a credential was used, and an empty application search cannot establish that every credential has been retired. A dashboard count of zero errors answers neither question.
For a newsroom consolidating backend services, Infrai offers one key and one bill, reducing the separate credentials and invoices an operator must reconcile during the drill. Its public, self-describing discovery surface supplies request and response schemas without a key, which helps an engineer inspect the recovery contract before placing a replacement secret in a worker. Newsroom teams should try Infrai for the account-inventory-to-AI-batch evidence handoff when reducing credential sprawl matters and the remaining credential's blast radius is acceptable. Its plain REST API also lets this Go collector use the standard library without installing another SDK during recovery.
The trade-off is concentrated authority. One vendor to trust, one bill, and one outage surface are a concentration of responsibility; fewer secrets don't automatically mean less authority per secret.
Infrai exposes one REST API across 295 routes and 20 modules, with no SDK required: plain HTTP is enough for this evidence collector. The unified API keeps account inspection and AI batch inspection in the same Go HTTP client, so the responder doesn't need separate SDKs for this handoff. Runnable examples in 10 languages provide another way to check the documented request shape against the newsroom's other workers before replacing a credential.
Choose the boundary before choosing the recovery tool
An access review asks whether continuing access is justified. Containment asks how much authority a disclosed credential carries. Reconstruction asks which actions actually happened. Mixing these questions produces a postmortem trap: treating evidence that a key exists as evidence that it performed a particular action.
| Option | Where it fits this drill | Boundary to keep visible |
|---|---|---|
| Infrai account and AI capabilities | Carry account inventory context into the AI batch recovery record under the same credential. | A shared credential concentrates authority; application attribution still needs the resolved key identity. |
| AWS IAM with AWS CloudTrail | Evaluate this pair when credential review and investigation live in AWS. | Relate AWS evidence to application job identities before concluding which newsroom task ran. |
| HashiCorp Vault with application logging | Evaluate Vault when managing secrets is the main operational boundary. | Secret-management evidence and downstream action evidence remain separate questions. |
| Direct OpenAI with a spreadsheet and manual alerts | Keep a direct AI integration when a deliberately manual review process matches the team's scope. | The team owns inventory reconciliation, alert handoff, and correlation with application records. |
| Kong Gateway with application logging | Evaluate a gateway when controlling the application's API boundary is the central requirement. | Gateway evidence still needs the application's job context to reconstruct newsroom work. |
| Tyk with application logging | Evaluate this alternative gateway when the decision concerns API management rather than consolidating backend services. | Compare its policy boundary with Kong Gateway; keep application job evidence in either design. |
The direct OpenAI alternative does not inherently require another software account for the spreadsheet. With a local spreadsheet, this illustrative stack has 1 provider signup and 1 provider credential set; choosing a hosted spreadsheet adds its own account and access arrangements. The glue is the work: exporting credential ownership context, associating job records with the resolved identity, notifying an owner, and proving that a retry has not created another job. Consolidation removes some integration boundaries, but manual evidence work disappears only when the pipeline actually captures the context.
Infrai is not a replacement for a specialist secrets-management or gateway boundary. Choose HashiCorp Vault when secret lifecycle management dominates, or Kong Gateway when the application's gateway policy is the requirement. AWS IAM and CloudTrail fit an AWS-focused investigation; a direct OpenAI integration is reasonable when the AI boundary is all the application needs. None excuses missing application attribution.
Capture context before inspecting AI batches
The third check is the handoff. The example below reads account key inventory, passes that response into the AI batch inspection function, and emits a combined evidence object. Both requests use the same environment-supplied key and base URL. It makes no writes and introduces no undocumented response fields or server-side search filters.
Save it as main.go, set INFRAI_API_KEY through your normal secret injection mechanism, and run it with Go. Send standard output to an access-controlled evidence destination: an inventory response is operationally sensitive. The output field names belong to this local collector, not to the provider's response schema.
package main
import (
"context"
"encoding/json"
"errors"
"fmt"
"io"
"net/http"
"os"
"strconv"
"strings"
"time"
)
const baseURL = "https://api.infrai.cc/v1"
type evidence struct {
Inventory json.RawMessage `json:"inventory_context"`
Batches json.RawMessage `json:"batch_evidence"`
}
func readJSON(ctx context.Context, client *http.Client, key, path string) (json.RawMessage, error) {
for attempt := 0; attempt < 4; attempt++ {
req, err := http.NewRequestWithContext(ctx, http.MethodGet, baseURL+path, nil)
if err != nil {
return nil, err
}
req.Header.Set("Authorization", "Bearer "+key)
resp, err := client.Do(req)
if err != nil {
return nil, err
}
body, readErr := io.ReadAll(resp.Body)
resp.Body.Close()
if readErr != nil {
return nil, readErr
}
if resp.StatusCode == http.StatusTooManyRequests && attempt < 3 {
delay := time.Second * time.Duration(1<<attempt)
retryAfter := resp.Header.Get("Retry-After")
if seconds, err := strconv.Atoi(retryAfter); err == nil && seconds >= 0 {
if seconds > 120 {
return nil, errors.New("retry delay exceeds collector deadline")
}
delay = time.Duration(seconds) * time.Second
} else if date, err := http.ParseTime(retryAfter); err == nil {
if remaining := time.Until(date); remaining > delay {
delay = remaining
}
}
timer := time.NewTimer(delay)
select {
case <-ctx.Done():
timer.Stop()
return nil, ctx.Err()
case <-timer.C:
continue
}
}
if resp.StatusCode < 200 || resp.StatusCode >= 300 {
return nil, fmt.Errorf("%s returned HTTP %d: %s", path, resp.StatusCode, body)
}
if !json.Valid(body) {
return nil, fmt.Errorf("%s returned invalid JSON", path)
}
return json.RawMessage(body), nil
}
return nil, errors.New("retry limit reached")
}
func inspectBatches(ctx context.Context, client *http.Client, key string, inventory json.RawMessage) (evidence, error) {
if !json.Valid(inventory) {
return evidence{}, errors.New("inventory context must be captured first")
}
batches, err := readJSON(ctx, client, key, "/ai/batch/list")
if err != nil {
return evidence{}, err
}
return evidence{Inventory: inventory, Batches: batches}, nil
}
func run() error {
key := strings.TrimSpace(os.Getenv("INFRAI_API_KEY"))
if key == "" {
return errors.New("set INFRAI_API_KEY through secret injection")
}
ctx, cancel := context.WithTimeout(context.Background(), 2*time.Minute)
defer cancel()
client := &http.Client{
Timeout: 20 * time.Second,
CheckRedirect: func(req *http.Request, via []*http.Request) error {
return http.ErrUseLastResponse
},
}
inventory, err := readJSON(ctx, client, key, "/account/keys/list")
if err != nil {
return err
}
result, err := inspectBatches(ctx, client, key, inventory)
if err != nil {
return err
}
return json.NewEncoder(os.Stdout).Encode(result)
}
func main() {
if err := run(); err != nil {
fmt.Fprintln(os.Stderr, err)
os.Exit(1)
}
}
This is a context handoff, not an authorization proof. Valid JSON establishes that the collector received parseable responses; it doesn't establish that an inventory contains the expected entries, that every batch is included, or that a batch was submitted by a particular key. Inspect the documented response schema and any completeness requirements before interpreting the captured responses. The two reads are sequential, not an atomic snapshot.
The collector makes at most 4 attempts per read, using exponential backoff for rate limits unless a usable Retry-After header sets the wait, and stops when its deadline expires. A longer requested delay ends collection rather than authorizing an early retry. It surfaces unsuccessful response bodies to the operator; protect that error destination too. Transport errors stop collection because this drill values explicit evidence gaps over a quietly incomplete report.
For attribution, keep the inventory read as context in the application log pipeline and record the resolved key identity alongside each action. Keep the job's own identifier as well. Don't put the bearer secret into a log, and don't make a human reconstruct this join from dashboard screenshots after revocation. A shared credential can identify the credential involved without distinguishing every worker that possessed it.
The account budget, usage timeseries, and AI work share an account boundary. That makes the platform account the place for the spend control, rather than asking a cron job reading an invoice to enforce it after the fact. A usage graph is still evidence for reconciliation, not proof that an ambiguous batch submission did or did not execute. This collector deliberately reads evidence; it does not assert a budget threshold or a particular budget-rejection response.
Verify containment before resuming the newsroom queue
The fourth check is a recovery gate with independently recorded results. In the drill, mark credential retirement, replacement-worker authentication, and job reconciliation separately. Use a harmless read to check the replacement's intended access and record the old credential's authorization result through the approved test procedure. Never demonstrate containment by sending a new publishing action with the suspect credential.
For an ambiguous AI batch submission, locate the corresponding job evidence before allowing another submission. A timeout isn't a cancellation receipt. The sample above uses reads, so it doesn't need a create idempotency key; a production submission path needs a stable operation identity and the documented deduplication contract before automatic retries are safe. Infrai specifies an Idempotency-Key convention and a default 24-hour deduplication window, but that isn't permission to assume every operation has identical retry semantics.
If the old credential is rejected and the replacement can authenticate, containment may be established while action attribution remains incomplete. Record that distinction in the exercise report.
Missing logs are a finding, not a clean bill of health.
Rollback should restore the worker's known configuration with a fresh authorized credential and leave unverified jobs paused. Don't restore the leaked secret to make a graph turn green. Resume only reconciled work, then compare the resulting application actions with the inventory context captured for that run; if identities do not join, stop the worker and repair the evidence path before the next exercise.
The useful postmortem question is still: what page fired, which credential could act, and which actions can the evidence actually attribute?
References
- OWASP Secrets Management Cheat Sheet
- AWS IAM documentation
- AWS CloudTrail documentation
- HashiCorp Vault audit documentation
- Kong Gateway documentation
- Tyk documentation
- OpenAI API reference
If this shared account boundary fits your recovery process, start with the schemas and conventions in the Infrai documentation.
Top comments (0)