Short answer: choose the simplest HTTP ingestion API that preserves request identity, deployment context, timestamps, and a searchable event body; add a managed search backend only after those fields are stable. For a startup dashboard, the hard part is reconstructing an incident, not collecting another stream of text.
Start small.
Which API should a startup use for centralized application logs ingestion and search?
Start with the evidence you need to recover. A request enters the service, the application emits structured events, an agent or small transport sends them to a central store, and the dashboard searches by time range, service, environment, trace or request ID, and severity. The API is only one link in that path. If an event cannot be tied to a customer action or deployment, a very fast ingestion endpoint still leaves you guessing.
For a B2B SaaS product, I would make the application write JSON to a narrow internal interface and keep the storage provider behind it. That keeps the first implementation shippable and leaves room to change the collector or search engine later. The interface should accept batches, return an explicit result, and make retry behavior visible.
Build the evidence path before choosing an endpoint
A useful log event answers five questions: what happened, where it happened, to which request, under which release, and when. Put those answers in fields rather than packing them into a sentence. A practical minimum is timestamp, level, service, environment, release, request_id, trace_id when available, tenant_id in a carefully access-controlled form, event, and a bounded error object. Never treat a customer email, token, cookie, or raw authorization header as ordinary context.
The timestamp needs a declared convention. Use UTC and record the event time at the source; ingestion time is useful too, because delayed delivery is a different failure mode from a late application action. Keep both if the backend permits it. Search results should make the distinction obvious.
A native crash report is a separate evidence stream. Electron's crashReporter documentation describes crash reports and minidumps, which can help investigate a native process failure, but that does not replace application logs for a request-level reconstruction. Keep crash identifiers linked to the same release and session vocabulary when the client can provide it.
A TypeScript ingestion boundary you can test
The following boundary deliberately knows nothing about a particular backend. It validates the fields that make search useful, caps the event payload, and returns a result that callers can test without a live search cluster. The production adapter can translate the batch into the chosen HTTP API.
type LogLevel = "debug" | "info" | "warn" | "error";
type LogEvent = {
timestamp: string;
level: LogLevel;
service: string;
environment: string;
release: string;
requestId: string;
traceId?: string;
tenantId?: string;
event: string;
error?: { name: string; message: string; stack?: string };
fields?: Record<string, string | number | boolean>;
};
type IngestResult = { accepted: number; rejected: number };
type LogSink = {
ingest(events: readonly LogEvent[]): Promise<IngestResult>;
};
const MAX_EVENTS = 100;
const MAX_EVENT_NAME = 120;
export async function publishLogs(
sink: LogSink,
events: readonly LogEvent[],
): Promise<IngestResult> {
if (events.length === 0) return { accepted: 0, rejected: 0 };
const bounded = events
.filter((item) => item.event.length > 0 && item.event.length <= MAX_EVENT_NAME)
.slice(0, MAX_EVENTS);
return sink.ingest(bounded);
}
The important behavior is outside the function too. A caller should attach a stable request ID at the edge, preserve it across asynchronous work, and log the same ID in success and failure paths. The sink adapter should use bounded batches, an idempotency key or equivalent duplicate strategy, deadlines, and a retry policy that cannot turn an outage into an application-wide retry storm. A dropped debug event is usually preferable to blocking a customer request on logging. An authentication failure or a queue saturation event deserves a different alert path.
Test this boundary with malformed timestamps, oversized batches, duplicate delivery, missing request IDs, and a sink timeout. I have seen teams test only the happy path and then discover that the incident search was missing the exact error fields needed to explain it. The test is cheap. The missing evidence is not.
Search is a reconstruction workflow, not a text box
Search should support a sequence, not one heroic query. First narrow the time window around the customer report. Then filter by environment, service, and release. Next pivot from request_id or trace_id to neighboring events, including retries and downstream calls. Finally compare the failing request with a successful request from the same release.
That workflow creates backend requirements: indexed fields for the pivots, full-text search for the human message, retention long enough to cover support and deployment cycles, and access controls that prevent one tenant's records from appearing in another tenant's investigation. A dashboard that can search text but cannot filter by release is a poor incident tool.
This is the pivot.
Do not hide ingestion lag. Display the event timestamp and the received timestamp, plus a freshness indicator. If logs arrive late, the investigator needs to know whether the absence is evidence or merely transport delay. Your mileage may vary with batch size and network conditions; measure those in your own deployment instead of borrowing a vendor benchmark.
The trade-offs that decide the backend
| Decision | Prefer the simpler option when | Pay for the heavier option when |
|---|---|---|
| Direct HTTP ingestion vs. local collector | You have one service, modest volume, and can tolerate coupling an adapter to the app | You need buffering, redaction, fan-out, or a common policy across many runtimes |
| Document search vs. indexed fields | The team is still learning which pivots matter | Incident response depends on predictable filters and saved queries |
| Synchronous send vs. async queue | You need a small first release and can drop noncritical events | Delivery guarantees matter more than request latency |
| Short retention vs. long retention | Support handles incidents quickly and storage access is limited | Customers report problems after long billing or release cycles |
The catch is that a backend optimized for easy setup may be unsuitable when you need strict tenant isolation, regional retention rules, or an audit trail for every accepted event. In that case, choose a design with an explicit collector or queue and a storage layer whose access model you can review. Stick with direct HTTP when the system is small, the event contract is tested, and losing a low-value event is an acceptable trade-off.
Cost matters, but event shape and retention usually matter first. A noisy payload with unbounded stack traces can overwhelm any pricing model. Sample debug events, keep errors useful, and make retention a deliberate product and support decision rather than a default nobody owns.
Before shipping, make one synthetic request and follow its ID through the edge, application, worker, and dashboard. Deploy a new release and confirm that old and new events can be separated. Inject a controlled error and verify that the error object remains searchable without exposing secrets. Measure delivery latency, rejection rate, duplicate rate, and storage growth.
Then write the incident query in plain language: “show failed checkout requests for tenant X in production after release Y, pivoted by request ID.” If the data model cannot express that sentence, the API choice is premature.
This approach is intentionally vendor-neutral. It gives a startup a small boundary to ship, a concrete evidence contract to evolve, and a decision rule for when buffering, stronger isolation, or longer retention justifies more infrastructure.
Top comments (0)