A customer called to say OTPs weren't reaching them.
We checked the logs. Every request came in. Every SMS went out. The provider said delivered. Nothing looked broken, which is about the worst state a bug report can be in.
So we asked for a screen recording. Then two days of guessing at timestamps and flipping between two tools that don't share a clock. We found it. It should not have taken two days.
The bug isn't the part I still think about. The two days are.
The seam
The replay showed what the user did. It didn't know what it called. The traces showed the calls. They didn't know what the user did.
The answer was sitting in the gap between two tools, and closing that gap was hand work every single time. Watch the video. Note a timestamp. Switch tabs. Search a window around it. Get back forty traces. Guess.
Both tools are good at their own layer and blind at the seam between them.
What I built
Syncline plays the session recording, the backend trace, and the SQL that ran on one scrubber.
Drag to 00:42 and you see the user click Checkout, the POST /api/checkout it fired, the four spans that fanned out from it, and the Prisma query in there that took 1.8 seconds.
How the correlation works
Probably the only part you care about, so it goes first.
The browser SDK records with rrweb and patches fetch and XHR to mint a W3C traceparent for every outgoing request. That trace ID gets written into the replay stream itself, as an rrweb custom event, at the frame the request fired. The recording is self-describing. There's no side table mapping time ranges to traces.
Your backend takes no SDK from me. OpenTelemetry auto-instrumentation already reads an incoming traceparent and continues the trace, so you point OTEL_EXPORTER_OTLP_ENDPOINT at Syncline and that's it. The viewer resolves player time to trace ID to span tree, and draws the backend lanes under the video.
Frontend:
import { startRecording } from 'syncline-browser';
startRecording({
key: 'pk_live_...',
endpoint: 'https://syncline.example.com',
traceOrigins: ['https://app.acme.com', 'https://api.acme.com'],
release: 'web@2.4.1',
user: { id: currentUser.id },
});
Backend:
OTEL_EXPORTER_OTLP_ENDPOINT=https://syncline.example.com
If you already run OTel, the server side is one env var.
Two decisions I'd defend
The link is by ID, not by time. Clock skew between a browser and a server can misdraw a lane by a few milliseconds. It can't put a request on the wrong trace. Timestamp matching gets both of those wrong, and it gets the second one wrong quietly, which is the exact thing that cost me two days.
Sampling is inverted. If a session is being recorded, the browser forces sampled=1 on the traceparent. The usual arrangement lets the backend decide, so sooner or later you open the replay of the one bug that matters and its spans were sampled away. I'd rather have no replay than a replay I trusted that can't explain itself.
Smaller things, all of which came from a real bug
The SDK never breaks the page. Every patched path is wrapped and falls through to the original fetch on any failure, and there's a test that makes both instrumentation hooks throw and asserts the request still completes. A recording tool that takes down checkout is worse than no recording tool.
It never injects cross-origin. traceparent goes only to origins you list in traceOrigins, which defaults to the page's own. Sending it to a third party leaks internal trace IDs and adds a header their CORS policy doesn't allow, so a working request turns into a failed preflight. Subdomains don't match either, because a third-party widget can be parked on one.
Masking is on by default. Query values are stripped out of recorded URLs and only the keys kept, so ?token=abc&page=2 becomes ?token&page. Fragments get dropped entirely, since that's where implicit-flow tokens live.
Errors are captured, console output isn't, unless you ask for it. An uncaught error is the thing the recording exists to explain. Console arguments are whatever the app decided to print, which on plenty of codebases means tokens and request bodies.
What it doesn't do
No SSO. No way to export a project before you delete it. The SDK is pre-1.0, so the integration surface is settled enough to build on but a minor version may still move it.
Everything else is there and running: ingest, trace stitching, the viewer, accounts and orgs, error and console capture, search, retention, an audit log.
Running it
Three Node processes (api, worker, web) plus Postgres, Redis, and anything that speaks S3.
cp .env.production.example .env.production # nothing has a working default
docker compose -f docker-compose.prod.yml --env-file .env.production up -d --build
Point DATABASE_URL, REDIS_URL and S3_ENDPOINT at managed services and that's the deployment. On a single box with none of those, --profile bundled brings up all three alongside.
AGPL-3.0. Self-host it, modify it. Run a modified Syncline as a network service and you share the modifications.
Go break it
It's early. I'd rather hear about a bug from you than find it myself.
- Demo: https://youtu.be/0yoq6Rpnt1g
- Live: https://syncline.co.in
- Code: https://github.com/Adithya-Adi/syncline
- SDK: https://www.npmjs.com/package/syncline-browser
What I actually want to know is what you do instead. Everyone I've asked about this has their own manual workaround for it, and they're all different, and I don't think any of us should need one.
Next time someone tells me the OTP never arrived, I want that to be a 30-second answer.
Top comments (0)