Remote video support that survives a technician's phone network

Remote visual assistance demos well on office Wi-Fi and fails the first time a technician climbs into a van. I have watched this happen with a working demo and a confused buyer, and the failure is almost never the codec. It is the network underneath it: the phone changes address mid-session, it sits behind a carrier NAT that no direct candidate can reach, and the server does not know its own public address.

This is the write-up I wanted when I started. Symptoms first, because that is what you search for when a field session has gone dark and the customer is waiting.

The whole thing in one line: field video fails in four different layers, and they look identical from the UI.

It works in the office and fails on cellular

ICE finds a path by gathering candidates. A host candidate is the device's own address. An srflx candidate is the address a STUN server saw after NAT. A relay candidate is an address on a TURN server that both sides send to when nothing direct works.

Consumer Wi-Fi is usually cone NAT, so two peers discover each other with STUN alone and the session connects with no infrastructure at all. A phone on cellular is frequently behind a carrier-grade NAT whose mapping is not reusable. The srflx candidate is still gathered, still looks valid, and every packet sent to it is dropped, because the mapping was opened for a different destination.

Symptom: on Wi-Fi the connection establishes in under a second; on cellular it never leaves checking.

What to look at: log the selected candidate pair. If it is never relay when the call fails, you need TURN before you debug anything else.

pc.oniceconnectionstatechange = () => {
  if (pc.iceConnectionState === 'failed') {
    // which pair did we actually end up on?
    pc.getStats().then((s) => s.forEach((r) => {
      if (r.type === 'candidate-pair' && r.state === 'failed') {
        console.log('failed pair', r.localCandidateId, r.remoteCandidateId);
      }
    }));
  }
};

The relay port answers but the allocation is refused

"No video through the relay" has two causes that are indistinguishable from the browser: the UDP port is unreachable, or the port is fine and the relay declined the allocation. Different fixes, so test them separately.

First, a STUN probe against the same host and port as your TURN server. If it returns an srflx candidate, the port is open and something is listening. Then a real allocation with the real credentials. A relay candidate coming back means TURN works and the problem is elsewhere. A refusal with the port open means credentials or the allocation range — usually a 401 in the coturn log, or a relay port range that was never opened even though the control port was.

# coturn: open the control port AND the relay range
listening-port=3478
min-port=49152
max-port=65535

# a cloud VM must be told its own public address
external-ip=203.0.113.10

The relay range is the part people forget. The control port on 3478 answers, the credentials are perfect, and the allocation still fails because the media ports coturn wanted are closed in the security group. This is the trap that makes a TURN config look correct while every cellular session dies.

Video freezes when the technician walks to the van

A WebRTC transport is bound to a 5-tuple: source address, source port, destination address, destination port, protocol. When the phone moves from Wi-Fi to cellular it gets a new source address and the 5-tuple the session was using is gone. The old candidate pair cannot be revived. On screen this is a freeze with no error, because the connection object has not given up yet.

The recovery is an ICE restart, which renegotiates connectivity on the existing peer connection instead of tearing the session down. A restart is not always enough after a long outage or a carrier change, so the practical chain is: ICE restart first, full rejoin as fallback, jittered backoff between attempts so a fleet of devices does not hammer the server at once when a tower comes back up.

Do not trust the connection state alone. A connection can read connected while frames have stopped. Watch the frame callbacks instead, and treat a missing frame as frozen even while ICE still says connected:

let last = performance.now();
video.requestVideoFrameCallback(function tick() {
  last = performance.now();
  video.requestVideoFrameCallback(tick);
});
setInterval(() => {
  if (performance.now() - last > 3000) restartChain(); // stalled, not disconnected
}, 1000);

ICE connects but no frame ever decodes

On a cloud VM the network interface usually has a private address and the public address is attached by the provider's NAT. If the server gathers candidates from its own interface it advertises something like 172.16.x.x in SDP, and the remote browser tries a private address it can never reach. ICE can even report connected on a local path while the media never leaves the host.

Tell the server its public address, or force the relay path. In coturn that is external-ip; in a Go server it is a -public-ip flag or an ICE transport policy of relay. If a session connects instantly between two machines on the same office network and fails from outside, check this first.

The tab says connected but nothing arrives

Signaling runs over a WebSocket and reverse proxies reap idle WebSockets, often around 60 seconds. The socket closes, the application does not notice, and the UI keeps saying connected because nothing told it otherwise.

The fix is a protocol-level ping from the server, not a JavaScript heartbeat. Background tabs throttle timers aggressively, so a client timer that fires every 20 seconds in the foreground can fire far less often when the tab is hidden — which is exactly when a technician has put the phone in a pocket.

The check order that saves time

1. Log the selected candidate pair. Never relay on a cellular failure means you need TURN first.
2. Probe the relay in two steps: STUN on the control port, then a real allocation. It separates "port unreachable" from "allocation refused".
3. Confirm the relay media port range is open, not just the control port.
4. Set the public address explicitly on any host whose interface is private.
5. Trigger an ICE restart on a freeze and fall back to a full rejoin, with jittered backoff.
6. Keep the signaling socket alive from the server, not from a client timer.

None of this is exotic once it is written down, and all of it is invisible in a demo on good Wi-Fi. The failure modes only show up in the field, which is where a field-assist product actually runs.

I packaged the working version of all of this — the Go signaling server and SFU, the browser publisher and viewer, the ICE restart and rejoin chain, the diagnostics panel that shows the selected candidate path, the coturn config and the two-probe relay test — as a self-hosted source kit, so you can prove each layer on your own server instead of inferring it from a black box:

https://lowlatencykit.com/field-assist/

The relay test page is free and needs no account, if you just want to check whether your own TURN server allocates:

https://lowlatencykit.com/test/turn/

Comments

[AiLaoHuYu]

Thanks - the frozen-frame case is the one that bit hardest, and you are right that the diagnostics are only half there.

What the kit does today: the viewer tracks performance.now() from requestVideoFrameCallback and flags "frozen" after 2000 ms even while iceConnectionState still reads connected, and the stats panel shows the selected candidate pair as localType -> remoteType (so host -> relay vs srflx -> srflx is visible) with RTT beside it.

What it does not do is exactly what you named: there is no single record tying session id + last healthy frame timestamp + selected candidate pair together at the moment of failure. fullRejoin() closes the PeerConnection and resets the previous stats sample, so the path the session was on just before it died is discarded - and that is the only evidence of which layer failed.

On the duplicate-request test: session identity here is the room name, not the connection. fullRejoin() closes the PeerConnection and reconnects the signaling socket but rejoins the same room, so a reconnect does not mint a new session. The kit has no work-order integration, so "one support request" stays an application-layer property - key it on the WebSocket connection instead of the room id and you get exactly the duplicate you describe, which is worth a regression test on the integrator's side.

The other half is ordering: capture the last good sample inside the bounded ICE-restart window, before teardown, not after. That gap belongs in the kit rather than in every integrator's app, so I am closing it there.

[오피셜메일]

현장 지원에서는 화면에 연결됨이 떠 있어도 실제 프레임이 멈춘 경우를 별도로 잡아야 한다는 점이 유용합니다. ICE 재시작 뒤 전체 재참여로 넘어갈 때 상담 세션 ID와 마지막 정상 프레임 시각, 선택된 candidate pair를 함께 남기면 장애 원인 확인과 고객 대화의 연속성을 모두 지킬 수 있겠습니다. 재접속 때문에 같은 지원 요청이 두 건으로 만들어지지 않는지도 회귀 테스트에 넣고 싶네요.

[AiLaoHuYu]

감사합니다. 말씀하신 세션 ID, 마지막 정상 프레임 시각, 선택된 candidate pair를 같은 장애 타임라인에 남기면 원인 판별이 훨씬 빨라집니다. 저는 재참여할 때 기존 지원 요청 ID를 유지하고, 새 연결은 'recovery attempt'로 별도 기록하는 방식을 선호합니다. 그래야 같은 지원 건이 재접속 때문에 두 건으로 중복 집계되지 않습니다. 혹시 이런 복구 흐름을 실제로 테스트하고 계신 스택이 있으신가요?