DEV Community: hariharan

What's New in Apple's Foundation Models Framework at WWDC 2026

hariharan — Thu, 18 Jun 2026 12:21:00 +0000

From a single on-device model to a hybrid, multimodal, agentic AI platform — and it's going open source

When Apple shipped the Foundation Models framework at WWDC 2025, the pitch was tidy: a Swift API onto the ~3B-parameter model that powers Apple Intelligence, running on-device, no API key, no network round-trip, no per-token cost. Great for privacy-sensitive, offline-friendly work — and deliberately narrow.

WWDC 2026 blows that scope wide open. The framework is no longer "the on-device model with a nice Swift wrapper." It's now a hybrid platform that spans on-device inference, Apple's own server models on Private Cloud Compute, and third-party frontier models like Claude and Gemini — all behind one session API. It gained vision. It gained agentic primitives. It got a Python SDK, a CLI, an evaluations framework, and it's going open source this summer, including on Linux.

This is a tour of what changed, with the Swift that matters.

The on-device model got rebuilt

The headline isn't a new feature — it's that the base model is better. Apple rebuilt the on-device model with stronger reasoning and tool calling, and refined the guardrails to cut false positives (a real pain point in the first release, where benign prompts sometimes tripped safety filters).

Two practical APIs landed (starting in iOS 26.4) that you'll reach for constantly: inspecting context size and counting tokens before you spend them.

let model = SystemLanguageModel()
print(model.contextSize)
// 8192

let count = try await model.tokenCount(
    for: "What are the Japanese characters for origami?"
)
print(count)

The on-device context window is 8,192 tokens. Knowing your token budget up front means you can decide — programmatically — whether a request fits on-device or needs to escalate to a server model.

Vision: the on-device model can now see

The on-device model is multimodal now. You attach an image to a prompt and ask about it — object identification, text extraction, screenshot understanding, receipt parsing — entirely on-device, no cloud hop.

let response = try await session.respond {
    "What animal is this?"
    Attachment(UIImage(...))
}

Attachments accept UIImage, NSImage, CGImage, Core Image, CVPixelBuffer, and file URLs, at any size. Larger images simply cost more tokens, so it's worth downsampling when you don't need full resolution.

Private Cloud Compute: Apple's server models, same API

For tasks the small on-device model can't handle, you can now route to Apple's larger server models running on Private Cloud Compute (PCC) — through the same LanguageModelSession.

let model = PrivateCloudComputeLanguageModel()
let session = LanguageModelSession(model: model)

The selling points: a 32K context window, configurable reasoning levels, and crucially no account setup, no auth, no API keys — it inherits the same privacy guarantees as the rest of Apple Intelligence. As of this release it's available on watchOS 27 too.

The business news that made this real for most teams: at the Platforms State of the Union, Apple announced free PCC access for developers with fewer than two million first-time App Store downloads. That removes the infrastructure cost barrier that would otherwise make cloud inference a non-starter for indie and mid-size apps.

You can inspect exactly what a request consumed, including cached and reasoning tokens:

let response = try await session.respond(
    to: "Recommend a craft that doesn't require scissors.",
    contextOptions: ContextOptions(reasoningLevel: .light)
)

print(response.usage.input.totalTokenCount)
print(response.usage.input.cachedTokenCount)
print(response.usage.output.totalTokenCount)
print(response.usage.output.reasoningTokenCount)

The model abstraction layer — and partner models

This is the architectural centerpiece. A new LanguageModel protocol lets any model — local or server — back a LanguageModelSession. Apple's own models already conform, and the framework ships open-source CoreAILanguageModel and MLXLanguageModel conformers for running local models on the Neural Engine and GPU.

The downstream consequence: swapping models is a Swift Package Manager change, not a code change. Anthropic and Google publish Swift packages for their frontier models that conform to the same protocol. Your session logic, your prompts, your tool definitions — all unchanged.

// On-device
let session = LanguageModelSession(model: SystemLanguageModel())

// Apple server model
let session = LanguageModelSession(model: PrivateCloudComputeLanguageModel())

// Third-party frontier model — same session API
let session = LanguageModelSession(model: ClaudeModel(/* ... */))

The partner packages handle auth and billing securely (OAuth, credentials in Keychain) and expose per-token usage including cache and reasoning tokens. The model becomes a configuration choice — prototype on the free on-device model, escalate to Claude or Gemini for the hard queries, ship without rewriting your inference layer.

One constraint worth flagging: you still can't bring your own fine-tuned weights to the on-device runtime. If you need a domain-specific model, that's Core AI's territory, not Foundation Models.

System tools: Vision and Spotlight, built in

The framework now ships tools the model can call directly. Two are Vision-backed — BarcodeReaderTool and OCRTool — letting the model reason over visual information without you wiring up Vision yourself.

The more interesting one is a Spotlight-powered search tool that enables fully local Retrieval-Augmented Generation. Queries route through Spotlight first, then the results feed into the model's context. This replaces the custom vector-database infrastructure teams were standing up for on-device search — for a lot of use cases, Spotlight already had the index.

Dynamic Profiles: the agentic primitive

This is the big new building block for agentic apps. A DynamicProfile is a declarative description of a session's instructions and tools — and a single session can swap between profiles as the app's mode changes, while preserving conversation history.

The WWDC sample is a crafts app that flips between a "craft analysis" mode and a "brainstorm" mode. You describe each profile as a struct:

struct CraftProfile: LanguageModelSession.DynamicProfile {
    var body: some DynamicProfile {
        Profile {
            Instructions {
                """
                You are an expert crafting assistant. \
                Record craft project image analyses   \
                using the recordImageAnalysis tool.
                """
            }
            RecordImageAnalysisTool()
        }
    }
}

let session = LanguageModelSession(profile: CraftProfile())

And here's where it gets powerful — you can vary the model and reasoning level per profile branch. Quick analysis stays on the fast on-device model; the open-ended brainstorm escalates to Private Cloud Compute with deep reasoning — all within one profile, history intact:

struct CraftProfile: LanguageModelSession.DynamicProfile {
    let states: CraftProjectStates

    var body: some DynamicProfile {
        switch states.mode {
        case .craftAnalysis:
            Profile {
                Instructions { /* ... */ }
                RecordImageAnalysisTool()
                SwitchModeTool(states: states)
            }
        case .brainstorm:
            Profile {
                Instructions { /* ... */ }
                BrainstormRecordTool()
            }
            .model(states.privateCloudCompute)
            .reasoningLevel(.deep)
        }
    }
}

A profile resolves to exactly one active branch at a time. The model itself can drive the transition by calling a tool — here a SwitchModeTool that mutates app state, which the framework observes to rebuild the session:

struct SwitchModeTool: Tool {
    let description = "Switch to a different mode."
    let states: AppStates

    @Generable
    struct Arguments {
        let mode: Mode
    }

    func call(arguments: Arguments) async throws -> some PromptRepresentable {
        appStates.mode = arguments.mode
        return "Successfully switched to \(arguments.mode)."
    }
}

This is the cleanest expression yet of multi-tool, multi-mode agent behavior on Apple platforms — declarative, type-safe, and with model selection as a first-class part of the design.

Evaluations: measuring quality, not vibes

Anyone who's shipped an LLM feature knows the trap: you tweak a prompt, it feels better, you ship, and you have no idea if you actually improved anything or just got lucky on three test prompts. Apple's new Evaluations framework is a Swift framework for quantifying accuracy as you iterate, so you can understand the statistical impact of a change before shipping. This is the boring-but-essential piece that makes the rest production-viable.

Tooling: an fm CLI and a Python SDK

Two additions that broaden where these models live.

In macOS 27, an fm command-line tool brings on-device and PCC models to the terminal — fm chat for interactive use, or pipe text through it in shell scripts to summarize, extract, or generate.

And there's a Python SDK exposing the same on-device model as the Swift framework, aimed at data scientists and researchers:

import apple_fm_sdk as fm

model = fm.SystemLanguageModel()

is_available, reason = model.is_available()

if is_available:
    session = fm.LanguageModelSession(model=model)
    response = await session.respond(prompt="Hello!")
    print(response)

Open source — and it runs on Linux

The quiet bombshell: the Foundation Models framework is going open source this summer, and because it's Swift, it runs wherever Swift runs — including Linux servers. Alongside it, Apple is shipping a Foundation Models framework utilities package, updated between OS releases, with emerging building blocks: transcript management, a skill API, and a chat-completions interface for interop.

For a framework that started life as strictly on-device, Apple-platform-only, this is a remarkable reframing. The session API you write against on iPhone is, increasingly, a portable abstraction.

What this means for how you build

Put it together and the recommended pattern for an app shipping in the iOS 27 cycle is a deliberate hybrid:

Use the on-device model for fast, free, offline, privacy-sensitive work — now with vision and better reasoning.
Escalate to Private Cloud Compute when you need a bigger context window or deeper reasoning, free under two million downloads, with the same privacy posture.
Route to Claude or Gemini for the genuinely hard queries — same session code, swapped via SPM.
Compose it all with Dynamic Profiles, and validate it with the Evaluations framework before you ship.

The model is no longer a fixed dependency baked into your architecture — it's a configuration choice you can change without touching your logic. That's the real story of WWDC 2026: not any single model, but the abstraction layer that finally makes model choice a runtime decision.

A couple of caveats to keep on your radar: there's still no bring-your-own-fine-tune path for the on-device runtime (that's Core AI), and Siri AI features won't be available in the EU or China at iOS 27 launch — EU developers can't even test them during development — which matters if your user base spans those regions.

Where to start

Download the Xcode 27 beta, run fm chat in Terminal to feel out the on-device model, then pull Apple's sample app and get hands-on with Dynamic Profiles and the Evaluations framework. The deep-dive sessions on PCC, evaluations, the Xcode instrument, and dynamic profiles are where the real detail lives.

Bridging RTSP to WebRTC on iOS: Feeding GStreamer Frames into a WebRTC Pipeline

hariharan — Mon, 15 Jun 2026 19:14:00 +0000

Most WebRTC tutorials on iOS assume your video comes from the device camera. You call RTCCameraVideoCapturer, point it at the front lens, and the framework handles the rest. But a large class of real applications doesn't have a camera as the source at all — it has an IP camera, a dashcam, a drone feed, or some other device speaking RTSP, and the iOS app's job is to ingest that stream and republish it over WebRTC for low-latency delivery (often into a managed SFU like Amazon Kinesis Video Streams).

That problem turns out to be more interesting than it looks, because the two halves of the system have fundamentally different ideas about where pixels come from. RTSP gives you an encoded elementary stream pulled off the network. WebRTC's iOS SDK wants to own the capture path and hand you an encoder. Getting one to feed the other means stepping outside the comfortable parts of both frameworks and meeting in the middle — at the level of raw I420 frame buffers.

This article walks through a working architecture for exactly that bridge: RTSP ingestion and decoding via GStreamer, conversion to raw YUV, and injection into the WebRTC pipeline through a custom capturer. The signaling and ICE negotiation are handled against Kinesis Video Streams, but the core technique is provider-agnostic and applies to any libwebrtc-based stack.

Why you can't just point WebRTC at an RTSP URL

The WebRTC native SDK has no concept of a network video source. Its capture abstraction, RTCVideoCapturer, is designed around the assumption that frames originate locally and synchronously — a camera, the screen, a file. The RTCCameraVideoCapturer subclass that most apps use wires AVCaptureSession output straight into a video source. There is no built-in "open this RTSP URL" path, and there's a good reason for that: RTSP involves session setup, RTP depayloading, jitter buffering, and decoding, none of which belongs inside a real-time media engine whose entire design goal is to push locally captured frames out the door as fast as possible.

So the ingestion side has to be solved separately, and GStreamer is the pragmatic choice. It has mature RTSP support, hardware-accelerated decoders where available, and a clean way to extract decoded frames from the end of a pipeline via appsink. The architecture becomes:

RTSP camera ──▶ GStreamer (depay + decode + convert) ──▶ appsink
│
raw I420 frames
│
▼
custom RTCVideoCapturer ──▶ WebRTC ──▶ SFU/peer

GStreamer handles everything up to and including "I now have a decoded YUV frame in memory." WebRTC handles everything from "here is a frame, encode and transmit it." The bridge between them is the part you have to build, and it's where most of the subtlety lives.

The GStreamer pipeline

The ingestion pipeline is constructed with gst_parse_launch, which lets you describe the whole chain as a string:

rtspsrc location=rtsp://<host>/<path> !
rtph264depay ! avdec_h264 ! videoconvert ! videoscale !
video/x-raw, format=I420, width=640, height=480 !
appsink name=mysink emit-signals=true sync=false max-buffers=1 drop=true

Reading left to right: rtspsrc handles RTSP session negotiation and produces RTP packets. rtph264depay extracts the H.264 elementary stream from RTP. avdec_h264 decodes it to raw frames. videoconvert and videoscale normalize the pixel format and dimensions, and the explicit video/x-raw, format=I420 caps filter forces the output into planar YUV 4:2:0 — which, not coincidentally, is exactly what WebRTC's RTCI420Buffer expects. Pinning the resolution at this stage (here 640×480) means the downstream copy logic can assume fixed plane sizes rather than re-reading caps on every frame.

The appsink properties are where the real-time character of the system gets enforced, and they deserve individual attention because the defaults are wrong for this use case:

sync=false tells the sink not to throttle to the pipeline clock. For playback you want clock sync so video plays at the correct speed; for a live bridge you want frames out the instant they're decoded, because the WebRTC layer downstream is doing its own pacing.
max-buffers=1 caps the internal queue at a single frame.
drop=true says that when that one-frame buffer is full, throw away the oldest frame rather than blocking the pipeline.

Together, max-buffers=1 drop=true implement a "latest frame wins" policy. This is the single most important latency decision in the whole ingestion path. If you let frames queue up, any momentary stall downstream — a slow encode, a main-thread hiccup — turns into a growing backlog, and the stream drifts further and further behind real time with no way to recover. By dropping rather than queuing, you trade the occasional skipped frame for a hard ceiling on accumulated latency. For a live monitoring or dashcam scenario, that's exactly the right trade: nobody wants a perfectly smooth video that's eight seconds stale.

Threading and the GLib main loop

One non-obvious requirement: GStreamer's bus and signal machinery need a running GLib main loop, and that loop blocks. You cannot run it on the main thread of an iOS app. The pipeline is therefore set up and run inside a dispatched background block, with its own GMainContext pushed as the thread-default:

context = g_main_context_new();
g_main_context_push_thread_default(context);
// ... build pipeline, attach bus watch ...
main_loop = g_main_loop_new(context, FALSE);
g_main_loop_run(main_loop);   // blocks here until quit

Creating a private GMainContext rather than using the default one matters in a mixed app: it keeps this pipeline's event sources isolated, so multiple pipelines (or other GLib-using code) don't fight over a shared context. Teardown then runs in the reverse order — quit the loop, pop and unref the context, set the pipeline to GST_STATE_NULL, and unref it.

A note on dynamic pads

rtspsrc doesn't expose its source pads until it has connected to the server and discovered the stream's contents, so you can't statically link it at construction time. If you build the pipeline element-by-element rather than with gst_parse_launch, you have to connect to its pad-added signal and do the linking in the callback:

static void on_rtsp_pad_added(GstElement *src, GstPad *new_pad, gpointer user_data) {
    GstElement *depay = GST_ELEMENT(user_data);
    GstPad *sink_pad = gst_element_get_static_pad(depay, "sink");
    if (!gst_pad_is_linked(sink_pad)) {
        gst_pad_link(new_pad, sink_pad);
    }
    gst_object_unref(sink_pad);
}

gst_parse_launch hides this for you — it handles the deferred linking internally — which is one reason the string form is worth preferring for a fixed pipeline. The manual approach is only worth the extra code when you need runtime control over individual elements.

The bridge: pulling frames out of GStreamer

Every decoded frame triggers the appsink's new-sample signal, wired to a C callback. The callback's job is to get the frame data out of GStreamer's memory and onto iOS's main queue for handoff to WebRTC, and the lifetime management here is where it's easy to introduce a crash or a subtle corruption bug.

objc
static GstFlowReturn on_new_sample(GstAppSink *sink, gpointer user_data) {
    GStreamerBackend *self = (__bridge GStreamerBackend *)user_data;
    GstSample *sample = gst_app_sink_pull_sample(sink);
    if (!sample) return GST_FLOW_ERROR;

    GstBuffer *buffer = gst_sample_get_buffer(sample);
    GstMapInfo map;
    if (gst_buffer_map(buffer, &map, GST_MAP_READ)) {
        void *copiedData = malloc(map.size);
        if (copiedData) {
            memcpy(copiedData, map.data, map.size);
            size_t dataSize = map.size;
            NSData *frameData = [NSData dataWithBytes:copiedData length:dataSize];
            dispatch_async(dispatch_get_main_queue(), ^{
                [self.delegate processRawNSDataFrame:frameData];
                free(copiedData);
            });
        }
        gst_buffer_unmap(buffer, &map);
    }
    gst_sample_unref(sample);
    return GST_FLOW_OK;
}

The critical detail is the defensive copy. The GstBuffer you get from gst_app_sink_pull_sample is only valid until you unref the sample, and the mapped memory is only valid between gst_buffer_map and gst_buffer_unmap. But the handoff to WebRTC happens asynchronously on the main queue — by the time that block runs, the sample is long gone. You cannot hand the mapped pointer across a dispatch_async boundary. So the frame bytes are memcpy'd into a heap allocation that the dispatched block owns and frees, and the GStreamer buffer is unmapped and unreffed synchronously before the callback returns.

This copy is the one unavoidable allocation per frame in the path. It's tempting to try to eliminate it, but the asynchronous boundary makes it necessary unless you redesign around a frame pool — and at 640×480 I420 the copy is a fraction of a megabyte, cheap relative to the decode that produced it. (If you do find this allocation showing up in profiling under high frame rates, a fixed-size ring of preallocated buffers is the standard answer, but measure before reaching for it.)

Crossing onto the main queue is a deliberate choice rather than an accident: it gives a single, predictable thread on which frames enter the WebRTC layer, which sidesteps a class of data races around the capturer's delegate. It does couple frame delivery to main-thread responsiveness, which is the cost side of that decision — if your UI thread stalls, frame injection stalls with it. For an app whose main job is displaying this video, that coupling is usually acceptable, but it's worth knowing it's there.

The injection side: a synthetic capturer

On the WebRTC side, the trick is to create a video source and a bare RTCVideoCapturer — not the camera subclass — and then feed it frames manually. The capturer normally has a real implementation behind it that produces frames; here it's essentially a handle whose only purpose is to carry the source's delegate.

private func createVideoTrack() -> RTCVideoTrack {
    let videoSource = WebRTCClient.factory.videoSource()
    videoSource.adaptOutputFormat(toWidth: 1280, height: 720, fps: 30)
    videoCapturer = RTCVideoCapturer(delegate: videoSource)
    return WebRTCClient.factory.videoTrack(with: videoSource, trackId: "KvsVideoTrack")
}

The video source is the capturer's delegate. That's the hook. Anything delivered to videoCapturer.delegate as an RTCVideoFrame enters the WebRTC encode-and-transmit path exactly as if a camera had produced it. So frame injection becomes a matter of building an RTCVideoFrame from the raw bytes and calling the delegate:

func processRawNSDataFrame(_ data: NSData) {
    let width = 640, height = 480
    let i420Buffer = RTCMutableI420Buffer(width: Int32(width), height: Int32(height))

    let base = data.bytes.assumingMemoryBound(to: UInt8.self)
    let ySize = width * height
    let uSize = ySize / 4

    memcpy(i420Buffer.mutableDataY, base, ySize)
    memcpy(i420Buffer.mutableDataU, base + ySize, uSize)
    memcpy(i420Buffer.mutableDataV, base + ySize + uSize, uSize)

    let timestampNs = Int64(Date().timeIntervalSince1970 * Double(NSEC_PER_SEC))
    let rtcFrame = RTCVideoFrame(buffer: i420Buffer, rotation: ._0, timeStampNs: timestampNs)
    videoCapturer?.delegate?.capturer(videoCapturer!, didCapture: rtcFrame)
}

The I420 memory layout is the thing to get right. A 4:2:0 planar frame stores all the luma (Y) samples first — one per pixel — followed by the two chroma planes (U then V) at quarter resolution, since chroma is subsampled both horizontally and vertically. So for a W×H frame, Y is W*H bytes, and U and V are each W*H/4 bytes. The three memcpy calls split the flat buffer GStreamer produced back into the three plane pointers WebRTC's RTCMutableI420Buffer exposes. Because the GStreamer caps filter already guaranteed I420 at this exact resolution, the offsets are fixed and known; this is the payoff for pinning the format upstream.

A guard on the incoming length is worth adding before any of the memcpys — a frame that's shorter than W*H*3/2 means something upstream changed format or truncated, and copying plane data out of a short buffer reads past the end:

guard data.length >= width * height * 3 / 2 else {
    print("Frame too small — unexpected format or truncation")
    return
}

A word on timestamps

The frame timestamp here is derived from wall-clock time at the moment of injection (Date().timeIntervalSince1970). That's the simplest thing that works, and for a live bridge where every frame is "now" it's defensible. But it discards the timing information GStreamer actually had — the PTS on the original buffer — and uses arrival time as a proxy for capture time. If the ingestion path ever introduces variable delay, monotonic-but-uneven wall-clock stamps can confuse the encoder's rate control and the far end's jitter buffer. A more robust version propagates the GStreamer buffer PTS through the bridge and converts it into the WebRTC timebase. For a first working system, wall-clock is fine; it's a known place to tighten up later.

Signaling and connection management

The media bridge is the novel part, but a working system still needs signaling and ICE, and a couple of details in that layer are worth calling out because they bite people.

The connection is established against Kinesis Video Streams, which means resolving a signaling channel ARN (creating the channel if it doesn't exist), fetching the WSS and HTTPS endpoints, signing the WebSocket URL with SigV4 using Cognito credentials, gathering ICE servers, and finally opening the signaling socket. The offer/answer and ICE candidate exchange then follow the standard WebRTC handshake over that socket.

The detail that catches people is ICE candidate ordering. Candidates can arrive over the signaling channel before the SDP offer/answer exchange has completed — that is, before there's a peer connection ready to accept them. Adding a candidate to a peer connection that hasn't had its remote description set yet is an error. The fix is a pending-candidate queue: if no peer connection exists for a given client ID yet, candidates get held in a per-client set; once the SDP exchange completes and the connection is established, the queue is drained in order.

func checkAndAddIceCandidate(remoteCandidate: RTCIceCandidate, clientId: String) {
    if peerConnectionFoundMap.index(forKey: clientId) == nil {
        // SDP exchange not done yet — hold the candidate
        var pending = pendingIceCandidatesMap[clientId] ?? Set<RTCIceCandidate>()
        pending.insert(remoteCandidate)
        pendingIceCandidatesMap[clientId] = pending
    } else {
        // connection is up — add directly
        peerConnectionFoundMap[clientId]!.add(remoteCandidate)
    }
}

This is the same problem every WebRTC implementation eventually solves; it just shows up early here because the signaling and media paths run at different speeds. Keying the queue by client ID means the same logic handles a session with multiple remote peers without them stepping on each other.

The peer connection configuration is otherwise conventional, with a few choices worth noting: unifiedPlan semantics (the modern default and required for newer SDKs), gatherContinually so candidates keep flowing as network conditions change, maxBundle to multiplex all media over a single transport, and tcpCandidatePolicy = .enabled so the connection can fall back to TCP candidates when UDP is blocked — relevant for cameras or clients behind restrictive networks.

What this architecture buys you, and what it costs

The strength of this design is separation of concerns. GStreamer is genuinely good at network ingest and decode, with broad codec and transport support you'd spend months reimplementing. WebRTC is genuinely good at adaptive, low-latency, NAT-traversing real-time transport. By meeting them at the raw-frame boundary, each does what it's best at, and the bridge between them is small enough to reason about completely.

The costs are real and worth stating plainly. There's a full decode-then-re-encode cycle: GStreamer decodes H.264 to raw YUV, and WebRTC re-encodes that YUV (typically to VP8/VP9/H.264 again) for transport. That's CPU and battery you wouldn't spend if you could forward the encoded stream directly — but forwarding encoded RTSP straight into WebRTC isn't generally possible, because WebRTC controls its own encoder for congestion-responsive bitrate adaptation. The per-frame copies (one out of GStreamer, three into the I420 planes) add allocation pressure that matters at high resolutions or frame rates. And the main-queue handoff couples media flow to UI responsiveness.

For the live-monitoring and dashcam-style use cases this targets — where the alternative is no low-latency path at all — those costs are the price of admission, and they're manageable. Where they'd start to hurt is high-resolution, high-frame-rate, multi-stream scenarios on battery-constrained hardware; that's the point at which you'd want to profile the copy and re-encode overhead seriously and consider a frame pool, PTS propagation, and moving the injection off the main queue.

Latency: where it actually comes from

A note on performance claims, since this is the question everyone asks. Glass-to-glass latency in this system is the sum of several stages: RTSP/RTP transport and jitter buffering, H.264 decode, the format conversion and copy, the main-queue hop, WebRTC encode, network transport to the far end, and finally far-end decode and render. The max-buffers=1 drop=true policy bounds the accumulated latency on the ingest side, and sync=false removes playback-clock delay, but the absolute number depends heavily on the camera's encoder GOP structure, network conditions, and the far-end client.

I'm not going to quote a figure I haven't measured on a fixed setup, because a latency number without a stated measurement method is noise. If you're reporting this, measure end-to-end with a known method — a millisecond clock or timestamped pattern visible in-frame, captured at both ends — and report the conditions alongside the number.

Closing

The reusable insight here is the frame-level bridge pattern: when two media frameworks each insist on owning their end of the pipeline, you can often join them at the raw-buffer boundary by creating a bare RTCVideoCapturer and driving its delegate yourself. The same technique works for screen capture, generated/synthetic video, ML-processed frames, or any source WebRTC doesn't natively support — anywhere you can produce an RTCVideoFrame, you can inject into WebRTC. The RTSP case is just the most common reason to need it.

The parts that took the most care weren't the parts the tutorials cover. They were the buffer lifetime across the async boundary, the I420 plane arithmetic, the appsink queue policy, and the ICE candidate ordering — the unglamorous edges where two systems with different assumptions actually have to agree.