DEV Community: ZedIoT

What Edge AI Models Can ESP32-P4 Actually Run? From ESP-DL Support to Real-Time Vision Validation

ZedIoT — Thu, 12 Nov 2026 12:20:00 +0000

An ESP-DL model listing may say "ESP32-P4 supported," but that does not answer the question a camera product has to answer: will a result arrive while it is still useful?

Chip support, successful loading, sufficient memory, fast individual inference, and acceptable response time with the camera, display, and network all running are separate checks. Passing the first one does not certify the others.

Start with a specific job, such as detecting a person near an entrance or reading a fixed instrument display. Get a minimal model path working, then measure the complete path on the intended board, sensor, resolution, and concurrent workload. This article uses Espressif's public documentation to explain that process. It does not claim measured FPS, power consumption, temperature, recognition accuracy, or production reliability for any untested device.

Choose the job before choosing a model

The current ESP-DL Model Zoo lists object detection, face and pedestrian detection, face recognition, classification, pose estimation, OCR, and other tasks with chip support markers. Those entries do not have interchangeable inputs or outputs. A detector returns candidate locations; a classifier returns categories; an OCR pipeline has to turn image regions into usable text. Asking only how many models the chip "can run" skips the interface your application actually needs.

Consider an entrance device. Face detection answers whether a face appears in a frame. Identity recognition adds feature extraction and matching, plus enrollment, thresholds, and a response to false matches. If the product only needs to wake a display when someone approaches, adding identity recognition creates privacy and error-handling work without improving that task. Define the action to be taken on a result before selecting a model with more features than the application requires.

OCR has a similar split. Reading digits at a known location on a panel is different from reading arbitrary labels at varying angles and under glare. The first task can narrow the crop and expected character set. The second needs to deal with localization, perspective, blur, and potentially multiple languages. An OCR entry marked for P4 in the official list is a credible integration starting point. It is not evidence that a particular camera, lens, and mounting angle will read your labels accurately. That conclusion needs actual images and a record of omissions, substitutions, and rejected reads.

The list also includes non-vision tasks. Their presence does not provide performance evidence for a vision pipeline. For a camera product, lock down the image source, the desired output event, and the maximum acceptable delay first. Only then does a model shortlist become meaningful.

What "supported" can — and cannot — mean

The first level is repository-level chip support: a model or example is marked for ESP32-P4. Record the model version, dependency versions, target chip, and file source. This prevents a later library update from silently changing what was actually assessed. It does not certify an arbitrary board configuration or an arbitrary input stream.

The next level is runtime compatibility. ESP-DL's .espdl model and TensorFlow Lite Micro's .tflite model are different artifacts. The ESP-VISION P4 guide describes separate execution paths and task-specific APIs. Copying a model that worked elsewhere to an SD card does not make it loadable by ESP-DL. Check the conversion toolchain, target chip, operator support, tensor dimensions, types, and input/output conventions rather than relying on the filename extension alone.

The ESP-DL dl::Model::test(), profile_memory(), and profile_module() functions are C++ model APIs. ESP-VISION's espdl and tflite examples show a separate Python application path. Both address related validation questions, but their calls cannot simply be mixed into one program. Choose the runtime that the product firmware actually uses, and instrument that runtime's own interfaces.

The third level is application fitness. A successful test() compares a model's result against test values exported with the model. It does not include exposure changes, color conversion mistakes, motion blur, business thresholds, or display lag from a real camera. Calling that result "validated entrance recognition" would claim much more than the test covers.

Decide where the model lives before finalizing updates

ESP-DL documents three broad loading routes: embed an .espdl file in application rodata, store it in a dedicated Flash partition, or load it from an SD card. Embedding is straightforward but ties the model to an application reflash and can enlarge the application partition. A dedicated partition allows a different model-update flow but requires partition sizing and compatibility planning. SD storage makes file replacement convenient, while adding mounting, file integrity, and read-speed conditions. These are release and maintenance trade-offs, not an inherent ranking of accuracy or speed.

Route	Convenient when	What it costs you
Embedded in app rodata	Model never changes independently of firmware	Any model change means reflashing the app; app partition grows
Dedicated Flash partition	Model updated separately from firmware	Partition sizing, compatibility checks, version negotiation
SD card	Frequent model swaps, large artifacts	Mount, integrity verification, read-speed and card-quality worst cases

If a deployed device must update its model without updating the main application, design that boundary before the firmware image and partition map are frozen. Decide how the device rejects a corrupted model, how it returns to the previous compatible version, and how the firmware checks model/API compatibility. The public API establishes that several loading options exist. It does not establish that any of them has passed your product's power-loss or update testing.

Keeping parameters in Flash or copying them into faster memory is another explicit trade-off. The ESP-DL documentation describes param_copy=false as a way to reduce RAM copies, with a possible inference-speed penalty. A model that starts only after disabling the copy has not automatically met a response-time requirement. Compare both configurations with the same input, clock, and concurrent tasks before accepting the memory saving.

File size is not the application's memory requirement

A common sizing mistake is to subtract a model file's size from total PSRAM and call the remainder "free for the camera." The file does not account for intermediate tensors, parameter copies, input and output buffers, captured frames, transforms, display buffers, or other tasks. Some allocations may also need a particular memory type or contiguous region. Plenty of total PSRAM does not guarantee that the next allocation a driver needs will succeed. If the product also runs a GUI, framebuffer allocations compete directly with the model workspace.

ESP-DL's profile_memory() reports model structures, copied parameters, variables, and other overhead across internal RAM, PSRAM, and Flash. Use it as a model-level measurement, not as the entire application's peak. Log free and peak memory before and after camera initialization, model loading, display initialization, and network activity. A failure observed after display startup is much easier to explain when the memory change at each boundary is known. The combined profile() call can also show where model-layer time is spent.

A simple lower bound helps expose hidden buffers. A 640×480 RGB565 frame with no extra stride contains 614,400 pixel bytes; two complete buffers contain 1,228,800 bytes, about 1.17 MiB. That calculation excludes alignment, driver-private allocations, a resized model input, and display-side copies. It says nothing about which memory region the buffers occupy. Its purpose is to flag that frames can consume memory on the same scale as a compact model's workspace — not to predict your board's measured peak.

The model input also deserves an explicit assumption. A tightly packed 224×224×3 tensor at one byte per element needs at least 150,528 bytes. A different camera format may require a transform buffer before inference. Layout, alignment, preprocessing, and runtime allocation policy can all raise the observed number. Write the assumptions beside the estimate; "AI needs roughly 2 MB" is not a reproducible memory plan.

An inference benchmark is not a response-time benchmark

A camera result travels through capture, format conversion, resize or crop, inference, post-processing, business rules, and finally display or event delivery. profile_module() reports latency inside the model's layers. It does not include that entire path. Reporting only its total cannot explain why a bounding box still trails a moving part on a screen.

The input may need a color conversion and resize before reaching the model. A detection result may then need thresholding, box decoding, and non-maximum suppression. Camera and display ownership can add copies or queueing. None of these stages is automatically dominant, but none is safely assumed to be free. Put timestamps around each stage in the same clock domain and record end-to-end percentiles, not only model time.

"Real time" has to come from the job. A meter reader that samples once per second cares about correct readings, worst-case response, and retry behavior. A device that controls access may care far more about tail latency and false activation. A live video overlay also needs input rate, processed rate, queued frames, and age of the output image. Mean inference time alone can look healthy while the frame queue grows and the displayed decision becomes stale.

Separate model bring-up from a real camera workload

First, use a fixed input to check that the model loads, tensors match, and the model produces the expected result. ESP-DL's test() works when test inputs and outputs were included during model export. A shipping model may omit those values to save space, so the absence of built-in test values is not itself evidence of a broken model. This stage isolates conversion and runtime errors.

Second, connect the intended camera. Record sensor, lens, resolution, pixel format, exposure range, crop, and channel ordering. Preserve failed frames for analysis. If the model behaves on public example images but misses objects under actual glare, motion, or viewing angle, shaving a few milliseconds from inference will not solve the product problem. Accuracy and timing must be accepted independently.

Third, enable the display, network, logs, and continuous capture. On the same model and board settings, compare isolated and concurrent operation. Record memory peaks, end-to-end latency percentiles, dropped frames, temperature, and recovery from faults. The duration of a meaningful sustained test follows the product's duty cycle. A short demonstration is not a reliability test, and public documentation cannot supply your device's pass criterion.

Quantization changes both fit and failure modes

The ESP-DL quantization guide describes .espdl as a quantized deployment format with 8-bit, 16-bit, and mixed-precision options exported through ESP-PPQ. A smaller quantized artifact does not by itself prove unchanged application accuracy. Small objects, low-contrast digits, and motion-blurred frames may be exactly the inputs that reveal a changed error distribution.

Hold the task, preprocessing, and evaluation set constant while comparing quantization choices. Check that each model exports and loads, that board-level correctness checks pass where test data is available, and that the relevant categories or characters still meet the application error budget. Then compare memory and timing.

If a smaller variant loses recall for a safety-relevant target, an attractive average accuracy figure will not rescue the decision. If a higher-precision variant exhausts working memory, changing a recognition threshold is not a memory fix.

There may also be a .tflite route for the task. ESP-VISION treats TensorFlow Lite Micro and ESP-DL as distinct runtimes. The TFLite model needs the enabled operators and correct quantized input/output handling. Verify equivalent input and output semantics before comparing their file sizes, working memory, and elapsed times. Otherwise the experiment compares two different models or preprocessing paths rather than two runtime options.

Carry the sample into product firmware one boundary at a time

An example often places capture, model invocation, and drawing in one demonstration loop. Product firmware may also own touch input, communications, logging, updates, and power management. Before copying the loop, decide which task owns each camera buffer, where a frame becomes a model tensor, and which component consumes the result. Unclear ownership can appear to work on the first frame and then fail when a reused buffer is overwritten or an allocation remains live.

Convert the camera image according to the model's actual tensor contract. Check width, height, channel order, data type, normalization, crop policy, and the coordinate mapping back to the original image. ESP-VISION exposes preprocessing parameters and tensor metadata queries that can help with this check. If boxes are consistently shifted although an object is found, inspect crop and resize coordinates before deciding that the model itself is inaccurate.

Keep the result interface separate from the business event interface. A detection box is transient frame state; an alarm may travel over a network and be persisted. Sending an event for every frame containing the same object binds network traffic and user notifications to inference frequency. A sensible design is to pass a timestamped candidate with confidence and source-frame identity to business logic, then apply deduplication and retry policy there.

Preserve checkpoints as integration becomes more complex: readable file, model load, fixed-input output, real-camera input, display overlay, business event, network concurrency, and sustained operation. Add one variable at a time. When the next checkpoint fails, the previous one remains a useful baseline. Enabling a new camera, model, display, and wireless stack all at once may feel faster but makes a memory fault difficult to attribute.

Write an acceptance record that can explain a failure

The useful deliverable is not a sentence saying "P4 supports YOLO." It is a record tied to board and PSRAM/Flash configuration, firmware and model versions, storage location, camera conditions, pixel transformations, threshold settings, concurrent tasks, clock rate, and failed samples. Keep model-only latency separate from application response time; keep mean values separate from high-percentile delays. This makes a later regression attributable rather than anecdotal.

The gates below add one new class of uncertainty at a time. They are a validation sequence, not results of testing performed for this article.

If the model will not load, verify conversion target, file version, storage path, partition, and filesystem before replacing the board. If it fails only after extended operation, inspect memory peaks, fragmentation, ownership, and restart behavior. If boxes lag behind the scene, time capture, preprocessing, inference, post-processing, and display separately, and check whether frames queue up. Treating all three symptoms as "P4 is too slow" wastes debugging time because they point to different parts of the system.

An approval decision can therefore be conditional:

The model has a compatible conversion or official component route.
The target board retains enough usable memory with all required functions active.
Real inputs meet the error budget.
Tail latency and sustained operation satisfy the application's deadline.

Passing an earlier condition cannot stand in for a later one. Mark an unmeasured condition "pending," not with an invented FPS estimate.

If the requirement is an open-ended large language model, long context, several high-resolution video streams, or a fixed frame-rate promise before any target-board testing, the Model Zoo's P4 support marker is not a sufficient basis for a product commitment. Narrow the task, consider a different compute platform, or budget a board-level benchmark first. Compact and well-defined vision tasks are a reasonable area to investigate on P4. Suitability for a particular product still rests on the actual model, memory map, camera path, and measured results.

Sources

Espressif ESP-DL Model Zoo
ESP-DL model loading, correctness testing, and profiling
ESP-DL quantization and export
ESP-VISION: introducing a model on ESP32-P4

Can an 8 GB Edge Device Run a 2 GB Model? Budget Memory at Peak Concurrency

ZedIoT — Tue, 10 Nov 2026 12:20:00 +0000

Eight gigabytes of RAM minus a 2 GB model file is not 6 GB of usable inference memory. Calculate the KV cache, then test the longest request and peak concurrency.

That subtraction cannot approve a device. The file is a storage artifact; inference also keeps weight pages resident, allocates KV cache and compute buffers, and competes with the operating system and any retrieval, audio, or display service. One successful short chat says little about two overlapping long ones.

You can obtain two useful numbers before touching a board: the exact bytes of a fixed model artifact, and a theoretical KV calculation from its architecture. Runtime workspace, system use, allocation policy, and recovery after a peak remain measurements for the target device. The 2 GB file and 8 GB device here are a hypothetical purchasing question, not a tested board result.

Why "8 minus 2" mixes incompatible accounts

"This model needs 4 GB" could refer to storage, resident weights, a process peak, GPU memory, or the entire device. Ask which one. The useful question is who occupies physical memory under the same model, configuration, and workload — not whether two numbers on different datasheets can be subtracted.

The layers below are conceptual, not proportional to measured capacity. Unallocated headroom is not another use of RAM. If the runtime allocates KV for a second session, that demand consumes some of the same headroom; nothing here establishes a fixed multiplier or a device result.

Account	What can establish it	What it does not establish
Weights and model metadata	Exact artifact version, format, file bytes, loader output	The process or device peak during inference
KV cache	Layer count, KV heads, head dimensions, allocated tokens, K/V types	Actual allocator overhead or a runtime's reservation policy
Compute and graph buffers	Runtime memory breakdown or a target-device measurement	A universal multiplier of model parameters
Operating system and companion tasks	Available memory after the real image and services start	The full advertised RAM as exclusive model capacity
Concurrent requests and headroom	Overlapping request lengths, error records, the system peak	A guarantee that every cache is shared — or that every byte is duplicated

These are diagnostic accounts, not necessarily additive measurements. A final capacity figure needs a consistent runtime configuration and observation window. If the only known value is the GGUF or other model-file size, you have screened storage, not signed off RAM.

A four-bit weight calculation is only a floor

Suppose a hypothetical model has three billion parameters and every parameter occupies exactly four bits. Its bare arithmetic weight floor is 3,000,000,000 × 4 ÷ 8 = 1,500,000,000 bytes, about 1.40 GiB. At eight bits the floor is about 2.79 GiB; at sixteen bits it is about 5.59 GiB. The calculation deliberately distinguishes decimal GB from binary GiB.

Real artifacts are not packed uniformly at that theoretical floor. Quantization blocks may carry scales or zero-points; some tensors may stay at higher precision; headers, alignment, embeddings, and output layers also consume space. Use the bytes of an artifact with a fixed hash, then inspect what the chosen runtime actually loads. "Q4" and "INT4" are not interchangeable byte-count promises across formats and backends.

More importantly, weight quantization does not shrink the operating system, the compute graph, or the KV cache by the same factor. The llama.cpp server reference exposes model-loading mode, context size, device offload, and K/V cache types as separate controls. If two measurements differ in several of those settings, the file format alone cannot explain the difference.

Memory mapping does not make RAM free

A runtime may map a model file rather than copying it into one obvious heap allocation. Pages accessed during inference can still become resident and may be shared between processes. A process's RSS can include shared pages, so RSS is not automatically the device's exclusive model cost. Linux documents RSS and proportional-set-size accounting in smaps_rollup. PSS helps describe shared mappings, but neither RSS nor PSS alone answers whether a separate GPU has enough memory or whether a device allocator can satisfy a large request.

For a target-device test, keep both process and system observations: available memory, any container limit, swap or OOM events, and device-memory figures where applicable. Record when each figure was captured. A healthy idle reading and an inference peak are different facts.

Where the 512 MiB KV estimate comes from

During autoregressive generation, the model reuses previously calculated keys and values instead of rebuilding them for every new token. The cache exists across attention layers and grows or is reserved according to the runtime's cache strategy.

KV bytes ≈ 2 × attention layers
         × KV heads × dimensions per head
         × allocated tokens per session
         × bytes per K/V element
         × simultaneous independent sessions

The leading two represents keys and values. Use the number of key-value heads, not the number of query heads printed elsewhere in a configuration. Grouped-query and multi-query attention may use fewer KV heads than query heads. If keys and values use different data types, calculate their terms separately and add them. A sliding-window layer, a hybrid architecture, or shared state across layers needs a model-specific calculation instead of this uniform expression.

A calculation, not a board measurement

Assume 32 standard attention layers, eight KV heads, 128 dimensions per head, and two bytes per K or V element. If a runtime allocates 4,096 tokens for one session:

2 × 32 × 8 × 128 × 4,096 × 2 = 536,870,912 bytes ≈ 512 MiB

Two independently allocated sessions at that same limit would have a 1,024 MiB theoretical KV baseline. Doubling the per-session allocation to 8,192 tokens doubles that term if the other assumptions remain fixed.

The 512 MiB figure is not a measurement of a commercial model or board. "4,096 tokens" here is an assumed allocation, not a number of visible characters. System instructions, retrieved passages, user input, and reserved output all count toward a request's token budget. Some runtimes reserve the entire configured context; others grow or reuse cache space, allocate blocks, or carry batching overhead. A 100-token chat cannot establish the peak for a 4,096-token configuration, let alone two overlapping long chats.

Two sessions do not always mean "multiply by two." Under the same hypothetical architecture, one session allocated 4,096 tokens and another allocated 2,048 tokens would have a 512 MiB plus 256 MiB, or 768 MiB, theoretical KV baseline if their caches are independently allocated at those lengths. A runtime reserving two full 4,096-token slots would not obey that 768 MiB figure. Record each slot's effective allocation and the runtime's reported allocation; "two users online" is not a memory measurement. Sharing the model's weights does not imply that two different conversation histories share their KV.

Cache quantization and offloading move the trade-off

Lower-precision K/V may reduce the nominal cache-data term, but scales, residual storage, conversion buffers, and latency can change the real saving. Offloading cache from a GPU to the CPU changes where memory is consumed and adds transfer work; it does not create free device capacity. The llama.cpp server options let an operator choose K and V cache types separately.

When comparing two runs, record the weight artifact, K and V types, context allocation, session count, offload setting, and runtime version together. A result labeled only "4-bit" is ambiguous: it might describe weights while the cache stays at sixteen bits.

Turn 4,096 tokens into an actual request

Allocated tokens are not visible characters. Suppose the longest business input takes 3,000 tokens, the answer allowance is 800, and instructions and templates consume another 200. That is already 4,000. A 4,096-token context leaves little room for a longer retrieved passage or tokenizer variation; an extra 150 tokens can cause truncation before an out-of-memory failure. Raising the context to 8,192 may avoid truncation but can make some runtimes reserve more KV memory.

That is why a 100-token demo cannot approve an 8 GB product. Separate a short request, the longest allowed single request, and requests that genuinely overlap. If the product searches local manuals or drives a display, leave those services enabled during the concurrent run. Multiplying a single-request peak by a thread count is not a concurrency test: weights may be shared, KV allocation may be reserved or reused, and a runtime may silently queue work.

Likewise, "two users online" does not mean two full contexts are active. Define a concurrency workload that another engineer can reproduce — for example, two longest inputs overlapping during prompt processing. If a product intentionally serializes long requests to save memory, document the wait time and timeout behavior as part of acceptance. A memory reduction that breaks response-time requirements is not a free win.

Keep the unmeasured cells blank

Fix the product workload before treating any estimate as a requirement: longest actual input, output-token allowance, concurrent users, companion retrieval or audio services, and acceptable failures and latency. Then fix the model artifact hash, tokenizer, architecture configuration, runtime and backend versions, context limit, and K/V type. Without those inputs, a range of candidates is reasonable; "8 GB is enough" is not.

A working budget should show four stages side by side: static estimate, loaded idle device, longest single request, and maximum intended concurrency. The static stage can contain measured artifact bytes and a calculated KV baseline. Mark compute workspace and system use "to measure" rather than inventing a safety percentage. Observe the loaded stage with all normal services running.

Copy this blank worksheet for each candidate configuration. Enter values, units, and capture times, and attach the model-file hash, runtime version, and raw logs. Leave cells blank until measured; a static calculation must not masquerade as a device observation. Record both the device peak and MemAvailable in the final row so process RSS is not mistaken for the whole-system answer.

Observation (MiB; rows are not additive)	Static estimate	Loaded idle	Longest request	Intended concurrency
Model artifact	File bytes converted to MiB: ____	Same artifact hash: ____	Same	Same
Model-file mappings, if separately identifiable	Not inferred from file size	Mapping RSS/PSS: ____	Mapping RSS/PSS: ____	Mapping RSS/PSS: ____
KV cache	Architecture formula: ____	Allocated: ____	Allocated: ____	Allocated: ____
Compute, graph, batch buffers, if reported	To measure	____	____	____
Inference process total RSS/PSS	Not the sum of rows above	____ / ____	____ / ____	____ / ____
System and companion services; container/device limits	To measure	State and limit: ____	____	____
Device peak / simultaneous MemAvailable	Not inferred from file size	____ / ____	____ / ____	____ / ____

The process-total RSS/PSS can already contain mappings, KV, and buffers attributed to that process. The detailed rows explain a peak; they are not separate charges to add together. File bytes and RSS/PSS use different accounting boundaries too. Keep effective input/output tokens, allocated context, truly overlapping sessions, container limits, swap/OOM, and failed requests on the same record. Capture a timestamp for each number so MemAvailable and the reported peak refer to the same moment.

The measurement sequence is short enough to repeat. Boot the target image with its normal services and record idle MemAvailable. Load the artifact identified by a hash and preserve the runtime's effective weight, context, KV, and workspace settings. Run the longest single request, then genuinely overlapping requests, then observe recovery over several rounds.

Useful Linux fields include /proc/meminfo MemAvailable and the inference process's smaps_rollup RSS/PSS; record container current usage and limits if containerized, and device-memory counters when the runtime provides them. If the runtime omits a field, write "not reported," not a number borrowed from the model file.

Suppose all you have today is an 8 GB device specification, a 2 GB model artifact, and the hypothetical 512 MiB KV calculation above. The static column can hold artifact bytes and a formula. The next three columns must remain blank. That record says storage has a candidate and cache has a theoretical starting point. It does not say roughly 5.5 GB remains: resident pages, buffers, services, container limits, and actual allocation policy have not been measured on one accounting basis.

Once the first target-device log arrives, an unexplained difference between theoretical KV, runtime-reported allocation, and the system peak is worth investigating. It may reflect a longer reserved context, batching buffers, another process, or a different memory device. Do not "fix" the worksheet by tuning a multiplier until numbers match. A missing or surprising field tells you which log to collect next.

Two comparable rows fix the same device, operating-system image, runtime version, and request set while changing one candidate configuration. Comparing a 3B and a 7B model requires reading each model's own layer and KV-head counts; the first model's cache-per-token constant does not transfer. Comparing two weight formats requires checking that the tokenizer, context, and K/V precision did not change in the background. There are no paired device runs behind this article, so it supplies a test design, not a ranking of models for 8 GB hardware.

Locate the failure before buying more memory

"The model won't run" can mean that input was rejected before inference, that loading or generation ran out of memory, or that only two overlapping requests fail. Start from an observable failure stage rather than assuming every failure calls for more RAM.

Preserve the last successful action before a failure. If loading completed but the first request failed, inspect input processing and its allocation. If short requests work but a long one is truncated, inspect the tokenizer and output allowance; more RAM may not address a context-policy error. If only concurrency causes OOM, inspect overlapping time ranges, per-session KV, and container or device limits. If the first round succeeds and later rounds deteriorate, inspect cache release and queued work. Change one variable and rerun the original load; otherwise a new success does not show what fixed the original problem.

Interpret counters against the memory topology. On unified-memory hardware, process RSS, a GPU API counter, and system availability may describe some of the same physical pages, so adding them can double-count. With discrete VRAM, system RAM can remain available after device memory is exhausted. A container can also hit its own limit while the host still reports headroom. One OOM line without the request shape and effective runtime settings does not prove that the product needs "at least 16 GB."

Even if quantizing weights or shortening context makes a configuration fit, either change can damage answers, time to first token, or the manual passage the product must retain. After capacity screening, judge the result with a task-relevant answer set and a separate TTFT, TPOT, memory, and power validation pass.

When can you call 8 GB enough?

Only for a fixed artifact, effective runtime configuration, and target workload: the longest request and intended concurrency must complete, and the device peak, failures, swap/OOM record, and recovery must meet the product's requirements. File bytes and theoretical KV can screen candidates; they cannot approve capacity.

If the product must search long manuals while serving a display and multiple users, and reducing context or concurrency would break that requirement, compare a larger-memory device or another deployment approach instead of adding a safety multiplier.

There are no target-board runs, paired model scales, or paired quantization formats behind this article. The three-billion-parameter, 512 MiB KV, and 2 GB/8 GB figures are worked examples, not performance results for any specific device. A useful purchasing conclusion today may be: "The theoretical account is complete; the whole-device concurrency peak still needs measurement." That is more informative than an unqualified "it runs."

Sources

Hugging Face Transformers: how caching works
Hugging Face Transformers: cache strategies
llama.cpp server options
Linux kernel documentation: /proc and smaps_rollup

Choosing an Edge LLM Runtime: Start with Devices and Model Artifacts

ZedIoT — Thu, 05 Nov 2026 12:20:00 +0000

Choosing an Edge LLM Runtime: Start with Devices and Model Artifacts

Suppose a team wants local question answering on a Linux gateway, then similar capability on an RK3588 device and in a phone app. A tempting plan is to run four runtimes, rank their tokens/s, and standardize on the fastest. That comparison skips an earlier question: can the team's model become an artifact the target runtime can load? It also skips backend support and who will rebuild, package, and release the model when it changes.

Start with the delivery target:

For a general Linux CPU or supported GPU and a compatible GGUF model, llama.cpp is a sensible first route to check.
If a supported Rockchip NPU is a product requirement, verify the RKLLM conversion, Runtime, and driver combination.
A team already maintaining PyTorch models and native mobile apps can evaluate ExecuTorch's export and backend path.
A team prepared to compile and distribute target-specific model libraries can evaluate MLC LLM.

These are not four interchangeable model players, and none has a universal speed rank independent of hardware, model, and version.

This guide uses public project documentation to compare delivery contracts, not benchmark results. It makes no claim about decoding speed, time to first token, peak memory, power, or production reliability. Backend lists can change; confirm every route against the version and device you intend to ship.

Identify the artifact before comparing speed

The first difference is what leaves the build machine.

llama.cpp commonly loads a compatible GGUF model and exposes CLI and server entry points. Its official README describes CPU, Metal, CUDA, Vulkan, and other backend paths. That breadth is useful for testing a local model on different general-purpose machines, but a backend name in a README does not prove support for a particular board or that every operation stays on its accelerator.
RKLLM has a different chain: Rockchip's PC-side RKLLM-Toolkit converts and quantizes a model, while a board-side Runtime and RKNPU driver run the resulting artifact. Its official repository lists RK3588, RK3576, RK3562, and RV1126B families. The supported model and quantization options still need checking against the selected SDK and firmware. Having a GGUF file and a Linux-based Rockchip board does not mean that file will use the board's NPU through RKLLM.
ExecuTorch exports a PyTorch model into a .pte artifact for its edge Runtime. The LLM path also needs tokenizer and generation components, quantization decisions, and a suitable target backend. A .pte is not a universal file that will accelerate on every phone. After choosing XNNPACK, Vulkan, Core ML, or a specific NPU route, the team still has to confirm export, operator coverage, and behavior on actual devices. Its attraction to a PyTorch-and-app team is an explicit export and integration path, not a promised speedup.
MLC LLM puts weight conversion, configuration, and compilation of a target-specific model library into the delivery chain. Its documentation describes explicit compilation and JIT compilation on some Python paths. An iOS or Android product must then decide whether weights ship with the app, download on first use, or update separately, and how those weights stay compatible with the library and app version. Renaming a GGUF or .pte does not create a usable MLC artifact.

Delivery question	llama.cpp	RKLLM	ExecuTorch	MLC LLM
Typical artifact	Compatible GGUF	Converted RKLLM model	Target-exported `.pte` plus generation components	Converted weights, config, and target model library
Device boundary to check	Actual CPU/GPU backend and model support	Rockchip chip plus Toolkit, Runtime, and driver versions	App platform, delegate, and operator coverage	Target compiler backend, runtime library, and packaging path
Often-missed maintenance	GGUF and quantization versions, service config	Model conversion and board software lockstep	Export regression, app API, backend compatibility	Per-target compilation, library and weight distribution

Use the table to eliminate routes that cannot ship, not to score performance. A model-family name does not make four artifacts equivalent: quantization, tokenizer, context settings, and generation code can differ. If procurement asks for one unchanged model file across all devices, test whether that requirement itself is realistic. A product may need several validated artifacts from one model source, each recorded with its own target and build provenance.

Put hardware and update ownership on the same decision sheet

A Linux gateway prototype is not an NPU deployment

Consider a gateway that answers questions about equipment manuals on site. No NPU has been selected yet. The team knows only that the network may be unavailable, a local API is needed, and the model must work with the retrieval flow. Starting with a supported model and a llama.cpp service can separate functional validation from later resource measurements. Once the hardware is fixed, measure CPU or a supported GPU backend on that target.

If the eventual product requires a Rockchip NPU, the prototype cannot simply be "moved to the NPU." It needs an RKLLM conversion path, driver compatibility check, and answer-consistency tests. Whether that extra chain pays off is a question for the target board, not an inference from a desktop benchmark.

A phone app owns the model release cycle

Now consider a team embedding offline summarization in Android and iOS apps. It already has PyTorch training and evaluation, and wants to release model and app versions together. An ExecuTorch minimum path is worth testing: export one supported model, assign ownership for tokenizer and generation code, and run it on target phones with CPU or a chosen delegate. If the team instead plans to build target GPU model libraries and can maintain the TVM/MLC compilation and packaging infrastructure, MLC LLM is another candidate. Both routes create app-size, first-download, offline-start, weight-update, and rollback questions. A support badge for "Android/iOS" does not settle those choices.

Even "the same model" can cease to be a comparable test object after different quantization or operator handling. Export can fail; a backend can fall back to CPU; or a model architecture may have to change. Record each candidate as a chain:

model source/version → conversion/export version → quantization → target backend → Runtime/driver → tokenizer and generation config

A ticket that says only "use RKLLM" cannot identify which layer should be rolled back when an update fails.

For OTA products, attach that chain to the release record. Weights, Runtime, driver, and app interface may not be independently upgradeable. Replay a fixed question set on a small target-device cohort before widening rollout. The point is to make artifact dependencies visible before choosing the runtime.

A tokens/s screenshot is not an architecture decision

Before a comparison, define "same conditions": actual device, power and thermal conditions, model source and weights, acceptable quantization differences, tokenizer, prompt, context length, concurrency, and output limit. Then record cold start, time to first token, sustained generation, peak memory, temperature, power, and answer quality. If conversion changes operators or quantization, describe the difference instead of presenting the numbers as a pure runtime race.

Three cheaper gates come first:

Can conversion or export complete reliably, and can the artifact load on the locked Runtime version?
Is the intended backend really executing the work, without an unacceptable CPU fallback?
Can the device return to a known-good model, Runtime, and configuration after failure?

A fast demo does not make a production route sound if any of those three checks fails.

Include user experience in the measurement plan. A field manual assistant needs a timely answer with traceable sources. A mobile app needs a controlled first download and useful offline behavior. An industrial terminal may share memory with camera, UI, and networking tasks. Average decode speed cannot stand in for first response, memory peaks, or long-running thermal stability.

When to abandon the first candidate

If a GGUF runs on the Linux prototype but the product contract specifies a Rockchip NPU, do not promise that llama.cpp will use that NPU merely because it has several general backends. Verify the actual board's backend; if the official Rockchip chain is required, assess RKLLM separately. In the opposite situation — hardware still uncertain, models changing frequently, and no capacity to maintain conversion matrices — binding to RKLLM early can turn model exploration into a firmware compatibility project.

ExecuTorch is a concrete candidate for a mobile team with a PyTorch export and native-app workflow. If the target model cannot export reliably or important operators have no suitable delegate coverage, "mobile-friendly" is not a reason to keep investing. MLC LLM's multi-target compilation can be attractive, but without ownership of compiler, model-library, and weight-package versions it is not a low-maintenance shortcut. The team can validate the product task with a simpler route while hardware and update frequency are still fluid.

The output of this selection should be a short responsibility record: target device, shippable model artifact, intended backend, rollback version set, and owner of conversion and app integration. If a field remains unknown, run a minimal test on one target device and record the unsatisfied condition.

Sources and scope

llama.cpp official README
Rockchip RKLLM repository
ExecuTorch LLM and backend documentation
MLC LLM compilation and packaging

These sources support claims about documented interfaces and delivery paths. They do not establish target-device benchmark results. Before a production decision, recheck exact project versions, model licenses, target OS and backend constraints, and keep failed test cases and rollback outcomes with the release record.

How to Choose an RK3588, RK3576, RK3568, or RK3566 Core Board

ZedIoT — Tue, 03 Nov 2026 12:20:00 +0000

How to Choose an RK3588, RK3576, RK3568, or RK3566 Core Board

Suppose a device needs two cameras, two wired network connections, and a display. A supplier sends back a one-page RK3588 or RK3576 specification headed "6 TOPS, eight cores." Neither number answers the first design question: can the particular core board and carrier board run those interfaces together? A SoC may expose a controller that the module does not bring to its connector, or the carrier may route a high-speed lane to a different peripheral. For a single-display controller with one Ethernet port and modest local inference, starting with RK3588 by default may buy capability the product cannot use.

The useful first pass is elimination, not a four-chip performance ladder. List the interfaces and software constraints that cannot be traded away. Use Rockchip's public SoC documentation to narrow the field. Then request evidence for the exact module revision and carrier design, and test sustained operation in the product enclosure. This article supports that documentation-stage shortlist. It is not a measured comparison of frame rate, power, temperature, unit price, or availability.

Start with interfaces that must operate at the same time

Write a simultaneous-use requirement, such as "two camera inputs + two physical Ethernet ports + NVMe storage + one touch display," rather than four independent checkboxes. The camera path still has to coexist with display, video processing, storage, and network traffic at the required resolution and format. A controller on the SoC block diagram is not proof that the module exposes it; a signal on the module connector is not proof that the carrier provides a usable port. Extra NPU capacity cannot create a lane already assigned elsewhere.

The distinction is especially important for shared high-speed resources. Rockchip's RK3576 product page lists USB 3.0, PCIe 2.1, and SATA among combo interface options. Those labels should not be counted as independent ports in the same configuration. The RK3566 brief datasheet likewise describes shared high-speed resources. Ask the board supplier to map the exact target peripherals to pins and mux selections before freezing a design.

Software can eliminate a candidate even when the wiring works. "Boots Linux" is weaker than "the supplier's current BSP supports this camera sensor, display panel, network PHY, and NPU runtime together." Record the module and carrier revisions, kernel branch, device tree, image version, driver ownership, and patch path. A boot demo proves only the first step. Products with field updates and component substitutions need a maintainable software combination, not an isolated demonstration.

The decision path separates documentation screening from product testing. Failure at either hardware gate ends the current board proposal; passing them only qualifies a module-and-carrier combination for a reproducible software check and sample test.

Where each SoC belongs on an initial shortlist

The following rows identify plausible evaluation tasks and the first board-level question for each chip. They do not rank four tested products under one workload.

Candidate	Reason to evaluate it	First question for the module vendor
RK3588	Multiple high-resolution vision inputs, complex displays, or substantial high-speed expansion	Which camera, display, PCIe/storage, and Ethernet paths are exposed concurrently, and how is the enclosure cooled?
RK3576	A display-and-vision terminal with industrial connections	How are combo lanes assigned, are both network paths and CAN FD exposed on this board revision, and which BSP is supported?
RK3568	A gateway, HMI, or lighter vision device that still needs networking and expansion	What do the module connector and carrier actually provide for Ethernet, PCIe/SATA, image support, and temperature range?
RK3566	A single-screen or lighter edge-control device where limited expansion is acceptable	Are its exposed peripherals, shared lanes, memory, and runtime headroom sufficient for the target application?

RK3588 and RK3576 share a headline NPU figure, not a system profile

The RK3588 product page describes four Cortex-A76 and four A55 cores, multiple camera inputs, high-resolution media capabilities, and a 6 TOPS NPU. Rockchip describes RK3576 with four A72 and four A53 cores, a 6 TOPS NPU, multiple display options, CAN FD, and two RGMII interfaces. Equal advertised NPU figures do not establish equal operator coverage, CPU preprocessing time, memory behavior, video throughput, or sustained performance.

For a product dominated by several camera streams, complex display output, and fast storage, include an RK3588 board in the test set. For a display-focused industrial terminal that also needs network and field-bus connections, an RK3576 board deserves evaluation. These are reasons to request pinout and BSP evidence, not promises about which finished device will perform better.

Do not transfer a carrier's connector list to every board using the same SoC. Firefly, for example, describes Core-3588JD4 together with the MB-JD4-RK3588 carrier in its AIO development board; HDMI, serial, camera, and other accessible connectors belong to that defined combination. Another RK3588 module can have a different board-to-board connector, power design, image, or manufacturing revision.

RK3568 and RK3566 need separate network and expansion checks

The RK3568 brief datasheet and RK3566 brief datasheet both describe quad-core A55 devices with media and edge-compute resources. That common ground puts them in the same early screening exercise, but does not make their boards interchangeable. If a gateway needs two independent wired networks plus high-speed storage and expansion, ask an RK3568 supplier for the exact port and lane map. If one physical network and modest peripherals meet the requirement, an RK3566 module may remain in scope. Neither conclusion establishes price or power savings; those need like-for-like quotations and measurements.

Firefly's Core-3568J and Core-3566JD4 documentation shows how a specific module is paired with a carrier to make a usable development system. It is a useful prompt for supplier questions, not a universal specification for all RK3568 or RK3566 boards.

Three pieces of evidence turn a chip shortlist into a board shortlist

First, request an interface and concurrency map for the exact core-board revision. A pinout, board-to-board connector definition, carrier schematic, or supplier-signed interface matrix should show camera lanes, display outputs, PHYs, PCIe, SATA, USB, CAN, serial, and debug paths in one place. Mark mux alternatives explicitly. A design supported only by a SoC block diagram, without evidence that the board exposes the required combination, is not ready for board freeze.

Second, request a reproducible software set: module and carrier revisions, bootloader, kernel and device tree, Linux or Android image, NPU runtime, relevant camera and display drivers, and the supplier's update channel. Ask for a minimal image and test steps for the functions the product will actually ship. For "two cameras, local inference, and network upload," a camera preview and an unrelated NPU demo are not an end-to-end validation. Acquisition, preprocessing, inference, output, and error recovery need to run in one versioned setup.

Third, define the sustained-operation boundary. Published operating-temperature ranges and heatsink recommendations do not predict temperature inside the final enclosure. At the intended ambient temperature, record frequency, power, dropped frames, restart behavior, and component temperatures while the actual peripherals run together. A gateway also needs tests for network loss, power loss, storage wear, and recovery. Until those records exist, the honest conclusion is "eligible for evaluation," not "production-ready."

Apply the evidence to the product, not to a chip hierarchy

Multi-camera industrial vision device. Give RK3588 and RK3576 board suppliers the same sensor, resolution, encoding, display, network, and storage workload. Compare their pinout and lane allocation first. Then run the same model and runtime version on candidate hardware inside a representative thermal setup. Decide whether RK3568 or RK3566 should leave the list based on the target chain, not a presumed market tier. Both six-TOPS labels leave end-to-end frame rate and cooling unresolved.

Dual-network gateway with a local HMI. Establish whether the networks must be physically independent and whether cellular, CAN, isolated serial, or maintainable storage is also required. RK3568 and RK3576 are reasonable documentation-stage candidates; a single-network, single-screen product may also retain RK3566. Require a board-specific PHY and connector map, then test the actual protocols and recovery behavior. CPU or NPU peaks are poor substitutes for wiring, power, and update evidence.

Single-display controller. Ask whether multiple video streams, fast storage, or complex inference are genuine requirements. If not, begin with specific RK3566 or RK3568 modules and add RK3576 or RK3588 only when an identified display, vision, or industrial interface exceeds their board-level resources. Do not infer lower price or power from the chip name. Quotations become comparable only after memory, storage, carrier, cooling, quantity, and support terms are held constant.

When none of the four is the right starting point

A product doing simple sensing and control, without Linux-class applications or media processing, may be better served by an MCU or a simpler platform. If the hard constraint is cellular certification, electrical isolation, deterministic control, or long-term supply, select around that system boundary first. Familiarity with Rockchip is not a reason to force one of these application processors into the design. Conversely, if a specific peripheral driver or NPU operator is mandatory and the available BSP cannot establish support, a generic "Linux supported" statement should not keep a board on the shortlist.

The useful output of this phase is a record of rejected candidates and why they failed, plus no more than two module-and-carrier combinations for sample testing. Attach each combination's interface map, software versions, power and cooling proposal, and the workload that must be measured. The next discussion can then be about a reproducible product configuration rather than an abstract claim that one chip is "stronger."

Sources and scope

Rockchip RK3588 product specifications
Rockchip RK3576 product specifications
Rockchip RK3568 brief datasheet
Rockchip RK3566 brief datasheet
Firefly Core-3588JD4, Core-3576JD4, Core-3568J, and Core-3566JD4

The conclusions above are limited to public SoC and board documentation. They do not cover the target module's complete connector definition, current BSP release notes, orderable part number, or a like-for-like workload measurement. The product scenarios are selection exercises, not results from a deployment.

ESP32-P4 Has No Wi-Fi: Choosing a C6 Co-Processor, Transport, and Dual-Firmware Boundary

ZedIoT — Thu, 29 Oct 2026 12:20:00 +0000

ESP32-P4 Has No Wi-Fi: Choosing a C6 Co-Processor, Transport, and Dual-Firmware Boundary

ESP32-P4 does not integrate Wi-Fi or Bluetooth. When a product needs P4's display, camera, or multimedia capabilities and wireless connectivity, one officially supported architecture is to use an ESP32-C6 as a wireless co-processor. P4 keeps running the application, UI, and TCP/IP stack; C6 does the actual Wi-Fi and Bluetooth work; SDIO, SPI, or UART connects the two chips.

That is more than attaching a serial Wi-Fi module. Control calls and network data cross the chip boundary. P4 and C6 each run firmware, must establish reset and handshake state during startup, and need a version and update policy that covers both images. A poorly chosen transport can constrain the network path. A demo that connects once, without a dual-firmware lifecycle, merely postpones the hardest failures until system integration or field upgrades.

This article uses Espressif's public documentation to define the architecture and a validation order. It contains no measurements from a target board, so it makes no project-specific claim about throughput, latency, power, recovery time, or production readiness.

Start with the responsibility boundary between P4 and C6

Espressif's ESP32-P4 Function EV Board guide describes the onboard ESP32-C6-MINI-1 as the Wi-Fi and Bluetooth communication module. That reference design makes the basic split concrete: P4 provides the high-performance MCU, display, imaging, and multimedia side; C6 provides the radios. C6 is neither a standby P4 nor the place where the whole product application must move.

ESP-Hosted divides the inter-chip traffic into two logical paths:

Control path — carries calls such as scanning, joining an access point, and starting SoftAP from P4 to C6, where the real radio operation happens.
Data path — bridges network frames from C6 to a network interface on P4. The P4-side lwIP, DHCP, sockets, and application protocols continue to work through that interface.

This split gives debugging a useful order. If the product cannot get online, do not begin with MQTT, HTTP, or certificates:

First prove that the P4-C6 transport is established.
Then prove that control calls can associate with an access point.
Only after that diagnose DHCP and application traffic.

If the INIT and capability exchange never completes, cloud-side changes only add noise.

SDIO, SPI, or UART: choose for the product, not the demo

The ESP-Hosted-MCU guide supports SDIO, SPI Full-Duplex, SPI Half-Duplex, and UART between an MCU host and an Espressif co-processor. All can establish a hosted link between P4 and C6, but they differ in wiring, board requirements, and the amount of traffic they are meant to carry. Public documentation can support a first elimination round; it cannot supply a throughput promise for a board and workload that have not been tested.

Transport	What the official guide establishes	A sensible first validation use	Cost to account for
SDIO	Highest-throughput option in the guide; C6 supports it	Products with image, audio/video, larger file, or sustained network traffic, where the PCB can support the bus	More signals, mandatory pull-ups, and a proper PCB for four-bit operation; signal and power integrity require validation
SPI Full-Duplex	Called the easiest and most robust bring-up path; C6 supports it	A first ESP-Hosted design with moderate traffic, where the team wants an observable baseline quickly	Handshake, data-ready, and reset signals still required; target-load throughput must still be measured
SPI Half-Duplex	One-, dual-, and quad-line modes available, but not simply interchangeable with a generic SPI peripheral	Teams that understand the ESP-Hosted SPI-HD implementation and need a more specialized path	Host and co-processor implementations must match; high-speed quad operation belongs on a PCB, not jumper wires
UART	Fewest signals; marked as the lowest-throughput choice and not for more than 1 Mbit/s	Basic control, low-data prototypes, or a quick proof of the control path	A design can pass a light demo and still fail under real traffic; hosted Wi-Fi and Bluetooth share the link

Do not copy the development-board pinout blindly

The official guide publishes example mappings for the P4 Function EV Board and C6. A custom board still needs its own pin-mux, power, pull-up, reset, and debug review. SDIO in particular requires external pull-ups on CMD and data lines; the guide also recommends length matching, controlled routing, and power-integrity work for a production PCB. A reference board proves that an architecture exists, not that arbitrary routing will behave the same way.

Both firmware images must also select the same transport, bus width, and pins. A C6 image built for SPI will not negotiate automatically with a P4 image built for SDIO. Store the configuration, board revision, and firmware commit for both sides in one release record — far easier to reason about than reconstructing the combination from logs after a field failure.

A practical order for choosing the first bus

If the product clearly carries video, large images, or sustained downloads, evaluate SDIO early and budget its signal, reset, and power work in the schematic.
If this is the team's first ESP-Hosted integration and the immediate goal is to prove control, data, and dual-firmware workflows, SPI Full-Duplex provides a lower-friction baseline.
UART is suitable for concept proof or genuinely small traffic. It is not a good starting point when the requirement already exceeds the public UART boundary.

The final decision must come from the target workload. Record transport settings, Wi-Fi mode, packet sizes, direction, concurrent UI or camera work, retries, and error counters. A successful ping or DHCP lease proves that basic control and data paths work — it does not prove sustained throughput or acceptable interaction latency for the complete device.

Startup, reset, and two firmware images are the real system boundary

A dual-chip design turns "firmware version" into a tested combination. The P4 host and C6 co-processor use corresponding ESP-Hosted code and configuration. Updating only one side may leave message formats, capabilities, or transport settings incompatible before the business application even starts. A support record should identify the P4 firmware, C6 firmware, ESP-IDF, ESP-Hosted revision, and the combinations that have passed acceptance.

P4 must be able to return C6 to a known state

The official guide requires a host-controlled signal to the co-processor's EN/RST pin and uses it to synchronize startup state. A product cannot assume both chips always power up or reset together. Watchdogs, wake from sleep, brownouts, and partial resets can leave one side holding state from an earlier session. The startup state machine therefore needs an owner for reset, an INIT timeout, a limited retry policy, and a clear degraded state when the wireless co-processor is unavailable.

During bring-up, stop at the first missing checkpoint. Confirm matching transport and GPIO settings, then the INIT and capability exchange, then AP association and an IP address, and only then the real application. If base transport is absent, inspect wiring, pull-ups, power, clocking, and configuration before blaming the access point or backend.

Updating C6 must be part of the product release, not a bench procedure

ESP-Hosted-MCU includes an example in which the host updates the co-processor through the existing transport. The official material also points to a separate UART and serial-flasher path for some designs. Those examples establish that update mechanisms exist — a product still has to define image origin, integrity checks, failure recovery, and version compatibility, and verify that an interrupted C6 update leaves a recoverable device.

A practical release can treat P4 and C6 firmware as two artifacts in one bundle. Record both hashes and the compatible pair. Check preconditions, update the selected side, wait for the transport and network checks to recover, and commit the pair only when the critical application path passes. If the product requires A/B images, rollback, or secure boot, verify those properties on each chip separately — P4-side OTA does not automatically provide equivalent protection for C6.

Where P4 + C6 is a good fit — and where it is unnecessary

P4 + C6 is a strong candidate when the product genuinely needs P4 peripherals or compute: a local vision terminal with MIPI camera and display, a complex LVGL HMI, a multimedia control panel, or a device that benefits from separating UI and imaging work from the radio stack. In those cases, the extra co-processor, bus, and firmware lifecycle purchase a useful combination of capabilities.

If the product is primarily sensing, relay control, or ordinary connectivity and does not need P4's camera, display, or multimedia interfaces, start by evaluating a wireless ESP32-S3, C6, or another single-chip option. One chip removes an inter-chip bus, a co-processor image, and a recovery path. "Newer" or "faster" is not a sufficient reason to select P4 — the design should name a local task that a simpler device cannot meet.

The architecture is also a poor fit when the PCB cannot support a reliable SDIO or SPI implementation for the required traffic, or when the team cannot maintain two firmware images. Forcing excessive traffic through UART or relying on manual C6 flashing is a reason to revisit the host or network architecture rather than a workaround to normalize.

Bring these inputs to the schematic review

Before choosing a bus, gather:

the UI, camera, audio, or local-inference tasks that run concurrently on P4;
the peak and sustained Wi-Fi traffic, plus whether Bluetooth runs at the same time;
GPIOs, reset, debug ports, and PCB resources available for the P4-C6 link;
startup-time, reconnect, offline, and field-update requirements;
the release relationship between P4 and C6 images, including signing, rollback, and factory traceability requirements.

That input is enough to choose a useful first SDIO, SPI, or UART validation and to expose whether dual-firmware ownership exceeds the team's support capacity. When the requirement is still only "add Wi-Fi to P4," the missing design work is usually not an API call — it is bus allocation, reset control, and lifecycle ownership.

References and validation boundary

ESP32-P4 Series Datasheet
ESP32-P4 Function EV Board User Guide
ESP-Hosted-MCU: Getting Started for MCU Host
ESP-Hosted-MCU: Troubleshooting
ESP-Hosted-MCU: Host Performs Slave OTA

This article validates only public contracts and examples. It does not measure target-board throughput, latency, power, reconnect behavior, or interrupted updates, and it does not validate a particular antenna, RF, EMC, or factory-flashing design. "Good fit" here means eligible for engineering validation, not production-proven.

Building a Multi-I2C Sensor Gateway with ESP32 and ESPHome

ZedIoT — Tue, 27 Oct 2026 12:20:00 +0000

Building a Multi-I2C Sensor Gateway with ESP32 and ESPHome

Connecting an SCD41, a BME680, a light sensor, and an electrochemical gas sensor to the same ESP32 can look like a simple ESPHome exercise. In real deployments, the hard part is usually not whether the first readings appear. It's whether the I2C bus, sensor warm-up behavior, power noise, deep sleep schedule, and Home Assistant entities are designed as one system.

The core conclusion is simple: a multi-I2C environmental gateway is not an ESP32 with more sensors attached. It's a constrained data chain from power and bus design to sampling, diagnostics, and publication. If every sensor is treated as an independent YAML block, the project is likely to fail on address conflicts, unstable first readings, drift after sleep, or unclear failure diagnosis.

Scope note. A multi-I2C environmental gateway here means an ESP32 running ESPHome that reads several environmental sensors and publishes temperature, humidity, CO2, VOC, light, or gas trend data to Home Assistant or another platform. It's useful for trend monitoring, automation triggers, and device-side diagnostics. It should not be treated as a lab-grade calibrated instrument or a safety interlock.

1. Why a multi-sensor gateway is more than a component list

ESPHome provides an I2C bus component and common environmental sensor integrations such as SCD4x and BME680. That makes prototyping fast, but a working example for one component is not the same thing as a production-ready gateway design.

A real multi-sensor node has to handle five constraints at once:

Addressing and topology: fixed sensor addresses can collide, and cable length affects bus reliability.
Warm-up and measurement cadence: CO2, VOC, and electrochemical sensors may not be trustworthy immediately after power-up.
Power and noise: sensor heaters, Wi-Fi transmission, long wires, and weak regulators can affect both analog front ends and I2C stability.
Deep sleep: saving ESP32 power changes the meaning of continuous measurement, baselines, and first readings.
Platform semantics: Home Assistant needs reliable entities and diagnostic signals, not every raw register.

When one ESP32 handles multiple I2C sensors, Wi-Fi publication, and low-power wake cycles, reliability depends more on sampling cadence and fault isolation than on the number of sensors the board can physically connect.

2. Recommended layering: separate bus, sampling, and publication

The important point is the boundary.

Hardware should solve power and bus risks.
Firmware should control sampling, filtering, and failure marking.
The platform should receive entities that already have operational meaning.

If these responsibilities are mixed together, every strange reading becomes a long argument about whether the sensor, the bus, or the automation rule failed.

3. I2C bus design: resolve conflicts before tuning sampling

ESPHome's I2C scan option is useful during development, but it should not be the main operational diagnostic. A deployable gateway should answer these questions before the enclosure is closed:

Do any sensors have fixed addresses that collide with other modules?
Will cable length, connectors, or enclosure placement weaken the I2C signal edge?
Are pull-up resistors duplicated across breakout boards?
Is an I2C multiplexer such as TCA9548A needed for same-address devices?

If every sensor is on the same small PCB and addresses do not collide, a single bus is usually enough. If sensors are distributed inside an enclosure, use long wires, or include same-address modules, bus segmentation or a multiplexer is more reliable than hoping a lower I2C clock will solve everything.

Scenario	Better approach	Avoid
Short traces, no address conflict	One I2C bus with fixed SDA/SCL and known pull-ups	Depending on boot-time scans as topology control
Same-address sensors	TCA9548A or separate buses	Parallel wiring and hoping software can distinguish them
Long or removable sensor cables	Reduce bus risk and add diagnostics	Treating I2C as a long-distance fieldbus
Low-power wake node	Fixed warm-up and measurement window	Publishing the first value immediately after power-up

I2C is a good board-level or short-distance module bus. It is not a replacement for an industrial fieldbus. If the sensors are remote, noisy, or frequently reconnected, evaluate RS485, CAN, Ethernet, or dedicated acquisition modules instead of adding more I2C devices.

4. Warm-up and deep sleep: power saving changes data meaning

SCD41-style CO2 sensors, BME680-style environmental sensors with gas estimation, and electrochemical gas sensors may need stable measurement windows or warm-up time. ESP32 deep sleep can reduce average power, but it also introduces three costs:

The first values after power-up may not be suitable for automation.
Continuous measurement or baseline algorithms can be disrupted.
Electrochemical sensors may need bias current and stabilization time that is much longer than the ESP32 wake window.

A practical design should separate sensor classes:

Temperature, humidity, and light: often acceptable with shorter wake windows.
CO2 trend: needs enough measurement time before publication.
VOC or gas trend: benefits from continuity and baseline stability.
Safety-related gas detection: should not rely only on a low-cost ESPHome node.

If the sensor needs warm-up or a continuous baseline, deep sleep must be evaluated against data trustworthiness. Saving power is not worth publishing unreliable readings.

5. ESPHome modeling: separate diagnostics from business entities

This is a structure example, not a complete production configuration:

i2c:
  id: bus_a
  sda: GPIO21
  scl: GPIO22
  scan: true

sensor:
  - platform: scd4x
    co2:
      name: "Room CO2"
    temperature:
      name: "Room CO2 Sensor Temperature"
    humidity:
      name: "Room CO2 Sensor Humidity"
    update_interval: 60s

  - platform: bme680
    temperature:
      name: "Cabinet Temperature"
    humidity:
      name: "Cabinet Humidity"
    pressure:
      name: "Cabinet Pressure"
    gas_resistance:
      name: "Cabinet Gas Resistance"
    update_interval: 60s

Before shipping, add:

Wi-Fi signal, uptime, reset reason, last valid reading time, and sensor timeout diagnostics.
Rules that mark first, impossible, or stale values instead of triggering automation immediately.
Home Assistant recorder retention decisions to avoid storing high-frequency noise.
Installation position, power design, and firmware configuration version for each sensor group.

6. When ESP32 and ESPHome are the wrong fit

Do not force this architecture into these situations:

Compliance monitoring, life safety, or hard interlock requirements.
Long sensor distances, strong interference, or frequently disconnected modules where I2C is the wrong link layer.
Battery life targets measured in months while the sensor itself needs long warm-up or continuous baselines.
Large industrial deployments that need calibration traceability and field maintenance workflows.

The realistic boundary is clear: ESP32 + ESPHome is a good fit for home, lab, small server room, cabinet, and device-adjacent environmental trend gateways. If the goal becomes compliant monitoring, industrial-scale acquisition, or safety action, move to industrial acquisition hardware, distributed buses, or dedicated sensor gateways.

7. References

ESPHome I2C Bus
ESPHome SCD4X CO2 Sensor
ESPHome BME680 Sensor
ESPHome TCA9548A I2C Multiplexer
ESPHome Deep Sleep Component

RK3566 Can Run YOLOv8 INT8 — But Only Within These Limits

ZedIoT — Thu, 22 Oct 2026 12:20:00 +0000

RK3566 Can Run YOLOv8 INT8 — But Only Within These Limits

Running YOLOv8 INT8 on an RK3566 is not just a model conversion task. It is a system-level alignment problem. This article walks through how quantization, operator compatibility, and detection head design define whether real-time inference is actually possible on the RK3566 — and under what strict conditions.

Why YOLOv8 INT8 Can't Just Run on RK3566

The real issue with combining RK3566 and YOLOv8

In many edge AI projects, the RK3566 is treated as "a cost-effective platform that can also handle some AI tasks."

Its positioning isn't aggressive: controllable power consumption, full peripheral support, and limited but non-zero compute power. That means one thing — it was not designed for complex models with compute headroom.

YOLOv8 Detection sits in a delicate spot. It is no longer a "lightweight, run-anywhere" model, but it is still far from server-level detection models. In theory, it lands in the "just about doable" category for the RK3566. In practice, this assumption often fails during deployment.

Many projects run smoothly at the model stage: ONNX exports fine, PC-side inference is normal, and the structure doesn't look too complex. But once converted to RKNN and deployed on-device, performance drops — unstable FPS, high CPU usage, and rapid system resource exhaustion.

This isn't one parameter being off. It's deeper: the RK3566 is not a platform that brute-forces inference. Usability depends on having a clean execution path.

Why floating-point inference has little value on RK3566

Running YOLOv8 Detection in FP16 or FP32 on the RK3566 usually leads to a predictable result: the model runs, but runs poorly.

This isn't an "implementation issue" of RKNN or the NPU — it's a design-logic issue of the platform. On the RK3566, the NPU is not a fully independent compute unit. If there are unsupported operators in the model, execution falls back to the CPU, and detection models often include such unsupported operations.

With floating point, the problems worsen:

Low NPU coverage for FP ops
Constant data movement between CPU and NPU
Fragmented inference with high scheduling overhead

Result: single-digit FPS with near-maxed system load. In this state, optimizing FP16 further is meaningless. It isn't under-tuned; it's the wrong execution path for this hardware.

INT8 isn't a "bonus" — it's the entry point

Switching to INT8 reveals the RK3566's true nature. INT8 isn't just about precision — it unlocks the most stable, fully supported execution path on the chip.

In INT8 mode:

Operator mapping success rate rises
Ops can stay in the NPU for longer
The CPU acts more like a scheduler than a compute unit

Now YOLOv8 Detection starts "really running on the NPU," not just "partially using the NPU."

But INT8 isn't zero-cost. Detection models are sensitive to quantization, especially in the Head. Poor quantization leads to missed detections or box jitter. So the real question isn't "should we use INT8" — it's: how far can YOLOv8 Detection go on the RK3566 with INT8, and what are the boundaries?

YOLOv8 Detection's structure determines RK3566 compatibility

YOLOv8 Detection isn't structurally complex, but its complexity concentrates in subtle areas.

The Backbone is usually fine. As long as channel counts and input sizes aren't extreme, the RK3566's NPU handles it stably. The real issues appear in the later stages:

Irregular scale changes during feature fusion
Unpredictable Concat and Upsample combinations
Unnecessary tensor ops in the Detection Head

These are legal in ONNX but can prevent the RKNN compiler from statically fixing the compute graph. Most failures aren't due to model size — they're structural elements the compiler can't resolve. If the graph can't be fully static, NPU advantages disappear quickly.

Structure first — a prerequisite for INT8 success

On the RK3566, if structure doesn't serve the execution path, quantization only helps partially. Repeatedly validated practices include:

Fixed input size is more important than flexibility
Dynamic shapes offer less benefit than cost here
Simpler Detection Heads yield more stable INT8 results

These aren't flashy choices, but they aim at one thing: keep the entire inference inside the NPU without interruption. If the path breaks, even aggressive quantization can't fix overall performance.

How YOLOv8 INT8 Actually Runs on RK3566

What happens between model and device

Placing a YOLOv8 Detection model onto the RK3566 isn't determined by the export — it's determined by the in-between steps that are often oversimplified. From PyTorch to on-device execution, three major changes occur:

The graph gets compressed into a statically analyzable form.
Data precision maps from float to fixed point.
The execution path splits between NPU and CPU.

Any "gray area" here nearly guarantees performance issues. INT8 adds clarity to this path, and the RK3566's INT8 support goes beyond compute — it affects compilation, scheduling, and caching.

INT8 quantization isn't a one-click step

Many first-time RKNN INT8 users think it's just a switch: enable INT8, feed a few images, done.

It's more like a filtering process. Calibration data doesn't "train" the model — it constrains value ranges. Detection models are highly sensitive to feature distribution shifts, especially with large object-size variation.

If calibration data poorly matches real scenes, expect:

Small-object confidence drops
Box jitter across frames
Higher false positives in complex backgrounds

These problems usually occur in the Detection Head, not the Backbone.

The Detection Head is the make-or-break point for INT8

On the RK3566, bottlenecks rarely lie in the Backbone. The real gap appears downstream. The Detection Head has:

Frequent resolution shifts
Wide numerical ranges
Extreme precision demands

That makes it the most prone to quantization distortion. Even if Backbone and Neck quantize well, aggressive Head quantization can degrade detection. That's why many RK3566 failures stem not from "large models," but from Head structures misaligned with quantization.

Execution continuity beats theoretical compute

Once on-device, another critical issue surfaces: can operators stay on the NPU continuously?

Ideally, input goes into the NPU, multiple layers execute without switching, and output returns to the CPU. But if an op can't run on the NPU, control switches to the CPU. On the RK3566, with limited compute and bandwidth, this cost is high.

INT8 greatly increases the odds of uninterrupted NPU execution. That's why INT8 often yields better-than-linear performance gains on the same model.

Quantization failures don't mean the model is bad

Often, models that work fine on PC fail post-quantization on the RK3566. This doesn't mean model choice was wrong. More likely:

Inadequate calibration data coverage
Input size mismatch
Detection Head too complex

In such cases, simplifying structure — reducing branches or compressing channels — is more effective than tweaking quantization parameters.

Real-World Limits of YOLOv8 INT8 on RK3566

Test conditions and constraints

To avoid misleading results, all tests follow the same setup:

Hardware: RK3566 (NPU enabled)
Model: YOLOv8 Detection (no pruning)
Input size: 640×640, fixed
Inference mode: single-frame, Batch=1
Post-processing: on CPU, not on NPU

The goal is to evaluate a "reusable in production" YOLOv8 Detection on the RK3566, ignoring extreme tuning and special trims.

FPS: INT8 vs FP16 isn't linearly different

The raw numbers tell the story:

Model Precision	Inference FPS (640×640)	Stability
FP16	3 – 5 FPS	Unstable, high CPU load
INT8	12 – 18 FPS	Stable, sustained

This isn't just "INT8 is faster" — it's a different execution mode. FP16 triggers frequent CPU fallback. INT8 allows sustained NPU-only execution. Only in INT8 mode does YOLOv8 Detection become close to real-time on the RK3566.

Accuracy loss: localized, not global

FPS boost is just the start. Detection accuracy defines usability. INT8 quantization doesn't degrade uniformly — it shows structural patterns:

Scene Type	Accuracy Change
Medium / Large objects	Mostly stable
Simple background	Almost unaffected
Small / dense objects	Much more sensitive
Complex textures	More false positives

INT8 doesn't "weaken everything" — it amplifies pre-existing weak points. If the Detection Head and calibration data match your real use case, losses are usually acceptable.

There is no "sweet spot" between FPS and accuracy

A common RK3566 myth: find a "perfect FPS with minimal accuracy drop."

Reality: you either choose INT8 and accept structured accuracy changes, or use FP16 and sacrifice real-time capability. It's not a quantization flaw — it's a hardware limit. As model complexity nears the RK3566's ceiling, you can't have both high FPS and full accuracy.

Turning test data into engineering judgments

Compressing the findings:

The usable FPS lower bound is ~10 FPS on the RK3566.
Below that, load spikes and stability plummets.
INT8 is a must — but not sufficient on its own.
Model structure and Head design define quantized accuracy.

When it's worth using, and when to switch plans

Suitable

Single-class or few-class detection
Medium and larger object sizes
FPS target of 10–15
Prioritize response speed over peak accuracy

Not suitable

Many small objects
High precision required
Complex post-processing
Expecting PC-level accuracy on the RK3566

In the unsuitable cases, pushing the RK3566 further yields little. You have to change model size, hardware, or task design.

YOLOv8 Detection feasibility on RK3566 (INT8)

Dimension	Acceptable Range	Typical Issue When Exceeded	Takeaway
Precision type	INT8	<5 FPS in FP16/FP32	INT8 is essential
Input size	≤640×640	Larger → nonlinear FPS drop	Fixed input preferred
Real FPS	12–18 FPS	<10 FPS → system overload	10 FPS = lower bound
NPU utilization	High (continuous)	Frequent CPU fallback	Path continuity > GFLOPs
Backbone	Light to medium	Rarely a problem	Acceptable
Detection Head	Simpler = better	Box jitter / missed detections	Decides success or failure
Small-object density	Low to medium	High → misdetects increase	Not an ideal use case
Calibration data	Scene-aligned	Misaligned → accuracy loss	Critical for INT8
Long runtime	Stable in INT8	FP16 fluctuates	INT8 is sustainable

Final judgment

If only one takeaway matters, it's this: the RK3566 can run YOLOv8 Detection — if you accept INT8 and understand its limits.

It's not a failure platform, nor "AI-ready by default." When model, structure, and expectations align, the RK3566 delivers stable, predictable results. Push beyond its limits, and both performance and accuracy collapse.

Quick decision guide

Your Need	RK3566 + YOLOv8 INT8 Recommended?
Few-class detection	Yes
Medium object sizes	Yes
Real-time (≥10 FPS)	Yes
Many small objects	No
High-precision localization	No
Complex post-processing	No

Key takeaways

INT8 is the starting point for YOLOv8 on the RK3566.
The FPS gain comes from an execution-path shift, not a compute boost.
Accuracy loss centers on the Detection Head and select cases.
Once platform boundaries are clear, decisions become simpler.

FAQ

Q: What is the optimal inference mode for YOLOv8 on the RK3566?
A: INT8 quantization is the only viable mode. It ensures maximum NPU utilization, minimizes CPU fallback, and enables 12–18 FPS, compared to 3–5 FPS in FP16.

Q: Why is the RK3566 not suited for floating-point inference?
A: Floating-point ops are only partially supported by the RK3566 NPU. Unsupported ops get routed to the CPU, leading to fragmented execution and low performance.

Q: Where does INT8 quantization most affect YOLOv8 accuracy?
A: In the Detection Head, due to frequent resolution changes and high precision needs. It's the most fragile area post-quantization.

Q: How can I make sure my YOLOv8 model survives INT8 quantization?
A: Use calibration data that mirrors deployment scenarios, fix input resolution, and simplify the Detection Head architecture.

Q: Is there a performance-accuracy sweet spot for YOLOv8 on the RK3566?
A: No. You either accept INT8's structured accuracy loss or fall back to FP16 with unacceptably low FPS. It's a binary choice dictated by platform limits.

Why TinyML on ESP32-S3 Bottlenecks on Memory, Quantization, and Real-Time Inference

ZedIoT — Tue, 20 Oct 2026 12:20:00 +0000

Why TinyML on ESP32-S3 Bottlenecks on Memory, Quantization, and Real-Time Inference

When people ask whether ESP32-S3 can run TinyML, the useful answer is not a simple yes or no.

ESP32-S3 has dual-core Xtensa LX7 CPUs up to 240 MHz, 512 KB of on-chip SRAM, SIMD instructions, and optional PSRAM. It is clearly more capable for lightweight edge AI than many basic microcontrollers. But a TinyML product succeeds or fails on something else: whether the model, memory, sampling task, wireless stack, and real-time response all fit into the same resource budget at the same time.

The core conclusion up front: ESP32-S3 is a reasonable TinyML target for small INT8-quantized models with bounded input windows and controlled inference frequency. It is not a good target for large models, continuous high-frame-rate vision, complex multi-model pipelines, or any edge AI workload without a clear latency and memory budget. If a team only proves that one Invoke() call works — without measuring tensor arena peak usage, PSRAM trade-offs, peripheral contention, Wi-Fi/BLE concurrency, and end-to-end latency — the prototype is a demo that cannot survive production.

Scope note. "ESP32-S3 TinyML" here means running small machine learning inference workloads on an ESP32-S3-class MCU with TensorFlow Lite Micro or a similar runtime. Typical examples: sensor anomaly detection, keyword spotting, simple gesture recognition, low-resolution image classification, and device-side state decisions. It does not mean moving cloud-scale AI models onto an MCU.

1. "Can run" is not the product question

1.1 A successful single inference proves very little

ESP32-S3 has real hardware strengths. Espressif's datasheet lists a dual-core Xtensa LX7, up to 240 MHz, 512 KB SRAM, a 128-bit data bus, and dedicated SIMD instructions. The hardware is genuinely more suitable for lightweight edge AI than a basic MCU.

But the evaluation should not start with "can the model be compiled into firmware?" It should start with a harder question: does the inference path have a closed resource budget?

1.2 Where ESP32-S3 fits, and where it does not

Use case	ESP32-S3 fit	Why
Vibration, temperature, current, or other low-dimensional anomaly detection	High	Small input windows and controlled sampling rates make this realistic
Keyword spotting, event sound detection, lightweight audio preprocessing	Medium	Works only if audio buffers, Wi-Fi contention, and RAM are controlled
Low-resolution image classification or presence detection	Medium	Possible, but input size, PSRAM, camera bandwidth, and latency must be tested together
Multi-stream video, complex object detection, continuous visual analytics	Low	Input and compute demand exceed the MCU boundary quickly
Multi-model pipelines, online learning, LLM or RAG-style workloads	Very low	Compute, memory, and storage boundaries are mismatched

If the input is low-dimensional, low-frequency, and technically bounded, ESP32-S3 TinyML can be valuable. If the task is really continuous vision, multimodal understanding, or large-model inference, the MCU should not be treated as the main edge AI compute node.

2. Bottleneck one: on-chip SRAM and the tensor arena

2.1 TFLM memory is not just a malloc problem

TensorFlow Lite Micro is built around the tensor arena. TFLM's memory documentation describes this arena as a shared continuous buffer split into Head, Temporary, and Tail sections — for shared tensor buffers, scoped scratch buffers, and persistent runtime data. A model's deployability is therefore not its file size. Intermediate activations, scratch buffers, operator state, and input/output tensors can dominate the peak memory requirement.

ESP32-S3's 512 KB on-chip SRAM has to serve all of these at once:

FreeRTOS task stacks
Wi-Fi / BLE protocol stacks
driver DMA and sampling buffers
the TFLM tensor arena
application state, logs, and communication payloads
OTA, file system, and configuration buffers

Assign too much SRAM to tensor_arena and inference may work while networking, logging, sampling, and OTA become fragile. Make the arena too small and model initialization or scratch allocation fails.

2.2 PSRAM adds capacity, but does not erase latency

Many ESP32-S3 modules include PSRAM. That helps with image inputs, audio buffers, and larger models. But PSRAM is not a transparent replacement for on-chip SRAM. It is usually accessed through an external memory interface and cache, which makes it better for large buffers or less time-critical data than for hot tensors, real-time scratch buffers, or strict latency paths.

A more reliable memory plan treats memory as tiers:

On-chip SRAM: real-time tasks, stacks, DMA-sensitive buffers, and hot tensors.
PSRAM: frame buffers, larger input windows, non-real-time caches, and data that can tolerate latency.
Flash: model constants, configuration, and versioned resources — but not random hot-path reads.

For ESP32-S3 TinyML, PSRAM solves capacity pressure, not deterministic latency. If hot tensors or input pipelines repeatedly fall onto a slower path, the final symptom is still inference jitter and task timeouts.

3. Bottleneck two: quantization changes the model boundary

3.1 INT8 is the default reality for MCU TinyML

On an MCU like ESP32-S3, INT8 quantization is usually not an optional optimization. It is often what makes deployment possible. It reduces weight and activation memory and makes it easier to use optimized kernels. But quantization changes numerical behavior, especially in these areas:

boundary samples near anomaly thresholds
noisy audio or vibration signals with device-to-device variation
low-light, blurry, compressed, or lens-dependent image inputs
decisions that depend on ranking or confidence thresholds

Looking only at average accuracy after quantization can hide the failures that matter in the field. A better acceptance process uses both a representative calibration set and a field replay set, with typical, boundary, and noisy samples tested separately.

3.2 Operator coverage matters more than file format

Being able to convert a model into .tflite does not mean it will run reliably on TFLM. TensorFlow Lite Micro is designed for microcontrollers, so its operator set, memory planning, and kernel support are more constrained than desktop or mobile TensorFlow Lite. Model architecture should favor operators that are supported by TFLM, benefit from ESP-NN where relevant, and have predictable scratch requirements.

Three practical checks should happen early:

Constrain the model architecture before training so it avoids MCU-hostile operators.
After conversion, run real AllocateTensors() and Invoke() tests with the micro runtime.
Use RecordingMicroInterpreter or similar allocation logging instead of guessing tensor_arena_size.

ESP32-S3 TinyML model design should be driven backward from deployment constraints. Training a general model first and trying to squeeze it into an MCU later is usually the expensive path.

4. Bottleneck three: real-time budget and peripheral contention

4.1 Single-inference latency is not the full metric

Many prototypes record only one inference duration. A real device cares about the full cycle:

sampling window → preprocessing → inference → postprocessing → local control or telemetry

If a vibration model runs once every 500 ms, an 80 ms inference may be acceptable. If a voice trigger pipeline needs continuous sampling and low-latency response, 80 ms may interfere with audio buffers and network upload. If the same device also runs Wi-Fi, BLE, a display, buttons, logs, and OTA, inference must be part of the scheduling model rather than a standalone benchmark.

4.2 Camera, I2S, ADC, and Wi-Fi compete for the same MCU

ESP32-S3 is useful for sensor-rich lightweight AI nodes, but every peripheral path consumes memory, DMA, CPU time, and interrupt budget. Typical failure modes:

Camera frame buffers use PSRAM, forcing the inference arena to shrink or access slower memory.
I2S audio capture and inference run at high load at the same time, causing audio gaps or inference jitter.
Wi-Fi uploads, logs, or OTA operations stretch the inference cycle.
Task stacks look sufficient in demos but fail during pressure tests.
Long synchronous inference or postprocessing triggers watchdog issues.

At minimum, an ESP32-S3 TinyML project should record these metrics:

Metric	Why it matters	Recommended acceptance method
`tensor_arena_size` peak	Determines whether the model initializes and runs reliably	Log allocation details after `AllocateTensors()`
Remaining on-chip SRAM	Determines safety for networking, stacks, and logs	Record lowest free heap during stress tests
`Invoke()` P50 / P95	Shows average latency and tail latency	Run thousands of iterations with real inputs
sampling-to-decision latency	Determines business usability	Measure the real peripheral path, not only the model
latency under Wi-Fi / BLE load	Shows online behavior	Run with real communication load
power and temperature	Affects battery and enclosure design	Test under the target duty cycle

If an ESP32-S3 TinyML proposal does not show memory peak, tail latency, and peripheral-concurrency behavior, it proves demo feasibility at most. It does not prove product readiness.

5. A safer implementation order

5.1 Lock the product decision before locking the model

The safer order is not "find a model, then find a board." It is:

Define the device-side decision: what must be inferred locally?
Define the input window: sampling rate, window length, feature count, image size, or audio segment.
Define the latency budget: how quickly must a result be produced, and what happens if it is late?
Build the smallest useful model first: prefer INT8, a small operator set, and explainable features.
Then test on hardware: arena, heap, peripheral concurrency, and power.
Finally decide whether the edge node needs stronger hardware.

This order prevents a team from spending weeks compressing a model into an ESP32-S3 only to discover that the business requirement needs higher resolution, lower latency, or continuous connectivity.

5.2 A practical gate before small-batch production

Before an ESP32-S3 TinyML design moves into a pilot or small production batch, it should meet these conditions:

The model is INT8 and has a representative calibration set.
tensor_arena_size, peak heap, and task stack usage are recorded.
Inference runs with real sampling, networking, and logging load.
P95 or P99 latency meets the business budget, not just average latency.
OTA, logs, and configuration were not sacrificed to fit the model.
Model version, thresholds, input features, and firmware version can be traced together.
Unsuitable cases are explicit, such as high-frame-rate vision, multi-model chains, or hard real-time control.

6. When to stop forcing ESP32-S3 and use a stronger edge platform

Changing platform is not a failure. It is often the correct system boundary. These signs mean compressing the model further is probably less useful than choosing stronger hardware:

The input itself is large, such as multi-stream images, high-rate audio, or long time-series windows.
INT8 quantization causes false positives or false negatives that affect the business decision.
PSRAM is used simultaneously for frame buffers, model input, and communication buffers, and tail latency becomes unstable.
The device needs multiple models or complex postprocessing.
The same unit also acts as a gateway, protocol adapter, UI device, or local database cache.
OTA, logging, and diagnostics are being reduced to make room for the model.

Final judgment: ESP32-S3 is a strong TinyML edge node, not a general-purpose edge AI host. It works best when it moves small, well-defined decisions closer to the device: anomaly screening, pre-trigger filtering, low-dimensional state recognition, and lightweight voice or image event detection. Once the workload becomes high-throughput, multi-model, multimodal, or strongly real-time, ESP32-S3 should return to its role as a sensing and control node while a stronger edge compute unit handles the main inference path.

References

Espressif ESP32-S3 Series Datasheet
Espressif ESP-NN Component Registry
TensorFlow Lite Micro Memory Management

ESP32-P4 LVGL HMI: Budget Frame Buffers Before Tuning Refresh and Tearing

ZedIoT — Sat, 17 Oct 2026 12:20:00 +0000

ESP32-P4 LVGL HMI: Budget Frame Buffers Before Tuning Refresh and Tearing

An 800 × 480 panel in RGB565 needs 768,000 bytes for one uncompressed frame. Two panel frames alone consume about 1.46 MiB, before LVGL draw buffers, fonts, decoded images, rotation workspaces, or other tasks enter the budget. A display demo that boots on an ESP32-P4 therefore says little about the memory available to the finished HMI. The first useful design step is to name every buffer, its owner, and the point at which it is read or written. Only then can you make a defensible choice between one, two, or three panel frames. The LVGL Adapter guide treats interface, frame count, tearing mode, and draw-buffer height as separate decisions.

This is a design and validation guide based on public APIs. It does not report frame rate, touch latency, tearing improvement, or sustained behavior from a particular board. Those outcomes require a specified panel, PSRAM configuration, firmware version, workload, and measurement record.

Put the three kinds of memory on different lines

Teams often call all display-related storage "video memory," then discover that enabling a feature added an entire frame they did not budget. These are different things:

The panel frame buffer holds pixels read by the display path. In MIPI-DSI Video mode, pixel data streams from a host-side frame buffer; the ESP-IDF RGB driver likewise reads a configured internal-memory or PSRAM frame.
An LVGL draw buffer serves a different purpose: LVGL renders an invalidated region into it. Depending on the render mode, it may be smaller than the screen.
Rotation, scaling, blending, and color conversion may need another input or output workspace.

Some adapter modes share addresses between layers; others do not. A memory worksheet must reflect the actual selected path rather than adding or merging buffers based on similar names.

Trace one update before selecting a frame count

During a partial update, LVGL renders the invalidated area into a draw buffer. An adapter or application flush then transfers those pixels into a writable panel frame while the display engine may still be scanning a different frame—or the very same one. If the scan and write overlap without a clear ownership rule, the image can contain portions from different update cycles. A second frame makes separation possible, but it does not by itself synchronize frame switching, keep unchanged regions consistent, or map touch coordinates after rotation. Espressif explicitly calls out synchronization between two PSRAM frames in its RGB double-buffer description.

With esp_lvgl_adapter, select RGB or MIPI-DSI first. Then pick a supported tearing-avoidance mode and pass the count returned by esp_lv_adapter_get_required_frame_buffer_count() into the panel configuration. DOUBLE_DIRECT, DOUBLE_FULL, TRIPLE_PARTIAL, and TRIPLE_FULL address different update-area and rotation needs at different memory costs. Three frames are not a universal "smoothness" setting. The adapter also rejects nonzero rotation with the RGB/DSI NONE mode. If you compose LVGL, esp_lcd, and PPA yourself rather than using the adapter, you own the equivalent flush, frame-lifetime, and synchronization contract. The adapter's behavior must not be assumed to appear automatically in a custom port.

Work backward from resolution and pixel format

The first-pass panel-frame calculation is horizontal pixels × vertical pixels × bytes per pixel. RGB565 uses two bytes per pixel; RGB888 uses three. In the table, MiB means 1,048,576 bytes. These numbers are raw pixel storage only. They exclude row-stride alignment, driver allocation, caches, icons, fonts, decoding, heap fragmentation, and conversion workspaces.

Assumed panel format	One raw frame	Two raw frames	Three raw frames
800 × 480, RGB565	0.73 MiB	1.46 MiB	2.20 MiB
1024 × 600, RGB565	1.17 MiB	2.34 MiB	3.52 MiB
1024 × 600, RGB888	1.76 MiB	3.52 MiB	5.27 MiB

The table is especially useful for ruling out an impossible starting configuration. Two 1024 × 600 RGB888 frames contain 3,686,400 raw bytes in total, about 3.52 MiB. If the total allocatable PSRAM remaining after camera, network, and UI resources is below that amount, a configuration switch cannot make the pair fit; revisit color format, resolution, frame count, or concurrency.

If the frames are separately allocated, each allocation also needs an available contiguous block of at least 1,843,200 bytes, plus actual alignment overhead. The 3.52 MiB total is not a single-allocation contiguous-block requirement. The reverse inference is invalid: enough capacity does not prove enough throughput. Display scanning, CPU rendering, PPA reads and writes, and other clients can contend for external memory. The RGB driver documents the tradeoff between saving internal RAM and sharing PSRAM access with EDMA; the DSI driver requires the actual panel timing and pixel format, not only the frame's byte count.

Budget the LVGL draw buffers separately. A hypothetical 800-pixel-wide, 50-line RGB565 partial buffer contains 80,000 raw bytes, about 78 KiB; two such buffers double that component. The 50-line height is the adapter documentation's default example for RGB and DSI, not a product-wide recommendation. A taller buffer can reduce the number of flushes while using more RAM. A shorter one reduces allocation but can add calls and copies. The actual allocation depends on LVGL color format, memory capability, adapter mode, and whether draw and panel buffers share storage. Confirm the worksheet against runtime allocations; do not mistake the table for a predicted heap_caps_get_free_size() reading.

Put every buffer in one ownership worksheet

The following is an illustrative allocation, not a verified adapter mode or a board measurement: 800 × 480 RGB565, two distinct panel frames, one 50-line partial draw buffer, no rotation, and no PPA call. Raw pixel storage is 2 × 800 × 480 × 2 = 1,536,000 bytes for the panel frames plus 800 × 50 × 2 = 80,000 bytes for drawing: 1,616,000 bytes, about 1.54 MiB. It excludes driver allocations, alignment, assets, and concurrent peaks. A selected adapter mode might share physical storage between the draw and panel buffers, or allocate them separately. Do not order PSRAM from this example alone.

Buffer	Raw size / assumed location	Writer and reader	Handoff and synchronization check
Panel frame A	768,000 bytes / PSRAM	Refresh path writes when available; display DMA reads when scanning	Transfer at the driver- or adapter-defined frame switch; establish who synchronizes CPU/PPA writes before DMA reads
Panel frame B	768,000 bytes / PSRAM	Alternates with A as writable and displayed frame; do not overwrite during scan	Also verify how unchanged regions persist and when the frame becomes writable again
LVGL partial draw buffer	80,000 bytes / actual memory region to verify	LVGL writes; flush or adapter reads	Reuse only after flush completion; if physically shared with a panel frame, count that allocation only once
PPA SRM workspace	Zero in this baseline; budget separately if enabled	PPA reads input and writes a distinct output	Check input/output non-overlap and the CPU/DMA handoff for the chosen operation

Adding one separate, full-size RGB565 SRM output workspace is a conservative scenario, not a claim about an adapter's actual allocations. It would bring the illustrative raw total to 1,616,000 + 768,000 = 2,384,000 bytes, about 2.27 MiB. Fill in the real addresses, largest allocatable blocks, alignment overhead, and any shared physical storage before adding rows to get a product budget. Even when two uses share one allocation, record their separate ownership intervals; subtracting bytes alone does not make concurrent writes safe.

PSRAM adds a correctness contract, not just capacity

When a display DMA path reads pixels and a CPU or PPA path writes them, address reachability, alignment, and cache coherency can determine whether the correct image appears. The ESP32-P4 memory-synchronization documentation describes both directions of the problem: a CPU write may remain in cache when DMA reads external memory, or a CPU read may observe stale cached data after DMA has changed memory. If an application manages these buffers below the adapter, identify who allocates them, which DMA engine can reach them, what alignment applies, and where synchronization belongs. esp_cache_msync() is an official mechanism, but sprinkling a call before and after every flush is not a substitute for a defined read/write lifetime.

The largest contiguous allocatable block matters more than a headline PSRAM capacity. Fonts, icon decoding, networking, and other components can consume or fragment memory before a new screen or keyboard opens. A successful one-frame boot does not establish that later UI transitions can allocate the next working buffer. Record free memory, the largest block, and peak usage at boot, after entering the most demanding view, and during a transition. If the product also uses a camera, list camera frames separately. A camera-to-display pipeline follows video acquisition and transformation; this article concentrates on HMI rendering and panel-frame ownership.

RGB and MIPI-DSI are different refresh contracts

The ESP-IDF RGB driver offers internal-memory single-frame, PSRAM single-frame, PSRAM double-frame, and bounce-buffer modes. A single frame uses less storage but can still be scanned while an update is written. Double buffering separates the currently displayed frame from a writable one, at the cost of another full frame and a rule for keeping unchanged content consistent. Bounce buffering is not free double buffering: the documented RGB path combines a PSRAM main frame with two smaller internal buffers and line-level movement between CPU and DMA. It can tolerate some short bandwidth spikes, but uses more CPU and depends on external-cache availability. A working RGB recipe should not be copied without review to a DSI panel.

In MIPI-DSI Video mode, the host streams pixels from frame memory. Panel-control commands on the DBI path and video data on the DPI path have different jobs. The lane configuration, rate, color format, pixel clock, blanking intervals, and DPHY power must agree with the selected panel. esp_lcd_dpi_panel_get_frame_buffer() exposes addresses allocated by the driver. Before handing those addresses to LVGL or PPA, define when each may be written, when a completed frame becomes visible, and whether an adapter already owns that transaction. The ESP-IDF DSI driver exposes a DMA2D draw-bitmap hook and a custom-hook route for PPA operations; the existence of a hook does not mean rotation and compositing automatically use it.

Diagnose symptoms by layer rather than increasing clocks immediately:

A panel that never displays an image calls for power, initialization commands, lane wiring, and timing checks before LVGL buffer tuning.
Misaligned rows or wrong colors call for a review of stride, region coordinates, pixel format, byte order, and DMA alignment.
Tearing only during animation points toward the relationship among invalidated regions, writable frames, scanned frames, and VSYNC or transfer-completion callbacks.

The LCD FAQ distinguishes insufficient PSRAM fetch bandwidth from lane-rate and timing mismatch; one blue-screen symptom should not be labeled "LVGL is too slow" without evidence.

Give PPA a specific pixel operation

The ESP32-P4 PPA can accelerate scaling, rotation, mirroring, blending, and filling. It can take a specific pixel transform away from a software path. It cannot decide which LVGL objects need redraw, create PSRAM bandwidth, or settle cache and frame-switch ownership on behalf of the application. The PPA SRM operation requires different input and output picture buffers. A design that budgets only the pre-rotation frame but no output workspace can fail before measuring a single frame. Blend has different conditions for buffer reuse; check the API for the operation actually used rather than assuming all PPA transactions have the same memory rules.

A 90° or 270° orientation also changes width and height, touch mapping, invalidated areas, and intermediate-buffer calculations. In some adapter rotation and tearing-avoidance modes, Espressif requires three frame buffers. That requirement belongs to the documented mode; it does not imply that every custom driver needs exactly three or that any particular panel will benefit. Recalculate memory for the final orientation and pixel format before enabling rotation. If the panel controller can deliver the desired orientation without a full-frame transform, verify that route before accepting another frame-sized copy.

Move from a demo to a testable product configuration

Freeze the board inputs and assign memory ownership

Start with the panel's resolution, orientation, pixel format, interface, refresh requirement, and touch controller. Add the PSRAM part and capacity plus concurrent network, camera, audio, and logging work. There is no meaningful "best frame count" while these inputs are still moving. Lock the ESP-IDF, LVGL, and adapter versions used by the target firmware; macros and mode behavior can change between releases. A snippet from one release is not a cross-version product contract.

If wireless work is delegated to an ESP32-C6 companion, include P4–C6 data exchange and concurrent load among those inputs. The companion's responsibility boundary does not replace a PSRAM peak measurement on this device.

Then write a buffer-ownership worksheet: allocator, size, internal or external location, CPU/PPA/DMA permissions, handoff point, cache-synchronization owner, and release path for every buffer. Apply num_fbs, buffer_height, and color format to the capacity calculation, then add measured peak costs for fonts, textures, conversion, and concurrent features. If allocation fails, find the frame-sized consumer and any alignment overhead first. Enlarging an abstract "PSRAM size" setting does not explain which allocation was wrong or whether a full-size buffer could have been a region buffer.

Compare legal configurations and preserve a fallback

On the same target board, compare at least two valid configurations: for example, partial updates with one panel frame against an adapter-supported two-frame DOUBLE_DIRECT path at the same pixel format, or a two-frame path against a rotation mode that needs three. Change one variable at a time. Capture free and largest-contiguous memory, LVGL render time, flush or transfer time, actual displayed frame rate, CPU load, touch-to-visible-change latency, and the exact content that tears if tearing occurs. A mostly static demo screen is a weak workload. Include charts, popups, animation, language switching, and concurrent network activity relevant to the product. Public documentation gives the candidate modes; it cannot supply these board-level results.

Keep a low-complexity known-good display configuration so a failed optimization has a rollback. If enabling PPA produces color displacement, return to the untransformed path while comparing input and output buffers, format, stride, and alignment. If a second frame reduces throughput, inspect extra whole-frame copies and PSRAM contention before assuming a third frame will help. If an RGB bounce-buffer configuration fails during OTA or flash writes, check that mode's documented dependence on external cache. These are targeted investigations, not claims that this article reproduced those faults.

Product sign-off should be more precise than "the UI feels smooth." Name the board and panel revision, firmware and adapter versions, exact screen and concurrent load, test duration, sampling method, and blocking failures. Memory staying within bounds, acceptable interaction, no reproducible tearing under the chosen workload, and a workable fallback are separate conditions. This article offers a way to design that evidence. It cannot assert an FPS target, latency, or long-term stability for any device until those measurements exist.

Where this budget does not apply

This memory and refresh discussion is aimed at RGB or MIPI-DSI HMI paths on ESP32-P4. It must not be copied unchanged to SPI, QSPI, or I80 screens. Espressif's adapter groups those as OTHER interfaces and lists only NONE for its tearing-avoidance mode; their transfer path and any panel-local memory need their own analysis. Even on RGB or DSI, the panel timing, PSRAM part, power design, and concurrent load set the actual limit. "The adapter exposes a mode" and "our product passed in that mode" are different statements.

References

Espressif ESP LVGL Adapter
ESP-IDF MIPI-DSI LCD on ESP32-P4
ESP-IDF RGB LCD on ESP32-P4
ESP-IDF PPA on ESP32-P4
ESP-IDF ESP32-P4 Memory Synchronization

Inside the ESP32-P4 Camera-to-Display Pipeline: CSI, ISP, PPA, Buffers, and DSI

ZedIoT — Wed, 14 Oct 2026 12:20:00 +0000

Inside the ESP32-P4 Camera-to-Display Pipeline: CSI, ISP, PPA, Buffers, and DSI

A camera preview that reaches the screen is not the same as a reliable camera-to-display design. The critical question is what happens to each completed frame between the sensor and display scanout: its pixel format, the memory it occupies, who may read or write it, and the event that makes it reusable. If an encoder and a GUI share the path, those questions matter more than a nominal frame-rate target.

Start with a contract for every boundary. Record the sensor's output mode, what CSI receives, what ISP writes to memory, whether PPA has a specific pixel operation to perform, and which completed buffer the display or encoder consumes. Only then can measured latency, a dropped frame, or a torn image be attributed to a stage. The architecture and API responsibilities below follow public Espressif documentation; they are not performance results from a particular board.

Draw the frame path around memory, not peripheral names

ESP32-P4 has MIPI-CSI, ISP, PPA, JPEG and H.264 hardware, MIPI-DSI, and DMA-capable paths. Their presence does not imply that a frame flows between them without an intermediate write, copy, or queue. The ESP-IDF ISP guide describes an ISP that receives data from a camera controller or system memory and writes processed output to system memory through DMA. That write is an architectural handoff, not a minor implementation detail.

The frame path, in text form:

Sensor → CSI → (system memory) → ISP → (system memory) → [PPA branch, if a pixel op is required] → display scanout (DSI) / encoder

Two notes on that path:

The direct buffer-to-display edge is conditional. Output format and display timing must actually be compatible.
PPA is a branch for required pixel work, not a mandatory stop between ISP and DSI. Likewise, the GUI is an input to a blend operation; it is not a claim that every GUI must be composited into a video frame.

At the central buffer pool, define four events for each frame: allocation, producer completion, consumer completion, and return to the free pool. A frame cannot be recycled just because one task has finished with it if another task, DMA engine, encoder, or display scanout still reads it. This contract often disappears when integration is described only as driver initialization calls.

Freeze the sensor-to-CSI contract before tuning images

A supported CSI format is only the receiver's capability

The ESP32-P4 datasheet lists a two-data-lane MIPI-CSI interface and input formats including RGB888, RGB666, RGB565, YUV422, YUV420, RAW8, RAW10, and RAW12. It does not configure a sensor for you. A valid receiver format says little about the selected sensor register table, lane rate, virtual channel, clocking, or exposure-control sequence.

Keep one reviewable mode record per sensor configuration: active resolution, line and frame lengths, CSI data type and bit depth, lane count and rate, external clock, and exposure and gain controls. If the sensor emits RAW10 but the display ultimately scans RGB565, name the stage that unpacks RAW, performs demosaicing and color processing, and quantizes the result. Otherwise an apparent "CSI works" milestone may conceal a color-order error or line-alignment problem that surfaces only after the display is added.

Use a sensor test pattern, if the sensor provides one, while establishing this boundary. A stable pattern removes scene movement and automatic exposure as competing explanations. Check frame completion and short-frame or line errors before adjusting ISP color parameters. If the pattern itself arrives with unstable rows, downstream image-quality tuning cannot repair the input contract.

CSI's 2.5 V supply belongs in the board bring-up checklist

The ESP-IDF Camera Controller guide calls out a stable 2.5 V supply for the ESP32-P4 CSI peripheral. An internal adjustable LDO can be used, but its configuration and connection must be complete before CSI driver initialization. Treat that as a schematic and power-sequencing prerequisite, not an optional software optimization.

On a custom board, check supply and clock first, then lane continuity and sensor register access, and only then complete image frames. An official development-board example is useful for API ordering, but it does not establish that a different board copied the same electrical path or that a new sensor's mode table is correct.

Make ISP output a useful downstream contract

For RAW input, the ISP can provide the image-processing stages that turn sensor samples into a usable frame: black-level correction, bad-pixel handling, lens-shading correction, Bayer filtering, demosaicing, white balance, and color correction, among others. Which stages to enable and how to tune them depend on the sensor and optics. Driver support for a stage is not a production image-quality calibration.

There is an initialization distinction worth preserving. In the stable v6.1 ISP documentation, using MIPI-CSI or ISP_DVP as the camera controller still requires an ISP processor created with esp_isp_new_processor(), even when ISP image algorithms are not needed. The bypass_isp configuration bypasses the selected algorithm pipeline; it is not a substitute for installing the driver. Recheck these names and requirements against the exact ESP-IDF release used by the product.

Work backward from the consumers

Before selecting an ISP output format, write down each consumer's requirements. The DSI/LCD path has a pixel format, line stride, address-alignment and scanout contract. A PPA operation has supported input and output formats and buffer rules. An encoder has an input format and may hold the frame after the display has moved on. A CPU algorithm may need temporary read access or a separate working copy.

An intermediate format shared by several consumers can remove a conversion, but it is not automatically optimal. Producing RGB for a display and converting it back to YUV for encoding adds memory traffic and queueing. Forcing the GUI to render against an awkward format to save that conversion can shift cost and complexity into composition. Compare complete paths—including memory reads and writes—rather than choosing a format based on one peripheral's advertised capability.

The handoff record should state whether the ISP output is immutable once marked complete. If PPA will modify an output in place where its operation permits it, distinguish that destination from the still-shared source frame. An encoder reading the original frame must not race with a GUI blend writing into it. This is an ownership decision before it is an API choice.

A buffer pool is a latency and correctness policy

Size allocations with stride, not width alone

For an uncompressed frame, begin with frame_bytes = stride_bytes × height. If stride happened to equal width × bytes_per_pixel, RGB565 would need approximately two bytes per pixel and RGB888 approximately three. Real allocations must include line alignment, DMA and driver constraints, and any format-specific planes or padding. The formula is a capacity check, not an FPS prediction.

Count all simultaneously live frames: capture or ISP outputs, PPA inputs and outputs, display framebuffers, GUI draw buffers, and encoder inputs. Then count memory transactions. A full-frame memory-to-memory conversion reads a source and writes a destination; scanout, GUI composition, encoding, and any copy add more traffic. "Does PSRAM fit the bytes?" and "Can the memory system sustain the simultaneous access pattern?" are separate questions requiring target-board measurements.

Single buffering saves capacity but exposes the same memory to producer and consumer contention. Two buffers let a producer work on a next frame while a consumer reads a completed one; they do not establish a safe display swap point. A third buffer can absorb a burst or slow consumer, yet it can leave an old frame waiting in a longer queue. More buffers are not a cure for an unspecified release rule.

Define ownership transitions explicitly

A useful design record names frame states, even if the application uses a different implementation: FREE → WRITING → COMPLETE → IN_USE → FREE. The producer alone owns a WRITING buffer. After its DMA completion and any memory-visibility work required by the configured memory region, it publishes a COMPLETE frame. Each consumer acquires a read claim; the pool recycles the frame only after the last claim is released and display scanout no longer references it.

That last condition matters when preview and H.264 encoding fork from one source. Reference counting, explicit per-consumer acknowledgments, or another equivalent mechanism can implement it. A frame should not be returned on "display finished" if encoding is still reading, nor on "encoder submitted" if encoding has only queued asynchronous work. Define whether the completion event means accepted, processed, or no longer reading memory.

Cache consistency belongs at the same transition. If the chosen CPU, DMA, and external-memory arrangement requires cache clean, invalidate, or barriers, perform the operation appropriate to the handoff direction before publishing the buffer or reusing it. Do not apply a generic cache recipe without checking current ESP-IDF memory and DMA requirements. When faults appear only with cache enabled, PSRAM placement, or higher concurrent load, first test visibility and premature reuse before labeling the symptom "insufficient bandwidth."

For a low-latency preview, an explicit policy may discard an expired completed frame and show the newest complete one. For per-frame analysis or archival, silently overwriting work is incorrect; backpressure, reduced input rate, or a documented degraded mode is needed. Log which policy fired. A display may scan the last complete frame again when its refresh rate exceeds capture rate; it must not scan a frame still under construction.

Give PPA a named pixel job

The ESP-IDF PPA guide describes scale/rotate/mirror, blending, and filling. PPA can move a required pixel operation out of general CPU code, but it still reads memory and writes a result. Its value depends on the transformation and resulting buffer traffic, not the mere fact that it is hardware acceleration.

Use it when sensor orientation differs from display orientation, when the preview window must be resized, or when a video frame and GUI surface must be blended. The PPA guide specifies separate input and output picture buffers for a scale/rotate/mirror operation; blending has different aliasing rules. Those distinctions matter when sizing a pool and deciding whether the original frame remains available to an encoder. Validate formats, block dimensions, offsets, alignment, and completion semantics for the operation actually selected.

If sensor output already matches the display's size and orientation, a whole-frame PPA pass may be unnecessary. For GUI overlays, compare two architectures: compose UI into the video frame, or let the display path manage video and UI separately where driver and hardware support that mode. Composition yields one final frame but may require substantial pixel work after each UI change. Separate layers or partial updates can avoid rewriting video pixels, but require compatible display features and careful synchronization. A claim of reduced CPU use alone does not settle the end-to-end tradeoff.

Split display and encoding into different consumer contracts

The display needs a complete frame available at the scanout deadline. The MIPI-DSI LCD example shows driver-managed framebuffer use and an LVGL draw-buffer arrangement in PSRAM. The MIPI-CSI–ISP–DSI example is a public reference for wiring modules together. Neither example proves a particular panel timing, sensor mode, UI load, or product memory layout.

Changing an address or writing a framebuffer while the display scans it can put two generations of pixels on one screen. The swap therefore needs a defined synchronization point and a buffer the producer will not overwrite. The right mechanism depends on panel timing, DSI driver behavior, and framebuffer mode; allocating two buffers without specifying the swap event is not a tearing test.

An H.264 encoder has a different deadline and output responsibilities: accepted input format, ordered frames, rate control, encoded-output buffers, and downstream storage or network consumption. Hardware encoding does not make its input lifetime zero. If preview and encoding share a source, the slow branch extends that source's lifetime. If encoding receives a separate copy, budget the extra allocation, read/write traffic, and delay. The display may prioritize freshness while the encoder requires continuity; one unbounded queue cannot satisfy both goals by accident.

Choose the contract that matches the product

Requirement or observation	Design response	Cost or boundary to verify
Lowest useful preview age	Keep the newest complete frame; document when older frames are dropped	Measure capture-to-scanout age and drop reasons, not just average FPS
Every frame must be analyzed or retained	Preserve per-frame ownership and apply backpressure or an explicit degraded mode	Queue growth, capture pressure, and recovery behavior
Rotation, scaling, or video/UI blend is required	Put the named operation on a PPA branch after checking formats and buffer rules	Extra memory transaction, destination buffer, and completion handoff
Preview and H.264 share a frame	Hold the source until both consumers have stopped reading	Encoder-induced pool starvation and display freshness
Camera format already matches display needs	Evaluate the direct completed-buffer route	DSI compatibility, scanout lifetime, and safe swap timing

This table is a decision aid, not a promise that all options can run together at a given resolution. Datasheet capabilities, reference examples, and measured end-to-end product behavior are different evidence classes.

Diagnose the first boundary that corrupts a frame

Similar screen artifacts can have unrelated causes. A color cast can originate in sensor output order, ISP color processing, RGB/BGR convention, or display interpretation. Repeating horizontal displacement points toward line length, stride, or starting address. An occasional old image suggests queue age, a missed swap, or incorrect return to the free pool. Adjusting the last module until the screen looks acceptable can hide an upstream error.

Place a comparable observation at each boundary:

With a sensor test pattern, record CSI completion and error counters.
At ISP memory output, inspect fixed rows and pixel positions along with recorded width, height, format, and stride.
Save a small set of pre- and post-PPA frames to see whether rotation, scale, or blend changed unintended areas.
At DSI, correlate the chosen framebuffer address with swap and refresh events.

Preserve the failing frame's ID, buffer address, stage, and event order so an intermittent failure can be traced to one handoff.

Do not treat visual correctness as latency evidence. Timestamp capture completion, ISP completion, PPA completion where used, display swap/refresh, and encoder completion where used. Record buffer-pool high-water mark, queue depth, dropped-frame reason, buffer location, and cache/DMA properties alongside those timestamps. A system can show an attractive average FPS while displaying increasingly stale frames or occasionally reusing a buffer too early.

Bring up the shortest observable route first

Control the sensor. Read its identity, fix one mode table, and save resolution, data type, lane configuration, clock, and power assumptions.
Prove CSI capture. Prefer a known test pattern. Log frame completion, short frames, line errors, and DMA status before adding a complete UI.
Fix ISP output. Start with unnecessary automatic adjustments disabled. Verify dimensions, stride, pixel order, and color; enable image-quality functions one by one.
Prove display ownership. With no GUI or encoder, identify producer completion, display acquisition, the swap event, and release event. Use timestamps, GPIO markers, or a logic analyzer where appropriate.
Add one PPA operation at a time. First test a required scale or rotation, then any GUI blend. Record its destination buffer, completion, and incremental memory traffic.
Add optional encoding or network consumption last. Exercise a deliberately slow consumer and observe queue high-water marks, drops or backpressure, and recovery.

At every step, retain ESP-IDF version, board and chip revision, sensor and panel part numbers, input/output formats and resolutions, buffer count, and memory placement. Without those identities, a later measurement cannot be compared fairly. Public references span stable v6.1 ISP documentation, v6.0 PPA documentation, latest Camera Controller documentation, and master-branch examples. They establish public responsibilities, not a tested single-SDK integration matrix.

Know when this is the wrong single-chip architecture

ESP32-P4 can fit products where MCU-level control, a camera input, local image processing, and an HMI must be tightly integrated. Its peripheral list alone is not a selection argument. A product needing complex multi-stream media handling, mature container support, heavy GPU composition, or large vision models may be better served by a Linux SoC and its media stack. At the other extreme, an occasional still image and simple display may not justify the power, routing, driver, and verification work of the full CSI–ISP–PPA–DSI path.

Sensor and panel ecosystems also matter. Missing stable drivers, timing data, or supply continuity cannot be fixed by the MCU's interface specification. If requirements simultaneously insist on no dropped frames, minimal preview delay, continuous encoding, and a complex GUI, first allocate explicit memory, power, thermal, and degradation budgets. Those goals may compete for the same buffers and bandwidth.

Before product review, assemble one stage-by-stage table covering format, resolution, stride, buffer size and count, memory region, DMA/cache attributes, producer, consumer, and release event. Then measure capture-to-display and capture-to-encode behavior under the fixed target workload. Public documentation supports the module responsibilities and validation method presented here; it does not establish frame rate, end-to-end latency, absence of tearing, memory bandwidth, power, temperature, or long-run stability on your hardware. Those are acceptance results to produce, not conclusions to import from an example.

Official references

ESP32-P4 Series Datasheet
ESP-IDF Camera Controller Driver
ESP-IDF Image Signal Processor, stable v6.1
ESP-IDF Pixel-Processing Accelerator, v6.0
ESP-IDF MIPI-DSI LCD Panel Example
ESP-IDF MIPI-CSI Camera via ISP to DSI Example
ESP-IDF DVP Camera Display via DSI Example, for DVP comparison only

Tuning ESPHome Voice Satellites for Low Latency and Stability

ZedIoT — Thu, 08 Oct 2026 12:30:00 +0000

Tuning ESPHome Voice Satellites for Low Latency and Stability

When an ESPHome voice satellite feels slow or choppy, it is tempting to blame the wake word model or Home Assistant itself. That instinct is usually wrong. Most problems come from the audio path, Wi-Fi jitter, or resource contention on the node — and they are fixable, if you first figure out where the delay actually sits.

1. Start by Classifying the Lag

ESPHome voice satellite problems usually fall into four different buckets.

Symptom	Likely area	Check first	Do not start with
Wake word feels slow or misses	Microphone, noise, wake model	Mic position, gain, room noise	Replacing TTS or an LLM
Long pause after speaking	Wi-Fi, Assist pipeline, STT/TTS	End-to-end timestamps, Home Assistant load	Changing random I2S pins
Choppy playback	TTS return, speaker, buffer	Speaker path, power, media player setup	Raising mic gain again
Reboots or disconnects	RAM/CPU, BLE, logs, Wi-Fi	Minimal config soak test	Adding more features to the same node

The routing rule is straightforward:

If the problem happens only in one room, inspect acoustics and Wi-Fi first.
If every room is slow, inspect the Assist pipeline.
If capture and playback interfere with each other, inspect I2S/PDM, speaker output, and resource contention.

This produces better fixes than searching for one magic latency option.

2. I2S and PDM Are About Timing, Pins, and Buffers

ESPHome's I2S audio component is used for audio input and output on ESP32-family chips. For a voice satellite, a compiling I2S or PDM configuration only proves that audio may be captured. It does not prove the audio path will stay stable under room noise, Wi-Fi jitter, and TTS playback.

A more reliable sequence is:

Fix the microphone and speaker GPIO choices before deeper tuning.
Keep wires short and avoid running audio lines next to power, relay, or LED wiring.
Validate capture, wake, and playback with a minimal voice configuration before adding lights, sensors, or other components.
Record timestamps for capture, STT start, STT completion, intent match, TTS start, and playback start.

Without these timestamps, teams mix up broken capture, network delay, Home Assistant load, and slow TTS retrieval as one vague problem. A tunable system needs to say where the delay actually sits.

3. Device Buffering Is the Stability Boundary

ESP32-S3 is a good fit for ESPHome voice nodes, but it is not an unlimited platform. A voice satellite can already include microphone capture, wake handling, API connectivity, status LEDs, a button, a speaker, logs, and sometimes BLE scanning. CPU, RAM, Wi-Fi, and audio tasks compete quickly.

For a stable satellite, keep three boundaries clear:

Do not make the voice node also serve as a BLE scanning gateway, display controller, and high-frequency sensor hub.
Reduce logging during audio tests — logging can become part of the latency problem.
Keep status LEDs functional, but do not turn the satellite into a continuous UI animation device.

This is not over-engineering. Once an ESPHome node is continuously handling audio, treating it like a generic multi-purpose ESP32 node usually sacrifices stability before it adds meaningful value.

4. Draw the Latency Path Before Tuning It

The latency path is not a conceptual overview — it is a debugging map:

If audio reaches Home Assistant late, inspect the ESPHome node and Wi-Fi.
If audio reaches Home Assistant quickly but TTS returns slowly, inspect STT, intent recognition, and TTS.
If TTS returns but playback stutters, inspect the speaker path, media player setup, amplifier, and power.

5. Wi-Fi Jitter Matters More Than Average Signal Strength

Many voice satellite tests only look at Wi-Fi signal strength. That is not enough. Voice interaction is sensitive to short jitter bursts from microwave ovens, TVs, walls, weak access points, and many ESPHome nodes sharing the same network.

Test the satellite where it will actually be installed, using fixed short commands and end-to-end timing. A successful test beside your desk does not prove the device will work on the far side of the kitchen.

If one room is unreliable, do not immediately replace the STT engine. Instead:

Move the node closer to the access point
Test another power supply
Remove non-voice tasks
Test at different times of day

Then decide whether the hardware or the architecture needs to change.

6. A Practical Debugging Sequence

First, run the smallest possible voice configuration: microphone, speaker, voice component, basic status LED, and API connectivity. Confirm that capture, upload, and playback are stable.

Second, test microphone input alone. Use one fixed command in quiet, normal, and noisy conditions. Occasional success is not enough; the capture path has to be consistent.

Third, log pipeline timings. At minimum, record the end of user speech, Home Assistant processing start, intent match, TTS return, and device playback start. Without timing data, there is no clear optimization target.

Fourth, test TTS and speaker output separately. Play fixed audio or a fixed response to validate power, amplifier, and speaker behavior.

Fifth, add features gradually. Only after the minimal path is stable should you add BLE, displays, extra sensors, LED effects, or automations. Retest latency and long-running stability after each feature class.

7. When ESPHome Voice Satellites Are the Wrong Tool

ESPHome voice satellites are useful for near-field control, room-level entry points, push-to-talk devices, and low-cost experiments. The following requirements should not be forced onto a basic ESP32-S3 node:

Far-field living-room pickup with TV noise and multiple speakers
Commercial spaces, kitchens, or workshops with high noise
Echo cancellation and continuous conversation close to a commercial smart speaker
A single node that also acts as a BLE gateway, display controller, and sensor hub
Long-term deployment maintained by non-technical users

These scenarios need dedicated voice hardware, a microphone array, a Linux-based satellite, or at least a split between voice duties and gateway duties. When voice controls important home actions, stability matters more than low cost or feature stacking.

8. Conclusion: Stabilize the Audio Path Before Optimizing Intelligence

Tuning an ESPHome voice satellite is not about finding one universal YAML snippet. It is about turning the audio path into a measurable system. Microphone capture, I2S/PDM timing, device buffering, Wi-Fi, the Assist pipeline, TTS, and speaker playback all need separate validation.

When those boundaries are unclear, every problem looks like a model issue, a wake word issue, or a Home Assistant issue. When the boundaries are clear, an ESPHome voice satellite can become a reliable local smart home entry point. It may not replace every dedicated voice device, but it is very good at becoming a customizable, maintainable, room-level interaction node.

Sources

ESPHome Voice Assistant
ESPHome I2S Audio Component
Home Assistant Voice Control
Home Assistant Voice Preview Edition

If you've tuned a voice satellite, where did your latency actually live — the mic path, Wi-Fi jitter, or the Assist pipeline? And at what point did you decide a basic ESP32-S3 node simply wasn't the right hardware anymore?

Why Industrial Edge Gateways Need Store-and-Forward

ZedIoT — Tue, 06 Oct 2026 12:30:00 +0000

Why Industrial Edge Gateways Need Store-and-Forward

Many teams still treat an industrial edge gateway as a simple protocol bridge: collect data from the field, forward it to the cloud, and relay commands back down when they arrive. If the link works while the network is healthy, the design looks good enough.

That assumption usually breaks in real deployments. Industrial links are not just "online or offline." A gateway may face:

intermittent 4G or private WAN instability
broker reconnect storms after a short outage
API throttling on the cloud side
telemetry that keeps arriving from PLCs while upstream delivery is blocked
commands that were accepted by the gateway but not yet confirmed by the platform
delayed replay mixed with newly collected realtime data

The core argument here: if an industrial gateway sits between stable field collection and unstable upstream delivery, it should not behave like a transparent forwarder. It should behave like a reliable delivery boundary.

That means implementing store-and-forward with local persistence, ordering control, deduplication, replay recovery, acknowledgment tracking, and expiration policy. Without that layer, the platform does not merely receive delayed data — it loses the ability to explain whether a record is complete, duplicated, stale, or still pending confirmation.

1. Why direct forwarding is not enough

Outages create explainability problems, not just latency

A network break interrupts more than transmission time. It breaks a chain of operational meaning:

when the value was produced
when the gateway received it
whether the platform ever acknowledged it
whether the record was already consumed downstream
whether the replayed record should still affect alarms or reports

If the gateway does not preserve that chain, recovery produces a pile of late data with no trustworthy context. The result is usually:

reports polluted by late-arriving values treated as current state
duplicate alarm evaluations after replay
operations teams unable to tell whether a gap came from field collection failure or upstream delivery failure

In practice, the first thing you lose without store-and-forward is not throughput. It is interpretability.

The real engineering goal is integrity, not connectivity

An industrial gateway usually sits between components that move at different speeds:

field polling loops or event subscriptions
normalization and mapping logic
local queueing and replay behavior
cloud brokers, APIs, and databases

Those layers do not fail or recover at the same pace. A PLC may still respond every second while the upstream broker rejects writes or the WAN link flaps. Without a local durability boundary, the system usually falls into one of two bad choices:

pause or degrade field collection
drop upstream failures and accept unrecoverable holes

For energy data, machine state history, production traceability, or cold-chain monitoring, those holes are operationally expensive. They are not a cosmetic delay.

2. What a usable store-and-forward design must solve

Local buffering must be durable, not informal

The queue must answer concrete questions:

Does every record have a stable message_id?
Is the record persisted before the collection loop moves on?
Can the queue recover after process restart or power loss?
Can the system distinguish ready, inflight, acked, and expired?

If data only lives in memory or in an ad hoc text file, a restart leaves the gateway unable to tell what already succeeded and what should be retried.

Acknowledgment must advance state

A practical send state machine usually includes at least:

State	Meaning	Exit Condition
`ready`	persisted and waiting to send	sender picks it up
`inflight`	sent and waiting for confirmation	ACK or timeout
`acked`	platform confirmed durable handling	safe to remove
`retry_wait`	transient failure, waiting to retry	backoff expires
`dead_letter`	expired or exhausted retry budget	operator action or cleanup

The key point is structural: confirmation must move local state forward.

This is why protocol-level acknowledgment is often insufficient. MQTT QoS 1, for example, only proves the broker accepted a publish. It does not prove the business pipeline persisted it or deduplicated it correctly. If the system needs ingestion guarantees, it also needs a business ACK.

Replay order must be explicit

One of the most common replay failures is mixing backlog with fresh realtime flow:

the gateway buffers ten minutes of telemetry during an outage
connectivity returns
field polling continues
replay and new collection are pushed upstream at the same time

Now the platform may receive a 10:10 sample before a 10:03 replayed sample. Without ordering rules, charts, aggregations, and alarm transitions become unreliable.

A safer design usually means:

a monotonic sequence or event-time rule per device_id + point_id
shared ordering checks across backlog and realtime paths
explicit support for late-arriving data on the platform side

3. Four failure patterns teams often underestimate

Weak dedupe keys create duplicate telemetry after restart

If a record identifier is built from "current timestamp plus an in-memory counter," a reboot can recreate the same key space. A stronger dedupe key normally includes:

gateway instance identity
source object identity
event time or collection window
a locally persisted increasing sequence

That gives the platform a stable basis for distinguishing replay from duplication.

No expiration policy eventually fills the disk

Not every message deserves infinite retention.

One-second temperature readings may only need a bounded replay window.
Billing or settlement records may need to wait until explicit confirmation.
High-frequency vibration payloads may need edge aggregation before uplink.

Store-and-forward therefore needs more than retry logic. It also needs:

a retention window per message class
explicit drop or aggregation rules
an audit event when valuable data is discarded

Without those rules, the gateway eventually chooses between uncontrolled disk growth and silent loss.

Commands and telemetry cannot share the same semantics

Telemetry replay usually aims for eventual persistence. Commands carry side effects. If both travel through the same retry behavior, the system can easily become unsafe:

losing a telemetry sample affects reporting
replaying a command may trigger a duplicate physical action

A safer split is:

telemetry supports idempotent replay
commands carry a distinct command_id
command status moves through accepted / executed / failed / expired
non-idempotent commands do not reuse ordinary telemetry replay paths

This is a common reason gateway teams succeed on telemetry buffering but fail when remote control is added later.

Event time and ingest time both matter

After a replay window, the platform often needs multiple timestamps:

event_time
gateway_received_at
platform_ingested_at

If only ingest time is stored, replay distorts historical truth. If only event time is stored, operators cannot measure recovery lag or delivery delay. A durable design keeps both.

4. A practical layering model for gateway reliability

A practical gateway usually benefits from four clear layers:

Protocol collection — only handles field connectivity and raw reads or subscriptions.
Normalization — maps raw values into stable objects, units, and quality semantics.
Local queue — owns durability, deduplication, retry behavior, expiration, and replay order.
Transport and confirmation — delivers upstream and applies platform acknowledgments back into local state.

The reason this split matters is simple: recovery complexity stays inside the queue boundary instead of leaking into every driver and every cloud-facing integration.

5. When store-and-forward should be a hard requirement

It should usually be in scope from the first version when:

the system needs production traceability or audit history
the WAN path depends on cellular, VPN, or unstable remote links
the cloud side applies rate limits or burst restrictions
data loss affects alarms, KPIs, or settlement logic
the gateway will later support commands, config sync, or fleet operations

A lighter forwarding model may be enough for:

labs and demos
single-device proof-of-concept setups
non-critical dashboards
workloads where short aggregation gaps are acceptable

The decision is not "industrial or not." The decision is whether integrity and explainability have business consequences.

6. Minimum implementation checklist

If you are adding store-and-forward to an industrial gateway, the first release should usually include:

a durable local queue
stable message_id generation
an explicit send state machine
bounded retries with backoff
queue depth, retry count, and expiry metrics
platform-side idempotency or business ACK support
preserved event and ingest timestamps

If the design only says "we buffer some records locally," it is usually not enough.

7. Important boundaries

Store-and-forward matters, but not every payload should be retained and replayed forever. Examples that need boundary decisions:

raw high-frequency waveforms that should be aggregated at the edge
commands with physical side effects that need a dedicated control-state model
short-lived debug logs that only matter during diagnostics

So the real goal is not "never lose anything." The real goal is to create a controlled, auditable delivery path for the messages whose loss, duplication, or reordering would actually matter.

8. Conclusion

Industrial edge gateways need store-and-forward because they sit exactly where timing mismatches become expensive: stable field collection below, unstable upstream delivery above. If the gateway only forwards, the system eventually loses control of replay order, deduplication, acknowledgment status, and recovery semantics.

A better model is to treat the gateway as a reliable delivery node — giving it explicit responsibility for local persistence, state progression, deduplication, ordering, expiration, and confirmation handling. Once that exists, the platform can still answer the question that matters most after an outage: what happened to this record, and can we still trust it?

A question for anyone running gateways in the field: have you been burned more by replay ordering, by business-ACK gaps, or by command paths that silently inherited telemetry retry semantics? And where did you end up drawing the line between "must never lose" and "acceptable to drop"?