DEV Community: Israel Negrete Lepe

Night shift: a local Gemma writes my 7 a.m. lab report, and I measured where it makes things up

Israel Negrete Lepe — Sat, 24 Oct 2026 19:45:36 +0000

This is a submission for the Hacktoberfest Open-Source AI Challenge Week 3: Slow and Steady.

What happened in the lab overnight? Most mornings the answer is nothing, and finding that out still means opening logs. Some mornings it is something, and logs are a slow way to find out.

So I built a night shift. Two small machines send a heartbeat all night, fixed rules go over the night at 6:30 a.m., and before 7 each machine leaves a short report written by Gemma 3, running on the machine itself, with no internet and no API key. The lazy part: nobody has to read logs in the morning. The careful part: I did not hand the model that job until I had measured where it gets things wrong.

What I built

The lab is an Intel N100 mini PC and a Raspberry Pi 5 on the same network.

A heartbeat. Every 10 seconds each machine sends a small record: a sequence number, when it was sent, when its process started, CPU load, available memory, temperature, and how many heartbeats it failed to send. Each machine also keeps its own copy on disk, in case the link is what fails.
A collector on the N100 that stores what arrives, with the arrival time. Both machines send to it, the N100 to itself.
A detector that runs once, at 6:30 a.m., over the whole night (10 p.m. to 6:30 a.m.) with fixed rules. It writes a JSON file with every incident: start, end, duration, when it was detected, whether it recovered on its own, and the evidence.
A writer. Gemma 3 4B in Ollama reads only that JSON and writes the morning report in Markdown. It does not decide whether something failed. That was decided before it woke up.

Each machine writes its own report from the same JSON: the N100 at 6:30, the Pi at 6:40. That lets me compare an x86 mini PC and an ARM board doing the same job with the same model. One known gap: everything goes through the N100, so if the N100 is down, there is no report.

Demo

This is part of the Pi's report after a night with eight injected faults, exactly as Gemma wrote it (the prompt is in Spanish; lines wrapped, two of the eight incidents shown):

# Turno de noche: 2026-10-21 22:00
**Salud general:** mala. El 99.9% de los latidos recibidos provienen del
equipo N100, mientras que el 87.2% provienen del Raspberry Pi 5.

## Incidentes
*   **N100:** servicio reiniciado, 22:39:51 - 22:40:30, 39 s, detectado
    22:40:21, no confirmado
    Hechos observados: Ningún latido recibido de 22:39:51 a 22:40:30 (39 s;
    se esperaba uno cada 10 s). El latido siguiente viene de un proceso
    nuevo, arrancado a las 22:40:30. El vigilante del equipo registró la
    caída del proceso a las 22:40:00 (código -15). / Posibles explicaciones:
    Fallo en el sistema operativo. / Causa confirmada: no confirmada
*   **Raspberry Pi 5:** el equipo no envió, 23:29:51 - 23:35:01, 310 s,
    detectado 23:30:21, no confirmado
    Hechos observados: Ningún latido recibido de 23:29:51 a 23:35:01 (310 s;
    se esperaba uno cada 10 s). El mismo proceso antes y después; faltan 30
    latidos en la secuencia. El propio equipo registró 30 latidos como no
    enviados en ese lapso. / Posibles explicaciones: Problema de red. /
    Causa confirmada: no confirmada

Translation. Night shift: 2026-10-21 22:00. Overall health: bad. 99.9 % of the heartbeats received come from the N100, while 87.2 % come from the Raspberry Pi 5.

N100: service restarted, 22:39:51–22:40:30, 39 s, detected 22:40:21, not confirmed. Observed facts: no heartbeat received from 22:39:51 to 22:40:30 (39 s; one expected every 10 s). The next heartbeat comes from a new process, started at 22:40:30. The machine's watchdog logged the process going down at 22:40:00 (code -15). Possible explanations: operating system failure. Confirmed cause: not confirmed.

Raspberry Pi 5: the machine did not send, 23:29:51–23:35:01, 310 s, detected 23:30:21, not confirmed. Observed facts: no heartbeat received from 23:29:51 to 23:35:01 (310 s; one expected every 10 s). The same process before and after; 30 heartbeats missing from the sequence. The machine itself logged 30 heartbeats as unsent in that span. Possible explanations: network problem. Confirmed cause: not confirmed.

The observed facts are right. Several things around them are wrong, and finding them is what this post is about.

Code

Everything that ran is in github.com/israel0negrete-crtdrops/night-shift, MIT licensed: heartbeat, watchdog, relay, collector, detector, writer, fault injector, fault plans and checker, all standard-library Python plus Ollama. It is the code exactly as it ran on these nights, not a cleaned-up version: the files on both machines match the repository byte for byte. It also has the raw heartbeats of the two fault nights; replaying the detector on them gives exactly the same JSON, and the writer can be replayed on any of the nights.

I built it with Claude Code as a pair: I decided what to measure, what the lab could and could not be put through, and what the report had to say; Claude wrote most of the code and ran it on the machines with me.

How I built it

Why the model does not read the logs

The first version was the obvious one: give Gemma the last 40 warnings from the N100's journal and ask for two lines. I gave the same 40 to both machines.

Both took about 2 minutes, wrote a paragraph instead of two lines, and described the desktop's background noise (GNOME, PipeWire, a keyring) as “system instability”; one recommended investigating it. On a desktop machine most warnings are not failures, and the model has no way to know which ones are.

So the job got split. Rules decide what happened; the model only puts it into words. The detector is under 200 lines. And to know whether the rules were right, I needed nights where I knew exactly what had failed and when.

Three fields that make a gap readable

The sequence number, the process start time and the machine's own count of what it failed to send. A new start time after a gap means the process restarted. The same process with missing sequence numbers means heartbeats went missing. And if the machine itself counted them as unsent, they never left; if it did not, they left and never arrived. Without those three fields, all of these look like the same hole in the chart.

Nights with known faults

Two quiet nights first, with no faults, as a baseline. Then two nights with eight controlled faults each, injected by a script at fixed times and logged with their real start and end; the second night moved each fault to the other machine and to another hour:

Fault	What it does	What the detector should say
Interruption	kills the heartbeat process; a watchdog restarts it 30 s later	service restarted
Silence	the process stays alive but stops sending	the machine did not send
Cut	the relay drops that machine's heartbeats	heartbeats lost on the way
Latency	the relay delays them 3–5 s	latency
CPU	four busy processes on four cores	CPU saturation
Memory	holds memory until 15 % is left	low memory
Pause	stops the process until after the window closes	no heartbeat, unresolved

Most faults touch only the experimental flow. CPU and memory put bounded load on the machine itself: the CPU fault stops at a temperature limit, and the memory fault never leaves less than 600 MB. No reboots, no changes to the machines' real network, no production service stopped.

A checker for the writer

A script compares each report with the JSON it was given. It looks for incidents left out, times and figures that are not in the JSON, causes stated as confirmed, adjectives like “high temperature” with no incident behind them, incidents invented on a quiet night, and a missing or invented comparison with the previous night. The real fault log grades the detector, not the report.

The checker runs on my laptop, not in the lab, and it misses things. Most of what is wrong in the demo above I found by reading the reports.

Results: detection

Night	Faults injected	Detected	Detection delay
Quiet 1	0	0	—
Quiet 2	0	0	—
Faults, night 1	8	8	21–30 s; CPU 111 s
Faults, night 2	8	8	20–30 s; CPU 112 s

On the quiet nights, 3,060 of 3,060 heartbeats arrived from each machine. With faults, the rules found all sixteen, each with the right type. The last fault of each night, a pause that runs past 6:30, came out as unresolved, which is correct.

The second night also had an incident nobody planned. The CPU fault pushed the N100 to 84 °C, over the 80 °C rule, for 50 heartbeats, and the detector reported high temperature. My checker counted it as a false alarm because it was not in the plan, but the temperature really did cross the line: it is a side effect of the fault, and the rule was right to catch it.

The detector runs once, at 6:30, so “detected” is when the rule would have fired, reconstructed from the data, not a live alert. The CPU fault shows up late because the one-minute load average needs about a minute and a half to pass 3.0 (75 % of four cores), plus 20 seconds for the rule's three heartbeats in a row.

The detection is boring, and it should be.

Results: what Gemma wrote

On the first quiet night, both machines invented incidents. The JSON said zero incidents. Each report listed one per machine anyway, built from that machine's peak temperature (73 °C on the N100, 50.1 °C on the Pi, no rule fired), gave them a duration of “00:30” that appears nowhere in the data, and recommended checking the temperature or the ventilation. A quiet night is exactly when a summarizer is tempted to find something to say.

On the second quiet night, the Pi correctly wrote that there were no incidents, but left out the comparison with the previous night, which was in the JSON. The N100 included the comparison but listed normal readings under “Incidents”. And both put the start of the night in the title (2026-10-20 22:00) instead of the morning date the template asks for.

On the first fault night, it got the structure and most of the numbers right. Neither machine left out an incident or stated a cause as confirmed, and the checker found no time or figure that is not in the JSON. But the checker only looks for numbers that appear nowhere in the JSON, and that is how the worst error got through:

The Pi invented a fact with a real number. For the last incident, the pause still open at 6:30, it wrote that “the machine itself logged 30 heartbeats as unsent in that span”. That sentence belongs to the silence incident six hours earlier. The JSON gave this incident two facts, and the Pi dropped one of them: that no heartbeat had arrived by the end of the window. It could not even be true, because the pause kills the heartbeat process, so nothing was counting. The 30 passed the checker because 30 appears elsewhere in the JSON.
The health line misreads the numbers. 99.9 % and 87.2 % are the share of each machine's own heartbeats that arrived, not where the heartbeats came from.
The possible explanations contradict the evidence. The Pi logged its 30 heartbeats as unsent, so they never reached the network; the real cause was the silence fault, and “network problem” is ruled out by the facts two lines above. The N100's process was stopped with a normal termination signal (code -15), which is not an operating system failure.
“Not confirmed” where the template asks whether it recovered on its own. The JSON said it did.

The checker did catch one: the Pi skipped the comparison again, and this time so did the N100, writing that there was no previous night to compare while the JSON carried its figures. The rest is smaller: the N100 reduced each incident to one generic line (“The N100 restarted the service.”) and dropped the evidence, the Pi wrote “null” for the end of the incident still open at 6:30, and both titles carried the start of the night again.

On the second fault night, both machines swapped two incidents between machines. The 22:29 interruption was the Pi's, and both reports put it under the N100; the 23:20 latency was the N100's, and both put it under the Pi. Same mistake on x86 and on ARM, and the checker cannot see it, because both machine names are in the JSON. The N100's health line also gave the Pi 87.2 %, which was the previous night's figure, copied from the comparison block; that night the Pi had 98.5 %. It wrote 23:40:22 as the detection time of an incident detected at 00:40:22, which the checker did catch. And both skipped the comparison once more: three nights in a row for the Pi.

Same model, same JSON, same prompt, temperature 0 and seed 42 on both machines, and the text still came out different on each. At temperature 0 the seed barely matters; what differs is the arithmetic. The x86 and ARM builds round slightly differently, and one different token changes everything after it. I did not test repeat runs on the same machine, so I cannot say that pinning the hardware is enough, only that pinning the model is not.

N100 or Pi 5?

	N100	Raspberry Pi 5
Report with 8 incidents	294.8 s	468.8 s
Reading the prompt (2,565 tokens)	20.7 tokens/s	26.3 tokens/s
Writing the report	5.7 tokens/s (934 tokens)	3.3 tokens/s (1,199 tokens)
Report on a quiet night	86–128 s	76–101 s

On the second fault night it was 303.4 s on the N100 and 520.9 s on the Pi. The Pi reads faster and writes slower, and with faults it also writes more, because it copies the evidence that the N100 drops. Three tokens per second sounds slow, and at 6:40 a.m. it does not matter: nobody is waiting, and both reports are ready well before 7. Slow and steady is enough for a night shift.

Why does open innovation matter?

The logs never leave the lab. Ollama listens on 127.0.0.1. Nothing about my machines goes to anyone else's server, and the night shift keeps working if the internet does not.
It costs nothing per night. No tokens to pay for, no key to rotate. It runs on hardware that was already on.
I can pin it. The model is gemma3:4b, id 68ceab7e468a, Q4_K_M, with temperature 0, seed 42 and an 8,192-token context, on Ollama 0.40.6. A hosted model can change under you between one night and the next; this one only changes when I change it. That matters when the point is to measure it.
I can run the same model on x86 and on ARM and see the difference, which is how I found that the same settings do not give the same text on two processors.

To be precise about “open”: Gemma's weights are open under the Gemma Terms of Use, not an OSI license; Ollama is MIT. What matters here is that both run entirely on my machines.

The lazy payoff

Nobody has to open logs in the morning: read the health line, and if it is not “good”, the incidents under it. The health word comes from the detector's JSON; Gemma copied it correctly every night, but that is Gemma copying, not a guarantee, so the next change is to have the detector write that line itself.

What I do not do is trust the report alone. The rules decide, I run the checker over what the model wrote, and its explanations are treated as guesses, because the fault night showed they can contradict the facts right above them. It has run for five nights, not months, and that is what five nights showed.

Prize categories

Best Use of Gemma: Gemma 3 4B runs locally in Ollama on both machines and writes every morning report, and this post measures where it gets things right and wrong.

Your turn

Where would you let a local model write without anyone reviewing it, and where would you not?

Israel Negrete Lepe is an electronics engineer. Twenty-five years making different systems work together under real conditions: from the sensor, the vehicle or the transaction to the backend, the data, security and operations. Today, increasingly, edge AI.

The right position at the wrong time

Israel Negrete Lepe — Fri, 23 Oct 2026 15:00:00 +0000

A truck loses its signal at ten in the morning and gets it back at noon. As soon as it is back, it sends two things at once: the positions it stored with no signal and the ones it is generating right then. All of them are correct. If the platform does not tell them apart, it mixes history with the present.

A position can be technically correct and still lead to the wrong operational decision if it is read outside its time context. That is what this note is about.

We have worked for years with GPS tracking, telemetry and embedded devices, over GSM/GPRS cellular networks and over LoRaWAN. On the move, you cannot assume the connection will always be there. What matters is not only how long a device is offline, but how it behaves meanwhile and what happens when it comes back.

The figures in the examples are illustrative: they come from a simulator with an assumed working day of a fleet, not from measurements of our platforms.

It is not a device connected to a database

The simplest way to think about telemetry is a device that sends data and a database that stores it. With that picture, store-and-forward looks like a device problem: give it enough memory and have it send what it stored.

Our evolution has been to stop thinking about it that way. Today we see it as a distributed system where each stage has its own job: receive, acknowledge, transport, store, interpret and present. The problem does not end when the device manages to resend its data. What the platform does with it matters too.

On the device side, there is no general rule either. How it stores, how much it holds and what it does when it fills up depend on the manufacturer, the firmware and the model. Some store and send later; others have memory limits or settings of their own. It has to be checked device by device.

Three different times for the same data point

Every position has three moments:

When it was generated, on the device.
When the server received it.
When it became available for operations, already processed.

On a day without failures, the three nearly coincide. After a disconnection, they drift apart: a position generated at 11:30 can arrive at 12:00 and, with the server's queue full, become available up to four minutes later.

The device clock deserves attention too. With GPS, the device can take the time from the positioning system, but not every device keeps reliable time under every condition. And if the event time is not reliable, nothing built on top of it is either.

Acknowledging is not processing

When a device gets its signal back, there is no guarantee that everything arrives, in order and without duplicates. That depends on the protocol and on how it was implemented. In TCP integrations with protocols such as JT808, you need to understand their sequence, acknowledgment and retransmission mechanisms.

And there is a critical difference: the gateway acknowledging a message does not mean the whole platform has already processed and stored it. If an acknowledgment is lost, the device sends the same thing again, and the server has to accept the duplicate without creating duplicate records or events.

Recognizing a duplicate is not trivial either. Matching device and time is not enough. Neither is the sequence number: in JT808 it is two bytes and wraps back to zero, so it repeats. The right approach is idempotency per protocol and event type: a unique message identifier when there is one and it is reliable, and composite keys when needed.

Some protocols help from the moment of reception. In JT808, the batch position upload, message 0x0704, says whether the positions are normal ones or were stored in an area with no coverage. With that flag, the gateway can recognize history as soon as it arrives. The flag applies to the whole batch, and some devices send stored positions as normal ones, without it. Whether each device model and firmware version uses it is something to verify, not assume.

Accepting connections is not processing them

Receiving one position from each device at regular intervals is not the same as receiving, all at once, the backlog of many devices while their current positions keep coming in. Accepting TCP connections is one capacity; processing and storing everything received is another.

In the simulator, a fleet of 50 devices sends one position a minute for eight hours. At 10:00 the network goes down for 41 of them, for two hours. Each device holds up to 100 positions and, when full, drops the oldest. Once back, it sends the present without waiting and resends what it stored at two records a second, and 2 % of the acknowledgments are lost. The platform processes 10 messages a second, twelve times the fleet's normal load.

The signal returns at 12:00, and in less than a minute the server's queue reaches 3,634 messages; the wait, 6.1 minutes. In that queue, 63 current positions waited more than five minutes behind the fleet's accumulated history. Along the way, 843 positions were lost because memory filled up, 436 duplicates arrived from lost acknowledgments, and 1,728 arrivals were out of order.

More memory does not fix everything. On the same day, with 500 positions per device nothing is lost, but the avalanche grows: the queue reaches 4,406 messages and 139 current positions wait more than five minutes.

History or present

The most delicate part is not only losing data. It is receiving valid data outside its time context.

If the platform treats everything that arrives as if it were happening now, history mixes with the present, and that hits routes, reports, geofences, alarms and indicators. With a geofence it is very clear: if a vehicle entered a restricted zone three hours ago and the position is only now reaching the server, an alert saying it has just entered makes no sense.

In the simulator there are 194 real geofence entries and exits during the day. A platform that treats everything as "now" raises 240 alerts: 140 on time, 54 about things that happened a while ago raised as if they were happening at that moment, and 46 false ones, from history and duplicates mixing with the present and making the device seem to go in and out. It also stores the 436 duplicates. Those false alerts depend on how the device resends: if everything goes out in order, they drop to 30; with no lost acknowledgments, to 16; with both, to zero.

A platform that separates history and present drops the duplicates, stores the history in its place and only alerts on the present. It raises 156 alerts: 140 on time and 16 late ones that say they are late. No false ones, though that is by construction: the simulator only carries the failures we put in it. And the route, rebuilt with each position's time, recovers 186 of the 194 entries and exits correctly timed; the 8 missing ones had their records wiped by the device's full memory, and no platform can fix that.

What I would do today

Treat disconnection as a normal condition, not an exception. The design starts by assuming the connection will fail.

Split the responsibilities into layers:

Reception: accept, validate and identify the device's messages.
Messaging: decouple communication from processing with queues.
Persistence: store idempotently, without duplicates.
Temporal processing: tell historical from current.
Business rules: evaluate geofences, alarms and operating conditions while respecting the time context.
Presentation: show both the last known state and the history correctly.

Store the event time and the reception time separately. A historical position keeps its original time, if it is valid, and reception is recorded apart. That way the route can be rebuilt without altering the chronology.

Tell a current condition from a historical event detected later. History rebuilds the route; the present triggers alerts. If something happened with no signal and is discovered on reconnection, it is recorded as what it is.

Measure where it gets stuck. Separating reception, queue and processing makes it possible to see at which point in the flow a delay happens, instead of guessing.

In our current architectures we work with reception gateways, RabbitMQ, processing services, databases and event distribution over Socket.IO. That decouples reception from processing, although on its own it does not guarantee that a device kept everything during the disconnection.

To try it with another fleet, another network outage, another memory size or another processing rate, there is the store-and-forward simulator: it runs in the browser and compares both platforms with the three times of each data point.

Receiving information is not the same as receiving timely information, and processing a message correctly does not guarantee we are reading it in the right time context. A good telemetry platform does not just survive devices going offline: it recovers their history without confusing it with the present.

Your turn

When a device reconnects, how does your platform tell history from the present: by a flag the device sends, by the event time, or not at all? And does a late geofence entry still raise an alert as if it were happening now? If you want to try your own fleet and outage, the simulator is at crtdrops.mx/reenvio.

The geofence computed on the fly

Israel Negrete Lepe — Tue, 20 Oct 2026 15:00:00 +0000

A truck spends the week between the yard, a customer and an underground lot. The report says that twelve times, with the engine off and nobody touching it, it left the yard and came back in.

For years we used geofences for almost everything: yard entries and exits, no-go zones, customer arrivals, routes. Circles and polygons; polygons were a more interesting challenge than a plain circle. And for years we fought the same failures: alerts that matched nothing, arrivals that never showed up, times that did not add up.

This note is about those failures, about the fixes almost everyone applies, and about a design mistake that stays hidden until someone changes a value: computing the geofence on the fly.

The figures in the examples are illustrative: they come from a simulator with a typical week of a truck, not from measurements of those fleets.

In or out looks like an easy question

For a circle, one distance is enough: if the position is closer to the center than the radius, it is inside. For a polygon, the classic test draws a line from the position off to one side and counts how many times it crosses the edge: odd, inside; even, outside. An L-shaped yard, a zone that follows an avenue, a customer with two gates: all of that is a polygon.

The geometry is the easy part. The hard part is that the question is put to a GPS position, and the position does not always say where the truck is.

What failed

Parked, the position jumps farther than the geofence. With the truck standing still, accuracy sometimes falls apart and a position lands a hundred or two hundred meters away. The simulator's yard is eighty meters across. To the geofence that is an exit, and the next good position is an entry. In the history, the truck seems to cover large distances in a few minutes without having moved.

Positions that arrive late. A 7:10 position that arrives at 7:30, after the 7:20 ones, puts the truck back where it no longer was. If the geofence is evaluated in arrival order, that is one more entry and one more exit that never happened.

The device stops transmitting right before it arrives. In an underground parking lot the signal is lost on the ramp. The last position stays on the street and the next one comes in once the truck has already left. The entry never shows up.

And the interval makes all of it worse. With the ignition on, our devices reported every 3 minutes; off, every 120. At 40 km/h, the truck covers 2 km in 3 minutes. A 400 m no-go zone is crossed in 36 seconds, so only one crossing in five leaves a position inside.

In the simulator, one week of that truck: 205 reports, 70 real entries and exits. Deciding with the raw point, position by position, gives 45 false alerts: 31 with the truck parked in the yard, and 14 from positions that arrived late. It sees 3 of the 14 zone crossings and none of the 7 entries into the underground lot.

A bit of all three

The fixes we applied, and that almost everyone running a fleet applies, were a bit of three things:

A margin to exit. The geofence is entered at the edge, but to exit the truck has to get a certain distance away. That removes the dithering of a truck parked by the fence. Against a two-hundred-meter jump it does nothing: with a 30 m margin, the 45 false alerts drop to 39.
Waiting for several reports in a row. A lone jump confirms nothing. With two positions in a row, false alerts drop to 13. But with the ignition off each position comes two hours after the previous one: the seven returns to the yard are reported two to six hours late, and seven of those 13 are real arrivals at the wrong time. And a single late position in between breaks the streak: it sees 3 of the 7 customer arrivals.
Using the ignition or the speed. With the engine off, the truck does not move. That removes almost all the yard ones; 21 remain, mostly from late positions and from the stretch when the engine warms up with the truck parked.

Each one helps, and each one is paid for somewhere else. I would sum up what we learned like this: any intervention or change of values eventually causes false positives or altered information.

On the fly

And the worst part was none of those. On that platform, a geofence entry or exit was not stored: it was computed in the web client, on the fly, every time someone opened the map or the report.

As long as nobody touches anything, that seems to work. The problem arrives with the first complaint. Someone enlarges the yard so the alerts stop, or raises the margin, or changes the customer's polygon. From that moment on, last week's report is recomputed with the new geofence and says something else.

In the simulator, enlarging the yard by 100 m brings the false alerts from 45 down to 31: it does not silence them, because the late positions and the longest jumps are still there. And in that same week's report, computed on the fly with the new yard, 23 alerts change: 16 disappear, 3 move to a different time and 4 appear that were not there before. If someone asked on Tuesday about a yard alert, the one they were shown may no longer be there, or may be at another time.

A history that changes when the configuration changes is not a history.

What I would do today

Store the event, once. The entry or exit is computed when the position arrives and stored, with the version of the geofence and that day's values. If the yard grows tomorrow, it grows from tomorrow on.

Stamp it with the position's time. Not the time it arrived. A late position is slotted into its place in the trip, and the alert carries the time it happened, even if it is sent later. If that position changes an event already stored, the event is corrected and the record says it was corrected and why: a late data point explains the change; a new configuration does not.

Drop the position, not the report. A position flagged as bad by its own quality data is discarded. But the report still says the device spoke and, if it is the ignition-off report, when the truck stopped.

Confirm with more than one signal. Two positions in a row, or the ignition-off report, or a position whose GPS speed is above 15 km/h. On most devices speed is measured separately, from the Doppler effect, and barely jumps with the position: a truck doing 40 km/h is not jumping. With the ignition off, a jump does not take the truck anywhere; if the GPS says it is moving, that is a different alert, the tow alert.

Check the segment, not just the point. If the straight line between two positions goes through the no-go zone, the truck most likely crossed it, even if no report fell inside. The time is interpolated along the segment, and the crossing is stored as inferred. With real streets, the line is checked against the street layout.

Three states, not two. Inside, outside and unknown. When the device goes quiet near a place known to have no signal, the entry is inferred, and stored marked as inferred. Whoever reads the report knows what was seen and what was deduced.

With as much of that as the simulator can model, in the same week: no false alerts, all 70 entries and exits, 36 of them inferred and marked as such (the zone crossings and the underground lot), and the time 1.1 minutes off on average. Not because it is perfect: the simulator only carries the failures I put in it, and the street has others. But each of those rules answers something that fails.

To try it with another week, with more bad or late positions, another margin or a bigger yard, there is the geofence simulator: it runs in the browser and decides in or out five ways.

Geofences did not fail because of the geometry. They failed because a lone position was asked something only the truck's history can answer, and because that answer was not stored.

Your turn

Where does your platform decide in or out: when the position arrives, or every time someone opens the map? And if someone changed a geofence today, would last month's report change with it? If you want to try your own margin and rules on a full week, the simulator is at crtdrops.mx/geocerca.

The top-up nobody knows went through

Israel Negrete Lepe — Fri, 16 Oct 2026 15:00:00 +0000

A customer asks for a prepaid airtime top-up at a corner store. The system debits the store's balance and sends the request to the carrier. The carrier does not answer.

Did the top-up reach the phone? At that moment nobody knows. And with the carrier, nobody can ask: the answer arrives tomorrow, in the reconciliation.

For years I worked on an electronic top-up platform. Requests came in from terminals, apps and points of sale, and reached the carrier after two, three and up to four hops, each with its own protocol and its own timeout. Everything happened: double top-ups, charges with no top-up. This note is about why, and about the state almost no system designs in from the start.

The figures in the examples are illustrative: calculated to show the mechanism. They are not that platform's numbers.

There are three answers, not two

A system that moves money is usually thought of with two outcomes: the operation went through, or it did not. On a network there is a third: unknown. The request went out and the answer never came back. The request may have been lost, and the top-up does not exist. Or the answer may have been lost, and the top-up is already on the phone.

From this side of the wire, the two cases look identical.

Every hop is one more place where an answer can get lost. If each one loses one in two thousand, with a hundred thousand top-ups a day, one hop leaves 50 in doubt; four hops, 200. Every day. It is not a bug you fix once: it is a probability you manage forever.

And the hardest case is at the end of the chain: the carrier does not answer, or its answer gets lost, but the internal system has already debited.

Everyone decides what to do with "unknown"

When the screen hangs, somebody decides, and every decision is a bet with someone else's money:

The cashier presses the button again. If the first one did go through, the customer gets two top-ups and the store pays for two.
The terminal retries on its own. Same thing, without anyone noticing.
The platform retries toward the carrier. Same thing, one hop deeper.
The platform marks the operation as failed and refunds the store. If the carrier did apply it, the platform gave away a top-up.
The platform marks it as good. If the carrier did not apply it, the customer paid and got nothing, and is going to complain.

None of them is right. They are all bets as long as nobody knows what happened.

An ID that travels end to end

The first defense is for every top-up to have a unique identifier from the moment it is born, in the terminal or the app, and for that same identifier to travel through every hop. We called it the externalID.

With it, a retry stops being a new top-up. If a request arrives with an externalID that was already processed, the system does not run it again: it answers with the result of the first one. The cashier can press ten times and the store pays once.

And it makes it possible to ask. Backwards, from the platform toward the points of sale, you could: the terminal could ask what happened to its externalID instead of sending another top-up.

What you cannot ask

With the carriers, you could not. There was no status query. The only thing was batch reconciliation: a file every 24 hours with what the carrier says it applied, against what the platform says it sent.

That means a top-up in doubt stays in doubt until the end of the day, or until the next day. Meanwhile, the store's balance has already been debited, the customer may be complaining, and the platform has nothing to tell them.

And reconciliation, when it comes, is torture: cross-matching files, finding the differences, deciding each one, and sometimes resolving them by hand.

Why not reverse

The way out that looks obvious is the reversal: when in doubt, cancel the operation with the carrier and return the money. In this business reversals are not desirable: they kill the business.

The margin on a top-up is very small. Every reversal costs work, time and arguments with the carrier, and eats the profit of many sales. And if you reverse something that did go through, the loss is total.

What I would do from the start

Design "unknown" as a real state. In the database, on the cashier's screen and in the reports: in progress, not failed or successful. A screen that says "in progress, do not repeat it" is worth more than any later validation.

The externalID from the origin, mandatory, and honored at every hop. A request with no identifier does not get in. A repeated identifier returns the stored result and never runs again.

No human retries blindly. The retry button checks the status first. If there is no answer, it shows that the operation is in progress; it does not send another.

Staggered timeouts. Each hop has to wait longer than the next one. If the terminal gives up before the platform does, the terminal retries while the first request is still alive inside.

Automatic reconciliation from day one. Not as a spreadsheet someone puts together once there are problems, but as part of the system: the carrier's file comes in, gets matched on its own, and a person only gets the differences.

Measure the doubts. How many top-ups stay in progress, per carrier, per hour and per hop. When that number goes up, something broke, and it is better to know today than in tomorrow's reconciliation.

To see what each way of deciding costs with your own numbers (top-ups a day, hops, timeouts), there is the top-up simulator: it runs in the browser and counts the doubled, given-away and charged-without-top-up operations of each policy, and what waits for reconciliation with an externalID.

None of these measures eliminates "unknown". What they do is stop it from being a bet: the money sits in a state everyone can see, nobody duplicates it out of desperation, and reconciliation only has to resolve what truly could not be known. Mitigating it was a lot of programming and infrastructure work. There was no shortcut.

Your turn

How does your system handle an operation stuck in "unknown": do retries carry an idempotency key end to end, or does someone decide by hand? If you want to try your own hops and timeouts, the simulator is at crtdrops.mx/recarga.

What failed in the field that no lab test would have caught?

Israel Negrete Lepe — Thu, 15 Oct 2026 15:00:00 +0000

I have spent 25 years taking telemetry and tracking systems from the lab to the field, and the failures that cost the most were never the ones we tested for.

A battery that was supposed to last ten years and would not make five, because the real wake-ups were longer than the math. A parked truck that kept adding kilometers, because a stationary GPS is never quite stationary. A Raspberry Pi that slowed down while Linux swore it was running at full speed.

None of them showed up on the bench. All of them showed up in production.

What is yours? The failure that only appeared once the device was out there: in a cabinet, on a pole, in a truck, in the sun. What gave it away, and what do you test for now that you did not before?

The parked truck that kept racking up kilometers

Israel Negrete Lepe — Tue, 13 Oct 2026 15:00:00 +0000

A truck spends the whole night in the yard. Nobody touches it. In the morning, the tracking platform says it drove thirteen kilometers.

For years we ran tracking platforms with devices from several manufacturers and with our own designs. They all reported more or less the same things: position, GPS speed, number of satellites and fix quality, ignition input and, on the better ones, an accelerometer. And all of them, sooner or later, had parked trucks that moved.

This note is about why that happens, about the filter almost everyone adds to make it stop, and about what that filter breaks without anyone noticing.

The figures in the examples are illustrative: typical values, calculated to show the mechanism. They are not measurements from those fleets.

A GPS at rest does not always give the same point

Every position a GPS reports carries an error of a few meters. Under open sky and with many satellites, the error is small. Under a sheet-metal roof, between buildings, in a parking garage or with few satellites in view, it grows: the signal arrives bounced, the satellite geometry is poor, and the position jumps ten, twenty or fifty meters while nothing moves.

On top of that come the gaps. An area with no cellular coverage makes the device store positions and send them late, all at once. A tunnel or a basement makes it lose the fix, and when it gets it back it appears somewhere else, with a straight line between the last good point and the first new one.

To the platform, all of that is movement.

Adding up points never gave the distance

The obvious way to compute how far a vehicle went is to add up the distance between each position and the next. In our experience, that figure was never reliable. And it fails in both directions.

Parked, it always adds too much. GPS error does not cancel out when you add it up: a distance is never negative, so every jump, in whatever direction, is added to the total. A device at rest for ten hours, with a position every thirty seconds and a typical error of about six meters, gives 1,200 jumps of about eleven meters: about thirteen kilometers without the wheels turning once.

Moving, it adds too little. Between two positions, the calculation draws a straight line. A ninety-degree curve driven between two points is counted as its chord, ten percent shorter than the real path. With spaced-out positions, in a city full of turns, the real route is always longer than the computed one. And across a coverage gap, the straight line cuts through whole blocks.

The two failures do not offset each other: they depend on how much time each unit spends parked and how much on the road, and that changes from one vehicle to another and from one day to the next.

The filter everyone adds

The fix we applied, and the one almost anyone running a fleet applies, is to clip the speed:

Below 5 km/h, the speed becomes zero and the point does not move. The parked truck stops jumping.
Above 130 km/h, it is clipped too. For most fleets that speed is unlikely; if the GPS reports it, it is most likely a jump.

The dashboard comes out clean. Phantom kilometers drop, false alerts drop, and customers stop calling.

But it cuts both ways.

What the filter hides

Real movement below the threshold. A truck crawling in traffic, one maneuvering in a yard, one moving slowly to line up at a loading dock. And the worst case: a truck being towed with the ignition off, which is exactly what the platform should catch. With the filter, all of that is zero. It happened to us many times.

Real speeds above the threshold. If the unit really was doing 140, the clip hides a speeding event the customer wanted to see. And if it was a jump, it now shows up as a believable 130 in the middle of the trip.

The trip stops being trustworthy. That is the underlying cost. Once the filter has changed the data on arrival, there is no way to know what really happened. The filter did not remove the error: it moved it somewhere it can no longer be seen.

What I would do today

Decide with several signals, not one threshold. GPS speed alone cannot tell a jump from slow movement. Ignition, accelerometer, number of satellites and fix quality, together, can: with the ignition off, no vibration and poor fix quality, a twenty-meter jump is noise. With sustained vibration and the position drifting away in one direction, it is movement, even at 3 km/h and even with the ignition off. That combination is the towing alert.

Anchor the point, do not erase the speed. When the unit parks, pin its position and ignore jumps within a radius. If the position leaves that radius and stays out, that is real movement, and it gets reported.

Keep the raw data. The filter belongs in the layer that displays, not the one that receives. If the criteria change tomorrow, or a customer asks what happened on Tuesday at three in the morning, the original data is still there to recompute.

Separate "zero" from "unknown". A speed clipped to zero says the vehicle was stopped. A position with poor fix quality should say nobody knows. They are not the same, and the report should not mix them.

Do not measure distance with GPS when there is something better. If the vehicle provides its odometer over the data bus, that is the number. If there is only GPS, add up only what was covered while moving and with good fix quality, and say so in the report.

To see it over a full night, with the truck parked, a slow maneuver, a tow with the ignition off and the morning trip, there is the parked truck simulator: it runs in the browser and counts the kilometers three ways: adding up points, with the clip and with the anchor.

Today devices and networks are much better than when we started, and the jumps are fewer. But a GPS under a sheet-metal roof still cannot see the sky, and a clipped speed still does not say what happened. What changed is the amount of error, not the decision about where to filter it.

Your turn

How does your platform tell a GPS jump from slow real movement: a speed threshold, ignition, the accelerometer, or something else? If you want to try your own thresholds on a full night, the simulator is at crtdrops.mx/gps.

Why your telemetry platform loses frames and nobody notices

Israel Negrete Lepe — Sun, 11 Oct 2026 15:00:00 +0000

There is an assumption that almost every backend receiving devices makes without knowing it makes it. It survives every test that occurs to whoever wrote it, and it can be stated in one line:

Every time I read from the socket, exactly one frame arrives.

It is false. Not sometimes, not under rare conditions: it is false by definition of the transport. And yet a system that assumes it can run for months without anyone noticing, because the conditions that disprove it are precisely the ones that do not happen on a bench.

What TCP promises and what it does not

TCP delivers a sequence of bytes, in order and without loss. That is all it promises.

It does not promise to preserve the boundaries of what the sender wrote. If a device makes two writes of twenty-nine bytes, the receiver has no way of knowing, from the transport, that there were two and not one of fifty-eight. Message boundaries do not exist in TCP. They exist in the protocol that travels on top, and reconstructing them is the receiver's job.

A read returns whatever is available at that instant. And what is available depends on things neither the sender nor the receiver controls: how the stream was segmented, whether two small writes traveled in the same segment, whether a segment arrived before the receiver read the previous one, whether the link retransmitted.

With a slow sender and an idle receiver, each read usually brings exactly one write. That is the lab condition. It is the exception, not the rule.

Why the assumption survives every test

A device on the bench transmits every ten seconds over a local link. The receiver is idle, it reads as soon as something arrives, and what arrives is a whole frame, because nothing split it and nothing had time to stick behind it. The test passes. Repeat it a hundred times and it passes a hundred times.

None of that happens in the field.

A device in a vehicle transmits over a cellular link, with variable latency and intermittent coverage. When it loses the link it does not discard what it had to send: it stores it. When the link comes back, it sends everything it stored back to back, one frame after another, with no pause. On the backend's side that arrives as a single block of bytes with six, ten or forty consecutive frames, or with the last one cut in half because the segment ended there.

And nothing has to go down for this to happen. A link with high latency and small writes tends to group them: the transport waits a little before sending, for efficiency, and two frames two hundred milliseconds apart arrive together. Conversely, a frame longer than the maximum segment size arrives in two pieces by construction.

The assumption does not fail through bad luck. It fails as soon as the traffic looks like real traffic, and that is the first thing that happens when the fleet leaves the lab.

The three things a parser that assumes it will do

The usual summary, "frames get lost", hides what matters, which is which ones and how. A backend that treats each read as a frame does one of these three things when what it read does not measure what it expected.

It discards it. It checks the length, it does not match, it ignores it. This is the most common variant and the worst, because it is silent. No error, no close, no log entry. The forty frames the device stored during the loss of coverage disappear, and the device takes them as delivered. Weeks later someone notices gaps in a history and there is no longer any way to reconstruct why.

It closes the connection. It reads the unexpected size as an invalid protocol and cuts off. This is noisier and therefore more benign: the device reconnects and the logs keep a close that someone may eventually look at. But if the device resends the same thing on reconnecting, the cycle repeats until by chance the read matches a frame.

It processes it anyway. It takes the first bytes as a header, reads the fields where it expects to find them, and carries on. If what it read was a frame and a half, the first comes out right and the second is lost. If it was half a frame, the fields come out as garbage. And if the parser never resynchronizes, every subsequent read interprets payload bytes as if they were a header, and the system records positions, speeds and states that no device ever emitted.

That last case is what a checksum is there to catch. But that presupposes the backend validates before processing, which is another assumption worth not taking on trust.

The fix is not sophisticated

Accumulate. What arrives is appended to a buffer, and complete frames are extracted from the buffer according to what the protocol lets you know.

If frames are fixed length, extract every time the buffer reaches that length and let the remainder wait. If they end in a delimiter, extract up to the delimiter. If they carry a length field, read that field, wait until you have that many bytes, and extract.

In all three cases the rule is the same: the buffer decides when there is a frame, and the read only contributes bytes.

A receiver built this way cannot tell the difference between a frame that arrived whole, one that arrived in two pieces half a second apart, and forty that arrived stuck together. To it they are bytes coming in and frames going out.

How it looks from outside, when you have no access to the backend

A device cannot see the buffer on the other side. It only sees what it is answered.

If the protocol acknowledges, a frame silently discarded is a frame with no reply. A frame that triggered a close is a connection that drops without the device closing it. And a frame processed halfway is not visible from outside at all: the acknowledgment arrives, and what was stored on the other side is wrong.

Of the three failure modes, the one that does the least damage is the only one detected with certainty from the client's side. Of the other two, the discarded frame leaves a lead (it gets no acknowledgment), and that lead is either confirmed in the backend's logs or it is not; the half-processed one leaves none. The two are worth keeping apart: what you observe from outside is half of what happens.

Who this happens to

It would be comfortable to present this as a beginner's mistake. It is not.

I wrote a tool to provoke exactly this defect in other people's backends: a generator that emits frames, splits them, coalesces them and counts the replies. Its own client had the bug. It counted one acknowledgment per read. When the backend replied twice in a row and both replies arrived in the same segment, the read brought four bytes, the client counted one acknowledgment, and gave the second up for lost.

The backend was fine. The instrument judging it was not.

I found it because I ran the tool against a receiver that behaved correctly and refused to believe the result. That is the only reason I can write about this today without feeling like a fraud.

Your turn

What is the strangest frame-boundary bug you have hit in a telemetry or industrial protocol? Mine are in the note; I would like to hear yours.

If you operate or build a platform that receives trackers, data loggers or meters over TCP, there is a good chance this describes a bug your platform has today and nobody has seen yet. The way to find out is not to read the code: it is to produce the traffic that disproves the assumption and watch what happens.

See it happen: the in-browser demo on crtdrops.mx feeds the same byte stream to two receivers, one that treats each read as a frame and one that buffers. Split the frames, simulate a reconnection burst, and watch the counters.

What you have just read is chapter 4 of a guide. If you want to pass it on to your team, the complete chapter is free as a PDF, with the diagram that does not fit here:

israelnegretelepe.gumroad.com/l/a-read-is-not-a-frame

And if what you want is to produce that traffic against your own platform, the whole guide comes with the software that generates it: twenty-three chapters, eight appendices, and a virtual fleet generator that speaks your devices' exact protocol.

Virtual Fleets — israelnegretelepe.gumroad.com/l/virtual-fleets

The throttling your system does not show you

Israel Negrete Lepe — Fri, 09 Oct 2026 15:00:00 +0000

In the field I have seen SD cards damaged by heat, and Raspberry Pis that freeze up when the heat builds, especially when the heat of the place is added to the heat of the processing. In the lab, on the other hand, the Pi had never throttled: in the Round 1 tests the highest temperature was 65 °C.

But those tests lasted between three and thirty-one seconds. So I put the Pi and the N100 to work for seventy-two minutes without a break, recording everything, to see what happens when the model does not rest.

The test

Both devices with their usual cooling: the 8 GB Raspberry Pi 5 with a heatsink and fan inside a metal case, and the 8 GB Intel N100 in its mini PC with a fan. A ventilated room, no direct sun. Both freshly rebooted (the N100 still warm from another measurement: details below).

The model, YOLO26n in FP32 with ONNX Runtime, running non-stop in four phases: two minutes idle, thirty with one process using all four cores, ten cooling down, and thirty with four processes of one core each. Every inference is recorded, and every second, the temperature, the frequency the system reports and, on the Pi, two more things only its firmware knows: the real processor clock and the flags it uses to say it is throttling.

What happened on the Pi

It enters the throttling zone in eight minutes. The Pi 5 starts slowing its cores down between 80 and 85 °C. It reached 80 °C at minute 8, the Pi itself flagged its limit as active at minute 10.6, and it stayed there until the end of the phase, peaking at 85.1 °C.

The system says nothing is happening. The frequency Linux reports stayed at 2,400 MHz for all thirty minutes. The real clock, the one the firmware reports, was below that for half the phase: over the last five minutes it averaged 2,283 MHz, and at moments it dropped to 1,500. That is the gray line against the amber one in the chart.

And the model gets slower, a little. From 7.76 inferences per second at the start to 7.58 at the end: 2.3 % less. Latency went from 128.3 to 131.2 ms. With this cooling, the Pi sustains the work; it pays for it with a few points of speed that no system tool shows.

With four processes, less heat and less work. In the four-process phase, four single-core processes reached 80.7 °C, the clock did not drop, and the device did 6.7 inferences per second on average: fewer than the 7.7 of a single process on four cores. On the Pi, splitting the work that way does not pay.

What happened on the N100

The N100 runs much hotter. With one process it reached 98 °C; with four, 102 °C, close to the chip's limit of around 105. A caveat on this figure: the N100 started the test already hot, at 85 °C, because it came straight from another measurement; where it ends matters more than where it starts.

There the throttling does show in the frequency: in the four-process phase it dropped from 2,790 to 2,514 MHz, and the device's speed from 18.7 to 17.9 inferences per second, 3.9 % less, with the latency of each inference rising 4.3 %. Unlike the Pi, on the N100 four processes do a little better than one: 18.3 against 17.6 inferences per second on average.

What it says

Both throttle under sustained load, even with a fan and in a ventilated room. The cost over thirty minutes was small—between 2 and 4 %—and it keeps growing; it does not stay put.

A test of a few seconds does not see it. With the Round 1 runs, the Pi was at 65 °C and full speed. It is the same Pi.

On the Pi, the system does not show it. If you watch for throttling with the frequency Linux reports, you will see nothing. What does show it is latency, the firmware's real clock and its throttling flags.

What it does not say

Thirty minutes per phase, not days. A ventilated room, not a cabinet in the sun or a closed box on a pole. One unit of each device. In the field, with more heat and no pauses, all of this moves in the same direction, and further.

And one thing I could not explain: the first time I ran this test, the Pi stopped responding twenty-seven minutes into the four-process phase, with no unusual heat, and did not come back until it was rebooted the next day. In the clean round, with the system log now available to find out why, it did not happen again. Noted, not explained.

What I would do

Measure sustained before promising a latency for the field. Hours, in the enclosure where the device will live, until the temperature stops rising. What you measure there is what you can promise.

Record the temperature, the real clock and the throttling flags with every measurement. On the Pi, vcgencmd measure_clock arm and vcgencmd get_throttled. Without them, a benchmark cannot tell a slow model from a hot device.

Watch for throttling in production. The Pi keeps track of whether it has throttled since the last boot. Putting that in the device's periodic report costs a few bytes and warns you before the customer notices everything is slower.

Look after storage. An industrial-grade SD card or an SSD, and write to it as little as possible. Heat and writes together are what finish it off.

The data from this test—every inference and every second of temperature, clock and flags from both devices—is at crtdrops.mx/datos, under a CC BY 4.0 license.

Your turn

How do you detect throttling on devices already in production: firmware flags in the periodic report, latency, or not at all yet?

The ten-year battery that would not make five

Israel Negrete Lepe — Tue, 06 Oct 2026 15:00:00 +0000

A water meter is installed once. After that nobody opens it again, and the battery decides how long the product lasts. If it runs out early, no update fixes it: someone has to drive out to every meter.

We designed one with a ten-year target. Ultrasonic measurement, a report at least four times a day—one version over LoRaWAN, another over GPRS—and a single D-size lithium thionyl chloride cell of about 19 Ah. The spreadsheet said it was enough, with margin. Then the prototype's consumption was measured in the lab, with instruments, and projected over time: the GPRS version did not even reach five years. The LoRaWAN one did not have that problem.

This note is about that gap: what a battery spreadsheet has in it, and what it is missing.

The numbers in the tables are illustrative: typical datasheet values, arranged to show the mechanism. They are not the measurements of that device.

It all fits in one division

Nineteen thousand milliamp-hours spread over ten years, which is 87,660 hours, gives 217 microamps. That is the whole budget: everything the meter does in ten years has to fit, on average, in 217 µA. Per day, 5.2 mAh. That is with the nominal capacity; with the 80 % that can really be used, it is 173 µA: 4.2 mAh a day.

The word that matters is average. The meter spends almost all its time asleep, drawing microamps, and a few seconds a day awake, drawing hundreds of milliamps. The budget is built from states: how much current each one draws, how long it lasts, and how many times a day it happens.

The spreadsheet

This is what a reasonable budget looks like for the GPRS version, made with the datasheets in hand:

State	Assumption	mAh per day	Share
Sleep	8 µA all day	0.19	5 %
Measurement	one ultrasonic shot every 2 s	0.14	4 %
Radio	4 reports of 15 s at 200 mA	3.33	90 %
Valve	one move a week	0.04	1 %
Total	154 µA average	3.71

With 19 Ah, that gives 14 years. Four years of margin over the target. It gets approved.

You can already see where the weight is: nine out of every ten milliamp-hours go to the radio. Measuring, which is what the meter exists to do, is almost free.

What the spreadsheet did not have

None of the rows is wrong. What happens is that things are missing, and the ones that are there assume the best case.

Real sleep current is not the microcontroller's. The microcontroller's datasheet says a few microamps asleep. But the meter is not just the microcontroller: there is the regulator with its own draw, the modem that is off but not entirely off, the pull-up resistors, the measurement circuit on standby. Going from 8 µA to 25 is easy. It looks small, and it is 24 hours a day for ten years.

The radio in the worst coverage of the deployment, not at the office. A meter lives in a pit at ground level, sometimes under a metal lid. Down there, registering on the cellular network takes longer, and sometimes it fails. A report that took 15 seconds at the office takes 30, and now and then there is a failed one-minute attempt, which costs as much as two reports and delivers nothing.

The battery drains on its own. A lithium thionyl chloride cell loses around 1 % a year even unused, and more in the heat. On 19 Ah that is about 22 µA, permanently: a tenth of the budget, which shows up in no row because no circuit draws it.

The 19 Ah are not 19 Ah. That capacity is measured at a low, steady current, at room temperature. A cellular modem asks for pulses of up to two amps, which a cell of this kind cannot deliver on its own: that is why it is paired with a capacitor that absorbs the peaks. On top of that, after weeks at rest the cell builds a layer that makes it slow to answer the first pulse, and the heat inside the pit speeds everything up. Counting on 80 % of the nominal capacity is already optimistic.

The valve, on the other hand, does not matter. Moving it once a week is less than 1 % of the consumption. Not everything that draws a lot in the moment weighs in the budget, which is why you measure before you blame.

With that, the same table looks like this:

State	Assumption	mAh per day	Share
Sleep	25 µA all day	0.60	6 %
Measurement	same	0.14	1 %
Radio	4 reports of 30 s at 220 mA, and one failed attempt every two days	9.17	88 %
Valve	same	0.04	0 %
Self-discharge	1 % a year	0.52	5 %
Total	436 µA average	10.47

With 80 % of the 19 Ah, that gives 4 years.

Where the gap is

Almost all of it in one row. The radio went from 3.3 to 9.2 mAh a day, because each GPRS report costs on the order of 1.8 mAh in the field. The whole daily budget is 4.2 mAh with 80 % usable. With GPRS, four reports a day already eat more than the meter has for everything. Run the math backwards with the field numbers, and to reach ten years there is room for one report a day.

That leads to the conclusion that matters most: with GPRS, the number of reports per day is not a configuration setting. It is a design decision, and it is made with the budget, not with what the client asks for in the meeting.

With LoRaWAN the math is different. A message costs on the order of a hundred times less energy than a GPRS report, and the radio stops being the problem: that is the difference between the two versions of that meter. But the error does not go away: it moves. With a cheap radio, what dominates is sleep and self-discharge, exactly the two rows the spreadsheet underestimated or did not have.

What I would do from the start

A "measured" column next to the "datasheet" one. The budget is not approved until every row has a measurement from the prototype, with the whole device, not the microcontroller alone.

Measure the radio where the meter will live. In the pit, lid on, in the area with the worst coverage of the deployment. And count failed attempts as a row of their own. That meter's projection was made in the lab and already fell short; in the field it could only be worse.

Write down the losses no circuit draws. Self-discharge, usable capacity, temperature. If they are not in the table, the table lies in your favor.

Decide the reports with the budget. Measure often and transmit little: the meter can measure every two seconds and accumulate, and send several readings in a single message. Every GPRS report saved is days of battery.

Redo the projection once there is data. The first months of a pilot tell you how much each meter really draws. Finding out there is much cheaper than in year four, with thousands installed.

To build yours with your own numbers, over GPRS or LoRaWAN, there is the battery budget calculator: it runs in the browser, comes with these examples loaded, and tells you how many reports a day fit your target.

The spreadsheet was not badly made. It was made with what was known before measuring, and a ten-year battery is not designed with that.

Your turn

How do you estimate battery life for devices in the field: datasheet math, or measured wake time per cycle? If you want to check your own numbers, the calculator is at crtdrops.mx/bateria. I would like to hear where yours drifted from the estimate.

On the N100, OpenVINO was not faster. And I almost measured it wrong

Israel Negrete Lepe — Sun, 04 Oct 2026 20:41:44 +0000

If you work with models on Intel hardware, you know the usual advice: for CPU inference, use OpenVINO. It is Intel's runtime, built for Intel's hardware, and on a mini PC with an N100 it would seem the obvious pick over ONNX Runtime.

I measured it. On the N100, across the three models I compared, OpenVINO was between 11% and 22% slower than ONNX Runtime.

But the result is not the interesting part. The interesting part is that before I got there, I almost measured something else, and the wrong measurement would have passed every check.

What happened in the rehearsal

Before touching the N100, I rehearsed the protocol on my Mac, which has an ARM CPU. MobileNetV2 in FP32, OpenVINO with its default settings.

It came out fast, and it came out right: the outputs matched the reference. The same top class on all twelve test images, and a cosine similarity of 0.999 or more against the expected result.

Then I looked at something almost nobody looks at: the precision OpenVINO had decided to run the model in.

An FP32 model, and OpenVINO running it in FP16. Without being asked, and without saying so.

INFERENCE_PRECISION_HINT = float16

It is a design decision in OpenVINO, not a bug: on that processor, FP16 runs faster. But it is not the one I asked for, and it shows up nowhere unless you go looking for it. In FP16, the model's p50 was 1.46 ms. Forcing f32, the same model on the same machine took 2.39 ms.

Had I kept the first number, I would have measured a precision change and credited it to the runtime.

Why validation does not catch it

This is what I think matters most here. The usual check—does it give the same answers?—passed. And it had to: for image classification, FP16 is enough. The output does not move far enough for an accuracy test to notice.

So same results, and faster does not prove the comparison is fair. It proves the precision change was small enough not to show on that model. On another one—deeper, a regression model, one that is already quantized—it may well show. And it may show in production.

How I measured it on the N100

After the rehearsal, the protocol changed. In every FP32 case, precision is pinned to f32 when the model is compiled, and after compiling, the effective precision is read back. If it is not f32, the measurement does not count: it is marked invalid, even if everything else passed.

compiled = core.compile_model(model, "CPU", {"INFERENCE_PRECISION_HINT": "f32"})
compiled.get_property("INFERENCE_PRECISION_HINT")   # checked, not assumed

With that, on an 8 GB Intel N100 running Ubuntu 24.04, freshly rebooted, one hundred measurements after twenty warm-up runs, batch 1 and default threading:

Model	ONNX Runtime 1.30.0	OpenVINO 2026.4.0	OpenVINO vs ORT
MobileNetV2 FP32	5.7 ms	7.0 ms	22% slower
ResNet18 FP32	22.8 ms	26.9 ms	18% slower
YOLO26n FP32	57.6 ms	64.0 ms	11% slower

These are the p50 of model inference. OpenVINO's effective precision was f32 in all three cases, and in all three the outputs matched the reference. In detection, YOLO26n found the same objects as the reference, not one more, not one fewer.

I had not predicted this. Before measuring, I wrote down what I expected from each case, and for this one I left the answer as unknown. The measurement settled it in a direction I did not expect.

It is not noise

The p50 is a single number. This is every one of the hundred inferences of each case:

The clouds do not touch. Comparing every ONNX Runtime inference against every OpenVINO one—ten thousand pairs per model—ONNX Runtime was faster in 100 % of the pairs for MobileNetV2 and ResNet18, and in 99 % for YOLO26n. In the first two, the slowest ONNX Runtime inference was faster than the fastest OpenVINO one. The difference is not measurement noise: it is systematic.

The raw data—every inference, the temperature during the run and the exact versions of each environment—is at crtdrops.mx/datos/round1, so you can check it or compare it with yours.

What this result does not say

This is a measurement, not a verdict on OpenVINO. It holds for what was measured, and nothing more:

One device. One N100. On a processor with native FP16, or with the matrix instructions of Xeon chips, the story may be different.
CPU only. The N100 has an integrated GPU, and OpenVINO knows how to use it. I did not measure it.
Batch 1, latency. OpenVINO has a mode built for processing many inputs at once, with several parallel streams. I did not measure that either.
Three models, in f32. With reduced precision, or with OpenVINO's own quantization, the numbers change.

What it does say is more modest and, I think, more useful: on this device and with this setup, the advantage you would expect did not show up.

The questions I left open, measured

On 7 October I went back to the N100, freshly rebooted, with three of those questions: whether it also lowered the precision on its own there, whether its many-inputs-at-once mode closed the gap, and what the integrated GPU did.

On the N100's CPU, the precision does not drop. Compiled without asking for anything, all three models stayed in f32. That is not OpenVINO's doing: that processor has no FP16, so there is nowhere to drop to. On the Mac it did drop because its processor does have it. The guard is still needed as soon as you change hardware.

Many inputs at once does not close the gap. In its mode for processing several requests in parallel, OpenVINO opens two streams on the N100 and gains between 4 and 9 %. Comparing the best of each—that mode against ONNX Runtime with four processes—ONNX Runtime stays ahead by 14 to 20 %.

On the integrated GPU, OpenVINO does win. In FP16. The N100's GPU does run FP16, and OpenVINO uses it in that precision without anyone asking: the same trap as the rehearsal, now on the N100. That way, YOLO26n goes from 17.8 inferences per second with ONNX Runtime to 24.7, 39 % more; ResNet18, from 43.4 to 55.9. With precision pinned to f32, the GPU gains nothing: 16.3 and 37.2. And MobileNetV2 is slower on the GPU than on the CPU, in any precision.

And is it the same? I checked it the way Round 1 does: the twelve real images, against the reference output, with the same tolerances. Both classifiers give the same class on all twelve. YOLO26n passes validation—it asks for 95 % agreement and got 96.6 %—but it is not the same: it found the same objects with lower confidence. In one image, a person went from 0.829 to 0.696; in another, a pair of skis from 0.468 to 0.376. If your system only accepts detections above 0.7, that person disappears on the GPU and not on the CPU.

It is this note's lesson again, on different hardware: passing validation does not prove nothing changed. The GPU's advantage is real, and it comes with a precision change you have to decide on knowingly, not discover later.

And the numbers repeat. The CPU measurements, taken on three different days, changed by less than 3 % from one time to the next. The data for this part is at crtdrops.mx/datos.

What I would do differently when comparing runtimes

If you are choosing a runtime for an edge device, four things I would have done from the start:

Pin the precision, and read back what you got. Not what you asked for: what the runtime chose after compiling. It is two lines of code, and it is the difference between comparing runtimes and comparing precisions.

Do not use accuracy as proof that the comparison is fair. Matching outputs say the change did not show on that model. They do not say there was no change.

Write down what you expect, first. Even if the answer is I don't know. If you do not write it down, any result seems to confirm what you already thought.

Measure on the device, not on your machine. Freshly rebooted, warmed up, and with p50 and p95, not an average. The Mac helped me find the problem; only the N100 gave the answer.

The guard, in code

This is what I use now so it does not slip past me again. It compiles the model asking for a precision, reads back the one it actually got and, if they differ, refuses:

import openvino as ov


class PrecisionMismatch(RuntimeError):
    """The compiled model runs in a precision other than the one requested."""


def effective_precision(compiled: ov.CompiledModel) -> str:
    return compiled.get_property("INFERENCE_PRECISION_HINT").get_type_name()


def compile_pinned(model, device="CPU", precision="f32", core=None, **config):
    core = core or ov.Core()
    compiled = core.compile_model(model, device, {"INFERENCE_PRECISION_HINT": precision, **config})
    got = effective_precision(compiled)
    if got != precision:
        raise PrecisionMismatch(f"asked for {precision} on {device}, the compiled model uses {got}")
    return compiled

With OpenVINO 2026.4.0 on a Mac with an Apple M5 Pro, compiling MobileNetV2 without asking for anything gives f16; with compile_pinned it gives f32. And if one day it does not, the measurement stops instead of carrying on in another precision. It works the same for the integrated GPU (device="GPU"), where OpenVINO uses FP16 by default.

Your turn

Have you caught a runtime changing precision behind your back? Which runtime, which hardware? And if you run these models on your own N100 or Pi 5, the raw data is at crtdrops.mx/datos: I would like to compare.

The full table—five models on a Raspberry Pi 5 and on the N100, with ONNX Runtime and OpenVINO—is on crtdrops.mx, next to an ONNX model check that runs in your browser without uploading the file anywhere. If you open one of the models I measured, the tool recognizes it and shows you its numbers.

Having your own model measured this way, on both devices and with the same protocol, is now possible: it is in the same section, for 149 USD, with a sample report so you can see what you get.

Try it: the full results table and the in-browser ONNX model check on crtdrops.mx.

Updated on 5 October 2026 with the full distribution, the raw data and the code for the guard, and on 7 October with the open questions, measured on the N100.