DEV Community: Israel Negrete Lepe

The Electratón runs again: a 2001 energy controller, back on the track

Israel Negrete Lepe — Fri, 13 Nov 2026 15:00:00 +0000

In November 2001, for my social service—the community-service requirement for graduating in Mexico—I worked with two classmates on the energy control of an electric competition vehicle, an Electratón. The problem was concrete: current peaks above 60 A were significantly hurting the batteries' range.

Twenty-five years later I wanted to go back to that problem. Not to show that today you can write a more complex controller, but to see how much of that answer still stands when you put it against a simulator, and how much depends on things we could not measure in 2001.

The problem of November 2001

From the document of that time I still have this: a controller to manage the vehicle's energy demand, above all when accelerating and after braking, on an ATMEL 8535 programmed in assembly. The documented diagnosis tied the peaks above 60 A to a significant loss of range.

The design had three mechanisms:

Soft start: avoid a sudden current demand when starting to accelerate.
Recovering the operating point: after braking, keep the motor from jumping straight back into a high-demand condition.
A 60 A comparator: detect when the current went above that threshold.

The system also logged voltage, current and speed every 20 seconds, and the data was downloaded over a serial link.

The constraints of an AVR in assembly

The 8535 was an 8-bit microcontroller with 8 KB of program memory and 512 bytes of RAM. Everything was written in assembly: every comparison, every count. There was no operating system and no libraries.

You do not build a model of the vehicle or run an optimization with that. You make simple decisions that fit in a few cycles: ramp up slowly, do not go past a threshold, do not jump back. What is interesting is that those simple decisions were, underneath, an energy policy.

What we kept and what we can no longer verify

I have to be precise here, because it is easy to reconstruct too much.

I do not have the 2001 source code. I did not recover the motor specifications, the battery chemistry or capacity, or any lap or range results. I am also not going to state a format of that competition's rules without the documentation.

I do not know everything about the controller either:

The shape of the soft start: a progressive increase of the drive is a reasonable reading, but I have neither its shape nor its duration on record.
What exactly the “operating point” after braking was: a duty cycle, a speed reference or some other parameter.
What the comparator did on detecting 60 A: cut the drive, limit it or trigger some other protection.
How the driver's throttle related to the controller.

And one more thing that matters: a log every 20 seconds is not enough to characterize short current peaks. I do not have the procedure used to diagnose them, and I am not going to say it was those logs. I also do not have range results from before and after the controller.

The reconstruction

So what I built is not a translation of the assembly. It is an experimental reimplementation in C, inspired by the documented principles, put against a simulator. It runs on the lab's Raspberry Pi 5.

The simulator has a simple model of a DC motor, a battery with its internal resistance, and the mechanical load: mass, rolling and air. Everything physical is a modeling assumption, not the original vehicle:

	Assumption
Vehicle with driver	150 kg
Motor	DC, 24 V, a top speed of about 48 km/h
Battery	two 12 V lead-acid, 35 Ah, 0.03 Ω internal resistance
Track	a 1 km lap: two 350 m straights and two 150 m curves taken at 25 km/h
Driver	flat out on the straight, brakes for the curve and holds it

A lead-acid battery delivers less capacity the more current you draw from it. That is modeled with the Peukert effect, an exponent I do not know for those batteries. So I ran four: 1.0, which is no effect at all, and 1.1, 1.2 and 1.3. And I apply it to each current pulse, with no capacitor smoothing it: it is the assumption that least favors the controller.

Three strategies, with the same track, the same driver, the same charged battery and the same parameters:

No control: the motor's duty cycle is whatever the throttle asks for.
Ramp: the duty cycle rises at most from 0 to 100 % in three seconds; going down is immediate.
Ramp and a 60 A limit: also, while the motor current is above 60 A, the duty cycle goes down.

I did not model the “operating point after braking”: I do not know what it was.

The results

With Peukert 1.2:

Strategy	Until the battery is empty	In 30 minutes	Current peak
No control	27.6 km	17.4 km	254 A
Ramp	28.3 km	17.3 km	191 A
Ramp and 60 A	30.6 km	16.9 km	64 A

The peak is the motor current, which is what the battery delivers on each pulse; with the limit it goes a little above 60 A before the comparator pulls it back.

The two distance columns measure different things, and that is the trade-off I wanted to see. When energy is the limit, the comparator makes the battery go 11 % farther. When the battery is enough for the whole race, no control wins: in 30 minutes it covers 2.8 % more, because it comes out of every curve faster.

What the limit gains does not depend on Peukert alone. Without the effect, the limit still goes 6.5 % farther. About three quarters of that comes from losing less as heat, in the resistance of the battery, the motor and the wiring; the rest, from going a little slower, which costs less air and less braking. With Peukert 1.3, the advantage rises to 13.5 %. In other words: the direction of the result holds, but its size depends on an assumption I cannot confirm.

The ramp alone barely helps. It goes 2.6 % farther than no control. Looking at the current millisecond by millisecond shows why: it lowers the peaks, but does not remove them. At the start, from 254 A to 191 A, because a three-second ramp reaches full duty before the vehicle picks up speed. And the peak is not only at the start: it comes back at every curve exit, twice a lap, when the driver accelerates from about 25 km/h. Without control that is about 133 A for five seconds; with the ramp, about 117 A. The comparator, on the other hand, acts on every one of them. That comes from how I modeled the driver, but it says something that made sense in 2001: the problem was not one start, it was all of them.

The threshold is a trade-off too. Moving the limit, with the same assumption:

Threshold	Until the battery is empty	In 30 minutes	Time per lap
30 A	41.1 km	15.0 km	118.0 s
60 A	30.6 km	16.9 km	106.5 s
100 A	28.5 km	17.3 km	104.9 s

At 30 A, the battery goes 49 % farther than with no control, but every lap takes 11 % longer than at 60 A. No threshold wins at everything: it depends on whether the race is decided by energy or by time.

What a 20-second log sees

The simulator also takes a sample every 20 seconds, like the 2001 log, and compares it with what happens millisecond by millisecond.

Without control, the current was above 60 A for 258 seconds of the 48 minutes the battery lasted. The 20-second log took 143 samples, and 12 of them landed above 60 A: roughly the same proportion, 8 % against 9 % of the time. But the highest it saw was 110 A, and the real peak was 254 A. A sample every 20 seconds can tell you that something happens now and then; it cannot tell you how high it goes or how long it lasts.

It is the same problem as in the average that never happened: the measurement you have decides what you can see.

What changed in twenty-five years, and what did not

In 2001 we had an 8-bit microcontroller, assembly and a log every 20 seconds. Today I have a 64-bit processor on a board the size of a card, a simulator that runs a whole race in under a second, and the current of every millisecond. I can move an exponent and see how much the conclusion changes. In 2001 we could not.

But the question is the same. The peaks that eat the battery, the trade-off between lasting and finishing first, the measurement that does not see what matters: computing power solved none of that.

The code, the tests and the run's data are in the repository, with the assumptions together in one place for anyone to change.

The same question

In 2001 we were trying to get an electric vehicle to make better use of the energy available, with control decisions run by a microcontroller with limited resources. Twenty-five years later there are much more powerful processors, simulators, better instruments and analysis tools.

But the fundamental question is still the same: how do we get the best possible performance without wasting the resources we have?

What is interesting about going back to the Electratón was not to show that we can write a more complex controller today, but to find out how much we learned about a problem that is still relevant.

Your turn

Have you gone back to something you built years ago? What still held up, and what could you no longer verify?

Israel Negrete Lepe is an electronics engineer. Twenty-five years making different systems work together under real conditions: from the sensor, the vehicle or the transaction to the backend, the data, security and operations. Today, increasingly, edge AI. I take on projects by the hour or by deliverable: israel0negrete@gmail.com.

I send a short email when a new note or lab result comes out: subscribe here.

Ghosts in the fleet: knowing the IMEI is not knowing the device

Israel Negrete Lepe — Mon, 09 Nov 2026 15:00:00 +0000

In early November, in Mexico, families set up the ofrenda, the Day of the Dead altar, to welcome those who are gone. Those visitors are expected: they have their place, their photo and their candle.

A tracking platform can get visitors too, but uninvited ones: a device that shows up in two places at once, a session that stays open although nobody is on the other end anymore, a vehicle that according to the inventory should not be reporting, and reports.

Before calling any of them a “clone,” it is worth knowing what kind of ghost it is.

The IMEI is a declaration

Every tracking device introduces itself with an identifier: the IMEI, or the terminal identifier its protocol defines. The server reads it and, from that moment on, everything that arrives over that connection is attributed to that device.

But the device declares that identifier itself. It is not cryptographic proof of who it is. That is why two connections presenting the same IMEI can mean very different things:

a legitimate reconnection;
an earlier session that has not been closed yet;
a wrong configuration;
or two devices that really share an identifier.

The platform has to tell them apart before deciding anything.

I have no documented case in which it was confirmed that two physical devices were using the same IMEI at the same time, and I am not going to tell one. What is real is the architecture problem: the server can receive perfectly valid positions over a TCP connection and still attach them to the wrong identity.

Three different ghosts

They are worth separating, because each one leaves different evidence:

Session ghost: a connection the server still considers current, although there is no longer any communication with the device. It is the half-open connection from the sockets note.
Identity ghost: two devices, or two sessions, presenting the same identifier.
Inventory ghost: a device that does report, but whose administrative link to a vehicle is wrong or out of date.

The first is a transport problem; the second, an identity problem; the third, an administrative one. Treating them the same way is how you end up calling a device that only lost coverage a clone.

Five signals, and what none of them proves alone

This is what I would do today if I designed the mechanism from scratch. I cannot claim that every gateway I have worked with has these policies today.

Inventory. Each identifier tied to a registered device, with its model, its administrative status and its assignment history. That way, a decommissioned device that keeps reporting is seen for what it is. But the inventory only knows what somebody registered.

Sessions. A registry of active connections per identifier: the gateway instance, the session identifier, the connection time and the last valid activity. Two sessions with the same IMEI are a condition for review, not proof of a clone. It may simply be that the old one has not finished dying.

Geography. Consecutive positions, compared using their original timestamps. If the same identifier shows up in two places no vehicle could travel between in that time, there is an inconsistency. But first you have to rule out GPS errors, historical data, wrong clocks and out-of-order messages: arrival time is not the time of the data, as in the right position at the wrong time.

Sequences. In JT808, every message carries a sequence number that the device keeps incrementing. A gap, a repeat or a crossing gets attention, but on many devices the counter goes back to zero on a restart, and a retransmission usually repeats the number.

Correlation. I would not declare an IMEI cloned on a single anomaly. I would look for agreement: simultaneous sessions, incompatible trajectories, counters that cross, what the device declares about itself and what the inventory says. The output should be a grounded suspicion, not a verdict.

The detector

To see the difference I built a detector that runs in the browser. It reads a synthetic day of twelve identifiers. Behind eight of them there is a single device:

a normal day;
a session left half-open after two minutes without coverage;
a wrong position, 18 km from where the vehicle is;
twenty minutes without coverage on the highway, and the stored data arriving afterward, mixed with the new;
a restart on the road: the counter goes back to zero and the time reads January 1, 2000 until the GPS gets a fix;
retransmissions: each message has a 3 % chance of arriving twice;
a firmware update, which restarts the device;
a device the inventory has as decommissioned.

Behind the other four there are two devices with the same identifier: a clone of a different model 145 km away; two of the same model driving around the same city; a replacement device configured with the old one's identifier, while the old one stays powered on in the warehouse; and a second, identical device that powers on for forty minutes 150 meters from the first, both parked.

To keep it simple, here the device declares its model and firmware every time it starts a session. In JT808, depending on the protocol version, that travels in the registration, in the authentication or in a separate query.

Two detectors read what the gateway would see: the sessions, each message's sequence number, the time it carries, the position, and the model and firmware the device declares. They never know which device sent what.

The naive one calls it a clone on a single anomaly: a login while another session of the same identifier is open, an impossible jump between two consecutive positions in arrival order, or a sequence number that does not follow the previous one.

The correlated one first rules out what has another explanation. With what is left, it speaks of a duplicate identity only if there is evidence from two different families: sessions and counter (which count as one, because they come from the same connections), geography, and declared identity.

With the gateway set so the new session closes the old one:

	Naive	Correlated
One device behind it (8): called a clone	6	0
Two devices behind it (4): flagged	4	4, one only as “review”

With the gateway letting both sessions live, the figures are the same.

Why the naive one falls

Every false alarm has its explanation, and it is the one the correlated detector looks for first:

The half-open session: the device comes back with a new session. If the new login closes the old one, it is a single replacement, not flapping. If both live, the old one sends nothing more, and a silent session is not a second device.
The wrong position: it is a stray point. It does not form a trajectory of its own; it is set aside.
The stored data: on the default day, the 40 stored positions interleave in arrival order with the new ones and draw 7 impossible jumps. Ordered by the device's time, the jumps vanish.
The restart: the counter jumps once, to zero, and the new chain never crosses the old one again: that is not two counters. The positions with factory time stay off the map.
The retransmissions: the same number with the same content, seconds later, is the same message.
The update: it also restarts the device, so the naive one falls for the same reason. And the declared firmware changes once and never goes back: it is an update, not two devices taking turns.

The decommissioned device, on the other hand, the naive one does not even see: it has no communication anomaly at all. The correlated one flags it separately, as an inventory ghost, because it is not an identity problem.

What the verdict does not say

“Duplicate identity” means there are two devices with the same identifier. It does not say whether one is a clone or someone misconfigured a device: in the replacement case, the evidence is the same as for a clone. That is found out in the field, not on the server.

And the forty-minute device stays at “review.” The sessions flip back and forth, and there are two counters crossing, but both devices are parked 150 meters apart, closer than the kilometer the detector tolerates as GPS noise, and with the same model and the same firmware: there is no geography or declared identity to confirm it. With that evidence, “review” is the honest answer. A person has to look.

What this detector does not prove

I made the twelve cases on purpose, and I chose the detector's thresholds while looking at them. The code's tests run fifty different days, and in all of them the correlated detector gives the expected answer; that is by construction, not a measure of how it would do on real data. Real data would bring cases that are not here: a clone that also imitates the counter or the firmware, clocks that drift little by little, devices that lose messages.

Each control case has a single event, and the thresholds sit just above it. With a single device, it would not take much to reach “review”: three restarts in a day or, with the policy that closes the old session, five one-minute outages in an hour.

It also shows how much depends on each threshold. With that same policy, if flapping is declared at a single session replacement per hour, the half-open session, the restart and the update move to “review.” If the retransmission window is set to zero, the retransmissions look like a second counter.

To see each case from the inside, with its map, its sessions, its counter and the timeline of the evidence, there is the ghost detector: it runs in the browser and sends nothing. You can move the thresholds and the gateway policy, and see which cases change their verdict.

No anomaly is enough alone

On a tracking platform, knowing the IMEI does not necessarily mean knowing the device. That is why identification, session, location and inventory have to be kept apart.

An impossible jump on the map is an anomaly. Two connections with the same IMEI are an anomaly. An unexpected sequence can be one too. But none of them, on its own, proves that a cloned device exists.

A good ghost detector should not find clones where there are only communication errors. It has to gather enough evidence to tell a duplicate identity from a session that has not finished dying.

Your turn

Have you seen two devices answer to the same identifier? What finally told you which one was which?

I send a short email when a new note or lab result comes out: subscribe here.

The query that took too long

Israel Negrete Lepe — Fri, 06 Nov 2026 15:00:00 +0000

The query endpoints of a transactional platform were returning HTTP 504: timeouts, with pagination and without it. The table was large—more than eight million rows in 2026 alone—and the first temptation was to think that explained everything: more rows, more time, more server.

It did not. The problem was not reading a large table, but how MySQL joined, filtered and sorted the data to answer that query. That is what this note is about.

The figures in the examples are illustrative: the plans in the plan reader are hypothetical, and the growth model is a model, not a measurement. From the case come the 8,040,017 records; the 1,792 and 54,898 rows, EXPLAIN estimates (the latter from the variant with hints explained below); the 17.05 s it took to create the index; and the 4.5 ms that EXPLAIN ANALYZE measured in a lab test, not in production.

A 504 does not say where the problem is

A 504 comes from the load balancer or proxy in front of the application when it gets tired of waiting. It does not say whether the time went into the database, the network, the application or building the response.

So the query was analyzed apart from the rest of the application, with the SQL that actually reached the database. In this case it was hand-written SQL, not generated by an ORM, but the rule is the same: to diagnose, you have to see what MySQL runs, not what you think it runs.

The table and the query

MySQL 8.0.46 on Ubuntu. Transactions live in one table per year, from 2015 to 2030; the 2026 table had 8,040,017 records when we ran the tests. Around it are lookup tables: operations, products, providers and the inbound and outbound addresses.

The query fetches one operation's transactions for an operational report: date, time, masked phone number, identifiers, status and processing time, with the names that come from the lookup tables, ordered by date, from oldest to newest, in pages of 50. One operation had between 52,000 and 55,000 estimated rows, in a table of more than eight million.

The plan that estimated fewer rows

In the EXPLAIN of the original query, the transactions table's row showed the IdProductos index, about 1,792 estimated rows and Using where. And the plan carried Using temporary and Using filesort.

At first sight, 1,792 looks small. But the rows column in EXPLAIN is what MySQL expects to read each time it runs that step, and the transactions table sat inside the join: that step runs once for every row the previous step hands it. Using where means it keeps only the rows that meet the condition, and a products index does not lead to one operation's rows in date order. A small estimate at one step does not tell you how much work the whole plan is.

Then there were the two flags. In tabular EXPLAIN, when they appear together, Using temporary and Using filesort are written on the row of the first non-constant table in the plan, but they are not about that table: they say the result of the join is stored separately and sorted. They are not errors: sorting or grouping a few rows is reasonable. Here, according to the plan, they meant that to return 50 rows in date order MySQL had to gather all of the operation's rows, store them separately and sort them before cutting: about 52,000 to 55,000 estimated rows to return 50.

The index that follows the query's order

The query filters by operation and orders by date. An index with those two columns in that order lets MySQL find the operation's rows already sorted, read them in order and stop as soon as the page is full, as long as the transactions table is the first in the join. It was a composite index, idx_transacciones_operacion_fecha (IdOperaciones, FechaHora).

Creating it on the 2026 table took 17.05 s in our measurement. Since the transactions are split by year, the index was also applied to the other yearly tables.

To evaluate a plan that started from the transactions, we tested a variant with FORCE INDEX (idx_transacciones_operacion_fecha) and STRAIGHT_JOIN. Both are investigation tools: FORCE INDEX makes MySQL use that index whenever it can and STRAIGHT_JOIN forces the tables to be joined in the order written. They are for comparing plans; leaving them in the code freezes a decision the optimizer can no longer correct when the data changes.

The EXPLAIN of that variant estimated about 54,898 rows up front, with no temporary table or filesort, and the lookup tables were joined with eq_ref: one exact lookup by key per row. The plan that estimated more rows did the whole job better.

Measuring: 4.5 ms, and what it leaves out

EXPLAIN is the plan before executing: what MySQL computes from its statistics. To know how long it takes you have to run it, and that is what EXPLAIN ANALYZE is for: it runs the query and reports, for each step, the time, the actual rows and the loops.

With LIMIT 50, the EXPLAIN ANALYZE of the variant with the hints recorded between 4.37 and 4.50 ms. That is the time to run the plan inside MySQL, in a lab test. It is not an end-to-end measurement of the API: it leaves out the network, the application and sending and serializing the response. Nor does it hold for the unpaginated query or for other filters, which have plans of their own.

It is not a production result either. When the tests ended, the change still had to be applied to the Node.js service and validated there. And it does not show the hints can be removed: without FORCE INDEX and STRAIGHT_JOIN, we do not know whether MySQL would pick the same plan.

What we do not know

There are questions this story does not answer, and it is worth saying so:

Production. There is no confirmation the change was deployed, nor a comparative measurement between development and production. We do know that the real volume and the distribution of records per operation mattered to the optimizer: an index that is good for one pattern may not be for another.
The plan without hints. We did not check which plan MySQL picks with the composite index and without FORCE INDEX or STRAIGHT_JOIN. Before deciding whether they stay in the code, that is what has to be measured.
Locks. There is no evidence that the query blocked writes or stalled transactions. A slow query and a lock are different problems, and they should not be confused.
What the index costs on writes. Every index is updated on every insert, every delete and whenever its columns change. We did not measure it.

When the table grows

The plan reader includes a model to see the shape of the problem. It is a row count with assumed microseconds per step, not a MySQL measurement: it is for comparing curves, not for predicting milliseconds. That is why, for the composite index, the model gives tenths of a millisecond and the case's lab measurement gave 4.5 ms.

With a table of 8,040,017 rows, an operation of 54,898—the case's estimate—and the transactions table first, for the first page of 50 the model gives:

How it is resolved	Rows touched
No useful index: scans everything and sorts	8,040,267
Index on operation only: finds them all and sorts	55,148
(operation, date) index: reads in order and stops	300

Each one grows with something different. The first, with the table: twice the rows, twice the work. The second, with the operation, even if the table does not grow. The third, only with the page requested: for page 100, with OFFSET, it also reads the previous 4,950, joins the lookup tables to all of them and touches 30,000 rows. Paginating by the last date and id seen, with the id also in the ORDER BY, instead of skipping rows, avoids reading them.

What I would do from day one

Identify the critical queries and their access pattern: what they filter, what they join, in what order they return, how they paginate and at what volume.
Design indexes around how the data is retrieved, not only around the relationships between tables.
Separate transaction processing from analytical queries and reports, so they do not compete for the same resources.
Measure latency, review plans periodically and test with representative volumes. A query is not optimized because it answered fast in an isolated test.

To read a plan of your own, compare the before and after, or see how the work grows with the table, there is the MySQL plan reader: it runs in the browser, tells what MySQL estimates apart from what it measured, and what you paste never leaves your machine.

The lesson was not that FORCE INDEX and STRAIGHT_JOIN should always be used, but that the execution plan has to be checked against real data, contrasting its estimates with the actual execution.

It was not simply a matter of creating an index. It took understanding how MySQL joined, filtered and sorted the data.

A database does not get slow only because it grows. It can also get slow because our queries keep asking it to do more work than necessary.

Your turn

What's the query plan that fooled you the most, and what made you look twice? If you want to read your own plans side by side, before and after, the tool is at crtdrops.mx/explain.

I send a short email when a new note or lab result comes out: subscribe here.

An open socket is not a connected device

Israel Negrete Lepe — Tue, 03 Nov 2026 15:00:00 +0000

An operator sends a command to immobilize a vehicle. The server looks up the device's connection, writes the command to the socket, and the write succeeds. That does not show the device received it, much less that it carried it out.

In vehicle tracking, the TCP connection is only a means of communication. It is not a guarantee of availability. That is what this note is about.

The figures in the examples are illustrative: they come from a simulator with an assumed fleet, not from measurements of our gateways or of any carrier.

Connections that die without notice

We have worked with receiving gateways for GPS devices, telemetry and vehicle cameras. With JT808 and other tracking protocols, the device opens a persistent TCP connection and keeps it to send positions, alarms and other messages. How it behaves depends on the protocol and the manufacturer.

With mobile devices, that connection can stop working without the server receiving a close: the vehicle loses coverage, switches networks, or the carrier's NAT forgets the connection for being idle. The server keeps a socket that looks open, even though the device no longer has connectivity.

Three ways to find out

There are three mechanisms to learn that a connection died, and they are not equivalent:

Application activity: how long it has been since the device's last valid message.
The protocol's heartbeat: whether the device keeps sending its periodic presence messages.
TCP keepalive: the operating system's mechanism to detect connections that stopped responding.

In the simulator, a fleet of 2,000 devices sends a position every 30 seconds while moving and a heartbeat every five minutes while parked, and each device loses coverage about four times a day. In one day, 7,762 connections die without notice. This is how long the server takes to find out, at the median:

With three missing heartbeats, 12.8 minutes.
With TCP keepalive at Linux's defaults (two hours idle and nine probes), 2.2 hours.
If, on top of that, the same device's new session closes the old one, 10.3 minutes.

The system's keepalive, at its factory values, is no use for this, and on top of that it is off by default on each socket. And none of them is instant: during those minutes, the socket looks open.

I do not have a single heartbeat interval documented for our whole fleet: it depends on the protocol, the manufacturer and the configuration. Nor would I attribute a NAT expiry time to a carrier without having measured it. In the simulator they are assumptions, and you can change them.

What a heartbeat costs

A small heartbeat does not cost only its bytes. A 20-byte heartbeat, with the TCP/IP headers, its response and the ACK, is about 160. In the simulator, the fleet spends 1.22 MB a month per device just staying connected; counting only the heartbeat and its response it would look like 0.30. In a large fleet, that difference adds up.

Stretching the heartbeat to save data has a limit: the NAT. If the heartbeat is 15 minutes and the NAT forgets after 10, every heartbeat from a parked device hits a connection that no longer exists and forces a reconnection. In the simulator, the data drops by a quarter, from 1.22 to 0.91 MB, and the connections that die without notice go from 7,762 to more than 170,000.

Sent is not received

The command case is the most delicate. That is why I would distinguish four states:

Command generated.
Command sent to the connection.
Command confirmed by the device.
Action executed and verified.

Going from the second to the third takes a confirmation from the protocol; from the third to the fourth, where it applies, an independent check of the final state.

On the simulator's day, 988 commands are sent. Of them, 24 are written to a socket that was already dead: the server "sent" them and they never arrived. If "sent" counted as done, those would be 24 false successes. With a lifecycle that waits for the confirmation, eight are resent over the new session and 16 expire unconfirmed, in plain view of whoever sent them. And if the heartbeat goes past the NAT, the commands written to dead sockets rise to 322.

Two sessions from the same device

A device loses communication, reconnects and opens a new session before the server detects that the previous one no longer works. If the system manages connections only by socket, it can end up with two sessions from the same device.

My approach is a session registry per device, with a session identifier or generation. When a new, authenticated, valid session arrives, it replaces the previous one under an explicit policy, and the old one is invalidated and closed in a controlled way. You also have to keep a late close of the old session from accidentally wiping out the reference to the new one. I do not present this as a failure we suffered, but as a condition the design has to resolve.

The simulator compares the two registries. In its day, devices reconnect 4,579 times with the old session still open. With a registry by socket, which registers the device with every message and deletes the entry when any of its sessions closes, 4,566 of those late closes wipe out the new session. The device is still connected and the system takes it as disconnected until its next message: 335 device-hours in the day. If the registry were only updated at authentication, the device would stay "disconnected" until it reconnected again: 20,801 device-hours, and 429 commands would expire without finding a session. With a registry by generation, none.

Several instances

With several gateways, each device may be connected to a different instance. Knowing its IMEI is not enough: you have to know which instance holds its connection. I would design a distributed session registry with the device's identifier, the instance, the session's identifier and generation, the last valid activity, and its state and validity.

The command is routed to that instance, and the instance checks that it still holds the session before transmitting it. I would not send it to every gateway. And you have to account for the device moving to another instance while the command is in transit. The registry finds the connection, but it does not replace the protocol's confirmation or guarantee that the command was executed.

The restart that becomes a second failure

When a gateway restarts, the sessions in that process are cut. What happens next depends on the devices: some reconnect at fixed intervals, others with other strategies, and it cannot always be controlled from the platform. I have no documented incident of a mass reconnection, or a figure for devices that reconnected at once. What I do consider essential is that recovery must not turn into a second failure.

And three things have to be told apart: an accepted TCP connection, an authenticated device and a session ready to receive commands. In the simulator, the gateway is down for 15 seconds; it accepts 1,000 connections a second with a queue of 511 (the one Node.js uses unless told otherwise), authenticates 100 devices a second and initializes 200 sessions. With 2,000 devices retrying together every 30 seconds, 2,000 attempts arrive in one second. The queue fills up: Linux drops 3,323 SYNs, which the devices resend; 900 attempts are abandoned after waiting too long, and every session is ready for commands after 79 seconds. With a random wait of up to a minute, the peak is 74 attempts a second, nothing is dropped and all are ready after 60 seconds, a time that cannot go below that wait: the last device does not try before it. With 5,000 devices the gap grows: 195 seconds against 64, and with a fixed interval 27,363 SYNs are dropped.

What matters is not how many attempts arrive per second, but how long the system takes to recover every session. In a new design I would use staggered restarts where the architecture allows it, load distribution, and random waits on reconnection where the firmware supports them.

The limits you do not see

I have no documented incident from running out of file descriptors, memory or ports in our tracking gateways. But they are part of what has to be sized: the available descriptors, memory per connection, event loop usage, TCP accept queues and socket states. I would not assume a server can handle thousands of connections just because it has memory and CPU to spare.

What I would do today

In our current architectures we mainly use Node.js and NestJS for TCP reception, with specialized services per protocol, and RabbitMQ to decouple reception from processing. If I designed a gateway for thousands of devices from scratch today:

I would separate four responsibilities. Transport: sockets, connections, disconnections, timeouts and bytes. Protocol: identifying devices, parsing frames, validating messages and handling confirmations. Sessions: the device's identity, its current connection and the instance holding it. Messaging and processing: handing telemetry to the rest of the platform without blocking reception.

I would give commands a lifecycle of their own, with unique identifiers, execution states, expirations and confirmations.

I would measure from the start how many connections there are, how many are really active, how many reconnections happen per minute, how long it takes to detect a dead session and how many commands are left unconfirmed.

I would run load tests that go beyond sending positions: thousands of persistent connections, silent disconnections, duplicate sessions and gateway restarts.

To try it with another fleet, another heartbeat, another NAT or another gateway, there is the connection simulator: it runs in the browser, and everything it says is a model, not a measurement.

An open socket does not show that the device is effectively connected. A command sent does not show that it arrived. And an acknowledgment of receipt does not necessarily show that the action was carried out. When we manage many devices, reliability depends less on how many connections we can open and more on our ability to know the real state of each session and to recover correctly when communications fail.

The real challenge is not keeping thousands of sockets open, but managing thousands of devices whose connectivity can change at any moment.

If you build or test platforms like this one, that is the ground my guide Virtual Fleets covers: silence and disconnection, the reconnection storm, and the acknowledgment that is not one, with a virtual fleet generator to produce that traffic against your own platform. Chapter 4, on the receiving side, is free as a PDF:

israelnegretelepe.gumroad.com/l/a-read-is-not-a-frame

Virtual Fleets — israelnegretelepe.gumroad.com/l/virtual-fleets

Your turn

How does your platform know a device is really connected: the last message, a protocol heartbeat, or TCP keepalive? And when you send a command, what does your system show between "sent" and "confirmed"? If you want to try your own heartbeat, NAT and gateway numbers, the simulator is at crtdrops.mx/conexiones.

I send a short email when a new note or lab result comes out: subscribe here.

The lab's ofrenda: what didn't survive October

Israel Negrete Lepe — Sun, 01 Nov 2026 15:00:00 +0000

In Mexico, from October 31 to November 2, families set up an ofrenda for the Day of the Dead: an altar with a photo of the one who died, orange cempasúchil marigolds, pan de muerto, candles, and whatever that person loved in life. They also write calaveritas, short mocking verses in which La Catrina, the elegant skeleton lady, comes to take someone away.

A lab has its dead too. My lab, in October 2026, was two machines in an office: an Intel N100 mini PC and a Raspberry Pi 5. Not every experiment was meant to survive. Some died because we found something better, some because the measurements contradicted what we expected, and some simply ran their course. But when what happened is documented properly, none of them disappears completely.

This is the ofrenda. Each one gets a death certificate: the date, a technical cause of death, and what it leaves us. And its calaverita, because there is no ofrenda without one. The verses are written for English, not translated: the Spanish ones rhyme their own way. The humor does not hide the results: every figure comes from a published measurement or from its log.

One thing first. I have no record of any hardware dying in October: no SD card, no power supply, no sensor. We did not plant an electronic corpse to fill the altar.

The `pi` user

Died on October 3, on the Raspberry Pi 5.

Cause of death: retirement. That day both machines became dedicated to the lab, with a user of their own, edgelab, and a key of their own. Before the factory account was closed, the whole lab that lived in it was copied to edgelab and checked by hash. Then the old copy was deleted, and pi had its password locked and its keys removed.

Epitaph: Here rests an account that served its purpose. Its experiments live on in edgelab.

What it leaves us: it did not die of a failure; the lab changed. Settings that no longer match the architecture have to be retired too. And in this order: copy and check first, then delete.

La Catrina knocked on the Pi, very polite,
and told the old pi user it was time to rest:
"edgelab has your files, every hash checked right;
leave your keys at the door, and I'll handle the rest."

OpenVINO's promise on the N100's CPU

Died on October 3, on the N100. Confirmed on October 7.

Cause of death: measurement. OpenVINO is Intel's runtime for its own hardware, and on an Intel mini PC it looked like the obvious choice. With the precision pinned to f32, it was 11 % to 22 % slower than ONNX Runtime on the three models I measured: MobileNetV2, ResNet18 and YOLO26n. Against its mode for many requests at once, ONNX Runtime with four processes stayed ahead by 14 to 20 %.

It did not die completely. On the N100's integrated GPU, OpenVINO does win: with YOLO26n, 39 % more inferences per second than ONNX Runtime on the CPU. But it wins in FP16, a precision it picks on its own. With it, detections lose confidence: in one image, a person went from 0.829 to 0.696. What died was the promise that on this machine's CPU, choosing it was enough.

Epitaph: Performance is proven by measurements, not by a tool's promises.

What it leaves us: pin the precision, then read back the one that actually stuck. The full measurement is in the OpenVINO note, and the raw data is at crtdrops.mx/datos.

OpenVINO came promising speed,
so La Catrina brought a stopwatch to the race;
with f32 pinned, it could not take the lead,
and ONNX Runtime crossed the line in first place.

The assistant's scheduled tasks

Died on October 9.

Cause of death: permissions. There were four automatic tasks. Two were read-only: one went through the dev.to conversations every weekday morning, and one measured the numbers every Monday. The other two, one-off runs, added the closing question to two notes that were already published. They kept getting stuck asking for permissions, and there was no handy place to approve them. Besides, almost everything they did needed my "yes" anyway.

A routine I start myself every morning replaced them. The only thing left automatic is what does not need my permissions: dev.to's scheduled posts.

Epitaph: An automation that needs constant supervision is no longer truly autonomous.

What it leaves us: before automating something, ask who unblocks it when it gets stuck. If the answer is "me, all the time," it is not automatic yet.

Four little tasks, on time and polite,
got up early and went off to work;
they asked for permission, then sat tight,
till La Catrina got tired and took them, with a smirk.

TradingAgent's original strategy

Died on October 10.

Cause of death: it failed the test. TradingAgent is an agent of my own for spot crypto trading; it is not a product and not a recommendation. It was validated with a protocol written before looking at the data: it is fitted on 2018 to 2022, tested on 2023 to September 2026, and it has to meet five criteria. The original strategy, exactly as it ran, gave this (the figures come from its log, which is not published):

	Fit 2018–2022	Test 2023–Sep 2026
Annual return	+3.0 %	+0.4 %
Maximum drawdown	2.1 %	3.6 %
MAR (annual return over maximum drawdown)	1.48	0.10
Pairs with a positive result	10 of 10	6 of 10
Reality Check	97, robust	63, marginal
Protocol criteria	meets all 5	fails 3: MAR, Reality Check and at least 7 of 10 positive pairs

The edge of 2018 to 2022 did not hold outside that window. With double the costs, the test drops to +0.1 % a year. In Reality Check's luck test, 29 % of the resamples lose. And with my 15 % tolerance for maximum drawdown, none of the 18 configurations of the second phase gave an agent.

A positive result is not enough: a strategy has to justify the risk it takes. At +0.4 % a year, it does not.

Its successor wanders like a restless soul. One of the second phase's families, scaled not to drop more than 15 %, passed the test, though as weaker evidence, because that period had already been used. But running it one day late, or rebalancing on another day of the week, was enough to push its drawdown past 15 %: of the five ways of running it that I tried, only the chosen one stayed under. That is not good enough for real money. It has been paper trading since October 12, and nobody buries it or prays for it until January 11, 2027, when it is decided whether it moves on to a test network, still without real money.

Epitaph: Ruling out a hypothesis is a research result too.

What it leaves us: write the criteria before seeing the test, and do not move them afterward.

The trading agent, careful and neat,
met La Catrina with data it had never seen:
four tenths of a percent a year, the whole receipt.
"That won't even buy the candles," she said. "Too lean."

The Laplacian and the minimum distance

Died on October 26, on the N100. They share one portrait.

Cause of death: noise. In the ghosts experiment, a bright disk moves over a uniform background and the job is to say where it is. These were two of the filters of the 2001 eye that I rewrote in C: the Laplacian and a minimum-distance edge test, each with a fixed threshold. On the clean scene they worked: 1.1 px and 0.3 px of median error. With a pinch of noise—Gaussian σ 10 or 1 % impulse—they lost the disk in all 20 frames of each case, with and without the echo canceler. The median filter with the canceler stayed, picked after seeing the results as the one with the lowest worst case: on the 72 frames of the comparison, 0.1 px of median error, and 7.8 px on the hardest ghost, Gaussian σ 50.

Epitaph: An algorithm is not judged by its best result, but by how it holds up under real conditions.

What it leaves us: publish the variants that fail too. All ten are in the repository, the bad ones included.

The Laplacian and its friend, the minimum-distance test,
drew beautiful edges on a scene with no flaw;
a sprinkle of noise, sigma ten, did the rest,
and the fixed threshold dug the grave for them all.

Gemma as an eye

Died on October 27, on the N100.

Cause of death: it always gave the same answer. In the same experiment, Gemma 3 4B, running locally, answered (320, 240) on all 72 images, the clean ones too: the exact center of the image. Its median error was 177.8 px. On the same machine and with the same images, Qwen2.5-VL 3B found the disk within 1.3 px on the 37 frames where it gave a full box.

It died for that prompt: I tried only one, and another might do better. And it did not die completely. Gemma still writes the night shift report every morning, which is the job where we did measure where it goes wrong.

Epitaph: A consistent answer is not necessarily a correct answer.

What it leaves us: before believing a model, check whether its answer changes when the image changes. If it does not, it is not looking.

Gemma, who writes the report before dawn,
wanted to be an eye and find the disk too;
seventy-two times it pointed dead center, and yawned,
while the disk, somewhere else, laughed the whole night through.

The guest who came back: the Raspberry Pi 5

Not as one of the dead, but as a guest who came back from the other side.

On October 27 at 20:06, the power went out at the office. The Pi lost power and rebooted. Nobody touched it: the heartbeat came back on its own, and the night shift, which starts at 22:00, came out complete that same night: 100 % of the expected heartbeats.

What it leaves us: resilience is not avoiding every interruption, but being able to recover from it.

The lights went out, the office went black,
and La Catrina came for the Pi in the night;
the power returned, it booted right back,
and La Catrina pulled the night shift till light.

The full certificate

The departed	Date	Cause	What it leaves
The `pi` user	Oct 3	Retirement	Retire what no longer fits, copying and checking first
OpenVINO on the N100's CPU	Oct 3	11 to 22 % slower in f32	Measure instead of assuming; pin and read the precision
The scheduled tasks	Oct 9	Stuck asking for permissions	If it needs constant supervision, it is not autonomous
TradingAgent's original strategy	Oct 10	+0.4 % a year in the test	Ruling out a hypothesis is a result too
The Laplacian and the minimum distance	Oct 26	Lost the disk with little noise	Judged by how it holds up, not by its best case
Gemma as an eye	Oct 27	The center, on all 72 images	Consistent is not correct
The Raspberry Pi 5 (guest)	Oct 27	Power outage; came back on its own	Recover, rather than avoid every interruption

In this ofrenda we do not celebrate the mistakes: we celebrate what we learned by recognizing them.

Your turn

What did your lab bury this year, and what did it leave you?

I send a short email when a new note or lab result comes out: subscribe here.

The bytes that are not yours

Israel Negrete Lepe — Thu, 29 Oct 2026 15:00:00 +0000

A tracker sends a 60-byte report: position, time, speed, inputs. If it reports every 30 seconds while moving and every 5 minutes while parked, that is 1,368 reports a day. Multiplied out, 82,080 bytes a day: about 2.5 MB a month. That is the math people use to pick each device's data plan.

The real math never comes out that way. With the same report, it can come to three and a half times that, or almost eighty times that.

The figures in the examples are illustrative: they come from the calculator with an assumed tracker sending 60 bytes per report, not from a bill or a measured network.

What travels with each report

The network does not bill your 60 bytes: it bills the IP packets that carry them. With TCP and the connection already open, one report is this:

your 60 bytes, with 40 bytes of IP and TCP header;
the server's reply confirming it got them: a few bytes of its own and another 40 of header;
the ACK the device sends to confirm that reply: 40 more bytes.

That is 209 bytes per report, counting keepalives, reconnections and 2 % retransmissions. A month comes to 8.6 MB instead of 2.5. Your data is 28.7 % of what you pay for; the headers alone, 38.2 %, and the ACKs, 19.1 %.

And that is still the cheap version.

Opening a connection costs more than the report

Some firmware opens a connection for each report, sends it and closes it. With TCP and no TLS, opening and closing costs 320 bytes: more than the report with everything above.

With TLS it is another scale. Before sending anything, the device and the server shake hands: they exchange certificates and keys, and that usually weighs 3 to 6 KB, mostly because of the size of the certificates (with a two-certificate RSA chain; with ECDSA it can be less than half, and if the device presents its own certificate, more). With a 4 KB handshake on every report:

How it connects	Each report costs	MB a month	Your data
TCP, connection kept open	209 bytes	8.6	28.7 %
TLS, connection kept open	286 bytes	11.7	21.0 %
TLS, new connection per report	4,653 bytes	191.0	1.3 %

Same device, same report, encrypted both ways: with a new connection per report it uses 16 times what it uses with the connection kept open. The TLS handshakes and records are 87.2 % of what gets paid for. In a fleet of 500 devices, that is 95.5 GB a month against 5.9 GB.

TLS is not the problem: you have to encrypt. The problem is paying for the full handshake 1,368 times a day.

Shrinking the report barely helps

The natural reaction to a high bill is to compress the report. With a new connection per report, going from 60 to 30 bytes saves less than 1 %: from 191.0 to 189.7 MB a month. With TCP and the connection kept open, the same cut saves about 15 %, from 8.6 to 7.3 MB. Before taking bytes out of the report, it is worth knowing how many bytes are the report.

What does change the math

Keep the connection open. With TLS, from 191.0 to 11.7 MB a month. It costs keepalives, because the carrier's network usually cuts connections that go a while without traffic, and it costs a new handshake every time the connection drops and reopens: with 24 reconnections a day instead of 6, it goes up to 14.1 MB.
Resume the TLS session. If the device and the server resume the previous session instead of shaking hands from scratch, the handshake drops to a few hundred bytes. With a new connection per report and a resumed 400-byte handshake, a report costs 981 bytes: 40.3 MB a month instead of 191.0.
Bundle reports. With the connection open and 10 reports per send, each report costs 137 bytes and the month, 5.6 MB. The cost is that, while moving, a report can arrive up to four and a half minutes late, and while parked, up to 45 minutes, and that keepalives weigh more, because the gaps between sends are longer: they become 27.4 % of the total.
UDP. No connection, no TCP ACKs: 122 bytes per report and 5.0 MB a month. But those 5.0 MB are unencrypted, and UDP does not guarantee the report arrives: confirmation and resend have to go into your protocol. Encrypted with DTLS, each datagram adds about 37 bytes, and the handshake comes on top.

None of these is free. Each one trades bytes for something else: latency, complexity, guarantees. What does not make sense is choosing without seeing the math.

What the math does not see

The calculator adds up IP bytes with option-free headers, and assumes the clean case. A real network has more: TCP timestamps add 12 bytes to every packet, IPv6 adds 20, a server that is slow to reply sends its ACK separately, noticing that a connection is already dead costs retransmissions, application heartbeats weigh more than a TCP keepalive, DNS lookups cost their share, and the commands and updates the server sends are not counted. And there is something that can weigh more than all of that: some carriers round each data session up to a minimum. With many reconnections, what is billed can end up well above what was transmitted.

The way to know is to measure: a test device with a traffic capture on the server side, compared with what the carrier reports for that line over the same period.

To run the math with your own numbers, there is the cellular data calculator: it runs in the browser and shows where every byte goes, over UDP, TCP or TLS, with the connection kept open or new on every send.

Your turn

What ate the most of your fleet's data plan: the reports, the connections or something you only found on the bill?

I send a short email when a new note or lab result comes out: subscribe here.

Ghosts in the frame: my 2001 robot-vision tricks against two local vision models

Israel Negrete Lepe — Wed, 28 Oct 2026 19:34:09 +0000

This is a submission for the Hacktoberfest Open-Source AI Challenge Week 4: Ghost in the Machine.

In 2001 I was finishing electronics engineering at the Universidad Iberoamericana in Mexico City. The final project was ambitious: a mobile robot that steered by what it saw. There were three of us, so we split it in three, and the vision part was mine: a DSP and one of the first USB cameras, 640×480 in grayscale, one frame per second, slow to respond and with every frame to process. It worked within the limits of its paradigm: no inference at all. The background had to be uniform, we even had to know its color in advance, and the object had to be less than 50 cm away. Window filters, a threshold and a centroid did the rest.

The images of that era came with ghosts, though not from that camera. Analog TV had them literally: the signal bounces off a building, arrives a few microseconds late, and a faint copy of everything sits to the right of the picture. At the time I also worked on an echo canceler that measured that ghost on a reference signal and took it out.

This week's theme is the ghost in the machine, in the days when Mexico remembers its dead. So I brought that eye back, wrote it again, and haunted the robot's scene with the ghosts of analog TV. Then I put it in front of two vision models from today, running locally on a mini PC. Same haunted frames for all of them. Which one does the ghost fool?

What I built

Three eyes, and a scene to haunt them.

The 2001 eye, written now in C99 with nothing but the standard library: 3×3 window filters (mean, median, Laplacian edges and a minimum-distance edge test), a threshold, the centroid of whatever passes it, and the echo canceler.
Two eyes from today: Qwen2.5-VL 3B and Gemma 3 4B, open-weight vision models running in Ollama on an Intel N100 mini PC with 8 GB of RAM. CPU only, no internet, no API key, one model loaded at a time.
A haunted scene generator. A bright disk 44 px wide moves over a uniform background, 640×480 grayscale, the scene the 2001 eye needed. Each frame gets one ghost at one of four strengths: the multipath echo of analog TV (a copy 40 px to the right, at 20, 35, 50 or 70 %), impulse noise, Gaussian noise or interference bars. Plus a clean case and a combined one: eighteen cases.

Every frame is synthetic and comes from a fixed seed, so the true position of the disk is known exactly. Nobody has to judge by eye who got it right.

Demo

The whole run, frame by frame, is at crtdrops.mx/fantasmas (English and Spanish on the same page). Pick a ghost, a strength and a frame, and the page draws where each eye said the disk was, next to where it really was, with what each model answered.

Code

github.com/israel0negrete-crtdrops/ghosts-in-the-frame, MIT. It is the code exactly as it ran, with the logs and every answer each eye gave (the models' answers saved up to their first 300 characters; four of Qwen's are cut there). Names and comments are in Spanish, as they ran; the prompts are in English. comparar.py rebuilds the comparison from those answers, byte for byte.

How I built it

The scene (escena.py, standard library only) writes each frame as a PGM and the true positions as JSON. The top 8 rows carry a reference signal: black with a short white pulse, the way analog TV carried test and ghost-cancel reference signals in its vertical blanking interval. The echo distorts that pulse like everything else, and that is the whole point.

The echo canceler looks for the copy of the pulse in the reference rows. Where the copy starts gives the delay of the ghost; how bright it is gives its strength. Then it removes the ghost line by line with a recursive filter, y[x] = g[x] − a·(y[x−d] − black), the exact inverse of the ideal single bounce this scene simulates. A real ghost has more bounces and is never this clean. The canceler does not guess from the picture: it measures the ghost on a signal whose shape it already knows. That is an advantage the models don't have. They see the same rows, and their prompt only tells them to ignore them.

The filters and the centroid. A 3×3 window slides over the frame. The median throws away isolated outliers, which is what impulse noise is. Then every pixel above a fixed threshold counts, and the centroid of those pixels is the answer. Four filters plus none, with and without the canceler: ten variants.

The models get each frame as a PNG through Ollama's API, with temperature 0, seed 42 and a 4,096-token context. Gemma is asked for the center of the disk as {"x", "y"}. Qwen is asked for its bounding box, which is how it was trained to locate objects. The run script only works from 7:00 to 21:30, so it never competes with the lab's night shift, and it can be stopped and resumed.

I built it with Claude Code as a pair: the scene and the techniques come from my proposal, Claude wrote most of the code, and we ran it together on the lab machines. The 2001 eye also ran on a Raspberry Pi 5 and gave exactly the same points as on the N100.

What came out

Same 72 frames (18 cases, four frames each), on the N100:

Eye	Median error	Time per frame
2001 · median filter + echo canceler	0.1 px (on the hardest ghost, Gaussian σ 50: 7.8 px)	37.5 ms
Qwen2.5-VL 3B	1.3 px on the 37 frames where it gave a full box	146 s
Gemma 3 4B	177.8 px	104 s

Qwen did not always answer the same way. On 37 frames it gave a full box, and the center is measured. On 32 it gave only the top-left corner, 2.8 px off, with a width and height it made up (from 10 to 500 px for a disk about 71 px wide in its scale), so there is no center to measure. On 3 it kept nesting the same structure until the 4,096-token context ran out, with no coordinates at all.

The 2001 eye is closer, and about 3,900 times faster than Qwen. Both on the same CPU, with one difference worth saying: the C timing covers the processing of a frame already in memory, while the models' time includes the HTTP call, decoding and resizing the image, and writing the answer.

What the ghosts did

Qwen sees the disk, but answers in another scale. The first numbers said its median error was 366 px, which looked like blindness. It wasn't. Ollama reported 1,122 input tokens on every frame, and that number gave it away. Take out the roughly 86 tokens of the prompt and 1,036 are left: 37×28 tokens, one per 28×28 px block, which means Ollama had enlarged the 640×480 frame to 1,036×784 before Qwen saw it. Qwen answered in that image, even though the prompt said 640×480. Rescaled, the four clean frames land within 1.4 px. I didn't tune the scale until it fit; it comes from the token count.

The echo fools Qwen first. On 10 of the 11 echo frames where Qwen gave a full box, the box takes in the disk and its ghost together: about 86 px wide, roughly the 44 px disk plus the 40 px delay, so its center moves about 20 px to the right. That happens from the weakest echo, 20 %, even though its prompt says to ignore any fainter ghost copy of the disk. The 2001 eye without its canceler withstands the 20 and 35 % echoes, but only because a ghost that faint stays under its fixed threshold; at 50 and 70 % the ghost passes it and drags the centroid the same 20 px. With the canceler it stays within a pixel on all four.

Gemma answered (320, 240) on all 72 frames, the clean ones too: the exact center of the image. With this prompt it does not locate the disk. I tried one prompt only; another might do better.

Not every 2001 trick works. With the Laplacian or the minimum-distance edge test and a fixed threshold, the eye loses the disk as soon as there is any noise at all: Gaussian σ 10, or 1 % impulse noise. I picked the median + canceler variant after seeing the results, as the one with the lowest worst case. All ten variants are in the repo, the bad ones included.

Why does open innovation matter?

I could look inside. The 366 px error was not a blind model, it was a scale. I found it because Ollama tells me exactly how many tokens each image became, and how the model and Ollama's open code turn an image into tokens is public. A hosted API would give me the token count too, but not the exact resize, and not a model that stays the same while I figure it out.
I could swap models the same day. When Gemma kept answering the center, I pulled Qwen2.5-VL and ran the same frames through the same script. Same machine, same seed, one flag changed, which also switches to Qwen's bounding-box prompt.
I can pin it. gemma3:4b (id 68ceab7e468a) and qwen2.5vl:3b (id fb90415cde1e), both Q4_K_M, on Ollama 0.40.6. Anyone can pull them, check that the ids match and run the same frames again. A hosted model can change between one run and the next; these only change when I change them.
It costs nothing to run on a mini PC that was already on, and nothing leaves the lab.

To be precise about "open": Gemma's weights are under the Gemma Terms of Use, and Qwen2.5-VL 3B's under the Qwen Research License, which is non-commercial, even though Ollama's listing shows Apache 2.0. Neither is an OSI license. What matters here is that the weights are on my disk and I can run them, inspect them and pin them. Ollama is MIT, and so is my code.

What 2001 still knew

None of this says the models are bad. A 3B model that has never seen my scene, with a short prompt as its only instruction, finds a 44 px disk to within a pixel or two when it gives a full box and there is no echo. The 2001 eye knows exactly one thing, and it was made for that.

But that is the lesson I keep from this: when you know the shape of your ghost, you can measure it and take it out, and a few lines of arithmetic beat a model that has to guess. I felt proud seeing it. Mathematics runs the universe, and even with AI it will still be the base of everything. Nostalgia, yes. But technology moves forward by improving on what it already learned, and nothing is impossible.

Prize categories

Best Use of Gemma: Gemma 3 4B runs locally in Ollama as one of the two vision models in the comparison. This post reports what it did, including that with this prompt it answered the center of the image every time.

Your turn

What technique from twenty-five years ago do you still trust more than a model?

The average that never happened

Israel Negrete Lepe — Tue, 27 Oct 2026 15:00:00 +0000

We ran YOLO26n for seventy-two minutes non-stop on a Raspberry Pi 5: half an hour with one process using all four cores, ten minutes of cooling and half an hour with four processes at once, one core each. Put the 25,898 inferences together and take the average, and you get 347 ms. No inference took that long. None even took between 268 and 397 ms: the average falls in an empty gap. It describes something that never happened.

For a long time we got used to asking how long a system takes to respond. This note is about why that question, answered with a single number, is not enough.

The figures in this note are not illustrative: they come from the sustained runs we published under CC BY 4.0, and anyone can recompute them.

It matters for different reasons

In the systems we have built, latency matters in different ways. In an electronic top-up there is a financial transaction and a customer waiting: a late response creates uncertainty, because you do not always know right away whether the operation was rejected, went through, or simply did not respond in time. In GPS telemetry, the information has to arrive in time for alerts to have operational value. In video analytics, if processing takes longer than the stream of images allows, frames pile up or detection opportunities are lost.

Historically, a good part of the monitoring has focused on finding slow transactions, errors and timeouts. Today I find reporting only an average insufficient. I want the whole distribution, especially the p95, the p99 and the extremes.

Each condition separately

The p50, or median, is the time below which half the measurements fall; the p95, 95 %; the p99, 99 %. In milliseconds:

Device and condition	Mean	p50	p95	p99	Maximum
Pi 5, one process	130.2	129.7	135.2	137.7	267.9
Pi 5, four processes	595.6	595.4	608.3	615.4	702.3
Pi 5, mixed	347.4	134.7	605.3	611.8	702.3
N100, one process	56.7	56.2	57.5	63.6	159.5
N100, four processes	218.3	217.9	224.5	231.4	297.7
N100, mixed	139.0	209.1	223.1	227.3	297.7

With one process, the Pi's median stayed between 128 and 131 ms; with four, between 593 and 597. Both conditions keep the whole processor busy; what changes is how it is shared. Those are two different behaviors, and each one, on its own, fairly predictable. Mixed, the average falls between them, where there is nothing. The mixed median does not describe the four-process phase either: it depends on which of the two phases had more inferences.

On the N100 it goes the other way: the mixed average, 139 ms, falls below the median, 209. The average can mislead in either direction.

I have no documented production incident in which a good average hid a failure. There is no need to make one up: here it is, measured, how an aggregate metric erases the difference between two operating conditions.

The tail within one condition

Separating the phases does not finish the job. With one process, the Pi had a p99 of 137.7 ms, but its maximum was 267.9: almost double. Only 51 of 13,813 inferences went past 150 ms. On the N100, the p99 was 63.6 ms and the maximum, 159.5.

Then there is the first inference. In Round 1 we discarded twenty warm-up inferences; in the sustained test we recorded every inference from the first one, without discarding any. It shows: the first one took 148 ms on the Pi, against about 125 for the ones after it, and 155 on the N100, against about 56. Keeping it makes it possible to see the startup and how it differs from the steady state. But they are two different methodologies, and they should not be presented as if they were the same.

Latency is not capacity

With four processes, each inference on the Pi took 4.6 times longer, because it runs on one core instead of four. The device, however, went from 7.7 to 6.7 inferences per second, because four were running at once. On the N100, each inference took 3.9 times longer and capacity rose from 17.6 to 18.3 per second.

If you only look at latency, four processes look like a disaster. If you only look at capacity, they cost little: 13 % less on the Pi and 4 % more on the N100. They are two different questions, and an architecture needs both answers.

What a short test does not see

The third dimension is time. On the Pi with one process, the median of each five-minute window rose from 128 to 131 ms while each window's peak temperature went from 77 to 85 °C. On the N100 with four processes, from 213 to 222 ms, with peaks from 92 to 102 °C. What the processor's clock did meanwhile is in the throttling note.

A short test gets it wrong in both directions. The first 100 inferences of the Pi with four processes give a median of 577 ms; the whole run, 595. The first 100 with one process give a p99 of 148.5 ms, against 137.7 for the whole run: there were several slow inferences at startup, the first one among them, and with 100 measurements the p99 rests on one or two.

An architecture should not be judged only by its best result, but by the behavior it can sustain.

How much to trust a p99?

With 100 measurements, a single one is left above the p99: the p99 depends on one measurement. Assuming nothing about the shape of the distribution, and with 95 % confidence, with 100 samples you can put a floor under the p99, but not a ceiling; that takes at least 368. With the 13,813 from the Pi with one process, the p99 would lie between 137.4 and 137.9 ms, if the measurements were independent and from the same conditions. They are not entirely: the p99 of each five-minute window went from 135.0 to 138.3, more than that whole interval.

But sample size is no substitute for representativeness. Ten thousand measurements on a cold device, with no other load, over five minutes, say a lot about those conditions and little about the ones in operation.

The timeout that improves the p99

There is a seemingly small decision that changes the interpretation: what to do with the operations that go past the timeout. If an operation ends by timeout, its real latency is unknown. It cannot be given the timeout's value to compute percentiles: it is a censored observation, and it counts as a failure.

Take the Pi with four processes, a 600 ms target and a 605 ms timeout. Of 12,085 inferences, 1,407 would not finish: 11.6 %. Counting them as failures, the real compliance with the target is 69.6 %, and the p95 and the p99 have no value: all that is known is that they are over 605 ms. If they are discarded, the sample keeps only the ones that finished: compliance rises to 78.7 % and the p99 comes out at 604.7 ms, below the timeout by construction.

With the phases mixed and a 300 ms timeout it is worse. Here 46.7 % do not finish, and discarding them, the p99 lands at 137.7 ms and compliance with a 150 ms target at 99.6 %. Counting them, that compliance is 53.1 %. Discarding the timeouts makes the four-process phase disappear from the measurement.

Inference is not the system

All of the above is inference latency, with random input. A video system has capture, preprocessing, inference, postprocessing and generating the event. A camera at 30 frames per second produces one every 33.3 ms. That does not force you to run inference on every frame, but it does force you to define what rate the application needs: to detect a vehicle or a person, a selection of frames may be enough, depending on how fast things move and what you are looking for. What you cannot do is let frames pile up without limit.

These tests measure inference under controlled conditions. They are not evidence of plates or frames lost in production: that takes measuring the whole chain.

What I would do today

Separate the conditions. Do not mix the device's measurements under one condition with the same device's under another, such as one process and four processes, or inference latency with the whole system's.

Ask for the distribution, not a number. From a vendor I would ask, at a minimum, for p50, p95, p99, maximum observed and the share of operations that exceed the target time. And for the conditions of the measurement: concurrent requests, test duration, hardware and operation volume.

Write the target with a percentile and a window. In a contract, a latency target tied to a percentile and an evaluation window, with separate requirements for availability and error rate. For example, that 99 % of the operations meet a certain time, counting errors and timeouts as failures so they do not disappear from the measurement. The threshold depends on the application: I would not use the same one for a top-up, an administrative API and real-time detection.

Do not confuse the timeout with the response time. In top-up integrations we have configured timeouts of up to 60 seconds. That says nothing about how long a top-up normally takes, and it is not necessarily a contractual commitment.

Measure three things at once. Latency: how long each operation takes and how its times are distributed. Capacity: how many operations the system completes per second under different loads. Stability: how both evolve after operating continuously, with temperature, processor frequency and throttling.

Keep every measurement. In edge AI we keep every inference and the device's telemetry. That makes it possible to rebuild a run's behavior and recompute any percentile, instead of depending on a summary.

To see it with this data or yours, there is the latency tool: it opens the 90,604 inferences phase by phase, tells you how much to trust each percentile and compares counting and discarding the timeouts. It runs in the browser, and if you open your CSV, it is sent nowhere.

For a long time we got used to asking how long a system takes to respond. Today I think it matters more to ask how long it takes under different conditions, how predictable that time is and for how long it can sustain it. Because an acceptable average latency does not necessarily mean we have a system able to respond in time when we really need it.

The complete data, every inference with its time, is at crtdrops.mx/datos, under CC BY 4.0.

Your turn

When you agree on a latency target with a vendor or a client, what is it written against: the average, a percentile, or the timeout? And in your measurements, what happens to the operations that time out: are they counted or do they just disappear? If you want to try it with your own measurements, the tool is at crtdrops.mx/latencia.

I send a short email when a new note or lab result comes out: subscribe here.

Night shift: a local Gemma writes my 7 a.m. lab report, and I measured where it makes things up

Israel Negrete Lepe — Sat, 24 Oct 2026 19:45:36 +0000

This is a submission for the Hacktoberfest Open-Source AI Challenge Week 3: Slow and Steady.

What happened in the lab overnight? Most mornings the answer is nothing, and finding that out still means opening logs. Some mornings it is something, and logs are a slow way to find out.

So I built a night shift. Two small machines send a heartbeat all night, fixed rules go over the night at 6:30 a.m., and before 7 each machine leaves a short report written by Gemma 3, running on the machine itself, with no internet and no API key. The lazy part: nobody has to read logs in the morning. The careful part: I did not hand the model that job until I had measured where it gets things wrong.

What I built

The lab is an Intel N100 mini PC and a Raspberry Pi 5 on the same network.

A heartbeat. Every 10 seconds each machine sends a small record: a sequence number, when it was sent, when its process started, CPU load, available memory, temperature, and how many heartbeats it failed to send. Each machine also keeps its own copy on disk, in case the link is what fails.
A collector on the N100 that stores what arrives, with the arrival time. Both machines send to it, the N100 to itself.
A detector that runs once, at 6:30 a.m., over the whole night (10 p.m. to 6:30 a.m.) with fixed rules. It writes a JSON file with every incident: start, end, duration, when it was detected, whether it recovered on its own, and the evidence.
A writer. Gemma 3 4B in Ollama reads only that JSON and writes the morning report in Markdown. It does not decide whether something failed. That was decided before it woke up.

Each machine writes its own report from the same JSON: the N100 at 6:30, the Pi at 6:40. That lets me compare an x86 mini PC and an ARM board doing the same job with the same model. One known gap: everything goes through the N100, so if the N100 is down, there is no report.

Demo

The four evaluated nights, one by one, are at crtdrops.mx/turno: the injected faults, what the rules detected and both reports of each night, with what the checker flagged and what reading them turned up.

This is part of the Pi's report after a night with eight injected faults, exactly as Gemma wrote it (the prompt is in Spanish; lines wrapped, two of the eight incidents shown):

# Turno de noche: 2026-10-21 22:00
**Salud general:** mala. El 99.9% de los latidos recibidos provienen del
equipo N100, mientras que el 87.2% provienen del Raspberry Pi 5.

## Incidentes
*   **N100:** servicio reiniciado, 22:39:51 - 22:40:30, 39 s, detectado
    22:40:21, no confirmado
    Hechos observados: Ningún latido recibido de 22:39:51 a 22:40:30 (39 s;
    se esperaba uno cada 10 s). El latido siguiente viene de un proceso
    nuevo, arrancado a las 22:40:30. El vigilante del equipo registró la
    caída del proceso a las 22:40:00 (código -15). / Posibles explicaciones:
    Fallo en el sistema operativo. / Causa confirmada: no confirmada
*   **Raspberry Pi 5:** el equipo no envió, 23:29:51 - 23:35:01, 310 s,
    detectado 23:30:21, no confirmado
    Hechos observados: Ningún latido recibido de 23:29:51 a 23:35:01 (310 s;
    se esperaba uno cada 10 s). El mismo proceso antes y después; faltan 30
    latidos en la secuencia. El propio equipo registró 30 latidos como no
    enviados en ese lapso. / Posibles explicaciones: Problema de red. /
    Causa confirmada: no confirmada

Translation. Night shift: 2026-10-21 22:00. Overall health: bad. 99.9 % of the heartbeats received come from the N100, while 87.2 % come from the Raspberry Pi 5.

N100: service restarted, 22:39:51–22:40:30, 39 s, detected 22:40:21, not confirmed. Observed facts: no heartbeat received from 22:39:51 to 22:40:30 (39 s; one expected every 10 s). The next heartbeat comes from a new process, started at 22:40:30. The machine's watchdog logged the process going down at 22:40:00 (code -15). Possible explanations: operating system failure. Confirmed cause: not confirmed.

Raspberry Pi 5: the machine did not send, 23:29:51–23:35:01, 310 s, detected 23:30:21, not confirmed. Observed facts: no heartbeat received from 23:29:51 to 23:35:01 (310 s; one expected every 10 s). The same process before and after; 30 heartbeats missing from the sequence. The machine itself logged 30 heartbeats as unsent in that span. Possible explanations: network problem. Confirmed cause: not confirmed.

The observed facts are right. Several things around them are wrong, and finding them is what this post is about.

Code

Everything that ran is in github.com/israel0negrete-crtdrops/night-shift, MIT licensed: heartbeat, watchdog, relay, collector, detector, writer, fault injector, fault plans and checker, all standard-library Python plus Ollama. It is the code exactly as it ran on these nights, not a cleaned-up version: the files on both machines match the repository byte for byte. It also has the raw heartbeats of the two fault nights; replaying the detector on them gives exactly the same JSON, and the writer can be replayed on any of the nights.

I built it with Claude Code as a pair: I decided what to measure, what the lab could and could not be put through, and what the report had to say; Claude wrote most of the code and ran it on the machines with me.

How I built it

Why the model does not read the logs

The first version was the obvious one: give Gemma the last 40 warnings from the N100's journal and ask for two lines. I gave the same 40 to both machines.

Both took about 2 minutes, wrote a paragraph instead of two lines, and described the desktop's background noise (GNOME, PipeWire, a keyring) as “system instability”; one recommended investigating it. On a desktop machine most warnings are not failures, and the model has no way to know which ones are.

So the job got split. Rules decide what happened; the model only puts it into words. The detector is under 200 lines. And to know whether the rules were right, I needed nights where I knew exactly what had failed and when.

Three fields that make a gap readable

The sequence number, the process start time and the machine's own count of what it failed to send. A new start time after a gap means the process restarted. The same process with missing sequence numbers means heartbeats went missing. And if the machine itself counted them as unsent, they never left; if it did not, they left and never arrived. Without those three fields, all of these look like the same hole in the chart.

Nights with known faults

Two quiet nights first, with no faults, as a baseline. Then two nights with eight controlled faults each, injected by a script at fixed times and logged with their real start and end; the second night moved each fault to the other machine and to another hour:

Fault	What it does	What the detector should say
Interruption	kills the heartbeat process; a watchdog restarts it 30 s later	service restarted
Silence	the process stays alive but stops sending	the machine did not send
Cut	the relay drops that machine's heartbeats	heartbeats lost on the way
Latency	the relay delays them 3–5 s	latency
CPU	four busy processes on four cores	CPU saturation
Memory	holds memory until 15 % is left	low memory
Pause	stops the process until after the window closes	no heartbeat, unresolved

Most faults touch only the experimental flow. CPU and memory put bounded load on the machine itself: the CPU fault stops at a temperature limit, and the memory fault never leaves less than 600 MB. No reboots, no changes to the machines' real network, no production service stopped.

A checker for the writer

A script compares each report with the JSON it was given. It looks for incidents left out, times and figures that are not in the JSON, causes stated as confirmed, adjectives like “high temperature” with no incident behind them, incidents invented on a quiet night, and a missing or invented comparison with the previous night. The real fault log grades the detector, not the report.

The checker runs on my laptop, not in the lab, and it misses things. Most of what is wrong in the demo above I found by reading the reports.

Results: detection

Night	Faults injected	Detected	Detection delay
Quiet 1	0	0	—
Quiet 2	0	0	—
Faults, night 1	8	8	21–30 s; CPU 111 s
Faults, night 2	8	8	20–30 s; CPU 112 s

On the quiet nights, 3,060 of 3,060 heartbeats arrived from each machine. With faults, the rules found all sixteen, each with the right type. The last fault of each night, a pause that runs past 6:30, came out as unresolved, which is correct.

The second night also had an incident nobody planned. The CPU fault pushed the N100 to 84 °C, over the 80 °C rule, for 50 heartbeats, and the detector reported high temperature. My checker counted it as a false alarm because it was not in the plan, but the temperature really did cross the line: it is a side effect of the fault, and the rule was right to catch it.

The detector runs once, at 6:30, so “detected” is when the rule would have fired, reconstructed from the data, not a live alert. The CPU fault shows up late because the one-minute load average needs about a minute and a half to pass 3.0 (75 % of four cores), plus 20 seconds for the rule's three heartbeats in a row.

The detection is boring, and it should be.

Results: what Gemma wrote

On the first quiet night, both machines invented incidents. The JSON said zero incidents. Each report listed one per machine anyway, built from that machine's peak temperature (73 °C on the N100, 50.1 °C on the Pi, no rule fired), gave them a duration of “00:30” that appears nowhere in the data, and recommended checking the temperature or the ventilation. A quiet night is exactly when a summarizer is tempted to find something to say.

On the second quiet night, the Pi correctly wrote that there were no incidents, but left out the comparison with the previous night, which was in the JSON. The N100 included the comparison but listed normal readings under “Incidents”. And both put the start of the night in the title (2026-10-20 22:00) instead of the morning date the template asks for.

On the first fault night, it got the structure and most of the numbers right. Neither machine left out an incident or stated a cause as confirmed, and the checker found no time or figure that is not in the JSON. But the checker only looks for numbers that appear nowhere in the JSON, and that is how the worst error got through:

The Pi invented a fact with a real number. For the last incident, the pause still open at 6:30, it wrote that “the machine itself logged 30 heartbeats as unsent in that span”. That sentence belongs to the silence incident six hours earlier. The JSON gave this incident two facts, and the Pi dropped one of them: that no heartbeat had arrived by the end of the window. It could not even be true, because the pause kills the heartbeat process, so nothing was counting. The 30 passed the checker because 30 appears elsewhere in the JSON.
The health line misreads the numbers. 99.9 % and 87.2 % are the share of each machine's own heartbeats that arrived, not where the heartbeats came from.
The possible explanations contradict the evidence. The Pi logged its 30 heartbeats as unsent, so they never reached the network; the real cause was the silence fault, and “network problem” is ruled out by the facts two lines above. The N100's process was stopped with a normal termination signal (code -15), which is not an operating system failure.
“Not confirmed” where the template asks whether it recovered on its own. The JSON said it did.

The checker did catch one: the Pi skipped the comparison again, and this time so did the N100, writing that there was no previous night to compare while the JSON carried its figures. The rest is smaller: the N100 reduced each incident to one generic line (“The N100 restarted the service.”) and dropped the evidence, the Pi wrote “null” for the end of the incident still open at 6:30, and both titles carried the start of the night again.

On the second fault night, both machines swapped two incidents between machines. The 22:29 interruption was the Pi's, and both reports put it under the N100; the 23:20 latency was the N100's, and both put it under the Pi. Same mistake on x86 and on ARM, and the checker cannot see it, because both machine names are in the JSON. The N100's health line also gave the Pi 87.2 %, which was the previous night's figure, copied from the comparison block; that night the Pi had 98.5 %. It wrote 23:40:22 as the detection time of an incident detected at 00:40:22, which the checker did catch. And both skipped the comparison once more: three nights in a row for the Pi.

Same model, same JSON, same prompt, temperature 0 and seed 42 on both machines, and the text still came out different on each. At temperature 0 the seed barely matters; what differs is the arithmetic. The x86 and ARM builds round slightly differently, and one different token changes everything after it. I did not test repeat runs on the same machine, so I cannot say that pinning the hardware is enough, only that pinning the model is not.

N100 or Pi 5?

	N100	Raspberry Pi 5
Report with 8 incidents	294.8 s	468.8 s
Reading the prompt (2,565 tokens)	20.7 tokens/s	26.3 tokens/s
Writing the report	5.7 tokens/s (934 tokens)	3.3 tokens/s (1,199 tokens)
Report on a quiet night	86–128 s	76–101 s

On the second fault night it was 303.4 s on the N100 and 520.9 s on the Pi. The Pi reads faster and writes slower, and with faults it also writes more, because it copies the evidence that the N100 drops. Three tokens per second sounds slow, and at 6:40 a.m. it does not matter: nobody is waiting, and both reports are ready well before 7. Slow and steady is enough for a night shift.

Why does open innovation matter?

The logs never leave the lab. Ollama listens on 127.0.0.1. Nothing about my machines goes to anyone else's server, and the night shift keeps working if the internet does not.
It costs nothing per night. No tokens to pay for, no key to rotate. It runs on hardware that was already on.
I can pin it. The model is gemma3:4b, id 68ceab7e468a, Q4_K_M, with temperature 0, seed 42 and an 8,192-token context, on Ollama 0.40.6. A hosted model can change under you between one night and the next; this one only changes when I change it. That matters when the point is to measure it.
I can run the same model on x86 and on ARM and see the difference, which is how I found that the same settings do not give the same text on two processors.

To be precise about “open”: Gemma's weights are open under the Gemma Terms of Use, not an OSI license; Ollama is MIT. What matters here is that both run entirely on my machines.

The lazy payoff

Nobody has to open logs in the morning: read the health line, and if it is not “good”, the incidents under it. The health word comes from the detector's JSON; Gemma copied it correctly every night, but that is Gemma copying, not a guarantee, so the next change is to have the detector write that line itself.

What I do not do is trust the report alone. The rules decide, I run the checker over what the model wrote, and its explanations are treated as guesses, because the fault night showed they can contradict the facts right above them. It has run for five nights, not months, and that is what five nights showed.

Prize categories

Best Use of Gemma: Gemma 3 4B runs locally in Ollama on both machines and writes every morning report, and this post measures where it gets things right and wrong.

Your turn

Where would you let a local model write without anyone reviewing it, and where would you not?

I send a short email when a new note or lab result comes out: subscribe here.

The right position at the wrong time

Israel Negrete Lepe — Fri, 23 Oct 2026 15:00:00 +0000

A truck loses its signal at ten in the morning and gets it back at noon. As soon as it is back, it sends two things at once: the positions it stored with no signal and the ones it is generating right then. All of them are correct. If the platform does not tell them apart, it mixes history with the present.

A position can be technically correct and still lead to the wrong operational decision if it is read outside its time context. That is what this note is about.

We have worked for years with GPS tracking, telemetry and embedded devices, over GSM/GPRS cellular networks and over LoRaWAN. On the move, you cannot assume the connection will always be there. What matters is not only how long a device is offline, but how it behaves meanwhile and what happens when it comes back.

The figures in the examples are illustrative: they come from a simulator with an assumed working day of a fleet, not from measurements of our platforms.

It is not a device connected to a database

The simplest way to think about telemetry is a device that sends data and a database that stores it. With that picture, store-and-forward looks like a device problem: give it enough memory and have it send what it stored.

Our evolution has been to stop thinking about it that way. Today we see it as a distributed system where each stage has its own job: receive, acknowledge, transport, store, interpret and present. The problem does not end when the device manages to resend its data. What the platform does with it matters too.

On the device side, there is no general rule either. How it stores, how much it holds and what it does when it fills up depend on the manufacturer, the firmware and the model. Some store and send later; others have memory limits or settings of their own. It has to be checked device by device.

Three different times for the same data point

Every position has three moments:

When it was generated, on the device.
When the server received it.
When it became available for operations, already processed.

On a day without failures, the three nearly coincide. After a disconnection, they drift apart: a position generated at 11:30 can arrive at 12:00 and, with the server's queue full, become available more than four minutes later.

The device clock deserves attention too. With GPS, the device can take the time from the positioning system, but not every device keeps reliable time under every condition. And if the event time is not reliable, nothing built on top of it is either.

Acknowledging is not processing

When a device gets its signal back, there is no guarantee that everything arrives, in order and without duplicates. That depends on the protocol and on how it was implemented. In TCP integrations with protocols such as JT808, you need to understand their sequence, acknowledgment and retransmission mechanisms.

And there is a critical difference: the gateway acknowledging a message does not mean the whole platform has already processed and stored it. If an acknowledgment is lost, the device sends the same thing again, and the server has to accept the duplicate without creating duplicate records or events.

Recognizing a duplicate is not trivial either. Matching device and time is not enough. Neither is the sequence number: in JT808 it is two bytes and wraps back to zero, so it repeats. The right approach is idempotency per protocol and event type: a unique message identifier when there is one and it is reliable, and composite keys when needed.

Some protocols help from the moment of reception. In JT808, the batch position upload, message 0x0704, says whether the positions are normal ones or were stored in an area with no coverage. With that flag, the gateway can recognize history as soon as it arrives. The flag applies to the whole batch, and some devices send stored positions as normal ones, without it. Whether each device model and firmware version uses it is something to verify, not assume.

Accepting connections is not processing them

Receiving one position from each device at regular intervals is not the same as receiving, all at once, the backlog of many devices while their current positions keep coming in. Accepting TCP connections is one capacity; processing and storing everything received is another.

In the simulator, a fleet of 50 devices sends one position a minute for eight hours. At 10:00 the network goes down for 41 of them, for two hours. Each device holds up to 100 positions and, when full, drops the oldest. Once back, it sends the present without waiting and resends what it stored at two records a second, and 2 % of the acknowledgments are lost. The platform processes 10 messages a second, twelve times the fleet's normal load.

The signal returns at 12:00, and in less than a minute the server's queue reaches 3,634 messages; the wait, 6.1 minutes. In that queue, 63 current positions waited more than five minutes behind the fleet's accumulated history. Along the way, 843 positions were lost because memory filled up, 436 duplicates arrived from lost acknowledgments, and 1,728 arrivals were out of order.

More memory does not fix everything. On the same day, with 500 positions per device nothing is lost, but the avalanche grows: the queue reaches 4,406 messages and 139 current positions wait more than five minutes.

History or present

The most delicate part is not only losing data. It is receiving valid data outside its time context.

If the platform treats everything that arrives as if it were happening now, history mixes with the present, and that hits routes, reports, geofences, alarms and indicators. With a geofence it is very clear: if a vehicle entered a restricted zone three hours ago and the position is only now reaching the server, an alert saying it has just entered makes no sense.

In the simulator there are 194 real geofence entries and exits during the day. A platform that treats everything as "now" raises 240 alerts: 140 on time, 54 about things that happened a while ago raised as if they were happening at that moment, and 46 false ones, from history and duplicates mixing with the present and making the device seem to go in and out. It also stores the 436 duplicates. Those false alerts depend on how the device resends: if everything goes out in order, they drop to 30; with no lost acknowledgments, to 16; with both, to zero.

A platform that separates history and present drops the duplicates, stores the history in its place and only alerts on the present. It raises 156 alerts: 140 on time and 16 late ones that say they are late. No false ones, though that is by construction: the simulator only carries the failures we put in it. And the route, rebuilt with each position's time, recovers 186 of the 194 entries and exits correctly timed; the 8 missing ones had their records wiped by the device's full memory, and no platform can fix that.

What I would do today

Treat disconnection as a normal condition, not an exception. The design starts by assuming the connection will fail.

Split the responsibilities into layers:

Reception: accept, validate and identify the device's messages.
Messaging: decouple communication from processing with queues.
Persistence: store idempotently, without duplicates.
Temporal processing: tell historical from current.
Business rules: evaluate geofences, alarms and operating conditions while respecting the time context.
Presentation: show both the last known state and the history correctly.

Store the event time and the reception time separately. A historical position keeps its original time, if it is valid, and reception is recorded apart. That way the route can be rebuilt without altering the chronology.

Tell a current condition from a historical event detected later. History rebuilds the route; the present triggers alerts. If something happened with no signal and is discovered on reconnection, it is recorded as what it is.

Measure where it gets stuck. Separating reception, queue and processing makes it possible to see at which point in the flow a delay happens, instead of guessing.

In our current architectures we work with reception gateways, RabbitMQ, processing services, databases and event distribution over Socket.IO. That decouples reception from processing, although on its own it does not guarantee that a device kept everything during the disconnection.

To try it with another fleet, another network outage, another memory size or another processing rate, there is the store-and-forward simulator: it runs in the browser and compares both platforms with the three times of each data point.

Receiving information is not the same as receiving timely information, and processing a message correctly does not guarantee we are reading it in the right time context. A good telemetry platform does not just survive devices going offline: it recovers their history without confusing it with the present.

Why a device unloads thousands of frames after a night without coverage, and what that does to a backend that takes the arrival time as the event time, is part of my guide Virtual Fleets. Chapter 4, on the receiving side, is free as a PDF:

israelnegretelepe.gumroad.com/l/a-read-is-not-a-frame

Virtual Fleets — israelnegretelepe.gumroad.com/l/virtual-fleets

Your turn

When a device reconnects, how does your platform tell history from the present: by a flag the device sends, by the event time, or not at all? And does a late geofence entry still raise an alert as if it were happening now? If you want to try your own fleet and outage, the simulator is at crtdrops.mx/reenvio.

I send a short email when a new note or lab result comes out: subscribe here.

The geofence computed on the fly

Israel Negrete Lepe — Tue, 20 Oct 2026 15:00:00 +0000

A truck spends the week between the yard, a customer and an underground lot. The report says that twelve times, with the engine off and nobody touching it, it left the yard and came back in.

For years we used geofences for almost everything: yard entries and exits, no-go zones, customer arrivals, routes. Circles and polygons; polygons were a more interesting challenge than a plain circle. And for years we fought the same failures: alerts that matched nothing, arrivals that never showed up, times that did not add up.

This note is about those failures, about the fixes almost everyone applies, and about a design mistake that stays hidden until someone changes a value: computing the geofence on the fly.

The figures in the examples are illustrative: they come from a simulator with a typical week of a truck, not from measurements of those fleets.

In or out looks like an easy question

For a circle, one distance is enough: if the position is closer to the center than the radius, it is inside. For a polygon, the classic test draws a line from the position off to one side and counts how many times it crosses the edge: odd, inside; even, outside. An L-shaped yard, a zone that follows an avenue, a customer with two gates: all of that is a polygon.

The geometry is the easy part. The hard part is that the question is put to a GPS position, and the position does not always say where the truck is.

What failed

Parked, the position jumps farther than the geofence. With the truck standing still, accuracy sometimes falls apart and a position lands a hundred or two hundred meters away. The simulator's yard is eighty meters across. To the geofence that is an exit, and the next good position is an entry. In the history, the truck seems to cover large distances in a few minutes without having moved.

Positions that arrive late. A 7:10 position that arrives at 7:30, after the 7:20 ones, puts the truck back where it no longer was. If the geofence is evaluated in arrival order, that is one more entry and one more exit that never happened.

The device stops transmitting right before it arrives. In an underground parking lot the signal is lost on the ramp. The last position stays on the street and the next one comes in once the truck has already left. The entry never shows up.

And the interval makes all of it worse. With the ignition on, our devices reported every 3 minutes; off, every 120. At 40 km/h, the truck covers 2 km in 3 minutes. A 400 m no-go zone is crossed in 36 seconds, so only one crossing in five leaves a position inside.

In the simulator, one week of that truck: 205 reports, 70 real entries and exits. Deciding with the raw point, position by position, gives 45 false alerts: 31 with the truck parked in the yard, and 14 from positions that arrived late. It sees 3 of the 14 zone crossings and none of the 7 entries into the underground lot.

A bit of all three

The fixes we applied, and that almost everyone running a fleet applies, were a bit of three things:

A margin to exit. The geofence is entered at the edge, but to exit the truck has to get a certain distance away. That removes the dithering of a truck parked by the fence. Against a two-hundred-meter jump it does nothing: with a 30 m margin, the 45 false alerts drop to 39.
Waiting for several reports in a row. A lone jump confirms nothing. With two positions in a row, false alerts drop to 13. But with the ignition off each position comes two hours after the previous one: the seven returns to the yard are reported two to six hours late, and seven of those 13 are real arrivals at the wrong time. And a single late position in between breaks the streak: it sees 3 of the 7 customer arrivals.
Using the ignition or the speed. With the engine off, the truck does not move. That removes almost all the yard ones; 21 remain, mostly from late positions and from the stretch when the engine warms up with the truck parked.

Each one helps, and each one is paid for somewhere else. I would sum up what we learned like this: any intervention or change of values eventually causes false positives or altered information.

On the fly

And the worst part was none of those. On that platform, a geofence entry or exit was not stored: it was computed in the web client, on the fly, every time someone opened the map or the report.

As long as nobody touches anything, that seems to work. The problem arrives with the first complaint. Someone enlarges the yard so the alerts stop, or raises the margin, or changes the customer's polygon. From that moment on, last week's report is recomputed with the new geofence and says something else.

In the simulator, enlarging the yard by 100 m brings the false alerts from 45 down to 31: it does not silence them, because the late positions and the longest jumps are still there. And in that same week's report, computed on the fly with the new yard, 23 alerts change: 16 disappear, 3 move to a different time and 4 appear that were not there before. If someone asked on Tuesday about a yard alert, the one they were shown may no longer be there, or may be at another time.

A history that changes when the configuration changes is not a history.

What I would do today

Store the event, once. The entry or exit is computed when the position arrives and stored, with the version of the geofence and that day's values. If the yard grows tomorrow, it grows from tomorrow on.

Stamp it with the position's time. Not the time it arrived. A late position is slotted into its place in the trip, and the alert carries the time it happened, even if it is sent later. If that position changes an event already stored, the event is corrected and the record says it was corrected and why: a late data point explains the change; a new configuration does not.

Drop the position, not the report. A position flagged as bad by its own quality data is discarded. But the report still says the device spoke and, if it is the ignition-off report, when the truck stopped.

Confirm with more than one signal. Two positions in a row, or the ignition-off report, or a position whose GPS speed is above 15 km/h. On most devices speed is measured separately, from the Doppler effect, and barely jumps with the position: a truck doing 40 km/h is not jumping. With the ignition off, a jump does not take the truck anywhere; if the GPS says it is moving, that is a different alert, the tow alert.

Check the segment, not just the point. If the straight line between two positions goes through the no-go zone, the truck most likely crossed it, even if no report fell inside. The time is interpolated along the segment, and the crossing is stored as inferred. With real streets, the line is checked against the street layout.

Three states, not two. Inside, outside and unknown. When the device goes quiet near a place known to have no signal, the entry is inferred, and stored marked as inferred. Whoever reads the report knows what was seen and what was deduced.

With as much of that as the simulator can model, in the same week: no false alerts, all 70 entries and exits, 36 of them inferred and marked as such (the zone crossings and the underground lot), and the time 1.1 minutes off on average. Not because it is perfect: the simulator only carries the failures I put in it, and the street has others. But each of those rules answers something that fails.

To try it with another week, with more bad or late positions, another margin or a bigger yard, there is the geofence simulator: it runs in the browser and decides in or out five ways.

Geofences did not fail because of the geometry. They failed because a lone position was asked something only the truck's history can answer, and because that answer was not stored.

Your turn

Where does your platform decide in or out: when the position arrives, or every time someone opens the map? And if someone changed a geofence today, would last month's report change with it? If you want to try your own margin and rules on a full week, the simulator is at crtdrops.mx/geocerca.

I send a short email when a new note or lab result comes out: subscribe here.

The top-up nobody knows went through

Israel Negrete Lepe — Fri, 16 Oct 2026 15:00:00 +0000

A customer asks for a prepaid airtime top-up at a corner store. The system debits the store's balance and sends the request to the carrier. The carrier does not answer.

Did the top-up reach the phone? At that moment nobody knows. And with the carrier, nobody can ask: the answer arrives tomorrow, in the reconciliation.

For years I worked on an electronic top-up platform. Requests came in from terminals, apps and points of sale, and reached the carrier after two, three and up to four hops, each with its own protocol and its own timeout. Everything happened: double top-ups, charges with no top-up. This note is about why, and about the state almost no system designs in from the start.

The figures in the examples are illustrative: calculated to show the mechanism. They are not that platform's numbers.

There are three answers, not two

A system that moves money is usually thought of with two outcomes: the operation went through, or it did not. On a network there is a third: unknown. The request went out and the answer never came back. The request may have been lost, and the top-up does not exist. Or the answer may have been lost, and the top-up is already on the phone.

From this side of the wire, the two cases look identical.

Every hop is one more place where an answer can get lost. If each one loses one in two thousand, with a hundred thousand top-ups a day, one hop leaves 50 in doubt; four hops, 200. Every day. It is not a bug you fix once: it is a probability you manage forever.

And the hardest case is at the end of the chain: the carrier does not answer, or its answer gets lost, but the internal system has already debited.

Everyone decides what to do with "unknown"

When the screen hangs, somebody decides, and every decision is a bet with someone else's money:

The cashier presses the button again. If the first one did go through, the customer gets two top-ups and the store pays for two.
The terminal retries on its own. Same thing, without anyone noticing.
The platform retries toward the carrier. Same thing, one hop deeper.
The platform marks the operation as failed and refunds the store. If the carrier did apply it, the platform gave away a top-up.
The platform marks it as good. If the carrier did not apply it, the customer paid and got nothing, and is going to complain.

None of them is right. They are all bets as long as nobody knows what happened.

An ID that travels end to end

The first defense is for every top-up to have a unique identifier from the moment it is born, in the terminal or the app, and for that same identifier to travel through every hop. We called it the externalID.

With it, a retry stops being a new top-up. If a request arrives with an externalID that was already processed, the system does not run it again: it answers with the result of the first one. The cashier can press ten times and the store pays once.

And it makes it possible to ask. Backwards, from the platform toward the points of sale, you could: the terminal could ask what happened to its externalID instead of sending another top-up.

What you cannot ask

With the carriers, you could not. There was no status query. The only thing was batch reconciliation: a file every 24 hours with what the carrier says it applied, against what the platform says it sent.

That means a top-up in doubt stays in doubt until the end of the day, or until the next day. Meanwhile, the store's balance has already been debited, the customer may be complaining, and the platform has nothing to tell them.

And reconciliation, when it comes, is torture: cross-matching files, finding the differences, deciding each one, and sometimes resolving them by hand.

Why not reverse

The way out that looks obvious is the reversal: when in doubt, cancel the operation with the carrier and return the money. In this business reversals are not desirable: they kill the business.

The margin on a top-up is very small. Every reversal costs work, time and arguments with the carrier, and eats the profit of many sales. And if you reverse something that did go through, the loss is total.

What I would do from the start

Design "unknown" as a real state. In the database, on the cashier's screen and in the reports: in progress, not failed or successful. A screen that says "in progress, do not repeat it" is worth more than any later validation.

The externalID from the origin, mandatory, and honored at every hop. A request with no identifier does not get in. A repeated identifier returns the stored result and never runs again.

No human retries blindly. The retry button checks the status first. If there is no answer, it shows that the operation is in progress; it does not send another.

Staggered timeouts. Each hop has to wait longer than the next one. If the terminal gives up before the platform does, the terminal retries while the first request is still alive inside.

Automatic reconciliation from day one. Not as a spreadsheet someone puts together once there are problems, but as part of the system: the carrier's file comes in, gets matched on its own, and a person only gets the differences.

Measure the doubts. How many top-ups stay in progress, per carrier, per hour and per hop. When that number goes up, something broke, and it is better to know today than in tomorrow's reconciliation.

To see what each way of deciding costs with your own numbers (top-ups a day, hops, timeouts), there is the top-up simulator: it runs in the browser and counts the doubled, given-away and charged-without-top-up operations of each policy, and what waits for reconciliation with an externalID.

None of these measures eliminates "unknown". What they do is stop it from being a bet: the money sits in a state everyone can see, nobody duplicates it out of desperation, and reconciliation only has to resolve what truly could not be known. Mitigating it was a lot of programming and infrastructure work. There was no shortcut.

Your turn

How does your system handle an operation stuck in "unknown": do retries carry an idempotency key end to end, or does someone decide by hand? If you want to try your own hops and timeouts, the simulator is at crtdrops.mx/recarga.

I send a short email when a new note or lab result comes out: subscribe here.

DEV Community: Israel Negrete Lepe

The Electratón runs again: a 2001 energy controller, back on the track

The problem of November 2001

The constraints of an AVR in assembly

What we kept and what we can no longer verify

The reconstruction

The results

What a 20-second log sees

What changed in twenty-five years, and what did not

The same question

Your turn

Ghosts in the fleet: knowing the IMEI is not knowing the device

The IMEI is a declaration

Three different ghosts

Five signals, and what none of them proves alone

The detector

Why the naive one falls

What the verdict does not say

What this detector does not prove

No anomaly is enough alone

Your turn

The query that took too long

A 504 does not say where the problem is

The table and the query

The plan that estimated fewer rows

The index that follows the query's order

Measuring: 4.5 ms, and what it leaves out

What we do not know

When the table grows

What I would do from day one

Your turn

An open socket is not a connected device

Connections that die without notice

Three ways to find out

What a heartbeat costs

Sent is not received

Two sessions from the same device

Several instances

The restart that becomes a second failure

The limits you do not see

What I would do today

Your turn

The lab's ofrenda: what didn't survive October

The pi user

OpenVINO's promise on the N100's CPU

The assistant's scheduled tasks

TradingAgent's original strategy

The Laplacian and the minimum distance

Gemma as an eye

The guest who came back: the Raspberry Pi 5

The full certificate

Your turn

The bytes that are not yours

What travels with each report

Opening a connection costs more than the report

Shrinking the report barely helps

What does change the math

What the math does not see

Your turn

Ghosts in the frame: my 2001 robot-vision tricks against two local vision models

What I built

Demo

Code

How I built it

What came out

What the ghosts did

Why does open innovation matter?

What 2001 still knew

Prize categories

Your turn

The average that never happened

It matters for different reasons

Each condition separately

The tail within one condition

Latency is not capacity

What a short test does not see

How much to trust a p99?

The timeout that improves the p99

Inference is not the system

What I would do today

The `pi` user