DEV Community

Cover image for UG35 payload logs in Python — make timestamp pairing respect clock uncertainty
UNITED UAV Official
UNITED UAV Official

Posted on

UG35 payload logs in Python — make timestamp pairing respect clock uncertainty

An image file and an event log each contain a timestamp. A developer sorts both exports, finds the closest event for every image and stores the pair. The resulting table looks complete even when the two clocks differ by several milliseconds. The algorithm also returns a pair when two events are equally plausible. A small timestamp residual can therefore hide a large uncertainty about which event belongs to an image.

This tutorial builds an offline pairing check for a payload integration project. The UNITED UAV UG35 mission-payload VTOL supplies the aircraft context. UNITED UAV publishes this tutorial and the linked listing. The program uses synthetic records and a timing contract supplied to the program. A team would need to establish that contract for the actual recorder, instrument and payload configuration before processing field exports.

The useful output is a decision with its evidence. Some frames have a unique candidate that stays inside the tolerance for every error allowed by the model. Other frames have a possible candidate whose interval crosses the acceptance boundary. A third group has several possible events. Keeping these outcomes visible lets the next stage request more evidence instead of silently accepting whichever row happened to sort first.

UG35 fixed-wing VTOL aircraft shown in the UNITED UAV product catalog

Define what each recorded time represents

Write down the event that opens the timestamp. An image record might identify the beginning of exposure, the end of exposure, the instant a trigger arrives or the time a file is written. A separate instrument might report acquisition time or publication time. Comparing those values requires knowing how the represented events relate. A clock correction cannot remove an unexplained delay between physical acquisition and a later software event.

For the example, assume the integration team has already converted both streams into integer nanoseconds against declared time origins. The timestamps may be relative to an epoch or a run start. They must use the same unit and have a defined mapping between clock domains. The code receives an offset that converts a frame timestamp into the event clock. Adding a positive offset moves the frame later on that clock. State this sign convention in the import contract and test it with a known pair.

Keep raw timestamp text and units in the archived export. Normalization should produce a new field while preserving the source value. A monotonic counter that resets at power-up needs a boot or run identity. Reusing the same counter range after a reset can make an old event appear to be a current candidate. Partition the imports by the identities that define a continuous clock domain before constructing the index.

Separate offset, error bound and tolerance

The offset is the estimated clock correction. The error bound describes how far the corrected timing difference could depart from its estimate under the stated model. The tolerance is the application's maximum accepted separation between represented events. These quantities serve different purposes. Increasing the tolerance to make a problematic export produce more matches changes the acceptance criterion.

Suppose the frame timestamp contributes a bounded error of one millisecond, the event timestamp contributes half a millisecond, and the clock correction contributes one and a half milliseconds. A conservative bound on their combined difference is three milliseconds. Summing the absolute bounds requires no assumption of statistical independence. Each component must itself be a justified bound over the interval being processed.

A standard deviation or a typical synchronization error needs a stated statistical model before it can support a different kind of decision. The code below expects a deterministic bound with coverage defined by the integration team. Its classifications remain conditional on that bound. A calibration observed once at the beginning of a recording also needs a model for how far clocks may drift afterward.

Derive possible and certain candidates

Let the corrected frame time be the frame timestamp plus the offset. Subtract an event timestamp to obtain the estimated difference. If the total error bound is R, the allowed difference lies between that estimate minus R and that estimate plus R. For a tolerance T, the accepted difference interval runs from negative T through positive T, including both endpoints.

An event is possible when its allowed difference interval intersects the accepted interval. Algebraically, the absolute estimated difference must be at most T plus R. This is the interval used to search for candidates. An event is certainly inside the timing tolerance under this bounded model when the absolute estimated difference plus R is at most T. That second condition asks whether the whole allowed interval lies inside the acceptance interval.

These conditions leave a useful middle region. An estimated difference of eight milliseconds, an error bound of three milliseconds and a tolerance of ten milliseconds produce an allowed difference from five through eleven milliseconds. The intervals overlap, so the event is possible. Part of the allowed difference exceeds ten, so the timing evidence is insufficient to classify the whole interval as acceptable.

Keep every possible event in the candidate set

Consider a corrected frame at one hundred milliseconds and events at one hundred and one hundred thirteen milliseconds. With a tolerance of ten and an error bound of three, both events are possible. The nearer event is entirely inside the tolerance. The second event touches the permitted boundary under one allowed timing error. A program that checks the nearest row alone hides that second candidate.

Returning several candidates is an honest outcome when the evidence permits several associations. An event sequence identifier, a hardware trigger count or another independently established relationship might resolve the association later. Record the rule and evidence used for that resolution. Choosing the smallest file name or the first row in an export would introduce a tie-breaking policy with no physical basis.

For this tutorial, a single possible event whose entire interval is acceptable receives a unique-within-bound status. A single possible event whose interval crosses the threshold receives an uncertain-time status. Multiple possible events receive an ambiguous status. No possible event receives a no-candidate status. These describe the relationship between the supplied records and timing model. They do not establish completeness of the event export.

Search an immutable sorted index

Construct the event index once for a batch of frames. Validate that event identifiers are unique and that times are sorted in nondecreasing order. Two different events can legitimately have equal timestamps when the recorder's resolution is coarse. Preserve both rows so the pairing decision can expose the ambiguity. Repeated event identifiers need reconciliation before indexing because two different rows cannot both stand for one stable identity.

For each frame, binary search locates the first event at or above the lower possible boundary and the first event strictly above the upper possible boundary. The second search includes an event exactly at the upper endpoint. Replacing it with a left-boundary search would lose that boundary case. Python's bisection functions define these insertion-point conventions. Python bisect documentation

The index copies the supplied events and timestamp keys into tuples. Each frozen event object preserves the values used during lookup. This avoids a caller changing the original list while frames are processed. After linear initialization, a lookup costs logarithmic search plus work proportional to the returned candidate count. If a very wide bound admits most of the batch, the large candidate set reflects the model's limited discrimination.

Implement the bounded join

The following Python 3.11 example uses integer timestamps throughout the comparisons. It rejects floats and booleans for time and bound fields. This catches accidental unit or parser conversions before they influence the decision. Integer nanoseconds are a convenient representation here, while the actual measurement resolution belongs in the import metadata.

The calibration validity interval is expressed in the original frame clock, including both endpoints. A frame outside that interval raises a validation error. Both the offset estimate and the combined bound must remain applicable throughout the interval. A calibration that needs several segments can produce a separate index configuration for each segment with the same event data and its own coverage. Handle the boundaries of those segments explicitly.

from bisect import bisect_left, bisect_right
from dataclasses import dataclass


@dataclass(frozen=True)
class Event:
    event_id: str
    time_ns: int


@dataclass(frozen=True)
class Match:
    event_id: str
    earliest_delta_ns: int
    latest_delta_ns: int
    certain_within_tolerance: bool


@dataclass(frozen=True)
class Pairing:
    status: str
    candidates: tuple[Match, ...]


def integer(value: int, name: str) -> int:
    if type(value) is not int:
        raise ValueError(f"{name} must be an integer")
    return value


class TimeJoin:
    def __init__(
        self,
        events: list[Event],
        offset_ns: int,
        error_bound_ns: int,
        tolerance_ns: int,
        valid_from_frame_ns: int,
        valid_to_frame_ns: int,
    ):
        self.offset = integer(offset_ns, "offset_ns")
        self.bound = integer(error_bound_ns, "error_bound_ns")
        self.tolerance = integer(tolerance_ns, "tolerance_ns")
        self.valid_from = integer(valid_from_frame_ns, "valid_from_frame_ns")
        self.valid_to = integer(valid_to_frame_ns, "valid_to_frame_ns")
        if self.bound < 0 or self.tolerance < 0 or self.valid_from > self.valid_to:
            raise ValueError("negative bound/tolerance or reversed validity interval")

        self.events = tuple(events)
        self.times = tuple(integer(event.time_ns, "event.time_ns") for event in self.events)
        identifiers = [event.event_id for event in self.events]
        if any(not name or name.strip() != name for name in identifiers):
            raise ValueError("event identifiers must be nonempty and trimmed")
        if len(set(identifiers)) != len(identifiers):
            raise ValueError("duplicate event identifiers")
        if any(left > right for left, right in zip(self.times, self.times[1:])):
            raise ValueError("event times must be sorted")

    def pair(self, frame_ns: int) -> Pairing:
        frame_ns = integer(frame_ns, "frame_ns")
        if not self.valid_from <= frame_ns <= self.valid_to:
            raise ValueError("frame falls outside calibration validity")
        target = frame_ns + self.offset
        radius = self.tolerance + self.bound
        lo = bisect_left(self.times, target - radius)
        hi = bisect_right(self.times, target + radius)
        matches = []
        for event in self.events[lo:hi]:
            delta = target - event.time_ns
            matches.append(Match(
                event.event_id,
                delta - self.bound,
                delta + self.bound,
                abs(delta) + self.bound <= self.tolerance,
            ))
        if not matches:
            return Pairing("no_candidate", ())
        if len(matches) > 1:
            return Pairing("ambiguous", tuple(matches))
        status = "unique_within_bound" if matches[0].certain_within_tolerance else "uncertain_time"
        return Pairing(status, tuple(matches))
Enter fullscreen mode Exit fullscreen mode

Exercise the decision boundaries

Use synthetic event times of one hundred and two hundred milliseconds, a zero offset, a three millisecond bound and a ten millisecond tolerance. A frame at one hundred six milliseconds has a single possible event at one hundred. Its difference interval is three through nine milliseconds, so it is unique within the bound. Moving the frame to one hundred eight makes the difference interval five through eleven, producing uncertain time.

A frame at one hundred fourteen milliseconds has no possible event in this example. Its nearest event differs by fourteen milliseconds and the sum of tolerance and bound is thirteen. The exact thirteen millisecond boundary still supplies a possible match because the intervals touch. Add another event at one hundred thirteen and process a frame at one hundred. The result becomes ambiguous even though the event at one hundred has an estimated difference of zero.

Test offset direction with a frame at ninety milliseconds, an event at one hundred and a positive ten millisecond correction. Then reverse the sign in a separate case and check the changed result. Include zero tolerance with a zero bound, negative timestamps for a relative origin and a bound larger than tolerance. When R exceeds T, no candidate can have its entire interval inside tolerance, even when its estimated difference is zero.

Validate the assumptions around the function

The class cannot infer the clock domain from an integer. The calling importer should bind the event clock identifier, frame clock identifier, calibration revision, time unit and run identity before constructing the index. Store those fields with the batch result. A timing outcome without the version of its correction and bound cannot be reproduced after calibration data changes.

Check the span of the event export as well. If the file ends early, a unique candidate near the end may be the only surviving record from several events that actually occurred. An importer can record the expected acquisition period, file sequence checks and any observed gaps separately. The pairer's output should preserve these coverage limitations so a downstream user can determine whether the batch meets its own acceptance requirements.

An offset that varies over a recording needs either a time-dependent correction or bounded segments. As an illustrative calculation, a relative rate bound of twenty parts per million accumulates as much as six milliseconds over three hundred seconds. Add that drift allowance to the applicable error model, or shorten the calibration coverage. This arithmetic describes the synthetic assumption. It supplies no clock-accuracy figure for the aircraft or a proposed payload.

Distinguish local pairing from batch assignment

The function considers each frame independently. Two frames may both return the same unique event. That can be valid if the event marks a state relevant to several images. If the application expects every trigger to produce exactly one frame, the batch has an additional cardinality constraint. Build the frame-to-event candidate relationships first and then evaluate whether a one-to-one assignment exists and whether it is unique.

Greedily assigning the nearest event to the earliest frame can consume a candidate required by a later frame. A batch assignment solver needs its own explicit policy for missing frames, extra events and temporal ordering. Keep that decision separate from the interval calculation so the evidence for possible correspondence remains inspectable. An assignment chosen to minimize total residual can still have alternatives with similar residuals.

Reprocessing should retain the original pairing report. A changed offset, drift allowance or tolerance can alter both candidate membership and status. Compare results by stable frame and event identities and explain why a previous association changed. If the batch supports inspection annotations, keep those annotations bound to the underlying image identity while flagging any changed event association for review.

Ask for a timing contract with the payload configuration

Before implementing a production importer, request representative unmodified image and event exports from the exact proposed payload configuration. Ask which physical event each timestamp represents, what survives a restart and which clock supplies the time. Request the evidence used to establish the correction, its sign, its coverage interval and the error bound under the operating conditions the integration will encounter.

Use the UG35 product reference to identify the aircraft platform during that discussion, then bind the timing contract to the actual recorder and instrument revisions. A product name alone supplies no export schema or synchronization method. Once the agreed fields are available, test deliberate clock shifts and duplicate timestamp cases alongside a normal batch. The resulting report can show which pairs remain supported, which require another identifier and which records have no admissible counterpart.

This article and its code were developed with AI assistance. The executable examples use synthetic data, with boundary cases checked locally before publication.

Top comments (0)