A punctuation split is a poor contract for a research-text handoff. In a source such as 1.25 µg/mL, the decimal point is part of the value. Treating every period as a sentence boundary can detach the value from its unit before an editor even proposes a change.
For a small, explicitly prepared input, one practical alternative is to let the author or an upstream reviewed step supply one review unit per line. Keep those boundaries and the exact original text. This is a line-preservation demonstration, not a general-purpose sentence tokenizer.
The input contract
The example uses two fictional laboratory-report lines. Neither is a measured result. The first line ends with CRLF, the second with LF. The micro sign is µ (U+00B5), so the UTF-8 byte offsets differ from Python string positions.
The task is narrow: return stable unit IDs, half-open spans and unmodified source text. Do not normalize whitespace, decimal punctuation or units while constructing the source side of the review. Any later proposed language edit should remain separate.
Runnable example
This standard-library example was executed locally with Python 3.14. It reads no files, calls no network service or model and starts no research task.
import json
source = (
"The assay recorded 1.25 µg/mL.\r\n"
"Values below 0.05 µg/mL were not quantified.\n"
)
rows = []
char_start = byte_start = 0
for i, unit in enumerate(source.splitlines(keepends=True), 1):
char_end = char_start + len(unit)
byte_end = byte_start + len(unit.encode("utf-8"))
rows.append({
"id": f"unit-{i}",
"codepoint_span": [char_start, char_end],
"utf8_byte_span": [byte_start, byte_end],
"source_text": unit,
"author_review": "pending",
})
char_start, byte_start = char_end, byte_end
reconstructed = "".join(row["source_text"] for row in rows)
assert reconstructed == source
for row in rows:
a, b = row["codepoint_span"]
assert source[a:b] == row["source_text"]
print(json.dumps({"units": rows, "exact_reconstruction": True},
ensure_ascii=False, indent=2))
Actual local output
{
"units": [
{
"id": "unit-1",
"codepoint_span": [
0,
32
],
"utf8_byte_span": [
0,
33
],
"source_text": "The assay recorded 1.25 µg/mL.\r\n",
"author_review": "pending"
},
{
"id": "unit-2",
"codepoint_span": [
32,
77
],
"utf8_byte_span": [
33,
79
],
"source_text": "Values below 0.05 µg/mL were not quantified.\n",
"author_review": "pending"
}
],
"exact_reconstruction": true
}
The first unit occupies Python string positions [0, 32) and UTF-8 bytes [0, 33). The second starts at those respective endpoints. Joining the retained units reconstructs the original string exactly, including both line endings.
Those positions are Python Unicode-code-point indices or UTF-8 byte offsets, as labelled. They are not visual columns, grapheme-cluster positions or JavaScript UTF-16 offsets. Do not pass one coordinate system to a consumer expecting another.
What this check does and does not establish
Exact reconstruction establishes that this particular segmentation preserved the supplied text. It does not validate the reported concentrations, units, assay interpretation or a subsequent editing candidate. It also does not prove the lines are linguistically complete sentences. A supplied review unit can contain several sentences, and a wrapped PDF line can contain only part of one.
For arbitrary PDF extraction, do not silently apply this one-line contract. First choose and review the unit boundaries with the source context available. A tool should say which input shape it accepts; a text-retention check should not be presented as scientific validation.
Keep the current source version attached to the handoff. If the author changes the source, the positions must be recomputed against that version. A candidate should be reviewed next to its retained source, with the reason for the change and unresolved author checks visible.
Implementation note
Python documents str.splitlines(keepends=True) as retaining line boundaries in the returned pieces. It recognizes more boundary characters than just LF and CRLF. The example deliberately uses the two stated conventions; integrations should make their accepted input and coordinate system explicit.
Python 3.14 built-in string documentation
We are the RigorMed team. This is an original developer teaching example, not a product feature claim, a client manuscript or a clinical finding. For an existing academic-text task, provide authorized source material and a narrow editing goal; model use follows the displayed credit terms. Author scientific review remains necessary.
Top comments (0)