When JavaScript and Python Disagree About Text Offsets

Disclosure: This article was generated with AI and was not reviewed by Martin Yovchevski before publication. Martin is a co-founder of Unplag.bg.

Imagine a teacher opening a report and seeing the wrong letter highlighted. No student did anything wrong: a browser and a Python service may simply be counting the same text in different units.

JavaScript string offsets use UTF-16 code units. Python indexes a string by Unicode code points. For many Bulgarian letters those positions coincide, but an emoji outside the Basic Multilingual Plane uses two UTF-16 code units and one Python code point.

For the fictional string А😀Б, JavaScript's indexOf('Б') returns 3; Python's index('Б') returns 2. Applying a JavaScript offset directly as a Python slice index points at the wrong boundary.

A small boundary converter

The function below converts a UTF-16 code-unit boundary into a Python index for the same, unchanged string. It accepts Unicode scalar values and rejects Python strings containing surrogate code points. It also rejects offsets inside a supplementary character’s UTF-16 surrogate pair and offsets outside the text.

def utf16_to_py_index(text: str, offset: int) -> int:
    if isinstance(offset, bool) or not isinstance(offset, int) or offset < 0:
        raise ValueError("offset must be a non-negative integer")
    if any(0xD800 <= ord(char) <= 0xDFFF for char in text):
        raise ValueError("text contains surrogate code points")

    units = 0
    for index, char in enumerate(text):
        if units == offset:
            return index
        units += 2 if ord(char) > 0xFFFF else 1
        if units > offset:
            raise ValueError("offset falls inside a surrogate pair")

    if units == offset:
        return len(text)
    raise ValueError("offset is outside the string")


sample = "А😀Б"
start = utf16_to_py_index(sample, 3)
end = utf16_to_py_index(sample, 4)
print(sample[start:end])  # Б

If an upstream offset is 2 in that example, it falls inside the emoji's two-unit representation. The function raises an error rather than silently guessing. A zero offset and the boundary at the end of the text are valid.

Limits that matter

This conversion is for boundaries in the same sequence of Unicode code points. It does not repair offsets after text is edited or normalised. It also does not count user-perceived characters (grapheme clusters), which can contain multiple code points. Keep the original text and offset convention together, and test the entire browser-to-server path before using this in a real application.

Use fictional data in tests. In a classroom review, a highlight is evidence to inspect in context; it is not proof of misconduct. AI-writing indicators can be wrong, too, and the educator remains responsible for the decision.

Update: The example now rejects Python strings containing surrogate code points, rather than treating those cases as valid boundaries.

References: MDN on JavaScript string length and Python's text sequence documentation.

Teachers, lecturers and education developers who want to discuss text-review workflows can write to [e-mail].