A CSV import can show an unexpected character before the first header name even when the visible text looks right. Check the file's opening bytes before changing the header or resaving the export. This read-only program reports whether the file begins with the three-byte UTF-8 signature.
1. Make the practice files
A byte is a small unit of stored data. UTF-8 is an encoding that turns text characters into bytes. Some programs put the bytes EF BB BF before the text as a signature, often called a byte order mark or BOM. UTF-8 does not need a byte-order choice, so the name does not mean these bytes control the order of the rest of the file.
Save this as MakeExamples.py in a practice folder:
from pathlib import Path
Path("Plain.csv").write_bytes(b"Id,Name\n001,Ada\n")
Path("Marked.csv").write_bytes(b"\xef\xbb\xbfId,Name\n001,Ada\n")
Path("Inside.txt").write_bytes(b"before\xef\xbb\xbfafter\n")
Path("Empty.txt").write_bytes(b"")
Run:
python3 MakeExamples.py
Use your installation's Python 3 command if it is not named python3. This creates or replaces four files, so use a folder without files you need to keep. Plain.csv has no signature. Marked.csv starts with one. Inside.txt has the same three bytes after the word before, and Empty.txt has no bytes.
The \x sequences in the setup code specify individual byte values. Both CSV files contain the same visible header and row after any initial signature.
2. Add the checker
Save this as CheckUtf8Signature.py:
import json as Json
import sys as Sys
from pathlib import Path
def CheckUtf8Signature(FilePath):
with Path(FilePath).open("rb") as Input:
Prefix = Input.read(3)
return {"PrefixHex": Prefix.hex(), "InitialUtf8Signature": Prefix == b"\xef\xbb\xbf"}
def Main():
if len(Sys.argv) != 2:
print("Usage: python3 CheckUtf8Signature.py input-file", file=Sys.stderr)
return 2
try:
Report = CheckUtf8Signature(Sys.argv[1])
except OSError as Problem:
print(f"Check stopped: {Problem}", file=Sys.stderr)
return 2
print(Json.dumps(Report, indent=2))
return 1 if Report["InitialUtf8Signature"] else 0
if __name__ == "__main__":
Sys.exit(Main())
open("rb") reads bytes instead of decoding text. read(3) takes at most three bytes, so the check does not load the rest of a large file. A file shorter than three bytes gives back only the bytes it has.
Prefix.hex() prints those bytes as hexadecimal, using two characters for each byte. The comparison tests the exact opening sequence, not whether the same bytes appear somewhere later.
The output uses JSON, a text format for named values and lists. It deliberately reports InitialUtf8Signature, not a claimed file encoding.
3. Compare the two CSV files
Run:
python3 CheckUtf8Signature.py Plain.csv
The output is:
{
"PrefixHex": "49642c",
"InitialUtf8Signature": false
}
Those are the opening bytes for Id,. Now run:
python3 CheckUtf8Signature.py Marked.csv
The output is:
{
"PrefixHex": "efbbbf",
"InitialUtf8Signature": true
}
The marked file's first three bytes are the signature, before the header starts. The checker leaves both files unchanged.
The exit code, a small result number another script can check, is 1 when the signature is present, 0 when it is absent and 2 for a command or file-opening error. A 1 here is a finding, not proof that the file is corrupt.
4. Check the position and the empty file
Run:
python3 CheckUtf8Signature.py Inside.txt
The output is:
{
"PrefixHex": "626566",
"InitialUtf8Signature": false
}
The opening bytes spell bef. Although the same EF BB BF bytes occur later, they are not an initial signature. When decoded as UTF-8, that later sequence is the character U+FEFF; this checker neither finds nor removes it.
Now run:
python3 CheckUtf8Signature.py Empty.txt
The output is:
{
"PrefixHex": "",
"InitialUtf8Signature": false
}
An empty file has no signature. It also has no header or records, so a 0 does not mean it is ready to import.
5. Try an incomplete prefix and an invalid tail
The checker needs all three signature bytes. Save this as MakeLimits.py beside it:
from pathlib import Path
with Path("ShortPrefix.bin").open("xb") as Output:
Output.write(b"\xef\xbb")
with Path("InvalidTail.bin").open("xb") as Output:
Output.write(b"\xef\xbb\xbf\xff")
The .bin extension labels these as practice byte files, not import-ready text. The setup creates each file only if its name is free. Use a fresh practice folder if either name already exists; a failure on the second file does not undo creation of the first.
Run:
python3 MakeLimits.py
python3 CheckUtf8Signature.py ShortPrefix.bin
The output is:
{
"PrefixHex": "efbb",
"InitialUtf8Signature": false
}
The exit code is 0. Two matching bytes are not a complete signature. The check reports their hex values without inventing a third byte or deciding whether the file was cut short.
Now check the other file:
python3 CheckUtf8Signature.py InvalidTail.bin
The output is:
{
"PrefixHex": "efbbbf",
"InitialUtf8Signature": true
}
The exit code is 1, exactly as for the marked CSV. But this file has a fourth byte, ff, which cannot occur in valid UTF-8. The checker never reads it. A complete signature therefore does not establish that the remaining bytes decode as UTF-8 or that any records are present. Both practice files stay unchanged.
6. Decide what to check next
This result checks only the opening three bytes. It does not validate UTF-8, inspect CSV columns, parse JSON or identify every other encoding. Invalid bytes later in the file will not change the result. A file without the signature could still be UTF-8, another encoding or non-text data. Use a saved file that will not change while it is read, and check the importing program's documented encoding rules.
When a format and its reader explicitly allow a UTF-8 signature, Python's utf-8-sig decoder skips it at the start. The ordinary utf-8 decoder keeps it as U+FEFF. That difference can affect the first CSV header name. It does not justify silently removing characters or rewriting the source file.
Do not apply a CSV workaround to every text format. Python's JSON parser rejects an initial BOM in a decoded string; JSON readers and the surrounding decoding step can have different rules. Check the whole import path before choosing how to decode it.
Keep the original export. If a conversion is needed, write a separate copy and confirm its headers and records before replacing anything.
Top comments (0)