DEV Community

Devil Scrapes
Devil Scrapes

Posted on

The US sanctions list still ships with a 1980s DOS end-of-file byte

Quick answer

The US Treasury's OFAC sanctions list still ships with a DOS end-of-file marker. The last three bytes of SDN.CSV are 0d 0a 1a — CR, LF, then 0x1A, the Ctrl-Z character MS-DOS used to mark end-of-text in the 1980s. Python's splitlines() treats that trailing \x1a as content and hands you a final one-column row that is not a record. If your parser validates column counts — and a sanctions parser absolutely should — it will reject the entire file on the last line, after correctly parsing 19,364 real ones.

Why is there a 1980s EOF byte in a live compliance feed? 🔍

Because the file format is old and stable, and stability is the point. SDN.CSV is a fixed-shape export that downstream compliance systems have consumed for decades, so nobody is going to modernise the trailer and break them. Here are the actual last bytes:

00000000: 2041 4e4f 4e49 4d20 5349 524b 4554 492e   ANONIM SIRKETI.
00000010: 220d 0a1a                                 "...
Enter fullscreen mode Exit fullscreen mode

That final 1a is the whole problem. It is invisible in a text editor, survives every naive .strip() that only removes whitespace (\x1a is not whitespace), and produces a phantom row precisely at the point where most parsers stop paying attention.

The fix is a guard clause, not a heuristic: strip the EOF marker explicitly before splitting, and keep the strict column-count check for every remaining row. What you must not do is relax the column-count validation to make the error go away — that check is the thing standing between you and silently mis-parsing a sanctions record.

What else is unusual about the SDN format?

-0- is null. OFAC does not use empty fields. A missing value is the literal three-character string -0-:

36,"AEROCARIBBEAN AIRLINES",-0- ,"CUBA",-0- ,-0- ,-0- ,-0- ,-0- ,-0- ,-0- ,-0-
Enter fullscreen mode Exit fullscreen mode

Note the trailing space after each one, too. If you load this into a dataframe without translating the sentinel, you get a column full of the string "-0-" that is not null, not empty, and will happily pass an if value: check. Every one of those fields then reads as populated.

The download is a redirect to a signed, expiring URL. A request to the export path answers 302 with a Location header pointing at a presigned S3 object carrying X-Amz-Expires=3600 and a signature. You cannot cache that URL — it dies within the hour. You have to re-request the export path each time and follow the hop. Worth also not forwarding your original headers onto the S3 leg: presigned URLs can reject requests carrying unexpected headers.

The OFAC SDN Sanctions List Scraper handles all three — the EOF marker, the -0- sentinel, and the redirect chain — and joins ALT.CSV (aliases) and ADD.CSV (addresses) onto the primary records by entity number.

How big is the list, and how fast does it move?

SDN.CSV is roughly 5.6 MB across about 19,365 rows as of 2026-09-09, with a separate alias file of comparable size. That matters for a practical reason people discover in production rather than in testing: if you pull the primary list, the aliases and the addresses concurrently through a single network egress, they share bandwidth, and a per-request timeout tuned on a laptop will be far too tight. A 30-second total timeout that passes locally every time will die in a datacenter having received 991,217 of 1,062,698 bytes — 93% of the way through the smallest of the three files.

The lesson generalises past this API: for multi-megabyte concurrent downloads, a total request timeout is not the same knob as a connect timeout, and the number you need is set by your slowest egress, not your fastest.

FAQ

Do I need an API key?
No. The OFAC sanctions list is published by the US Treasury as open data, with no key, no registration and no IP whitelisting.

Is screening a name against the SDN list enough for compliance?
No, and please do not treat it as such. The SDN list is one list among several — consolidated non-SDN lists, EU and UK regimes, and local requirements all exist. Fuzzy name matching also produces both false positives and false negatives, especially across transliterated names, which is why match scores need human review rather than being wired straight into a block.

Why do aliases live in a separate file?
Because one entity can have many. ALT.CSV maps alias records back to the primary entity by ent_num, so a single sanctioned individual can appear under a dozen spellings. Screening only against primary names in SDN.CSV will miss the alias that a counterparty actually uses — which is, in practice, the whole reason alias data exists.

How current is the file?
OFAC republishes on change, and the presigned URL path includes a dated segment. Treat the list as something to re-pull rather than cache; a stale sanctions list is worse than no sanctions list, because it feels like diligence.

The short version

A 5.6 MB CSV, a 1980s EOF byte on the last line, -0- where a null should be, and a one-hour signed URL in front of it. None of these are hard once you know; all of them are silent when you don't.

Byte-level details in this post were verified against sanctionslistservice.ofac.treas.gov on 2026-09-09.

Top comments (0)