At the start of each month we send an insurer the previous month's damaged parcel claims through their upload API, one HTTPS PUT of a CSV file. In October the upload failed on every attempt with 411 Length Required, and the claims went in two days late.
The job had changed in September. It used to read the whole file from S3 into memory and send it with Python's requests library. That month's file was nine hundred megabytes and the pod ran out of memory, so a colleague changed the job to stream instead, passing requests a generator that yielded the S3 body a megabyte at a time. Memory dropped to almost nothing and the tests passed.
requests decides how to frame a body by whether it can work out its length. Bytes, strings and real files it can measure, and for those it sends a Content-Length header. A generator has no length, so requests sends the body with Transfer-Encoding: chunked, in pieces each prefixed with its size and ended by an empty chunk. HTTP/1.1 servers are expected to understand that. The gateway in front of it did not accept chunked requests, and 411 is the status for a server that refuses a body without a declared length. Nothing in our code mentioned either header.
Our test used a mock HTTP server that accepted any request and returned 200. It checked that we had sent the file, not how we had sent it.
The job now downloads the file to the pod's local disk and passes requests an open file. requests reads the size from the file system, sends a Content-Length and streams from disk, so memory stays as low as it was with the generator and the gateway sees the request it has always seen. Our tests now run each upload against a local server set up to reject chunked bodies, and assert on the headers we send as well as the bytes. The client we share across integrations refuses a chunked upload unless the partner's configuration says it accepts one. And a rejected upload alerts the claims team on the first failure, not after the retries run out, because a 411 will not succeed on the fifth attempt either.
We changed how a program held a file in memory, and the wire format changed with it. Only the partner noticed.
– Sergey Shinder
Top comments (0)