DEV Community

Sergey Shinder
Sergey Shinder

Posted on

We started renaming our stock file into place and on the second night SFTP refused to overwrite it

Every night we send a marketplace partner our stock levels as stock.csv, uploaded over SFTP into their inbound directory. Their server is OpenSSH. Their importer reads the file at half past five, and leaves it there for us to replace the next night.

A while ago I wrote here about reading a partner's file while they were still writing it, and said the fix belongs to the sender: write to a temporary name, then rename. In October we finally did that for our own upload. The job, Python with paramiko, now puts stock.csv.part and then calls rename to stock.csv.

The first night was fine, because there was no stock.csv to replace. On the second night the rename raised an IOError whose whole message was Failure. The job retried three times, failed, and paged. By the time we understood it the partner had imported yesterday's stock and sold twenty three items we no longer had.

SFTP version 3, the one OpenSSH implements, says a rename is an error if a file already exists at the new name. OpenSSH's server follows that: it renames by making a hard link and removing the old name, and the link fails because the target exists. Version 3 has no status code for file exists, so the client is told only Failure. Paramiko's rename sends exactly that standard request. Plenty of people expect it to behave like rename on a local disk, which replaces the target in one step, and it does not.

OpenSSH adds an extension for that, posix-rename@openssh.com, which replaces the target atomically. Paramiko exposes it as posix_rename. That is what the job uses now, and it fails at the start with a clear message if a server does not offer the extension, rather than falling back to deleting first and leaving the partner a moment with no file at all. The failure message in our alert now includes the operation and both paths, not just the status. And the job checks before it finishes that stock.csv on their side has today's size and modification time.

We gave the partner advice about rename and then used a rename that meant something different. A word that sounds the same on two systems is not a promise.

– Sergey Shinder

Top comments (1)

Collapse
 
acemeridian profile image
Ace Meridian •

I reproduced this against a local OpenSSH 10.2p1 server with paramiko 5.1.0. Night one rename("stock.csv.part", "stock.csv") works, night two raises OSError('Failure',) with errno set to None, stock.csv still holds the old contents, and the .part file stays behind. So an except OSError as e: if e.errno == errno.EEXIST branch never fires; the message is all you get. posix_rename replaces the file, and a reader that already had the old file open keeps reading yesterday's contents, which is what you want for a 05:30 importer.

The part I'd look at again is "fails at the start if the server does not offer the extension". paramiko keeps no record of the extensions the server sends in its VERSION reply, so there is nothing to check before you call it. And when an admin turns the extension off (sftp-server -P posix-rename, which is common on locked-down partner boxes), the reply is not "unsupported":

posix_rename on server without it: PermissionError (13, 'Permission denied') errno 13
Enter fullscreen mode Exit fullscreen mode

An alert built from that sends someone off to check directory permissions. What I use at startup is a probe that renames a name that cannot exist:

def require_posix_rename(sftp, directory):
    ghost = f"{directory}/.probe-{uuid.uuid4().hex}"
    try:
        sftp.posix_rename(ghost, ghost + ".x")
    except FileNotFoundError:
        return  # server ran the extension and found no source: supported
    except OSError as e:
        raise RuntimeError(f"server will not do posix-rename@openssh.com in {directory}: {e!r}") from e
    raise RuntimeError("probe rename of a missing file succeeded?")
Enter fullscreen mode Exit fullscreen mode

Against the stock server that returns quietly; against the -P posix-rename one it raises with the real reason, before anything is uploaded. One more case where both calls give you only Failure: renaming onto a path that is a directory.

Does the partner's server log show which user hit the failure, or did you only have your side's message to go on?