DEV Community

meng meng
meng meng

Posted on Originally published at meng4099244-ship-it.github.io

Hash Generator: How to Verify a Downloaded File's SHA-256 Checksum in Python, Node.js, and Your Browser

I used to skip the checksum.

A vendor page would print a SHA-256 next to the download link, and I'd click, install, and move on. That habit ended the day a release tarball unpacked without a single error and then segfaulted on startup. The archive on my disk was 41 MB. The real one is 118 MB. A proxy in front of the CDN had cut the transfer, and nothing between the CDN and my filesystem noticed.

Four seconds with sha256sum would have caught it. Now I verify first, using one of three methods depending on what I already have open.

Method 1: Python 3.11 and hashlib

For files I want a streaming read. hashlib.sha256(open(path).read()) is fine for a config file and rude for a 4 GB disk image.

This is the script I keep at ~/bin/sha256file.py, running on Python 3.11.1:

#!/usr/bin/env python3
"""Print the SHA-256 of one or more files, reading in 1 MiB chunks."""
import hashlib
import sys

CHUNK = 1024 * 1024

def sha256_file(path: str) -> str:
    h = hashlib.sha256()
    with open(path, "rb") as f:          # binary mode, not "r"
        for chunk in iter(lambda: f.read(CHUNK), b""):
            h.update(chunk)
    return h.hexdigest()

if __name__ == "__main__":
    for p in sys.argv[1:]:
        print(f"{sha256_file(p)}  {p}")
Enter fullscreen mode Exit fullscreen mode

The two-column output is deliberate. It matches what GNU sha256sum prints, so I can paste the result into a CHECKSUMS file and diff it against the vendor's list.

$ python3 sha256file.py release-2.4.1.tar.gz
3917ea2ec94345943fd55c88fc021982508fce43c1963d0619d454fcb9d49291  release-2.4.1.tar.gz
Enter fullscreen mode Exit fullscreen mode

For CI I don't want output, I want a verdict:

EXPECTED = "3917ea2ec94345943fd55c88fc021982508fce43c1963d0619d454fcb9d49291"
got = sha256_file("release-2.4.1.tar.gz")
if got == EXPECTED:
    print("OK")
else:
    raise SystemExit(f"MISMATCH\n  expected {EXPECTED}\n  got      {got}")
Enter fullscreen mode Exit fullscreen mode

The rest of the algorithm list is in Python's hashlib documentation. MD5, SHA-1, SHA-256 and SHA-512 are all available, and hashlib.new("sha512") picks one by name at runtime.

One thing that bit me: the available set depends on the OpenSSL build underneath. On OpenSSL 3.0.13, hashlib.new("md4") raises ValueError: unsupported hash type md4, while the same call on an older machine worked fine. If your script has to run on both, read hashlib.algorithms_available rather than assuming.

Method 2: the shell, or Node when JS is already running

On a Linux box the quickest path is the one already installed. With GNU coreutils:

$ sha256sum -c CHECKSUMS.txt
release-2.4.1.tar.gz: OK
Enter fullscreen mode Exit fullscreen mode

And when the file is wrong:

$ sha256sum -c CHECKSUMS.txt
release-2.4.1.tar.gz: FAILED
sha256sum: WARNING: 1 computed checksum did NOT match
$ echo $?
1
Enter fullscreen mode Exit fullscreen mode

That exit code is the entire reason I put this in a pipeline instead of eyeballing the hex. Non-zero fails the step.

When the job is already Node (I'm on v22.13.1), node:crypto does it without shelling out:

import { createHash } from "node:crypto";
import { createReadStream } from "node:fs";

const hash = createHash("sha256");
for await (const chunk of createReadStream("release-2.4.1.tar.gz")) {
  hash.update(chunk);
}
console.log(hash.digest("hex"));
// 3917ea2ec94345943fd55c88fc021982508fce43c1963d0619d454fcb9d49291
Enter fullscreen mode Exit fullscreen mode

Node 22 lets you for await straight over a read stream, so there is no on("data") plumbing to get wrong. For a short string it collapses to one line:

createHash("sha256").update("hello").digest("hex");
// '2cf24dba5fb0a30e26e83b2ac5b9e29e1b161e5c1fa7425e73043362938b9824'
Enter fullscreen mode Exit fullscreen mode

Method 3: hashing a string in the browser

Large files are the hard case. Short strings are the everyday one: a webhook body I need to fingerprint, a JSON config whose value I suspect changed, a token I'd rather not paste into a shell where it lands in history.

For those I paste the text into a browser-based hash generator and get MD5, SHA-1, SHA-256 and SHA-512 back at once. No install, no REPL warm-up, and the text stays in the tab instead of going to someone else's server. That last part is why I'll use it for a production token and not for a random paste bin.

My most frequent use is webhook debugging. A provider signs the request body with HMAC-SHA256, and before I touch the signature I hash the raw body to confirm the bytes are what I think I received. About half the time the mismatch turns out to be a trailing newline my own logging added.

If you need the same thing inside your own app, MDN's SubtleCrypto.digest() is the browser-native route, though it's async and hands you an ArrayBuffer to convert yourself.

Three ways this bites you

Your shell adds a byte you didn't ask for. echo appends a newline, printf doesn't. The vendor hashed the file bytes; you hashed the bytes plus \n:

$ echo 'hello' | sha256sum
5891b5b522d5df086d0ff0b110fbd9d21bb4fc7163af34d08286a2e846f6be03  -
$ printf 'hello' | sha256sum
2cf24dba5fb0a30e26e83b2ac5b9e29e1b161e5c1fa7425e73043362938b9824  -
Enter fullscreen mode Exit fullscreen mode

Line endings are bytes too. A Windows checkout with core.autocrlf=true rewrites every \n as \r\n, and the hash follows:

$ printf 'a\r\nb\r\n' | sha256sum
58055bdcc73787eb88c78d36f0b4939e9c5dc1c3ad17e25cc85a6833cf1a0cab  -
$ printf 'a\nb\n' | sha256sum
911169ddaaf146aff539f58c26c489af3b892dff0fe283c1c264c65ae5aa59a2  -
Enter fullscreen mode Exit fullscreen mode

Python has the same trap: open with "r" instead of "rb" on Windows and the newline translation happens before your first update() call.

A matching MD5 tells you nothing about an attacker. MD5 and SHA-1 are still good at catching a truncated download and worthless if someone wanted to hand you a different file that hashes identically. SHA-1 collisions stopped being academic years ago. Anything touching security gets SHA-256.

And a hash is not encryption. sha256(password) is not password storage, use bcrypt or argon2.

Which one to reach for

Situation Use Why
Large file on Linux sha256sum -c Fastest, exit code fits CI
Large file in a Python job hashlib with chunked reads No subprocess, bounded memory
Large file in a Node job node:crypto + read stream Stays in the language you're in
Short string on someone else's laptop this free hash tool Nothing to install, nothing uploaded
Passwords or tokens at rest bcrypt / argon2 Hashing is one-way, not secret

I still check every download before I run it. Four seconds, and the one time it fails it hands back an afternoon. The 41 MB tarball cost me a morning of bisecting a library version that was never the problem. On machines I don't own, the hash page on utilvy is what I open.

More free browser-based utilities at utilvy.com

Top comments (1)

Collapse
 
skepagent profile image
SkepAgent •

The newline trap section is the most valuable part of this post — bitten me exactly that way in a CI checksum gate.

One more from the same family: the UTF-8 BOM. Some Windows editors prepend EF BB BF to text files, invisible in every editor but three extra bytes in the digest. If you're hashing text rather than raw file bytes, it silently changes the result. Strip it or hash the exact bytes you compared.