Hashes and Checksums: Verifying Files, and Choosing the Right Algorithm

Hashing is not encryption and is not encoding

Encoding is reversible by anyone, by design. Base64, URL percent-encoding and hexadecimal exist to move bytes through channels that expect text; there is no key and no secret, and calling an encoding obfuscation is generous.

Encryption is reversible with a key. Ciphertext plus the correct key yields the plaintext; that is the entire point. If you need to read the data later, you want encryption, and the security rests on key management rather than on the algorithm.

Hashing is not reversible at all, because it is not injective: the input can be any length and the output is fixed, so infinitely many inputs map to each digest. There is no key and no way back. What you get instead is a compact value that changes completely when the input changes at all — flip a single bit and roughly half the output bits change. That property, and not secrecy, is what makes hashes useful.

$ printf 'hello' | shasum -a 256
2cf24dba5fb0a30e26e83b2ac5b9e29e1b161e5c1fa7425e73043362938b9824

$ printf 'hello\n' | shasum -a 256
5891b5b522d5df086d0ff0b110fbd9d21bb4fc7163af34d08286a2e846f6be03

One extra newline, and the digests share nothing. This is also the
single most common reason two checksums "should match but don't":
echo appends a newline, printf does not.

Verifying a download, and what the verification proves

The mechanics are easy: compute the digest of the file you received with shasum -a 256, sha256sum or Get-FileHash, and compare the full string — not the first and last few characters — against the digest the publisher states. Now the part that is usually left out. A checksum published on the same web page as the download proves only that the bytes you received are the bytes that server sent. It detects a truncated transfer, a corrupt mirror, a flaky disk. It does not detect a compromised server, because whoever replaced the file also replaced the digest next to it. Integrity against an adversary requires a signature — a GPG signature over the checksum file, a signed release artifact, a package manager verifying a repository key — because a signature cannot be regenerated without the publisher's private key.

The practical rule: verify the checksum to catch accidents, verify the signature to catch attacks, and be honest with yourself about which one you actually did.

$ cat SHA256SUMS
2cf24dba5fb0a30e26e83b2ac5b9e29e1b161e5c1fa7425e73043362938b9824  greeting.txt

$ shasum -a 256 -c SHA256SUMS
greeting.txt: OK

# And the step most guides skip — checking that SHA256SUMS
# itself came from the publisher:
$ gpg --verify SHA256SUMS.asc SHA256SUMS
gpg: Good signature from "Example Project Release Key"

# Without that second step you have verified the download
# against the same server that served the download.

Which algorithm, and what each one is for now

Hash functions are asked for three resistance properties and fail them in order: preimage resistance, second-preimage resistance, and collision resistance. Collision resistance is the weakest and the first to break, because the attacker gets to choose both inputs.

That ordering explains an apparent contradiction in the table below. MD5 is comprehensively broken for collisions and yet there is no practical preimage attack on it. So MD5 remains adequate for detecting accidental corruption of a file nobody is attacking — and completely unsuitable anywhere an adversary supplies the input, which includes certificates, signatures, integrity checks on downloads, and deduplication of untrusted content.

Why MD5 and SHA-1 were retired

The theoretical bound for finding a collision by brute force is the birthday bound: roughly 2^(n/2) work for an n-bit digest, so 2^64 for MD5 and 2^80 for SHA-1. Both functions fell far below their bounds because of structural weaknesses, not because hardware caught up.

SHA-1's first public identical-prefix collision was demonstrated in 2017 by researchers at CWI Amsterdam and Google, who produced two distinct PDF files with the same SHA-1 digest. A chosen-prefix collision followed in 2020, at a cost then estimated in the tens of thousands of dollars of cloud compute — which is to say, within reach of anyone motivated. NIST has set an end date of 2030 for SHA-1 in all federal applications.

HMAC: what you need when a shared key is involved

A plain hash proves a message was not corrupted. It cannot prove who sent it, because anyone can compute a hash. When two parties hold a shared key — a webhook signature, an API request signature — you need a message authentication code instead.

The intuitive construction, hashing the secret concatenated with the message, is broken for the SHA-2 family and for MD5 and SHA-1. Those functions use the Merkle–Damgård construction, in which the digest is the complete internal state at the end of the input. Knowing H(secret || message) and the length of the secret, an attacker can resume from that state and compute a valid digest for a longer message without ever learning the secret. This is the length-extension property, and it has produced real vulnerabilities in home-made signing schemes.

HMAC, specified in RFC 2104, fixes this by hashing twice with two derived keys, which breaks the attacker's ability to resume the state. It is available everywhere, costs roughly two hash invocations, and requires no cleverness from you. SHA-3, BLAKE2 and BLAKE3 are not vulnerable to length extension and offer their own keyed modes, but HMAC-SHA-256 remains the safe default because it is universally implemented.

One implementation detail that matters as much as the construction: compare MACs in constant time. A normal string comparison returns as soon as it finds a differing byte, and that timing difference is measurable over enough requests, allowing an attacker to recover a valid signature byte by byte. Every platform provides the right function — crypto.timingSafeEqual in Node, hmac.compare_digest in Python, subtle.ConstantTimeCompare in Go.

Message: hello
Key:     shared-demo-key

HMAC-SHA-256 =
88edf304eb62855ca689db051a37e434b6b3b56773d944aaf068e9fdd636a880

Compare with the unkeyed digest of the same message:
2cf24dba5fb0a30e26e83b2ac5b9e29e1b161e5c1fa7425e73043362938b9824

Anyone can compute the second. Only a holder of the key can
compute the first — which is precisely the difference between
"this file is intact" and "this message came from you".

Password storage needs a slow hash, not a fast one

SHA-256 is designed to be fast, and hardware obliges: a commodity GPU computes on the order of 10^10 SHA-256 operations per second. Against a leaked table of SHA-256 password hashes, an attacker does not reverse the function — they guess. They run a breach corpus, a dictionary with mangling rules, and then a brute-force sweep, hashing candidates and comparing. Most real user passwords fall in the first few minutes. Adding a salt is necessary and not sufficient: it defeats precomputed rainbow tables and forces the attacker to work per-account, but it does not slow down a single guess.

Password hashing functions are built to be slow and, in the modern ones, memory-hard, so that the attacker's GPU advantage collapses. bcrypt has a cost factor that doubles the work each increment. scrypt and Argon2 additionally require a configurable amount of memory per computation, which is expensive to parallelise on a GPU or an ASIC. The output is an encoded string carrying the algorithm, the parameters and the salt, so the parameters can be raised later without invalidating existing hashes.

For new systems, use Argon2id. OWASP's current baseline is 19 MiB of memory, 2 iterations and 1 degree of parallelism; raise it until your login endpoint's latency budget complains. bcrypt with a cost of at least 10 remains a perfectly respectable choice and is more widely available — note only that it silently truncates input beyond 72 bytes, so pre-hash long passphrases if you accept them. PBKDF2-HMAC-SHA-256 is acceptable where a standards requirement demands it, at 600,000 iterations or more, but it is not memory-hard and therefore the weakest of the three against dedicated hardware.

bcrypt, cost factor 10 (a real htpasswd -B output):
  $2y$10$BGmm0BZWcCbLgebJ4eSabOpuY0/mHQl/iSB6wNIqSFwTnVn9J1cDW
   |   |  |                      |
   |   |  +-- 22-char salt        +-- 31-char digest
   |   +-- cost: 2^10 key-setup rounds
   +-- algorithm identifier

Argon2id, OWASP baseline parameters (layout, with the salt
and digest shown as placeholders rather than invented values):
  $argon2id$v=19$m=19456,t=2,p=1$<base64 salt>$<base64 digest>

Both encode the parameters, so a verifier reads them from the
stored value. Raising the cost for new passwords does not
break the ones already stored.

What a browser hash tool is good for

Our hash tool computes MD5, SHA-1, SHA-256 and SHA-512 in the page using the browser's Web Crypto implementation. Nothing is uploaded. That makes it convenient for the checks you do dozens of times a week: hashing a short string, confirming that two configuration blobs are byte-identical, generating a fingerprint for a cache key, or checking a digest you were sent against one you computed elsewhere.

It is not the right tool for verifying a large download, and we would rather say so than let you find out at 60 per cent of a four-gigabyte ISO. Hashing in a browser tab means loading the file into the page's memory, which is bounded and shared with everything else the tab is doing. Your operating system already has a purpose-built command that streams the file and will be faster and more reliable. Use shasum -a 256, sha256sum, or Get-FileHash and keep the browser for short inputs.

Two more limits worth stating plainly. First, a hash tool cannot verify a signature, which as discussed above is the check that actually resists an attacker; for that you need GPG or your package manager. Second, hashing a secret does not make it safe to handle — the digest of an API key is not the key, but the key itself was in a text field on the way in, with all the clipboard and extension exposure that implies. If the value is a live credential, hash it on your own machine.

Continue with related checks and tools