Gzip Compression for Developers

DEFLATE is the algorithm; gzip and zlib are wrappers around it

Three names are used almost interchangeably and they are three different things. DEFLATE, specified in RFC 1951, is the compression algorithm. zlib, RFC 1950, is a six-byte wrapper around a DEFLATE stream: a two-byte header and a four-byte Adler-32 checksum. gzip, RFC 1952, is a larger wrapper: a ten-byte header carrying a magic number, a timestamp and flags, optionally a filename, and an eight-byte trailer holding a CRC-32 and the uncompressed size.

The practical consequence is that a decompressor expecting one container will reject the others even though the compressed payload inside is identical. This is the cause of the classic "incorrect header check" error: a zlib stream handed to a gzip reader, usually because someone compressed with the wrong function in a two-function API. The first bytes tell you which you have — 1f 8b 08 is gzip, 78 9c is the most common zlib header, and a raw DEFLATE stream has no recognisable prefix at all.

The size difference between the containers is fixed and small, which matters only at the very bottom of the range. Compressing the single byte "a" produces 3 bytes of raw DEFLATE, 9 bytes of zlib, and 21 bytes of gzip. On a megabyte of HTML the 18 bytes of gzip overhead are irrelevant; on a 40-byte API response they are the whole story.

The compressed payload is the same in all three. Only the framing differs.

How DEFLATE actually saves bytes

DEFLATE combines two ideas. The first is LZ77: as the encoder walks the input, any sequence it has already seen within a sliding window is replaced by a back-reference — a distance and a length — instead of being written out again. The window is 32 KiB, which is the single most important number in the format. Two identical paragraphs 30 KB apart compress well; the same two paragraphs 40 KB apart do not, because the first has fallen out of the window by the time the second arrives.

The second idea is Huffman coding: the literals and back-references that remain are written with variable-length codes, so common symbols get short codes and rare ones get long ones. DEFLATE can use a fixed code table or build a custom one per block and include it, and the encoder chooses whichever is smaller.

This explains the whole behaviour of the format. Repetitive text compresses extraordinarily well: a hundred identical bytes become a literal, a back-reference and a length, which is why 100 copies of the letter a compress to 24 bytes including the 18 bytes of gzip framing. Structured data compresses well because it repeats its own keys: a JSON array of forty user records is mostly the strings id, name, email and role, over and over. Random data does not compress at all, because no sequence repeats and no symbol is more common than another — 10000 random bytes come out of gzip at 10023, slightly larger than they went in.

Compression level is a search-effort dial, not a different algorithm. Higher levels spend more time looking for longer and more distant matches. Going from gzip level 6 to level 9 on the JSON sample below saves 17 bytes for noticeably more CPU, which is why level 6 is the default nearly everywhere and why raising it is rarely the optimisation people hope it is.

Content-Encoding and Transfer-Encoding are not alternatives

Content-Encoding is a property of the representation. It says this resource has been transformed, and it stays transformed end to end: through every proxy, into every cache, until the client decodes it. The client advertises what it can accept with Accept-Encoding, the server picks one and names it in Content-Encoding, and Content-Length describes the compressed size because that is what is on the wire. An ETag is computed over the encoded representation, which is why a server that compresses must either vary the ETag by encoding or mark it weak.

Transfer-Encoding is a property of one hop. It describes how the message was framed between two adjacent parties and may be undone and redone by every intermediary. In practice only one value matters, chunked, which lets a server stream a response of unknown length. The specification does permit Transfer-Encoding: gzip, but essentially no client or server implements it, and attempting it is a reliable way to receive unreadable bytes. In HTTP/2 and HTTP/3 the header does not exist at all, because framing is handled by the protocol itself; Content-Encoding works exactly as before.

The header that gets forgotten is Vary: Accept-Encoding. Without it, a shared cache that stored a gzip-encoded response can serve it to a client that never asked for gzip, which is a real and long-standing source of corrupted pages. Any server that negotiates encoding must send it.

Request
  GET /api/users HTTP/1.1
  Accept-Encoding: br, gzip, zstd

Response
  HTTP/1.1 200 OK
  Content-Type: application/json; charset=utf-8
  Content-Encoding: gzip          <- end to end, survives every proxy
  Content-Length: 431             <- the COMPRESSED length
  Vary: Accept-Encoding           <- without this, caches serve the wrong body
  ETag: W/"c1f0a2"                <- weak, because the bytes depend on encoding

Transfer-Encoding: chunked is hop by hop and only describes framing.
It is not an alternative way to request compression, and it does not
exist in HTTP/2 or HTTP/3.

gzip, brotli and zstd on the same data

gzip is the universal floor. Every HTTP client made in the last twenty-five years accepts it, so it is the correct fallback and, for many APIs, the only encoding worth configuring. Its ceiling is set by the 32 KiB window and by a format that has not changed since 1996.

Brotli wins on ratio for web content, and by more than its window size alone explains. It carries a built-in dictionary of roughly 120 KB of common web strings — HTML tags, HTTP header names, common English and JavaScript fragments — so it starts with a head start on exactly the content the web serves, and its window can reach 16 MiB. On the 3370-byte JSON sample measured below, brotli at its default quality produced 254 bytes against gzip's 431. The cost is encoding time at the top quality levels, which is why brotli 11 belongs in a build step for static assets and brotli 4 or 5 belongs on dynamic responses, where it is roughly competitive with gzip 6 on CPU.

Zstandard trades a little ratio for a large decompression speed advantage and a much cheaper compression curve, and it supports trained dictionaries, which is transformative for many small similar payloads such as JSON API responses. It is now accepted as a Content-Encoding by current versions of the major browsers, but it is the newest of the three and the one most likely to be missing from an older client, a corporate proxy or an embedded HTTP library.

Forty user records with repeating keys — ordinary API output. Absolute numbers depend entirely on the input; the ordering is what generalises.

What compresses, and what only costs you CPU

Text-shaped data compresses, because it repeats. HTML, CSS, JavaScript, JSON, XML, SVG, CSV, plain text, source code and WebAssembly all benefit substantially, typically landing somewhere between a fifth and a tenth of their original size. The more structured and repetitive the format, the better it does — which is why a JSON array of uniform records compresses better than prose of the same length.

Already-compressed data does not compress, and attempting it costs CPU on both ends while making the payload slightly larger. That covers every common image format except uncompressed BMP and TIFF — JPEG, PNG, GIF, WebP and AVIF all compress internally — as well as MP4, WebM, MP3, AAC, ZIP, gz, 7z and every archive format. It also covers encrypted data, which is indistinguishable from random by design. The measurement is blunt: 10000 random bytes come back from gzip at 10023.

Fonts need a specific note because they are easy to get wrong. WOFF 2.0 already uses Brotli internally, so serving it with Content-Encoding: brotli compresses a compressed file and gains nothing. WOFF 1.0 uses zlib internally and is in the same position. Raw TTF and OTF do compress, which is why they should be converted to WOFF2 rather than gzipped.

If you are unsure about a particular payload, measure it rather than reasoning about it. Compressing a representative sample takes a second, and the Gzip tool on this site will report the compressed size for a pasted body directly. One measurement settles an argument that otherwise recurs in every performance review.

input                                  gzip output
--------------------------------------------------------------
100 copies of the letter "a"           24 bytes    (18 of that is framing)
3370-byte JSON, 40 uniform records     431 bytes
10000 cryptographically random bytes   10023 bytes  <- larger than the input
2 bytes, the string "OK"               22 bytes     <- 11x larger

gzip framing is a fixed 18 bytes: a 10-byte header plus an 8-byte
trailer holding the CRC-32 and the original size. Below roughly a
hundred bytes of input, the framing dominates the result.

When compressing is the wrong choice

The first case is size. Below a few hundred bytes the fixed framing and the poor match-finding on short inputs mean compression saves little or nothing, and below about 50 bytes it reliably makes things bigger. There is also a network argument: a response that already fits in a single TCP segment does not arrive any sooner for being smaller. nginx defaults gzip_min_length to 20, which is far too low to be useful; most teams raise it to somewhere between 256 and 1400 bytes.

The second case is CPU. On a high-throughput dynamic endpoint, compressing every response at a high level can cost more latency than the transfer saves, particularly on a fast internal network where bandwidth was never the constraint. Static assets should be compressed once at build time and served precompressed; dynamic responses should use a middling level. Compressing at level 9 on the fly is almost always the wrong trade.

The third case is security, and it is the one that catches people by surprise. Compression ratio leaks information about content. If a response contains both a secret — a CSRF token, an account number — and a string the attacker controls, the attacker can vary their input and watch the compressed length change: when their guess matches part of the secret, the two compress together and the response gets shorter. This is the BREACH attack, it works over TLS, and TLS does nothing to prevent it because the size is visible regardless of encryption. The mitigations are to not compress responses that mix secrets with reflected input, to mask CSRF tokens with a per-request random value, and to separate secret-bearing endpoints from ones that echo user input. Its predecessor CRIME attacked compression of request headers, which is why TLS-level compression was removed entirely.

The fourth case is decompression of untrusted input. If your service accepts gzip-encoded request bodies, a small upload can expand to an enormous buffer — the classic decompression bomb. Always decompress with a hard cap on the output size and a limit on the expansion ratio, and reject rather than truncate when either is exceeded.

Defaults that are right most of the time

Almost every service can adopt the same configuration and be done: compress text-shaped content types and nothing else, set a minimum length in the high hundreds of bytes, precompress static assets at build time with brotli at maximum quality and gzip as a fallback, and use a moderate level for dynamic responses.

The remaining decisions are measurements, not opinions. If someone proposes raising the compression level, ask for the before and after byte counts and the CPU cost. If someone proposes compressing images, point at the fact that they are already compressed. And if a client reports a corrupted response, check the container and the headers before suspecting the compressor — the overwhelming majority of those reports are a zlib stream in a gzip reader, or a cache that was never told to vary.

• Compress text, JSON, XML, SVG, CSS and JavaScript; never compress images, video, audio, archives or encrypted data.
• Set a minimum response size — nginx's default of 20 bytes is not a useful threshold.
• Always send Vary: Accept-Encoding when the response body depends on the requested encoding.
• Do not compress a response that contains both a secret and attacker-controlled text.
• Cap the output size when decompressing anything you did not produce yourself.

Continue with related checks and tools