URL Encoding Explained: Percent-Encoding, Components, and Punycode

Percent-encoding, precisely

A URI is defined over a restricted set of US-ASCII characters. Anything outside that set, and anything inside it that would be read as structure in the position where it appears, has to be represented indirectly. Percent-encoding does that by replacing a single octet with a percent sign followed by two hexadecimal digits: a space, octet 0x20, becomes %20.

The important word is octet. Percent-encoding operates on bytes, not on characters, so a character must be turned into bytes before it can be encoded — and for modern URLs that means UTF-8. The character é is one code point, two UTF-8 bytes, and therefore two percent-escapes: caf%C3%A9. The Chinese characters 中文 are two code points, six bytes, and six escapes: %E4%B8%AD%E6%96%87. If you ever see a single non-ASCII character producing a single escape, something upstream chose a legacy encoding and the value will not survive a round trip.

The hexadecimal digits are case-insensitive by specification, but uppercase is the normalised form and the one every modern library emits. Treat %2F and %2f as equal when comparing, and emit uppercase when producing. Encoding an already-unreserved character is legal but not normalised, so %61 and a mean the same thing while only one of them will compare equal as a string.

Reserved, unreserved, and the rest

RFC 3986 divides the ASCII range into three groups, and knowing which group a character is in answers most encoding questions directly. Unreserved characters never need encoding anywhere and should never be encoded. Reserved characters carry structural meaning — they delimit components — so they must be encoded whenever they appear as data rather than as a delimiter. Everything else, including space, quotes, angle brackets and every non-ASCII byte, must always be encoded.

The reserved set splits further into general delimiters, which separate the major components of a URI, and sub-delimiters, which separate fields inside a component. That second group is where the per-component rules come from: an ampersand is a delimiter inside a query string and merely a character inside a path segment, so the correct treatment depends entirely on where it sits.

Note one detail that catches people comparing implementations: JavaScript's encodeURIComponent predates RFC 3986 and leaves the characters ! ' ( ) * unencoded, even though the RFC classifies them as sub-delimiters. Most servers do not care, but a signature scheme that requires RFC 3986 canonical form — AWS Signature Version 4, for instance — does, which is why those libraries ship their own escaping function rather than using the built-in one.

The component decides the rule

There is no such thing as "URL encoding a URL". Each component of a URI has its own delimiter set, and encoding a whole URL with one rule either destroys the structure or leaves data that will later be misread as structure. The only reliable approach is to encode each piece of user data as you assemble the URL, never afterwards.

The distinction matters most for the two places data most often arrives from outside: path segments and query values. A slash inside a path segment must become %2F or it will create a new segment. An ampersand or equals sign inside a query value must become %26 or %3D or it will create a new parameter. A hash inside either must become %23 or everything after it becomes the fragment and is never sent to the server at all — which is how a value silently disappears between the browser and the application log.

The practical version of this rule in JavaScript is to build the URL with the URL and URLSearchParams objects rather than string concatenation. URLSearchParams applies the form-encoding rules to every key and value it serialises, which removes the entire class of defects where a value containing a delimiter escapes its component.

The right-hand column lists the characters that will change the meaning of the URL if they are left raw in that position.

encodeURI versus encodeURIComponent

JavaScript gives you two functions, and choosing the wrong one is the most common percent-encoding bug in front-end code. encodeURIComponent escapes everything except the unreserved set plus the five legacy exceptions; it is the function for a single piece of data. encodeURI leaves every reserved character alone because it assumes it has been handed a complete, already-structured URI and is only tidying up spaces and non-ASCII bytes.

Run both over the same input and the difference is immediate. Given a whole URL, encodeURIComponent destroys it by escaping the scheme separator and every slash; given a single value that happens to contain an ampersand, encodeURI leaves the ampersand in place and the value splits into two parameters at the server. Both outputs are legal URI text. Only one of them means what you intended.

The rule that survives code review is simple: use encodeURIComponent on every individual value, never on a URL; use encodeURI only when you have been handed a complete URI as a string and cannot rebuild it. If you are reaching for encodeURI on something you assembled yourself, you assembled it wrong.

const url = "https://example.com/a b?x=1&y=2";

encodeURI(url);
// "https://example.com/a%20b?x=1&y=2"        structure preserved

encodeURIComponent(url);
// "https%3A%2F%2Fexample.com%2Fa%20b%3Fx%3D1%26y%3D2"   now a single value

const value = "q=1&r=2";

encodeURI(value);            // "q=1&r=2"        unchanged, and broken
encodeURIComponent(value);   // "q%3D1%26r%3D2"  correct for a query value

encodeURIComponent("中文");   // "%E4%B8%AD%E6%96%87"
encodeURIComponent("café");   // "caf%C3%A9"
encodeURIComponent("100%");   // "100%25"
encodeURIComponent("~_-.!*()'");  // unchanged: the five legacy exceptions

Plus signs, %20, and where the confusion comes from

A space can appear in a URL as %20 or as a plus sign, and which one is correct depends on a distinction that the syntax does not make visible. The application/x-www-form-urlencoded media type — originally an HTML form serialisation, now used for query strings almost everywhere — defines a space as a plus sign. Generic URI syntax, RFC 3986, does not: there a plus is just a sub-delimiter character and a space is %20.

The consequence is that a plus in a query string is ambiguous unless you know which parser will read it. Most web frameworks apply form-decoding to the query string, so ?q=a+b arrives as the value a b. A path segment is not form-decoded, so /search/a+b arrives with a literal plus. And a genuine plus sign in data must always be escaped as %2B in a query, or it will be read as a space — this is why email addresses with plus-addressing, and base64 values carrying the standard alphabet, break in query strings so reliably.

In practice, %20 is the safer thing to emit because every decoder accepts it in both modes. Note that JavaScript's built-in helpers disagree with each other here: encodeURIComponent produces %20, while URLSearchParams produces a plus, because the former implements RFC 3986 and the latter implements form encoding. Both are right for their own specification, and mixing their outputs in one URL is how you end up with a literal plus in a user's name.

encodeURIComponent("a b");   // "a%20b"
encodeURIComponent("a+b");   // "a%2Bb"

new URLSearchParams({ q: "a b", r: "c+d" }).toString();
// "q=a+b&r=c%2Bd"
//  space became "+", and the literal "+" became "%2B"

Decoding side, both round-trip correctly:
  new URLSearchParams("q=a+b").get("q")   // "a b"
  new URLSearchParams("q=a%20b").get("q") // "a b"
  decodeURIComponent("a+b")               // "a+b"  <- not a space

Double encoding, and how to recognise it

Double encoding happens when an already-encoded value is encoded again, usually because it passed through two layers that each assumed they were the one responsible. The percent sign of the first escape is itself a character that must be escaped, so %20 becomes %2520, and the value now decodes to the literal text %20 rather than to a space.

It is easy to spot once you know the shape. Any %25 followed by two more hexadecimal digits is almost certainly a double encoding rather than a genuine percent sign in the data. Sequences like %2520, %253A and %252F are the fingerprints. The user-visible symptom is a URL that works when pasted into a browser and fails through your application, or a filename that arrives with visible escapes in it.

The fix is never to add a decode step at the end; that only papers over the layer that is encoding when it should not. Find the boundary. Typically a client encodes a value, a framework encodes the query string it is placed in, or a proxy rewrites and re-encodes a path. Decide which layer owns the escaping and make the others pass the value through untouched. Blindly decoding twice is also a security problem: a path-traversal payload written as %252e%252e%252f becomes ../ after two decodes, which is exactly how filter-then-decode ordering gets bypassed.

encodeURIComponent("a b");        // "a%20b"
encodeURIComponent("a%20b");      // "a%2520b"     <- encoded twice

decodeURIComponent("a%2520b");    // "a%20b"       <- still encoded
decodeURIComponent("a%20b");      // "a b"

Fingerprints of a double-encoded value:
  %2520   was a space
  %253A   was a colon
  %252F   was a slash
  %2526   was an ampersand

Hostnames are different: IDN and punycode

One part of a URL is never percent-encoded, and it is the part people most often assume is. DNS labels are restricted to letters, digits and hyphens, and percent escapes are not valid in a hostname. Internationalised domain names therefore use a completely separate mechanism: IDNA, which normalises each label and then transcribes any non-ASCII label with Punycode into an ASCII form prefixed by xn--.

The transformation is per label, not per domain, so only the labels that need it are converted and the dots stay where they are. münchen.de becomes xn--mnchen-3ya.de: the ASCII letters are kept in order, and a compact suffix after the double hyphen encodes where the non-ASCII characters belong. A fully non-ASCII domain converts every label, so 例子.测试 becomes xn--fsqu00a.xn--0zwm56d.

This matters beyond curiosity for two reasons. First, comparison: a browser may display the Unicode form while logs, certificates and configuration files contain the xn-- form, so a naive string comparison between them fails. Second, security: because many scripts contain characters that look like Latin letters, a homograph domain can render almost identically to a legitimate one. Browsers mitigate this by displaying the punycode form when a label mixes scripts suspiciously, which is why an unexpected xn-- in the address bar is worth reading carefully rather than dismissing.

Continue with related checks and tools