Most developers treat strings as a solved problem: a string is text, text is easy, move on. Then one day a bug ticket arrives that says the word café breaks the search feature, or an emoji splits a username in half, or the API rejects a 10-character password that the UI swears is fine. Every one of those bugs is the same root cause: the assumption that one character equals one byte, and that string length is simple. It never was.
This post is the map of the territory: how text actually lives in bytes, why Unicode works the way it does, and what “length” even means once you leave ASCII behind. It is foundational knowledge that pays off in every language you will ever use.
Before Unicode: the 128-byte world
ASCII fit the Latin alphabet, digits, and punctuation into 7 bits — 128 slots, one byte per character, and a parity bit to spare. It worked because it was designed for American English in the 1960s. The rest of the world squeezed into the leftovers: each region defined its own mapping for the upper 128 values of a byte. The byte 0xE4 might be ä in one code page, Ω in another, something else entirely in a third. The same bytes meant different text depending on which code page the reader assumed — the reason old emails and files sometimes arrive as mojibake. The Greek page, the Latin-1 page, the Cyrillic KOI8-R page: same bytes, different letters.
Unicode: one number per character, and then the hard part
Unicode’s core idea is disarmingly simple: every character in every writing system gets a number, a code point. U+0041 is A, U+03B1 is α, U+1F600 is 😀. Version 16.0 defines 154,998 of them across Latin, Greek, Cyrillic, Arabic, Hebrew, the CJK scripts, math symbols, and every emoji the consortium has blessed. Text becomes a sequence of code points; the standard then takes responsibility for making those numbers mean the same thing everywhere.
The encodings: how code points become bytes
Code points are abstract numbers; bytes on disk are concrete. The encoding is the bridge, and the choice of encoding shapes everything about how strings behave.
UTF-32: honest and wasteful
The naive encoding: four bytes per code point, always. Random access is trivial — the fifth character is at byte 20. The cost is memory: an ASCII-heavy workload stores 4× the necessary bytes, and nobody ships it. Its real role is as a reminder that the obvious design is rarely the right one.
UTF-16: the compromise with a tail risk
Two bytes per code point was enough for a while — until CJK unification pushed the repertoire past 65,536 and the fixed dream died. The fix was surrogate pairs: code points above U+FFFF are encoded as two 2-byte units, and no code unit value is ambiguous. UTF-16 is what Java, JavaScript, and C#/.NET use internally, which means their length counts code units, not characters. It also comes in flavors: little-endian and big-endian, plus a byte-order mark. Windows API is UTF-16. It works, mostly quietly, until astral characters show up.
UTF-8: the variable-length winner
The encoding that won. Bytes 0x00–0x7F are ASCII, one byte each, unchanged — every valid ASCII document is already valid UTF-8. Higher code points use 2 to 4 bytes with a self-synchronizing design: lead bytes announce how many continuation bytes follow (0xC0 range leads 2-byte sequences, 0xE0 range 3-byte, 0xF0 range 4-byte), and continuation bytes always look like 10xxxxxx. The price is variable length, and the payoff is compatibility with the entire installed base of ASCII tooling. The web runs on it (over 98% of pages), files run on it, Go and Rust strings are UTF-8 natively, and POSIX filesystems treat it as the default lingua franca.
One more property worth knowing: because the lead-byte structure is unambiguous, a decoder can resynchronize mid-stream, and byte-wise functions like strcmp ordering and substring search preserve code-point order. The design is older than the web’s dependence on it and has held up better than anyone had a right to expect.
Grapheme clusters: what “one character” actually means
Here is where intuition breaks. What a user perceives as one character — one press of backspace, one cursor advance, one thing selected — is a grapheme cluster: one or more code points that combine into a single visual unit. Consider the family emoji 👨👩👧👦: that is four emoji code points joined by three zero-width joiners, seven code points in total. It is one grapheme. Or é: one code point (U+00E9), or two (e + U+0301 combining acute accent) — both render identically, and both are one grapheme.
The consequences are not theoretical:
- String “length” is ambiguous: bytes, code units, code points, or graphemes give four different answers for the same string.
- Slicing by index can split a grapheme in half, producing broken rendering and corrupted comparisons.
- Reversing a string naively puts combining marks before their base character, scattering diacritics across the word.
- Username and hashtag limits enforced as code-unit counts overcharge multilingual users and undercharge emoji users.
The fix is to operate on grapheme clusters wherever users are involved. Modern libraries expose it directly: Swift made Character a grapheme cluster by default — arguably the most user-semantic string model in a mainstream language; Rust’s grapheme handling lives in the ecosystem rather than std; Go’s range over strings iterates runes but grapheme segmentation needs a library. The rule of thumb: length checks and slicing for human-facing limits belong at grapheme granularity; byte-level ops are for storage and transport.
Normalization: same text, different bytes
Because é can be composed or decomposed, two visually identical strings can differ in bytes. Compare them byte-wise and they are unequal; store one and search for the other and the match fails. Unicode defines normalization forms to reconcile this: NFC composes (combining marks merged into precomposed forms where possible), NFD decomposes (everything split into base + marks). There are also compatibility forms that additionally fold visually similar characters — the ligature fi becomes fi, full-width ASCII becomes ASCII — at the cost of losing information.
Practical guidance: normalize to NFC on the way in for storage and comparison, and be aware that case-insensitive search is its own science — case mapping is locale-dependent (the Turkish dotless i being the classic trap) and the Unicode default case folding does not cover every language’s rules. Protocols that compare strings for identity — package names, cache keys, dedup — should say explicitly which normalization they assume, because “it works for ASCII” is not a specification.
What this means in your language
The trapdoors, language by language:
- JavaScript: strings are UTF-16.
lengthand[]index code units — emoji outside the BMP get split. UseArray.from(s)for code points orIntl.Segmenterfor graphemes. - Python:
stris a sequence of code points, solen("😀")is 1 — but grapheme clusters still need segmentation, and files must be opened with explicit encoding, because the default depends on the platform. - Go: strings are UTF-8 byte sequences;
lenis bytes,[]rune(s)converts to code points. Iteration withrangedecodes runes for free. - Rust:
Stringis guaranteed UTF-8;lenis bytes, and slicing panics on non-character boundaries — the compiler choosing safety over convenience. - C#/Java: UTF-16 underneath.
Lengthcounts code units;StringInfo(C#) andcodePointCount(Java) give you better units.
Notice the pattern: every language answers “what is a character?” differently, and every one of them is lying to you slightly. The honest answer lives one layer down.
Wrapping up
Text is a stack: bytes, an encoding that maps them to code points, Unicode’s number-for-every-character registry, and grapheme clusters that model what users actually see. Most bugs labeled “Unicode problems” are really “I assumed one byte equals one character” problems. Know which layer each operation lives on — bytes for transport, code points for processing, graphemes for people — and the weird tickets stop being weird. In a career of debugging, few fundamentals return value as reliably as this one.