The fracture becomes computation

Begin with a cupboard of labels. At first it contains one label for every byte or character. Training a subword vocabulary adds labels for recurring adjacent sequences: perhaps th, then the, then payment. A string that matches a stored label can enter the model through one embedding lookup. A string that lacks such a shortcut arrives as several lookups whose relation must be built by later layers.

Byte-pair encoding became influential because it addressed an old practical conflict. Word vocabularies handle frequent words efficiently but fail on unseen words. Character vocabularies remain open but create long sequences. Sennrich, Haddow and Birch adapted BPE for neural machine translation so rare words could be represented through subword units rather than an unknown-word symbol.^[1] SentencePiece later made BPE and unigram tokenisation trainable directly from raw Unicode sentences, including whitespace as part of the representation.^[3]

Those methods are often presented as compression devices. Their deeper effect comes from deciding which patterns receive dedicated coordinates in the embedding table. A token has no natural meaning merely because it is stored. Yet repeated exposure gives the model a stable identifier on which gradients can accumulate. Fragmented spans distribute their evidence across several identifiers and positions. Atomicity is therefore an allocation of learning opportunities, not a declaration of linguistic truth.

Worked example: a vocabulary learns its shortcuts

Take a toy corpus containing lower, lowest, newer and wider. Start with characters. Count adjacent pairs and merge the most frequent one. If e r occurs most often, the first new unit is er. Later merges may produce low and then lower. Each merge spends one place in the vocabulary to shorten every future occurrence of that pair.

The spending decision is distributional. Replace most of the corpus with Hindi, Tamil or source code and the high-frequency pairs change. Keep the corpus but impose a rule that no token may cross whitespace, or that every digit must remain separate, and the candidate shortcuts change again. The final sequence is caused by at least four choices: fitting corpus, pre-tokenisation rules, vocabulary construction and segmentation policy.

Algorithm choice also changes the inventory. Bostrom and Durrett found that unigram tokenisation aligned more closely with morphology than BPE in their analysis and matched or outperformed BPE in matched English and Japanese language-model experiments.^[4] That result supports comparison, not a universal winner: morphology, compression and downstream utility remain distinct criteria.

A tokeniser has a typed contract

Let Σ* be all finite strings over an input alphabet and V* all finite sequences over a vocabulary V. A deterministic tokeniser is a map from a string to token identifiers:

Here x is the surface string, τ(x) is its token sequence and n_τ(x) is the number of positions it consumes. Lossless tokenisation additionally requires a decoder δ such that δ(τ(x)) = x. Libraries such as tiktoken explicitly describe their BPE encoding as reversible and able to represent arbitrary text.^[16]

Losslessness protects the return trip. It does not make all journeys through the model equivalent. Two token sequences can decode to the same visible string while presenting different identifiers, lengths and neighbourhoods to the network. Kudo's subword regularisation exploits this ambiguity deliberately by sampling alternative segmentations during training, reporting gains especially in low-resource and out-of-domain translation settings.^[2] The intervention works only because segmentation changes the examples from which the model learns.

A useful causal diagram is therefore longer than “text becomes token IDs”. The fitting distribution and tokeniser design create a sequence geometry. The embedding table turns each unit into a learned vector. Attention and feed-forward layers coordinate information across positions. The output head then predicts another token. Boundaries matter when they change the number or arrangement of coordination steps needed to represent a task relation.

Thought experiment 1: move only the boundary

Hold the model weights, prompt wording, numerical values and decoding policy fixed. Add punctuation that causes right-aligned numeric grouping but carries no new arithmetic information. If accuracy changes systematically, the difference cannot be credited to more facts or more parameters. It isolates the interface between representation and procedure. The addition case is unusually clean because decimal place value supplies a known relational structure.

Now repeat the intervention on a task whose useful structure does not align with the new boundaries. Randomly insert separators inside ordinary words while preserving the recoverable characters. If the model becomes worse, that is unsurprising. If it remains unchanged, boundary sensitivity may have been learned away for that regime. This paired control prevents a convenient result from becoming a universal story about “meaningful tokens”.

The practical implication is immediate. A tokeniser specification is not merely a vocabulary file. It includes Unicode handling, normalisation, pre-tokenisation, special-token rules, vocabulary construction, merge or unigram scores, segmentation, decoding and version identity. Change one element and the model's executable input contract changes.

flowchart LR
  A["Opening case, one addition,<br/>two fractures"] --> B["The fracture becomes computation"]
  B --> C["Where the tax appears"]
  C --> D["Why compression is not quality"]
  D -. "re-test" .-> A
  classDef first fill:#0a1424,stroke:#007D68,stroke-width:2px,color:#e6f3ff
  classDef step fill:#0a1424,stroke:#3f7da0,stroke-width:1px,color:#e6f3ff
  class A first
  class B,C,D step

Where the tax appears

The tax is easiest to see when a domain has structure that token frequency does not reliably preserve. Numbers have place value. Names carry identity despite being rare. Code has grammar and identifier conventions. Multilingual services promise comparable access across scripts. Security controls care about code points, rendered glyphs and parser behaviour. Tokenisation can interact with each without being the sole cause of failure.

Numbers: boundaries can disagree with place value

Decimal notation is positional. The meaning of a digit depends on its distance from the right edge of the integer and, for fractions, from the decimal point. A learned subword vocabulary sees recurring character sequences rather than this rule. It may store one-, two- or three-digit spans, dates, common years, currency fragments and leading-zero patterns according to corpus frequency.

This creates several possible geometries. Single-digit tokens preserve a regular base-ten interface but lengthen sequences. Fixed right-origin triads preserve thousands groups. Frequency-learned chunks can compress familiar values while splitting nearby values differently. None guarantees arithmetic competence, because the model must still learn carries, signs, decimal alignment and the requested operation. Yet an irregular interface increases the number of cases the procedure must absorb.

A number audit should therefore include more than average tokens per value. Test equal-length values, leading zeros, signs, decimal separators, scientific notation, dates, identifiers that only resemble numbers, and the output format expected by downstream code. If exact arithmetic matters, use a calculator or typed numerical tool rather than asking tokenisation to rescue an unsuitable execution mechanism.

Names: the rare string is often the important string

Names expose another conflict. A person or organisation may appear once in pretraining yet be the decisive entity in a document. A frequent common word can receive one stable token, while an unfamiliar name is fractured into letters, syllable-like spans or bytes. This does not erase the name. It changes how evidence about the span is distributed and how much exact copying the model must coordinate.

The original subword argument explicitly included names and other rare forms among the cases where compositional units improve open-vocabulary translation.^[1] The modern failure mode is subtler than an unknown token. The system can represent any string, but spelling variants, transliteration, diacritics and script changes may yield very different sequences. That matters for entity linking, deduplication and exact output. Open vocabulary removes an absolute failure and leaves a graded exposure problem.

Do not infer that a one-token name is always safer. A token that occurs rarely during training may have a poorly estimated embedding. A whole-name token can also hide useful internal regularities. The right test is matched entity performance: copy fidelity, retrieval, linking, transliteration and discrimination between near neighbours, stratified by fragmentation and frequency.

Multilingual text: one semantic message can buy unequal context

Suppose four customers submit translations of the same request. The service promises the same maximum context window, but the tokeniser produces five units for the English sentence, eighteen for Hindi, twenty-three for Tamil and thirty for Malayalam. The nominal limit is equal. The usable message length is not.

Petrov and colleagues measured tokenisation lengths for translated text and reported differences up to fifteenfold in some language comparisons. They also observed gaps above fourfold for some byte- and character-level encodings, showing that script encoding length can persist even without a learned subword vocabulary.^[7] Ahia and colleagues connected fragmentation to commercial token charging and multilingual utility across 22 languages.^[8] These are population-level results over studied tokenisers and datasets, not a guarantee about every sentence.

The mechanism is vocabulary competition. A shared inventory has finite capacity. High-resource languages contribute more frequent pairs and therefore win more compression shortcuts under a global frequency objective. Morphology and script also affect how far those shortcuts travel. Rust and colleagues found, across nine languages and five tasks, that replacing a multilingual model's tokeniser with a specialised monolingual one improved downstream results for almost every evaluated language and task, under their retraining setup.^[9]

Thought experiment 2: two ciphers, one meaning

Create two artificial scripts, Amber and Violet. Map every English character bijectively into each script, so sentences have identical semantics, grammar, length in characters and training frequency. Train one shared tokeniser and language model, but give Amber most of the tokeniser fitting corpus while keeping model pretraining examples balanced. Nothing about the language task favours Amber. Only the merge allocation does.

If Amber receives shorter sequences and better performance under a fixed token or compute budget, the result isolates tokeniser allocation. Reverse the fitting imbalance while holding model data constant. The advantage should move to Violet. If it does not, the model may have compensated, the scripts may differ in byte length, or the assumed mechanism is too weak. This is a discriminating experiment because it can fail.

The embedded lab later in the paper performs a small version with English, Hindi, Tamil and Malayalam strings. Its figures are synthetic, but the acceptance logic mirrors a real audit. The toy result is not evidence about those languages as a whole.

Parity need not require abandoning BPE. Foroutan and colleagues introduced Parity-aware BPE, which uses a fair-max rule to improve the currently worst-compressed language at each merge. Across their experiments, token-cost inequality measured by the Gini coefficient fell by up to 89 per cent relative to classical BPE, with negligible global compression impact and no evidence of systematic downstream degradation.^[10] That result is encouraging and bounded: it supports an alternative merge objective under the reported language sets, model sizes and tasks.

Code: frequency does not know the grammar

Programming languages already have lexical units: identifiers, keywords, operators, literals, indentation and delimiters. A general-purpose subword tokeniser follows corpus statistics instead. It may split verifyBeneficiary after verify, inside Beneficiary, or differently after a harmless rename. Whitespace can become part of a token even where the parser treats it as insignificant.

TokDrift applies semantics-preserving changes such as whitespace edits and identifier renaming. Across nine code language models, the authors found substantial behavioural shifts and traced the mismatch to early representations where subword boundaries fail to follow programming-language grammar.^[11] This does not show that grammar tokenisation dominates every code task. It supplies a negative control: when semantics are fixed but surface form and tokenisation move, output stability should be measured rather than assumed.

Context, latency and money: the meter counts the interface

Token count becomes an operating quantity when a model has a token-limited context or an API meters input and output tokens. Google documents that Gemini API request cost is partly determined by token counts and that each model has a maximum token context.^[17] Anthropic's current token-counting documentation similarly links counts to rate limits, cost, routing and prompt length.^[18] These product details can change, so systems should count against the selected model at request time rather than preserve a stale average.

For a fixed model and prompt, more input tokens usually mean less remaining context for evidence or output. Autoregressive generation also produces one token decision at a time, so output fragmentation can increase decoding steps. Input latency is more complicated because modern attention kernels, caching and batching create non-linear costs. The defensible statement is narrower: tokenisation changes the units by which many limits, queues and bills are enforced.

A routing layer that estimates cost from characters or English words can therefore discriminate accidentally. Count the actual serialised request after system instructions, tools, retrieved evidence and safety wrappers are included. Record counts by language and domain, not only globally. A model that looks cheaper per million tokens may be more expensive for a population whose text fragments heavily.

Security: identical-looking text can cross layers differently

Unicode permits several code-point sequences to represent canonically equivalent text. The Unicode Normalisation Forms specification defines normal forms so equivalent strings can have a unique binary representation within the chosen form.^[14] Unicode also contains visually confusable characters across scripts. UTS #39 specifies mechanisms for mixed-script and confusable detection, including examples such as a Latin-looking name containing Cyrillic letters.^[15]

Tokenisation does not create these properties. It can make a boundary mismatch operational. Imagine an allow-list checks the rendered string, a normaliser changes code points, a tokeniser sees the normalised bytes, and a downstream parser evaluates the original field. Each component may be locally reasonable while the composed system lacks one authoritative string representation.

Boucher and colleagues demonstrated imperceptible NLP attacks using invisible characters, homoglyphs, reordering and deletion, finding that small encoding manipulations could substantially reduce performance in tested systems.^[13] The security lesson is not “block Unicode”. It is to make canonicalisation, script policy, tokenisation and parsing order explicit, preserve the original evidence, and test adversarial variants end to end.

Why compression is not quality

The simplest tokeniser metric is corpus token count: apply the tokeniser to a reference corpus and count positions. Fewer positions can reduce context pressure and some compute. It is tempting to make this the optimisation target. The temptation should be resisted because compression can improve for the wrong structural reason.

Negative control: the perfect compressor that learns nothing reusable

Construct a vocabulary that stores every sentence in a small benchmark as one token. Its corpus token count is unbeatable: one position per sentence. Now change one amount, name or verb. The new sentence lacks a stored unit and falls back to fragments. The vocabulary has memorised benchmark surfaces while exposing little reusable internal structure. It compresses the measured corpus and may generalise poorly.

A less extreme version appears when a tokeniser creates units spanning arbitrary word boundaries simply because the phrase recurs. Schmidt and colleagues found that a method designed to minimise token count did not establish the expected downstream advantage. In one analysis, unconstrained pre-tokenisation produced units that crossed word boundaries, while more constrained variants achieved better overall accuracy despite using more tokens.^[5]

This negative control separates two hypotheses. Under the compression hypothesis, the shortest sequence should be best, other factors held fixed. Under the structural-allocation hypothesis, performance depends on what distinctions the shorter sequence preserves. A common token can save coordination. It can also entangle features that the task needs separately. Compression is a resource measure, not a sufficient account of representation quality.

The competing explanation: large models can learn around poor boundaries

A strong objection says that token boundaries are transient scaffolding. Given enough data and capacity, attention can compose fragments, recover character patterns and become robust to alternative segmentations. Several observations support this objection. Modern models often copy unseen strings despite fragmentation. Subword regularisation can train robustness to multiple decompositions. Larger contexts and models can absorb coordination costs that would dominate smaller systems.

This objection weakens any deterministic claim from fertility to accuracy. It does not remove the budget effect. Compensation consumes training evidence, parameters or positions, and may fail in the long tail. The empirical question is not whether a model can ever recover the relation. It is how much resource is required and whether the recovery survives domain shift, rare scripts and adversarial forms.

Byte- and character-level models provide a more radical counterexample to fixed subword vocabularies. ByT5 operates directly on bytes and reported competitive quality with greater robustness to noise and spelling-sensitive tasks, while documenting trade-offs in sequence length, FLOPs and inference speed.^[12] CANINE removes an explicit token vocabulary for an encoder and uses downsampling before a deep Transformer stack.^[19] These systems show that a static subword lexicon is not necessary for capable language modelling.

They do not eliminate segmentation economics. Raw byte sequences are longer, so architectures introduce downsampling, local encoders or learned patches. FLEXITOKENS, published in ACL Findings 2026, learns byte boundaries that can adapt to new languages and domains; its authors report reduced over-fragmentation and task improvements against studied BPE and gradient-based baselines.^[20] The boundary has moved from a fixed preprocessing table into a learned component.

The system boundary that keeps the claim honest

A tokeniser and a trained model form one coupled object. Token identifiers index a particular embedding matrix and output vocabulary. Replacing the tokeniser while leaving weights fixed usually maps text to the wrong learned coordinates. A controlled comparison therefore requires matched pretraining or a justified embedding-transfer and adaptation method. Counting tokens under several vendors is useful for cost analysis, but it does not isolate the causal effect on model quality.

The claim also stops at the string interface. Tokenisation cannot ensure factuality, authorisation, policy compliance or exact execution. It cannot decide whether a retrieved source is current. It may affect how evidence is represented, yet the reliability of a compound system still depends on retrieval, tools, typed validation, decision rights and outcome verification.

The strongest practical position is therefore conditional. Token boundaries deserve scrutiny where strings are structurally dense, distributionally rare, multilingual, metered or security-sensitive. Elsewhere, a well-supported standard tokeniser may be entirely adequate. The mechanism matters most when boundary debt concentrates on the cases the system cannot afford to mishandle.

Audit before commitment

Tokeniser decisions are unusually expensive to reverse after pretraining because the vocabulary is tied to the input embeddings, output head, cached datasets and serving interfaces. Procurement creates a similar lock even when the model is external: prompts, chunk sizes, rate budgets and cost forecasts are calibrated to a particular token meter. The useful control point is therefore before training, model selection or routing policy is frozen.

Worked scenario: a multilingual payment-review packet

A review assistant receives a customer instruction, a beneficiary name, an amount, two retrieved policy passages and a short code-like validation trace. English, Hindi, Tamil and Malayalam are supported. The model may summarise evidence and draft a recommendation, but a deterministic service validates identifiers and executes no payment without an authorised human decision.

The team initially compares two candidate models on an English benchmark and average price per token. Model A is slightly more accurate and cheaper per token. The tokeniser audit changes the decision. On matched translated packets, Model A uses 2.8 times as many tokens for the most fragmented supported language as for the least fragmented one. Long customer names and transliterated identifiers consume another tail of small units. Several packets cross the retrieval budget, so the context compiler drops the oldest policy passage. Model B has a higher listed token price but a lower end-to-end packet cost and fewer evidence truncations for the supported population.

Now inspect the trace field. The validation code is semantically unchanged by a rename from beneficiary_id to beneficiaryId, yet Model A's recommendation changes more often than the baseline tolerance permits. This does not prove tokeniser causation because the models differ in weights and training. It does establish a procurement risk. The team adds a semantics-preserving perturbation suite and treats stability as a selection criterion.

Finally, a mixed-script beneficiary alias contains a Cyrillic character that resembles Latin a. The language model flags nothing. The deterministic identity service rejects the alias under its script policy and preserves both original and normalised strings in the evidence record. The architecture does not ask the tokeniser to perform identity control. It uses the audit to discover where deterministic checks are mandatory.

The changed decision is not “choose the lowest fertility”. The team chooses a routing and context policy that meets parity and truncation thresholds, uses exact token-count endpoints for cost admission, sends arithmetic to a calculator, and keeps Unicode identity validation outside the model. The tokeniser audit changes the whole request contract, not only the model score.

The tokeniser acceptance record

Use a representative corpus whose units of analysis match the service. For a chat assistant, that may be complete serialised requests. For a code model, use repositories and semantics-preserving variants. For document intelligence, include page text, tables, identifiers, OCR noise and supported scripts. Keep a locked audit set outside tokeniser fitting so a favourable vocabulary cannot simply memorise the test strings.

Record the following evidence before acceptance. Thresholds are application choices, not universal constants.

A release decision should preserve denominators and tails. “Average fertility 1.3” hides whether one supported script sits at 4.8. “No round-trip failures” is incomplete without the number and construction of tested strings. “Stable on code” requires a declared set of whitespace, comment, formatting and rename transformations. The acceptance record should be versioned with the model because a tokeniser update can change counts even when the visible prompt remains the same.

Executable artefact: adversarial tokeniser cases

The browser lab below trains two tiny BPE vocabularies from synthetic text using the same merge budget. One fitting corpus is English-heavy; the other allocates more examples to Hindi, Tamil, Malayalam and names. Both use digit pre-tokenisation so the multilingual intervention is not confounded by learned number chunks. Select a case or enter text to inspect the resulting units.

The positive test asks whether balanced allocation reduces the spread of token counts across four matched payment instructions. The negative test supplies a decomposed accent or mixed-script confusable and checks whether normalisation or script analysis changes the view. These tests reveal logic. They do not estimate the quality of a production model.

function trainBPE(corpus, mergeBudget, digitPolicy = "digits") {
  const sequences = corpus.flatMap(text =>
    lexemes(text, digitPolicy).map(piece => Array.from(piece))
  );
  const merges = [];

  for (let step = 0; step < mergeBudget; step += 1) {
    const counts = countAdjacentPairs(sequences);
    const best = highestFrequencyPair(counts);
    if (!best || best.count < 2) break;
    merges.push([best.left, best.right]);
    applyMergeInPlace(sequences, best.left, best.right);
  }
  return merges;
}

// Positive case: rebalance fitting data, keep implementation and merge budget fixed.
const heavyRatio = languageSpread(heavyMerges);
const balancedRatio = languageSpread(balancedMerges);
console.assert(balancedRatio < heavyRatio);

// Negative case: compare the preserved string with its NFC representation.
const original = "cafe\u0301";
console.assert(original !== original.normalize("NFC"));

A compact decision rule

Accept the tokeniser only when four statements are simultaneously supportable:

• The fitting allocation represents the populations, scripts and formats the model will serve.
• Fragmentation and structural alignment remain within declared budgets for important tails, not merely the average corpus.
• Round-trip, normalisation and adversarial cases pass deterministic string and security controls.
• Matched downstream evaluation shows that any efficiency gain preserves the decisions and behaviours that matter.

This rule also disciplines model procurement. A buyer cannot retrain a proprietary tokeniser, but can measure the interface. Test representative serialised requests through official counting endpoints, compare truncation and costs by population, probe exact strings, and preserve model-specific counts in the router. When a model fails a boundary-sensitive domain, choose a better model, route to a tool, or narrow the task. Prompt tricks alone should not carry an architectural obligation.

stateDiagram-v2
  [*] --> Claim
  state "Figure 1. The same addition<br/>reaches the model in two<br/>geometries" as Claim
  state "Worked example: a vocabulary<br/>learns its shortcuts" as Test
  state "Figure 2. Every merge spends<br/>finite vocabulary on a<br/>recurring span" as Revision
  Claim --> Test: expose mechanism
  Test --> Decision: evidence survives
  Test --> Revision: rival survives
  Revision --> Claim: narrow or rebuild
  Decision --> [*]

Compact glossary

Token

A vocabulary identifier consumed or predicted by the model. It may represent a byte, character, subword, word, punctuation span or special control symbol.

Pre-tokenisation

Rules that create boundaries before vocabulary segmentation, such as whitespace handling, byte conversion or forcing digits to remain separate.

Fertility

The average number of subword tokens produced per word or another declared unit. Its denominator must be stated, especially across languages without whitespace-delimited words.

Canonicalisation

The deterministic conversion of text to an authorised comparison form. Unicode normalisation is one component; script and identifier policy may require additional rules.

Boundary alignment

The extent to which token edges coincide with relations important to a task, such as decimal place groups, code lexemes or stable entity segments.

Tokeniser parity

A comparison of token cost across matched content, languages or populations. It describes allocation equality, not complete equality of model quality.

sequenceDiagram
  participant Q as Question
  participant M as Mechanism
  participant E as Evidence
  participant D as Decision
  Q->>M: Opening case, one addition, two fractures
  M->>E: Figure 1. The same addition reaches the model in two geometries
  E-->>M: compare rival explanations
  E->>D: Move the decision upstream
  D-->>Q: revise scope or act

Move the decision upstream

A tokeniser is usually invisible after a model begins to answer. That invisibility encourages a category error. We treat boundaries as a clerical conversion even though they decide which strings receive dedicated learned units, how many positions a request occupies and where software layers may disagree about the text.

The durable intuition is a finite cupboard of shortcuts. Corpus frequency and design rules fill it. Common spans enter cheaply. Rare languages, names, formats and adversarial strings assemble themselves from smaller pieces. The model may learn that assembly well. When it does not, the cost appears downstream as irregular arithmetic, brittle code, reduced context, unequal pricing or a validation gap.

No single intrinsic score resolves the choice. Fertility measures length, not task structure. Morphological agreement helps some tasks and can hide useful cross-lingual sharing in others. Byte models avoid a fixed lexicon and pay elsewhere in sequence processing. Large models can compensate for awkward boundaries, but compensation is an empirical result rather than a procurement assumption.

The changed decision is to approve a tokeniser as part of the model architecture and release evidence. Fix its identity in lineage. Audit matched populations and important tails. Test numerical, entity, code and Unicode interventions. Count the complete request on the selected serving model. Require downstream evidence before translating compression into quality. Keep exact arithmetic, identity policy and effect validation in deterministic components.

Once these controls exist, tokenisation becomes less mysterious. It is neither the secret source of intelligence nor harmless plumbing. It is an allocation made before the first attention layer, with consequences that can be measured before the system pays them.