A language model never receives the sentence you typed. It receives a sequence such as [1842, 291, 77, 9081]. Each number selects an embedding. The model operates on those selected vectors and their positions. Text becomes available only through the tokenizer that produced the sequence and the decoder that can turn tokens back into text.

The usual reassurance is that modern tokenizers are lossless. If the decoder reconstructs the original string exactly, what could have changed? The hidden assumption is that recoverable information is equally learnable information. That assumption is false. A zip file and a printed paragraph can contain the same information while presenting radically different work to the reader.

Lossless does not mean learning-neutral

Let the raw string be x. A tokenizer T maps it to a token sequence z, and a decoder D maps the sequence back. Losslessness gives us D(T(x)) = x. That is valuable. It prevents unknown strings from disappearing. It does not say that two tokenizers produce equal workloads.

An autoregressive model learns a distribution over the token sequence. Its training objective contains one prediction term for each token position. Its embedding and output matrices contain one row for each vocabulary item. Its context limit is counted in tokens. A different segmentation therefore changes the number of prediction events, the location of boundaries, the frequency of each learned symbol and the amount of text that fits into a fixed sequence.

Here n is not an innocent index. It is the number of model steps used to represent the string. Change the tokenizer and n changes. The targets change. The evidence pooled into each vocabulary row changes. Even when two models reconstruct the same sentence, they solve different intermediate prediction problems.

How subwords are invented

Subword tokenisation became popular because whole-word vocabularies fail on rare and unseen words, while character sequences can be long. Byte pair encoding, adapted for neural translation by Sennrich and colleagues, starts with small symbols and repeatedly merges frequent adjacent pairs.[1] Frequent strings become single tokens. Uncommon strings remain composable from smaller pieces.

SentencePiece made a related family of methods operate directly on raw sentences rather than assuming a language-specific word splitter.[2] Other systems use a probabilistic Unigram model, byte-level BPE or handcrafted pre-tokenisation rules. The algorithms differ, but they all choose a finite vocabulary and a segmentation procedure.

The tokenisation exposure vector

Compression alone cannot describe the trade-off. A one-token word is efficient, but its embedding may be weakly estimated if the word is rare. A character sequence uses stable units, but spends many positions and forces later layers to assemble every word. A segmentation can also cut across the unit required by the task, such as a digit group, identifier segment or suffix.

A practical audit can keep three quantities separate. For a string x, tokenizer T, representative corpus C and a set of task-relevant spans U, define a tokenisation exposure vector:

L is the budget tax. B is the boundary tax. R is the estimation tax. They are not ingredients of a universal quality score. They are separate causal warnings. The correct balance depends on the workload.

The word visible is operational here. It means directly presented at the model interface, not consciously perceived. A transformer can infer sub-token structure, and a byte model can infer words. Tokenisation changes the initial coordinate system. It does not settle what representations later layers will learn, still less whether a system has awareness.

Gradient sharing follows the cuts

The cuts also decide which observations share parameters. When a token appears as an input, its embedding receives learning signal through every context in which it occurs. When it is the prediction target, its output row receives direct evidence. A coarse token pools evidence only with exact repeats of that token. A smaller piece can pool evidence across many words, but later layers must compose the pieces into the required structure.

Take a tiny corpus with 600 occurrences of refund, twelve of refundable and four of nonrefundable. A whole-word tokenizer gives nonrefundable four direct target events. A subword tokenizer might produce non | refund | able. The pieces now receive evidence from many contexts, but the model must learn that their ordered composition reverses the permission. Neither representation is free. One pays an estimation cost; the other pays a composition cost.

This explains why vocabulary size and corpus mix cannot be chosen independently. Adding a token creates a new place to store a directly addressable pattern, but it also divides the evidence that would otherwise train shared pieces. A token is both a compression decision and a statistical pooling decision. The useful question is not whether a split looks linguistically elegant. It is whether the split allocates enough repeated evidence to the units the task needs while keeping their composition tractable.

flowchart LR
  A["The cancellation clause that<br/>vanished"] --> B["The lens before the model"]
  B --> C["Three taxes on the same text"]
  C --> D["When the ruler becomes the result"]
  D -. "re-test" .-> A
  classDef first fill:#0a1424,stroke:#087A98,stroke-width:2px,color:#e6f3ff
  classDef step fill:#0a1424,stroke:#3f7da0,stroke-width:1px,color:#e6f3ff
  class A first
  class B,C,D step

The budget tax

Return to the payment policy. A context limit is expressed in token positions, not sentences, characters or propositions. If one language needs twice as many tokens for the same translated material, it receives roughly half the semantic span before truncation. It may also require more generation steps for the answer.

This is more than hypothetical. Petrov and colleagues measured large differences in tokenisation length for translated text, reaching a factor of fifteen in some comparisons. They also found that moving to character or byte units did not remove all disparity because scripts require different numbers of characters or UTF-8 bytes.[8] Ahia and colleagues examined commercial API cost and utility across 22 languages and found that token-based pricing could charge some language communities more while returning poorer results.[9]

Those studies establish unequal encoding length and associated cost. They do not prove that tokenisation causes every multilingual quality gap. Training-data quantity, data quality, script normalisation, evaluation translation and model capacity also matter. Controlled studies help separate them. Rust and colleagues trained matched models and found that a language-specific tokenizer played a major role alongside pre-training data size; replacing a multilingual tokenizer with a specialised one improved downstream performance for almost every tested task and language.[5]

Vocabulary size creates another cost. More vocabulary rows consume parameters in the input embedding and output projection. Too small a vocabulary lengthens sequences; too large a vocabulary gives many tokens sparse evidence. In a controlled study of 24 models at 2.6 billion parameters, Ali and colleagues found that tokenizer choice affected both downstream performance and training cost. They also found that common intrinsic metrics such as fertility and parity did not consistently predict downstream performance.[11]

Efficiency is therefore a configured-system property. Token count, vocabulary parameters, attention work, output steps, cache use and hardware kernels interact. A tokenizer can reduce sequence length yet worsen task quality, or increase vocabulary size enough to consume the capacity it was meant to save.

Equity begins as a resource allocation question

Two claims are often blended. The first is an allocation claim: equivalent content receives different token budgets, prices or latency. The second is a capability claim: the model answers one language less accurately. Token counts can establish the first directly. They cannot establish the second without a matched model test.

Suppose two parallel customer messages contain the same request. Tokenizer T produces 120 tokens for message A and 240 for message B. Their token ratio is 2. Under a fixed 240-token input allowance, A uses half the window and B uses all of it. Under per-token charging, B consumes twice the metered input units. Under a fixed generation-time envelope, the longer prompt leaves less room for output or additional evidence. No assumption about the model’s language competence is needed to identify that allocation.

Now ask whether accuracy differs. That requires parallel meanings, comparable evidence, careful translation and enough repeated trials to separate tokenizer effects from data and evaluation effects. A language can be expensive to encode yet well answered, or compactly encoded yet poorly learned. Representational parity and capability parity are different measurements. A responsible system records both, because correcting one does not guarantee the other.

The boundary tax

Consider addition. Humans usually align numbers by place value from the right. A tokenizer that groups a seven-digit number from the left may produce 830 | 208 | 0. The answer may require groups with a different offset. A token boundary now slices through the columns where carries must line up.

Singh and Strouse used this kind of intervention on specific 2023-era GPT checkpoints. Right-aligned three-digit grouping improved addition accuracy by as much as 20 percentage points in their setting. Error analysis connected much of the gap to token alignment and answer-length mismatch, while larger models reduced but did not uniformly eliminate the effect.[12] The result is narrow but important: a measured reasoning score moved when the arithmetic problem stayed fixed and the segmentation changed.

The same issue appears in spelling, sorting by letters, copying identifiers and manipulating code. A token may contain several characters, so a single model step does not directly expose each character as a separate position. This does not make the task impossible. It makes the model infer a finer structure than its input interface provides.

This paired design is stronger than showing that one prompt format works better. Formatting can introduce familiar training patterns, clearer visual cues or a different answer distribution. Those are rival generators. The diagnostic becomes persuasive when several interventions move the same boundary property while other surface features vary, and the error moves with the property.

The estimation tax

Coarse tokens can make a frequent phrase cheap. They can also create a long tail of rare vocabulary entries. Each rare token has its own embedding and output parameters, yet receives few direct updates. Fine units reuse evidence across many words, scripts and domains, but require later composition.

Subwords succeeded in part because they offered this middle ground for rare and unseen words.[1] Yet shared vocabularies are not always beneficial. Limisiewicz and colleagues found that cross-language vocabulary overlap could harm some syntactic downstream tasks.[10] The result warns against a simple story in which more shared pieces always produce better multilingual transfer.

Separate representation difficulty from computation difficulty

A model fails to reverse a string. Is it unable to implement reversal, or did the tokenizer hide the characters inside larger units? A model fails long addition. Is the learned procedure wrong, or are input and output groups misaligned? A multilingual answer omits a clause. Was the relevant text never inside the context?

These questions do not excuse model errors. They improve diagnosis. Representation-induced difficulty arises when the chosen units lengthen, fragment or misalign the task. Computation-induced difficulty remains after the task is presented in a suitable coordinate system. The two can coexist.

The clean test is a counterfactual intervention. Hold the raw task, training data, model recipe and evaluation constant. Change segmentation or an input format that predictably changes segmentation. Inspect not only accuracy but error type, sample efficiency and length sensitivity.

Three counterfactuals answer three different questions

A runtime formatting intervention changes the token sequence presented to a frozen model. It asks whether the existing system can perform the task when boundaries are rearranged. This is the cheapest diagnosis for numbers, spelling, delimiters and context packing. A positive result identifies interface sensitivity in that deployed configuration.

A matched retraining intervention changes the tokenizer before learning. Two models start from random initialisation with the same raw corpus and model recipe. This asks whether a different segmentation improves sample efficiency, final quality or robustness. It is a stronger test of architectural choice, but “same raw corpus” does not mean the models receive the same number of prediction targets or the same compute. Those quantities must be reported rather than hidden under the word matched.

A transfer intervention changes the vocabulary of an already trained model and learns a bridge into its existing representation space. This asks how much capability is recoverable without repeating pre-training. Success measures adaptability and lock-in, not the quality of the replacement tokenizer in a fresh model.

These counterfactuals must not inherit one another’s conclusions. Helpful comma formatting does not prove that comma-based pre-training is optimal. Better retraining does not prove a safe vocabulary swap. Successful transfer does not show that the original tokenizer was harmless. Naming the intervention identifies which causal layer has actually moved.

Token-free does not mean world-free

One response is to remove the fixed subword vocabulary. CANINE operates on characters and uses downsampling before deeper processing.[6] ByT5 operates on UTF-8 bytes with a largely standard Transformer and reported competitive quality, stronger noise robustness and advantages on spelling-sensitive tasks.[7] Charformer learns latent subword blocks inside a byte-level model rather than freezing them before training.[14]

These systems demonstrate that a fixed external vocabulary is optional. They do not abolish representation choice. Bytes, Unicode code points, grapheme clusters and learned patches are different units. Byte sequences are usually longer. UTF-8 uses different byte counts across scripts. Downsampling and patching introduce new boundaries. The engineering question moves; it does not disappear.

Adaptive approaches try to learn boundaries with the model. MAGNET uses script-specific internal boundary predictors to reduce multilingual over-segmentation.[13] Another line of work tries to change a tokenizer after pre-training by learning new embeddings. Zero-Shot Tokenizer Transfer showed that detaching a model from its original tokenizer is possible in controlled settings, but the method exists precisely because model embeddings are normally bound to the vocabulary used during training.[15]

Recent evidence also suggests that early tokenizer choices can affect later language adaptation. Abagyan and colleagues reported that a tokenizer trained on a broader set of languages improved post-training adaptation, including gains for some languages unseen in both tokenizer and pre-training data.[16] That result supports treating tokenizer coverage as an option on future capability, not just present compression.

The string that looks unchanged

Humans see glyphs. Tokenizers receive code points or bytes after a normalisation pipeline. Two strings can render almost identically while differing in composed accents, homoglyphs, zero-width characters or direction controls. A copied identifier can therefore fragment into unfamiliar tokens without an obvious visual warning.

stateDiagram-v2
  [*] --> Claim
  state "Lossless does not mean<br/>learning-neutral" as Claim
  state "How subwords are invented" as Test
  state "The tokenisation exposure vector" as Revision
  Claim --> Test: expose mechanism
  Test --> Decision: evidence survives
  Test --> Revision: rival survives
  Revision --> Claim: narrow or rebuild
  Decision --> [*]

A tokenizer should be selected against the workload it will serve, not against one aggregate corpus statistic. The smallest useful process is a stress suite run before expensive pre-training and repeated whenever the corpus, languages or action domain changes.

The suite needs parallel semantic cases, domain-critical spans and operational edge cases. Include every language that matters, rather than only every script. Include long documents near the context boundary. Include numbers, dates, monetary values, identifiers, source code, punctuation, whitespace and noisy text. Mark spans that should remain local for the task, such as account suffixes or three-digit place-value groups.

Choose what the comparison conserves

Tokenizer experiments often claim to be fair while conserving different resources. Hold raw training bytes and the number of corpus passes fixed, and a fine tokenizer creates more prediction targets and usually more sequence positions. Hold token count fixed, and the fine tokenizer sees less raw text. Hold floating-point operations fixed, and vocabulary size, sequence length and architecture determine how that budget is spent. Hold total parameters fixed, and a larger vocabulary takes capacity away from the transformer body.

There is no resource-neutral comparison because tokenisation helps define the computational task. The conserved quantity must follow the question. To study sample efficiency from a fixed data reservoir, conserve raw examples and report tokens and compute as outcomes. To study quality under a fixed serving budget, conserve latency or metered tokens and report how much semantic content survives. To study architecture at equal investment, conserve an explicit compute and parameter envelope, then allow sequence length and vocabulary allocation to compete inside it.

A useful experiment ledger records at least raw bytes, grapheme clusters, produced tokens, prediction targets, vocabulary parameters, training operations, wall-clock time and peak memory. For multilingual studies, report these by language rather than only in aggregate. For operational tasks, add the fraction of critical spans that remain inside the context and the number of output steps required.

Fair comparison is not one setting. It is an explicit conservation choice. Without that choice, a tokenizer can appear better because it received more raw evidence, more optimisation events or more model capacity. The stress suite below is an intrinsic screen. It decides which candidates deserve the cost of matched experiments; it does not settle the conservation question for them.

An executable tokenizer stress suite

The lab below trains a small Unicode-aware BPE tokenizer in the browser. It is intentionally modest. Switch between an English-skewed corpus and a balanced multilingual corpus, edit the training text, change the number of merges and inspect how each stress case is segmented.

It reports the three components of the exposure vector. Budget tax is approximated by tokens per grapheme. Boundary tax is the share of lexical or numerical units split into several tokens. Estimation tax is the share of produced tokens that occur at most twice in the toy training corpus. Dashed token chips are low-frequency pieces.

A positive result is not “the tokenizer is good”. It is more specific. Perhaps the balanced corpus reduces the budget and rarity taxes for Hindi without materially worsening English. That permits a more expensive matched-model experiment. A negative result means the merge policy or corpus mix did not improve the relevant slice. Try a different vocabulary size, normalisation policy, language allocation or architecture.

The lab cannot establish downstream accuracy, fairness or production cost. Its corpus is tiny, its boundaries are lexical approximations, and its BPE implementation omits many production details. Its value is procedural: it makes the causal assumptions inspectable before training.

const exposure = {
  L: tokens.length / Math.max(1, graphemeCount(text)),
  B: splitTaskUnits / Math.max(1, taskUnits.length),
  R: tokens.filter(t => corpusFrequency[t] <= 2).length /
     Math.max(1, tokens.length)
};

A matched decision protocol

Intrinsic diagnostics should choose which tokenizers deserve training runs. They should not replace those runs. For a consequential model, train at least two small matched candidates before committing full scale. Keep raw data bytes, ordering, architecture, optimiser and evaluation fixed as far as the design allows, then report every resource that tokenisation itself changes.

The final row matters because tokenizers are hard to change after training. The vocabulary indexes the embedding and output matrices. Swapping it is not like changing a text parser. Research on tokenizer transfer and broad multilingual tokenizers shows promising escape routes, but the default remains architectural lock-in.[15][16]

Do not apply the vector mechanically where the task unit is unknown or deliberately fluid. Poetry, creative generation and open-domain dialogue may benefit from multiple plausible segmentations. In such settings, subword regularisation or learned boundaries may be more informative than a single deterministic split.

sequenceDiagram
  participant Q as Question
  participant M as Mechanism
  participant E as Evidence
  participant D as Decision
  Q->>M: The cancellation clause that vanished
  M->>E: Lossless does not mean learning-neutral
  E-->>M: compare rival explanations
  E->>D: Choose tokenisation before purchasing scale
  D-->>Q: revise scope or act