Junoh Kang

Back

Modern LLMs through the lens of the bitter lesson#

general methods that leverage computation are ultimately the most effective, and by a large margin.

— Richard Sutton, The bitter lesson (2019) ↗

Modern LLMs illustrate the bitter lesson through training methods that obtain learning signals at scale:

  • Next-token prediction turns existing text into training examples: the next token is already there to serve as the target. This makes large text corpora usable for learning without separately labeling each example, as demonstrated by GPT-2 ↗.
  • Reinforcement learning with verifiable rewards (RLVR) uses automatically checkable outcomes, such as correct answers or passing tests, to reward generated solutions. DeepSeek-R1-Zero ↗ showed how this can improve a pretrained model’s reasoning without requiring human-written step-by-step solutions for its RL training.

However, a tokenizer still determines the units these models learn from. Chunking bytes into larger tokens shortens sequences and reduces the computational cost of standard Transformers. These merge rules remain fixed during language-model training: the model learns patterns within and across tokens, but cannot change how the text is divided.

The example below shows how GPT-2’s tokenizer represents hello world as two tokens instead of eleven byte positions.

GPT-2 encodes hello world as two tokens; · marks a space. Source: OpenAI.
How byte pair encoding (BPE) tokenizes text

Byte-level BPE builds a vocabulary by repeatedly merging the most frequent adjacent pair of units in a training corpus, starting from individual byte values. These merge rules are learned before language-model training and remain fixed throughout it. The interactive example below steps through how such merge rules are learned on a toy corpus, one merge per click.

BPE merges on a toy character corpus from Sennrich et al.[2]

This raises a question through the lens of the bitter lesson: should the model also learn how to form the units it processes?


Hierarchical networks (H-Nets)#

Cartesia’s answer is yes. In its view, intelligence means learning from raw data, and that includes learning how to group the data into useful units. This is similar to how people learn: as psychologist George A. Miller ↗ illustrated with Morse code, a learner first hears individual dots and dashes, and later recognizes whole letters and words.

H-Nets embody this perspective through three key features:

  • End-to-end learning: The model learns how to form units jointly with the prediction objective, rather than in a separate preprocessing step.
  • Learning from bytes: The model reads raw bytes directly, so its learning signal comes from the data itself rather than through a fixed tokenizer.
  • Hierarchical abstraction: The model can recursively chunk these units to build representations at multiple levels of abstraction.

Observed chunking behaviors#

The examples below come from Cartesia’s blog post on H-Nets.

In-context tokenization#

Unlike BPE, H-Net can adapt its chunk boundaries based on the preceding context. Thus, the same word can be chunked differently depending on what came before it.

Chunking repeated formats. Source: Cartesia, Figure 11.
Chunking recurring four-digit blocks. Source: Cartesia, Figure 11.

Semantic chunking#

H-Net tends to chunk input into semantically meaningful units. These units can cross boundaries between words, whitespace, and punctuation, as illustrated by the Python and Korean examples below.

Chunking Python code. Source: Cartesia, Figure 10b.
Chunking Korean text (each character is 3 bytes). Source: Cartesia, Figure 10a.

References#

  1. Cartesia, From tokens to bytes I: Scaling hierarchical networks ↗

  2. Rico Sennrich, Barry Haddow, and Alexandra Birch, Neural Machine Translation of Rare Words with Subword Units (ACL 2016) ↗

Search