Learning to tokenize (3): Experimental results of LH-Net
The previous post examined how LH-Net learns chunk boundaries. This post turns to the experimental design and training practices in Dynamic Chunking for End-to-End Hierarchical Sequence Modeling[1], asking how the comparisons isolate the benefits of hierarchy, learned chunking, and training choices.
Experimental design#
The experiments examine four design choices in LH-Net: (i) network architecture, (ii) training recipe, (iii) dynamic chunking, and (iv) hierarchy depth. To compare models that process BPE tokens, individual bytes, or chunks of different lengths, the authors match both bytes per batch and training FLOPs per byte.

- Architecture
SpaceByteuses Transformer layers in the encoder, main network, and decoder.SpaceByte++uses Mamba-2 in the encoder and decoder and a deeper Transformer in the main network.
Training recipe (Section 2.3)
- Norm balance. RMSNorm at the end of each encoder, decoder, and main network keeps the deeper network’s output from overwhelming the encoder’s residual signal when the two are added.
- Separation of two streams. A linear projection on the encoder-to-decoder skip connection lets the encoder features adapt to the decoder while preserving the direct path into the next stage.
- Learning rate modulation. Outer stages process more positions per batch and use higher learning rates, with the multiplier accounting for both sequence length and hidden width.
SpaceByte++uses the baseline training recipe.LH-Net (space)adds the training changes from Section 2.3.
-
Chunking boundaries
LH-Net (pool)starts a new chunk every six bytes.LH-Net (space)uses space-like delimiters.LH-Net (1-stage)learns where to split.
-
Hierarchy depth
LH-Net (1-stage)learns to group bytes into chunks.LH-Net (2-stage)also learns to group those chunks into larger units.
Results#
We focus on the XL models, which share the training compute budget of the 1.3B-parameter token-level Transformer.
LH-Net overtakes the token-level Transformer as training progresses#
On English text, two-stage LH-Net overtakes the BPE Transformer after about 30B training bytes.
On Chinese text, two-stage LH-Net overtakes the BPE Transformer after about 25B training bytes. On code, it leads throughout the interval shown.
Two-stage LH-Net achieves the highest average accuracy on clean and perturbed inputs#
For examples of how LH-Net groups text into chunks, see the examples in Part 1.