Junoh Kang

Back

LH-Net and H-Net#

The previous post explored the motivation behind H-Net and what it can do. In this post, we turn to Dynamic Chunking for End-to-End Hierarchical Sequence Modeling[1] to understand how it works.

Following Cartesia’s distinction ↗, we call the original model LH-Net, where “L” stands for “lazy” or “lookahead.” Although Cartesia has not disclosed the full architectural details of the updated H-Net, it highlights a key difference: LH-Net draws a boundary only after seeing the start of the next chunk. This gives it more context for chunking, but, according to Cartesia, hurts next-byte prediction. While LH-Net differs from the H-Net discussed in the previous post, studying it provides a technical foundation for understanding hierarchical sequence modeling.


A lens for understanding sequence modeling#

We can view tokenization and learned compression methods through a shared frame: encode raw inputs into a shorter sequence, process it with a main network, and decode the result into raw-level predictions. They differ in how they construct these compressed representations—and which parts of this process are learned jointly with the main network. Keeping this picture in mind, we will first look at earlier approaches, then examine how LH-Net learns to form and use these representations.

Raw representation passes through an encoder, a compressed representation, a main network, another compressed representation, and a decoder to produce raw-level predictions.
A conceptual view of sequence modeling. Encoder and decoder denote transformations, not necessarily learned networks; compression refers to sequence length.

Viewed through this lens, BPE and the prior approaches discussed in the paper differ in how they construct compressed representations and choose their boundaries. The limitations below follow the paper’s discussion.

ApproachHow compression worksLimitation
BPE[3]Merge bytes into vocabulary tokens.Segmentation stays fixed during language-model training.
Direct byte modeling
(MambaByte[4])
—Processing the full byte sequence remains costly.
Fixed-size chunking
(Hourglass[5], MegaByte[6])
Encode each group of bytes into a single representation.Boundaries do not adapt to content.
Heuristic chunking
(SpaceByte[7], BLT[8])
Form chunks using delimiter or entropy rules.The boundary-selection rule may not align with the main objective.
End-to-end chunking
(DPT’s Gumbel variant[9])
Learn chunk boundaries jointly with next-token prediction.Training instability limits scaling and deeper hierarchies.

Note. Entropy rules place boundaries where uncertainty about the next byte is high or rises sharply.

LH-Net follows the same end-to-end direction as the last approach, with architectural and training changes designed to make learned chunking stable and scalable across model sizes and hierarchy levels.


LH-Net architecture overview#

LH-Net follows an encoder → chunking layer → main network → dechunking layer → decoder structure. During autoregressive generation, the encoder and decoder run at every byte, while the main network advances only when a new chunk boundary is selected.

Crucially, the main network can itself be another LH-Net, allowing the same structure to repeat across multiple levels of compression. The paper’s Figure 1 shows a two-stage example. For simplicity, this post focuses on a single-stage architecture with one encoder, one main network, and one decoder.

Figure 1: two-stage H-Net architecture and the routing, downsampling, smoothing, and upsampling components.
Two-stage architecture (left) and dynamic chunking components (right). Source: Hwang et al., Figure 1.[1]

The architecture and its learning objective can be understood through the following principles:

  • Keep the encoder and decoder efficient. Since the encoder and decoder process long, uncompressed sequences, they use efficient state space models: four Mamba-2 layers in each encoder and decoder.
  • Preserve fine-grained information for the decoder. The encoder passes fine-grained information through residual connections.
  • Concentrate compute and model capacity in the main network. The main network operates on a shorter sequence of richer representations. The encoder builds these representations, and the chunking layer reduces the sequence length.
  • Use next-byte prediction as the main learning signal, including for chunking. Learn chunk boundaries that support next-byte prediction, without explicit boundary supervision.

Dynamic chunking#

In LH-Net, the encoder preserves sequence length so that the model can decide where to place chunk boundaries before compression. There are no ground-truth boundary labels; instead, boundary decisions are learned to improve next-byte prediction.

The chunking layer uses a routing module to determine boundaries and a downsampler to construct compressed representations. The dechunking layer uses a smoothing module to support stable boundary learning and an upsampler to restore the original sequence length. The decoder then processes these full-length representations to predict the next byte at each position.

Chunking layer#

For a single hierarchy level, the encoder first produces a representation at every input position, then the chunking layer returns a shorter sequence and routing scores:

Routing module#

Routing module, panel (a) of Figure 1.

The routing module builds on the intuition that meaningful boundaries tend to occur where context or meaning shifts. Each encoder output represents in the context of . The router compares learned projections of adjacent representations to detect these shifts:

Here, and are learned projections of the encoder output . The binary indicator determines whether the current position starts a new chunk. The first position is always selected, with .

Downsampler#

Downsampler, panel (b) of Figure 1.

The downsampler constructs a compressed sequence using the boundaries selected by the routing module. LH-Net selects encoder outputs at boundary positions, rather than pooling the representations within each chunk. The same selection is applied to the routing scores: retaining only the boundary positions from produces , which is passed to the smoothing module during dechunking.

Dechunking layer#

The dechunking layer takes the main network output and the routing scores , restores the sequence length, and adds the encoder residual before decoding: . The dechunking components are designed to provide a learning signal for chunk boundary decisions. This requires gradients with respect to the routing scores .

Smoothing module#

Smoothing module, panel (c) of Figure 1.

The smoothing module blends the main network’s output with the previous smoothed representation using the boundary probability , and passes the result to the upsampler.

Here, indexes the compressed sequence rather than the original byte sequence.

The smoothing module serves three roles:

  • Boundary learning: provide a differentiable path from the prediction loss to the routing scores.
  • Error correction: soften the effect of uncertain boundaries by incorporating information from preceding chunks.
  • Training stability: reduce early commitment to suboptimal chunking patterns.

Upsampler#

Upsampler, panel (d) of Figure 1.

The upsampler restores the original sequence length by repeating each smoothed chunk representation until the next boundary:

It then weights the result by the router’s confidence in the current decision, , passed through a straight-through estimator :

Here, denotes stop-gradient. The STE sets the multiplier to one in the forward pass, so , but keeps a gradient path to . The contribution through this multiplier is:

The inner product measures how the loss changes as the scale of changes, evaluated at the forward-pass scale of one. When strengthening would lower the loss, this gradient reinforces the current boundary decision; otherwise it weakens it.


Training objective#

LH-Net’s objective is next-byte prediction with compression regularization. The prediction loss provides the main learning signal for the entire model, including chunking, while the ratio loss encourages each stage to meet its target compression ratio.

for and . Under , the ratio loss is minimized at ; when , it can fall below this value.

Interpreting the ratio loss as a chi-square divergence

With target boundary rate , the ratio loss can be rewritten as

If were used in place of , this would become

where the second term is the chi-square divergence of the actual boundary rate from the target rate. The ratio loss replaces one factor of with because is computed from hard boundary decisions and has no gradient, whereas is differentiable with respect to the routing scores. The same structure appears in the load-balancing loss of mixture-of-experts models.

References#

  1. Sukjun Hwang, Brandon Wang, and Albert Gu, Dynamic Chunking for End-to-End Hierarchical Sequence Modeling (2025) ↗

  2. Cartesia Research, From tokens to bytes I: Scaling hierarchical networks (2026) ↗

  3. Rico Sennrich, Barry Haddow, and Alexandra Birch, Neural Machine Translation of Rare Words with Subword Units (ACL 2016) ↗

  4. Junxiong Wang et al., MambaByte: Token-free Selective State Space Model (2024) ↗

  5. Piotr Nawrot et al., Hierarchical Transformers Are More Efficient Language Models (Findings of NAACL 2022) ↗

  6. Lili Yu et al., MEGABYTE: Predicting Million-byte Sequences with Multiscale Transformers (2023) ↗

  7. Kevin Slagle, SpaceByte: Towards Deleting Tokenization from Large Language Modeling (2024) ↗

  8. Artidoro Pagnoni et al., Byte Latent Transformer: Patches Scale Better Than Tokens (2024) ↗

  9. Piotr Nawrot et al., Efficient Transformers with Dynamic Token Pooling (ACL 2023) ↗

Search