Submitted to arXiv on September 1, 2026 (paper 2609.01343) by Shaowen Wang, Ge Zhang, Kairong Luo, Yuhao Wu, Shaofan Liu, Jiaheng Liu, Wenhao Huang, Shen Yan and Jian Li. The paper opens with a methodological complaint that applies to most published results on looped Transformers: they compare architectures at fixed model size, which conflates a genuine architectural advantage with simply spending more FLOPs per token.
SMELT stands for Sparse MoE Transformer, middle layers Loop Twice, and the recipe is exactly that - iterate the middle half of the layer stack twice while holding three budgets matched against an unlooped baseline: per-token FLOPs, total non-embedding parameters, and KV cache. The authors scale the recipe across four model sizes up to 54 billion non-embedding parameters and fit a separate Chinchilla-style scaling law to each architecture rather than reading off a single comparison point.
Under that matched-budget discipline, the SMELT loss falls faster with compute, saving 6.8 to 18.0 percent of training FLOPs along the compute-optimal frontier. The gain does not stop at validation loss: it transfers to downstream benchmarks by more than the loss curve alone predicts, is largest on code, and grows with both sample length and the number of in-context examples. The mechanistic analysis offers a candidate explanation - the second visit reduces the attention sink and redirects attention mass toward content-relevant tokens, an inductive bias the authors suggest underlies the gains.
The reason this matters commercially is the budget matching. Depth reuse has long been an attractive idea because weights are shared, but “deeper is better” results are cheap when the comparison quietly buys more compute. Holding FLOPs, parameters and KV cache constant and still recovering a double-digit percentage of training compute is a real efficiency claim, and the fact that the advantage widens with longer samples and more in-context examples points the benefit at exactly the long-context, many-shot workloads that dominate agentic use today.