Latest Scaling Laws Research Papers

The newest Scaling Laws papers from across the field — arXiv, NeurIPS, CVPR, Nature, and more — refreshed daily and ranked by relevance. Distill AI tracks Scaling Laws so you don’t have to: get the standout work delivered to your inbox every morning, with 2-sentence summaries and the option to chat with any paper.

Recent papers

• Homebrewed SLMs: Autoregressive and Masked Diffusion Language Models Built at Home for Portuguese
  Pedro Carvalho Brom · arXiv · Oct 29, 2026

  Building a language model of one's own no longer requires a data centre. This tutorial trains one from scratch, tokenizer included, on one consumer GPU, and shows how to tell whether it is good for the tasks of a specialist domain. It propo…

• Scaling Laws for Tool-Use Agents via Controllable Environment Synthesis
  Andrew Szot, Samira Abnar, Omar Attia, Abbas Kazerouni et al. · arXiv · Oct 27, 2026

  We explore how to build performant large language model (LLM) tool-use agents that complete complex, multi-step tasks. Specifically, we study how data composition in synthetically generated tool-use environments affects the training of such…

• Kronecker Delta Rule for Long Context Linear Attention
  Anamaria-Roberta Hartl, Levente Zólyomi, Michael List, Günter Klambauer et al. · arXiv · Oct 19, 2026

  We propose KronLSTM, a linear attention architecture based on the Kronecker Delta Rule, a novel update rule for associative matrix memories. KronLSTM extends the xLSTM family with a memory-revision mechanism that corrects targeted associati…

• Decoupling Exploration from Optimization in RLVR
  Saif Punjwani, Micah Goldblum · arXiv · Oct 7, 2026

  Modern language models undergo reinforcement learning with verifiable rewards (RLVR) on top of already-trained checkpoints. A key promise of RLVR is the discovery of new reasoning strategies. In principle, a model can sample novel ideas abs…

• How to scale your HEP ML models: A recipe for robust architecture comparisons at scale
  Matthias Vigl, Nikita Pond, Jackson Barr, Alexander Froch et al. · arXiv · Oct 5, 2026

  Much of the recent progress in machine learning domains such as language models has come from scaling laws that predict performance as a function of training effort. In high-energy physics (HEP) similar behavior has now been observed. To ai…

• IdeaLens: Detecting AI Ideas in Long-form Writing
  Rishanth Rajendhran, Minjoon Choi, Jenna Russell, Ramya Namuduri et al. · arXiv · Oct 5, 2026

  While modern AI detectors identify who wrote the words, emerging policies on AI use increasingly hinge on a different question: who came up with the ideas? We introduce IdeaLens, a detector that identifies whether a document's ideas came fr…

• Scaling Laws for Looped Mixture of Experts
  Yanbei Chen, Anirudh Goyal, Raghuraman Krishnamoorthi · arXiv · Sep 30, 2026

  Looped transformers and Mixture-of-Experts (MoE) offer complementary routes to efficient scaling: recurrence increases computational depth at fixed parameters, while MoE sparsity expands total capacity at fixed active compute. Yet existing …

• How Much Is an AI Token Worth? Scaling Laws for Wild AI-Generated Web Text
  Jenna Russell, Ben Glickenhaus, Katherine Thai, John Wieting et al. · arXiv · Sep 30, 2026

  Web text makes up the majority of pretraining data and is increasingly AI-generated. After applying FineWeb quality filtering, we find that 27.5% of tokens from June 2026 web data are labeled as AI-generated by Pangram, rising to 31.1% by A…

• From Spectra to Joint Schedules in LLM Pre-training: 3+3(+2) Scaling-Law Regimes
  Yichen Wang, Fanghui Liu, Yudong Chen · arXiv · Sep 30, 2026

  Power-law learning curves are often treated as fixed properties of a model and its data, although learning-rate and batch-size schedules can change the observed loss. We study this dependence in noisy online SGD with linear random features.…

• When do data mixtures improve scaling laws? Insights from high-dimensional regression
  Diyuan Wu, Lehan Chen, Theodor Misiakiewicz, Marco Mondelli · arXiv · Sep 29, 2026

  Modern machine learning systems are trained on mixtures of data from different domains, and choosing the right mixture can substantially improve downstream performance. Despite an extensive literature on data mixing and reweighting, existin…

• ScAn-Bench: Evaluating Scaling Analysis Methodology
  Artin Sermaxhaj, Nastaran Alipour, Donat Sinani, Johannes Hog et al. · arXiv · Sep 28, 2026

  Recent progress in machine learning is driven by large-scale foundation models, where scaling laws and finding optimal scaling prescriptions for architecture, data, and hyperparameters are key in advancing the state-of-the-art. Therefore, i…

• MeqMuon: Matrix-Equilibrating Muon for LLM Pretraining
  Chang-Wei Shi, Xu Wang, Wu-Jun Li · arXiv · Sep 28, 2026

  The success of large language models (LLMs) has been accompanied by continued growth in model size and pretraining costs. Muon offers high accuracy and training efficiency in LLM pretraining. Recent work introduces row-wise normalization in…

• hyperbolix: Hyperbolic Deep Learning in JAX
  Timo Klein, Thomas Lang, Yllka Velaj, Sebastian Tschiatschek · arXiv · Sep 23, 2026

  We present hyperbolix, an open-source library for hyperbolic deep learning in JAX, built on Flax NNX. To our knowledge, it is the first comprehensive, general-purpose hyperbolic deep learning library in JAX. It includes six manifolds with a…

• Recursive self-improvement of AI research agents
  Dhruv Srikanth, Bingchen Zhao, Dixing Xu, Yuxiang Wu et al. · arXiv · Sep 22, 2026

  AI agents are beginning to automate research and development across the AI stack, from improving training efficiency to optimizing inference. A natural next step is to improve the research efficiency of the agents themselves. When an AI res…

• How Model Growth, Recursion, and Boundary Operators Influence Scaling Exponents
  Zixi Chen, Akshay Vegesna, Samip Dahal, Andrew Gordon Wilson · arXiv · Sep 16, 2026

  Scaling laws predict how loss decreases with increases in computation. We show, contrary to conventional wisdom, that architectural interventions can modify scaling exponents in pre-training, leading to exponential improvements in performan…

• Infinite-Parameter LLMs: Generating and Adapting Weights from Live Data
  Jinli Hu, Ross M. Clarke, Yichuan Zhang, José Miguel Hernández-Lobato · arXiv · Sep 16, 2026

  The scaling laws hold that a language model grows more capable with more parameters and more training data, and Mixture-of-Experts (MoE) architectures have ridden these laws to remarkable results, activating only a fraction of an enormous s…

• LoRA-TSD: Tangent-Space Spectral Descent for LoRA via Muon-Style Updates
  Dmitrii Andriianov, Andrey Veprikov, Aleksandr Beznosikov · arXiv · Sep 2, 2026

  Low-rank adaptation (LoRA) is the standard way to fine-tune large models, yet when its two factors are trained independently, the update ignores the geometry of the low-rank weight change it induces. We introduce LoRA-TSD, an optimizer that…

• Efficiently Estimating Optimal Hyperparameter Scaling Laws through Power-Law Entropy Search
  Zhiliang Chen, Sebastian Ament, David Eriksson, Maximilian Balandat et al. · arXiv · Sep 1, 2026

  Optimal hyperparameter scaling laws describe how the best hyperparameters for large language model (LLM) training change with model and data scale, enabling practitioners to predict optimal configurations at production scales without expens…

• Beyond Scaling Law: A Data-Efficient Distillation Framework for Reasoning
  Xiaojun Wu, Xiaoguang Jiang, Huiyang Li, Jucai Zhai et al. · arXiv.org · Aug 13, 2025

  Large language models (LLMs) demonstrate remarkable reasoning capabilities in tasks such as algorithmic coding and mathematical problem-solving. Recent methods have improved reasoning through expanded corpus and multistage training combinin…

• Scaling Law for Quantization-Aware Training
  Meng-Zhao Chen, Chaoyi Zhang, Jing Liu, Yutao Zeng et al. · arXiv.org · May 20, 2025

  Large language models (LLMs) demand substantial computational and memory resources, creating deployment challenges. Quantization-aware training (QAT) addresses these challenges by reducing model precision while maintaining performance. Howe…

• Parallel Scaling Law for Language Models
  Mou-Xiang Chen, Binyuan Hui, Zeyu Cui, Jia-Xin Yang et al. · Neural Information Processing Systems · May 15, 2025

  It is commonly believed that scaling language models should commit a significant space or time cost, by increasing the parameters (parameter scaling) or output tokens (inference-time scaling). We introduce the third and more inference-effic…

• Agent RL Scaling Law: Agent RL with Spontaneous Code Execution for Mathematical Problem Solving
  Xin-Ji Mai, Haotian Xu, Zhong-Zhi Li, W. Xing et al. · Semantic Scholar · May 12, 2025

  Large Language Models (LLMs) often struggle with mathematical reasoning tasks requiring precise, verifiable computation. While Reinforcement Learning (RL) from outcome-based rewards enhances text-based reasoning, understanding how agents au…

• A Survey of Slow Thinking-based Reasoning LLMs using Reinforced Learning and Inference-time Scaling Law
  Qianjun Pan, Wen-Kai Ji, Yuyang Ding, Junsong Li et al. · arXiv.org · May 5, 2025

  This survey explores recent advancements in reasoning large language models (LLMs) designed to mimic"slow thinking"- a reasoning process inspired by human cognition, as described in Kahneman's Thinking, Fast and Slow. These models, like Ope…

• a1: Steep Test-time Scaling Law via Environment Augmented Generation
  Lingrui Mei, Sheng-Hua Liu, Yi-Wei Wang, Bao-Long Bi et al. · Annual Meeting of the Association for Computational Linguistics · Apr 20, 2025

  Large Language Models (LLMs) have made remarkable breakthroughs in reasoning, yet continue to struggle with hallucinations, logical errors, and inability to self-correct during complex multi-step tasks. Current approaches like chain-of-thou…

• Unsourced Random Access in MIMO Quasi-Static Rayleigh Fading Channels: Finite Blocklength and Scaling Law Analyses
  Jun-Yuan Gao, Yongpeng Wu, Giuseppe Caire, Wei Yang et al. · IEEE Transactions on Information Theory · Mar 21, 2025

  This paper considers the unsourced random access (URA) problem with a random and unknown number of active users in multiple-input multiple-output (MIMO) quasi-static Rayleigh fading channels. We derive non-asymptotic achievability bounds on…

• Predictable Scale: Part I, Step Law -- Optimal Hyperparameter Scaling Law in Large Language Model Pretraining
  Houyi Li, Wen-Zheng Zheng, Qiu-Feng Wang, Han-Shan Zhang et al. · Semantic Scholar · Mar 6, 2025

  The impressive capabilities of Large Language Models (LLMs) across diverse tasks are now well established, yet their effective deployment necessitates careful hyperparameter optimization. Although existing methods have explored the influenc…

• L2M: Mutual Information Scaling Law for Long-Context Language Modeling
  Zhuo Chen, Oriol Mayn'e i Comas, Zhuo-Tao Jin, Di Luo et al. · Neural Information Processing Systems · Mar 6, 2025

  We present a universal theoretical framework for understanding long-context language modeling based on a bipartite mutual information scaling law that we rigorously verify in natural language. We demonstrate that bipartite mutual informatio…

• Unlocking Scaling Law in Industrial Recommendation Systems with a Three-step Paradigm based Large User Model
  Bencheng Yan, Shilei Liu, Zhiyuan Zeng, Zihao Wang et al. · Web Search and Data Mining · Feb 12, 2025

  Recent advancements in autoregressive Large Language Models (LLMs) have achieved remarkable progress, largely driven by their scalability—commonly formalized as the scaling law. Inspired by these successes, there has been growing interest i…

• Scaling Law for Intrinsic Fracture Energy of Diverse Stretchable Networks
  Chase M. Hartquist, Shu Wang, Qiaodong Cui, W. Matusik et al. · Physical Review X · Jan 8, 2025

  Networks of interconnected materials permeate throughout nature, biology, and technology due to exceptional mechanical performance. Despite the importance of failure resistance in network design and utility, no existing physical model effec…

• Predictable Scale: Part I - Optimal Hyperparameter Scaling Law in Large Language Model Pretraining
  Houyi Li, Wen-Zheng Zheng, Jing-Cheng Hu, Qiu-Feng Wang et al. · arXiv.org · Jan 1, 2025