Latest Foundation Models Research Papers

The newest Foundation Models papers from across the field — arXiv, NeurIPS, CVPR, Nature, and more — refreshed daily and ranked by relevance. Distill AI tracks Foundation Models so you don’t have to: get the standout work delivered to your inbox every morning, with 2-sentence summaries and the option to chat with any paper.

Recent papers

• Rethinking Neural Thickets Without Weight Perturbations
  Nicolas Dufour, Elouan Gardès, Amil Dravid, Patrick Pérez et al. · arXiv · Oct 29, 2026

  The Neural Thickets hypothesis posits that pretrained language models lie in dense neighborhoods of task-specific experts in weight space, which random Gaussian perturbations can uncover. We reproduce the reported gains, but show that they …

• Training Foundation Models with a Single Algorithm
  Yuda Song, Alvin Zhang, Fahim Tajwar, Daman Arora et al. · arXiv · Oct 29, 2026

  Foundation models are typically trained in stages: pretraining, midtraining, supervised fine-tuning (SFT), and reinforcement learning (RL), with changes in both data and learning algorithms. We show that this algorithmic separation is not f…

• TRACER, a time-to-event EHR foundation model for disease detection across health systems
  Mara Pleasure, Ekaterina Redekop, Vedrana Ivezic, Rushikesh Kulkarni et al. · arXiv · Oct 29, 2026

  Foundation models trained on longitudinal electronic health records (EHRs) can predict clinical diagnoses years in the future, making them a potential tool for disease screening. We developed TRACER (Temporal Risk Assessment with Clinical E…

• Homebrewed SLMs: Autoregressive and Masked Diffusion Language Models Built at Home for Portuguese
  Pedro Carvalho Brom · arXiv · Oct 29, 2026

  Building a language model of one's own no longer requires a data centre. This tutorial trains one from scratch, tokenizer included, on one consumer GPU, and shows how to tell whether it is good for the tasks of a specialist domain. It propo…

• A Pretrained Model for Bayesian Filtering in Heterogeneous Dynamical Systems
  Ke Fan, Wen Wu, Haitao Su, Yilin Wang et al. · arXiv · Oct 29, 2026

  Filtering has not yet had its foundation-model moment: state estimation is still dominated either by online Bayesian filtering algorithms that require known dy- namics as inputs, or by learning-based neural filters trained from scratch for …

• Continuous-Action World Models: Learning Cross-Embodiment Spatiotemporal Dynamics Priors
  Liangyu Li, Qingwen Liu · arXiv · Oct 29, 2026

  We introduce Continuous-Action World Models (CoAWM), a framework for learning cross-embodiment spatiotemporal dynamics priors that support robot action generation across embodiments and execution frequencies. At its core is a shared latent …

• MuonGS: Pretraining LLMs with Relaxed Weight-Norm Control
  Athanasios Glentis, Dawei Li, Mingyi Hong · arXiv · Oct 27, 2026

  Weight decay and explicit weight-norm constraints improve optimizer performance in language model pretraining, yet a unified understanding of why such control helps and how much is needed remains incomplete. We analyze norm control in Muon …

• A note on data-efficient finetuning of vision foundation models
  Niclas Hergenröther, Antonio Orvieto · arXiv · Oct 27, 2026

  Adapting foundation models to a new task within or beyond the training domain is the de facto standard for building specialized machine learning models. Recently, Visual Prompting (VP) has emerged as a promising technique for efficient know…

• LinMix: Data Mixing as Linear Domain Attribution
  Hubert Baniecki, Przemyslaw Biecek, Martin Pawelczyk · arXiv · Oct 27, 2026

  Language model performance depends strongly on the mixture of domains in the pretraining data corpus. Popular data mixing methods select these domain weights by training many small proxy models on random mixes and then fitting a regression …

• Work Agents: Systems, Capabilities, and Optimization
  Zishan Xu, Enjun Du, Mingqi Gao, Cheng Yang et al. · arXiv · Oct 26, 2026

  As foundation models evolve from information generation to digital work execution, agents are increasingly capable of completing complex tasks across tools, applications, and workflows. However, existing research lacks a unified perspective…

• Pretraining Knows What, Simulation Teaches How: Compositional Generalization in a Sim-Trained Robot Agent
  Benjamin Pikus, Daniel Saunders, Mircea Mironenco, Salman Mohammadi et al. · arXiv · Oct 26, 2026

  A robot policy needs to understand what a task is asking for in a scene, and how to move the robot to accomplish that task. To achieve these two skills, we introduce Zero-real Agent Policy (ZAP), a recipe that teaches a small off-the-shelf …

• Sustainability of TSFM Deployment and Pretraining: From Carbon Footprint Analysis to Mitigation via Pruning
  Cécile Luc, Nathanaël Lemonnier, Ievgen Redko · arXiv · Oct 23, 2026

  Time Series Foundation Models (TSFMs) offer strong zero-shot forecasting capabilities, but their high pretraining cost raises a practical sustainability question: does this upfront investment pay off over a full deployment lifecycle, compar…

• On-Policy Self-Training: Data-Free Self-Improvement in Large Language Models
  Leon Sick, Lukas Hoyer, Yannick Strümpler, Christian Reisswig et al. · arXiv · Oct 23, 2026

  On-policy self-distillation enables post-training foundation models without relying on stronger external teachers. However, existing approaches rely on ground-truth question-answer pairs, human-curated problem sets, or other supervised sign…

• Sensorimotor Alignment: A Platonic Proxy for Embodied Policy Success
  Yunfei Xie, Howard H. Qian, Shiyi Lan, Kaiyu Hang et al. · arXiv · Oct 22, 2026

  Robotic foundation models are typically evaluated through costly closed-loop rollouts, which motivates cheap offline proxies for model capability. The Platonic Representation Hypothesis (PRH) suggests that as representations improve, differ…

• Piano2Pop: Expanding Piano Demos into Pop Productions Without Aligned Supervision
  Yun-Chen Cheng, Chih-Pin Tan, Tzu-Hung Huang, Yi-Hsuan Yang · arXiv · Oct 22, 2026

  We introduce Piano2Pop, a music-expansion task that transforms piano MIDI into instrumental pop productions. Enabled by recent advances in music generation and polyphonic music transcription, our pseudo-supervised approach conditions a pret…

• RheOPT: Few-Shot Non-Newtonian Flow Prediction via Newtonian Pretraining of RheOFormer
  Maedeh Saberi, Amir Barati Farimani, Safa Jamali · arXiv · Oct 22, 2026

  Despite advances in numerical methods, non-Newtonian flow simulations remain computationally expensive because velocity and stress must be advanced together through rate- and history-dependent constitutive dynamics. Neural-operator surrogat…

• RangeNav: Task-Aligned LiDAR Range-Image Pretraining for Robot Navigation
  Samuel Foo Enze, Manthan Patel, Fan Yang, Linus Kramer et al. · arXiv · Oct 21, 2026

  Autonomous mobile robots operating in dense crowds and cluttered sites require $360^\circ$ perception to react to surrounding hazards. While panoramic LiDAR offers full-surround coverage, learning representations within reinforcement learni…

• Where Do LLMs Learn to Recommend AI?
  Benaya Trabelsi, Jonathan Shaki, Sarit Kraus · arXiv · Oct 21, 2026

  Asked what to study, which career to pursue, or what product to build, large language models (LLMs) often recommend AI-related options. We ask where this tendency comes from by comparing models before and after fine-tuning and by varying ho…

• Turning TS2Vec into a Foundation Model for Time Series Classification
  Vasilii Feofanov, Zhenghan Tai, Andrei Ermakov, Irina Rish · arXiv · Oct 20, 2026

  TS2Vec is one of the strongest self-supervised methods for time series classification, introduced in 2022. It is designed to be trained anew on every dataset: self-supervised pre-training followed by linear probing. Since then, the field ha…

• Continual Concept Personalization in Diffusion Models via Residual-Space Adaptation
  Yuzhi Liu, Enis Simsar, Giulia Lanzillotta, Thomas Hofmann · arXiv · Oct 20, 2026

  Text-to-image personalization aims to equip pretrained diffusion models with the ability to learn user-specific concepts from only a few reference images. Existing methods, however, typically train each concept independently or assume that …

• GraphCast-Mamba: Learning Non-Markovian Corrections for Weather Forecasting
  Ilya Vilkoviskiy, Liang-Hong Mo, Suozhi Huang, Dmitry A. Abanin · arXiv · Oct 20, 2026

  Unresolved atmospheric processes can introduce history dependence beyond the short input windows used by many weather forecasting models. We introduce GraphCast--Mamba, which augments a frozen, pretrained $1^\circ$ GraphCast backbone with a…

• Text-to-Image Generative Modeling Without Language Pretraining
  Keshigeyan Chandrasegaran, Yue Zhao, Armin W. Thomas, Ehsan Adeli et al. · arXiv · Oct 19, 2026

  Modern text-to-image (T2I) systems use internet-scale language pretraining for text-conditioned image generation. We ask whether language pretraining is necessary for strong T2I generation. We introduce Halcyon, a family of early-fusion mul…

• Kronecker Delta Rule for Long Context Linear Attention
  Anamaria-Roberta Hartl, Levente Zólyomi, Michael List, Günter Klambauer et al. · arXiv · Oct 19, 2026

  We propose KronLSTM, a linear attention architecture based on the Kronecker Delta Rule, a novel update rule for associative matrix memories. KronLSTM extends the xLSTM family with a memory-revision mechanism that corrects targeted associati…

• MemT-1.0: Towards Memory-Native Evolution
  Zhiyu Li, Zeyu Zhang, Linyun Liu, Yiwei Bu et al. · arXiv · Oct 19, 2026

  Memory-native evolution requires making memory an intrinsic capability of foundation models, enabling them to continuously accumulate and utilize information. To this end, we introduce MemT-1.0, a new architecture of memory foundation model…

• Scaling Parameter-Efficient Fine-Tuning for Brain Encoding
  Zhuofan Josh Ying, Guanglong Sun, Nolan Ho, Kieran Liu et al. · arXiv · Oct 19, 2026

  Brain encoding models aim to predict neural responses to stimuli. The most widely used approach fits readouts from frozen pretrained backbones. Using neural data to train large models from scratch or fully fine-tune them after task-pretrain…

• Teacher Supervision, Fixed Representations and Calibration: Exact Risk Calculations for Candidate-Scoring Decision Models
  Agus Sudjianto, Denis Burakov · arXiv · Oct 19, 2026

  Candidate-scoring decision models return a probability distribution over a presented set of answers. Public documentation of one such system, Jev, and of an open counterpart, JevAny, describes three training ingredients that determine what …

• Simulation Priors for Rapid Adaptation in Model-Based RL for Autonomous Racing
  Adrian Remonda, Jiajun Xi, Nicklas Hansen, Marcus Greiff et al. · arXiv · Oct 19, 2026

  Reinforcement learning enables learning high-performance control policies, but its reliance on extensive interaction limits deployment on physical systems where data collection is costly, and failures are unsafe. We introduce Prior-Informed…

• recAMPLIFY: Large-Scale Pretraining of Recursive Protein Language Models
  Saurav Jha, Qianli Liao, Robert Vernon, Sarath Chandar et al. · arXiv · Oct 19, 2026

  Protein language models (pLMs) are powerful tools for protein annotation and mutation-effect prediction, but progress has largely depended on scaling model size and data, making them expensive and prone to memorization. Test-time scaling of…

• PathFM: Fast Generative Forecasting of Multivariate Trajectories
  Jiachen Yao, Xiyuan Zhang, Annan Yu, Boran Han et al. · arXiv · Oct 15, 2026

  Time series foundation models are becoming increasingly popular for general-purpose forecasting. However, most leading models predict marginal quantiles, leaving the dependence structure of future trajectories unspecified. This distinction …

• GeoState: Learning View-Decoupled Geometry States for 3D World Modeling
  Hongchang Liu, Jian Liu, Yujuan Ding, Liang Peng et al. · arXiv · Oct 15, 2026

  Recent 3D world models increasingly use features from geometry foundation models (GFMs) as predictive world states, benefiting from their rich geometric priors. However, the view dependence of these features entangles scene dynamics with ca…