Latest Benchmarks & Evaluation Research Papers

The newest Benchmarks & Evaluation papers from across the field — arXiv, NeurIPS, CVPR, Nature, and more — refreshed daily and ranked by relevance. Distill AI tracks Benchmarks & Evaluation so you don’t have to: get the standout work delivered to your inbox every morning, with 2-sentence summaries and the option to chat with any paper.

Recent papers

• BatchBench: Benchmarking AI Robustness to Batch Effects and Domain Shift in Biomedical Images - A JUMP Cell Painting Leaderboard
  Sonja Topf, Ana Sanchez-Fernandez, Günter Klambauer · arXiv · Oct 21, 2026

  Despite recent advances, current AI systems remain highly susceptible to domain shifts, particularly in biomedical imaging, where images are usually acquired under substantially different experimental conditions. Although recent domain adap…

• Your Benchmark Is an Empirical Measure Over Difficulty
  Yifan Sun, Naicheng Yu, Jiawen Gong, Jingyan Shen et al. · arXiv · Oct 21, 2026

  A benchmark is not merely a collection of test questions; it naturally induces an empirical measure over difficulty. In this work, we develop this measure-based view of benchmark evaluation. This perspective allows us to identify and formal…

• Kronecker Delta Rule for Long Context Linear Attention
  Anamaria-Roberta Hartl, Levente Zólyomi, Michael List, Günter Klambauer et al. · arXiv · Oct 19, 2026

  We propose KronLSTM, a linear attention architecture based on the Kronecker Delta Rule, a novel update rule for associative matrix memories. KronLSTM extends the xLSTM family with a memory-revision mechanism that corrects targeted associati…

• Decoupling Exploration from Optimization in RLVR
  Saif Punjwani, Micah Goldblum · arXiv · Oct 7, 2026

  Modern language models undergo reinforcement learning with verifiable rewards (RLVR) on top of already-trained checkpoints. A key promise of RLVR is the discovery of new reasoning strategies. In principle, a model can sample novel ideas abs…

• IdeaLens: Detecting AI Ideas in Long-form Writing
  Rishanth Rajendhran, Minjoon Choi, Jenna Russell, Ramya Namuduri et al. · arXiv · Oct 5, 2026

  While modern AI detectors identify who wrote the words, emerging policies on AI use increasingly hinge on a different question: who came up with the ideas? We introduce IdeaLens, a detector that identifies whether a document's ideas came fr…

• Hybrid deep learning for three-way classification of human-written, AI-generated, and AI-rephrased text
  Najma Sadia, Muhammad Naeem Khan, Muhammad Shahzad Faisal, Muhammad Ali Iqbal et al. · Scientific Reports · Oct 5, 2026

  Abstract The rapid proliferation of large language models (LLMs) has made it increasingly difficult to distinguish AI-generated and AI-rephrased text from authentic human writing, posing serious risks to academic integrity, journalism, and …

• Quality, safety, and readability of consumer-facing generative AI responses to end-of-life questions relevant to surrogate decision-makers for older adults: a benchmark evaluation
  Zichen Liu, Xiaoli Zhou, Zhenliang Zhu, Jiayan Ling et al. · BMC Palliative Care · Oct 5, 2026

  As the population ages and the prevalence of multiple coexisting conditions increases, end-of-life decision-making for older adults is becoming increasingly complex in clinical practice. limited access to relevant medical information and di…

• Understanding Automated Program Repair Agents through the Lens of Traceability: An Empirical Study
  Ira Ceka, Hailie Mitchell, Saurabh Pujar, Luca Buratti et al. · Proceedings of the ACM on s... · Oct 1, 2026

  Automated Program Repair (APR) agents leverage large language models (LLMs) to autonomously diagnose and patch software bugs using planning, reasoning, and tools. Although these agents show strong performance on leaderboards such as SWE-ben…

• How Sensitive Are LLM Leaderboard Claims to Hidden Model Selection?
  Chen Yang, Xianyang Zhang, Jun Chen · arXiv · Sep 23, 2026

  LLM leaderboard gains can reflect selection among privately evaluated model variants, yet neither the number of variants nor their dependence is public. We ask how many hidden variants a published margin can support while retaining statisti…

• Quality and safety of large language model–generated medication review outputs in geriatric pharmacotherapy: a two-stage comparative vignette-based benchmark evaluation
  Kübra Çıngar Alpay, Duygu Ozata, Tugce Emiroglu Gedik, Suna Avcı et al. · BMC Geriatrics · Sep 16, 2026

  Abstract Background Polypharmacy and multimorbidity make medication review in older adults a high-stakes clinical task. Large language models (LLMs) may support medication review, but whether item-level concordance with explicit prescribing…

• Clean Engineering, Unstable Measurement: A Preregistered Reliability Failure of Black-Box LLM Observers on Shared Endpoints
  Haoyaun Zhu, Jie Zhang · arXiv · Sep 3, 2026

  Language-model judges now gate training data, score generations, and drive leaderboards. The judge is then a measurement instrument, resting on one rarely stated assumption: the same request, sent to the same model name, reads the same tomo…

• LoRA-TSD: Tangent-Space Spectral Descent for LoRA via Muon-Style Updates
  Dmitrii Andriianov, Andrey Veprikov, Aleksandr Beznosikov · arXiv · Sep 2, 2026

  Low-rank adaptation (LoRA) is the standard way to fine-tune large models, yet when its two factors are trained independently, the update ignores the geometry of the low-rank weight change it induces. We introduce LoRA-TSD, an optimizer that…

• CapBencher: Give Your LLM Benchmark a Built-in Alarm for Test-Set Overfitting
  Takashi Ishida, Thanawat Lodkaew, Ikko Yamane · Semantic Scholar · May 23, 2025

  Publishing a large language model (LLM) benchmark (especially its ground-truth answers) on the Internet risks contaminating future LLMs and enabling evaluation gaming: it may be unintentionally (or intentionally) used to train or select a m…