Latest Knowledge Distillation Research Papers
The newest Knowledge Distillation papers from across the field — arXiv, NeurIPS, CVPR, Nature, and more — refreshed daily and ranked by relevance. Distill AI tracks Knowledge Distillation so you don’t have to: get the standout work delivered to your inbox every morning, with 2-sentence summaries and the option to chat with any paper.
Get the latest Knowledge Distillation papers in your inbox — free →Recent papers
- An ODE-based Sampler for Discrete Diffusion ModelsSamson Gourevitch, Yazid Janati, Eric Moulines, Eric P. Xing et al. · arXiv · Oct 29, 2026
The marginals of a continuous diffusion model can be reproduced by a deterministic ordinary differential equation (ODE), known as the probability flow ODE, which provides a foundation for accelerated sampling and consistency distillation. I…
- Homebrewed SLMs: Autoregressive and Masked Diffusion Language Models Built at Home for PortuguesePedro Carvalho Brom · arXiv · Oct 29, 2026
Building a language model of one's own no longer requires a data centre. This tutorial trains one from scratch, tokenizer included, on one consumer GPU, and shows how to tell whether it is good for the tasks of a specialist domain. It propo…
- Soil Moisture Estimation in Precision Agriculture: A Knowledge Distillation ApproachMohammed Rakib, Vrinda Nagula, Arunkumar Bagavathi · arXiv · Oct 29, 2026
Accurate soil moisture estimation is critical for precision agriculture, influencing irrigation, fertilization, and harvesting strategies. Traditional methods primarily rely on meteorological and soil property data, while recent approaches …
- Minding Anomaly Patterns via Uncertainty-Guided Heterogeneous Knowledge Distillation with DenoisingYuan Li, Song Yang, Yajie Wang, Buyu Wang et al. · Big Data Mining and Analytics · Oct 28, 2026
- Cross-Modal Adaptive Feedback Knowledge Distillation for Enhanced sEMG-Based Semantic RecognitionJunfeng Wei, 阮桂红, Peiguang Jing, Yun Li · Big Data and Cognitive Comp... · Oct 28, 2026
In semantic recognition based on surface electromyography (sEMG), low signal-to-noise ratios and physiological differences across subjects limit the ability of unimodal sEMG features to fully represent speech semantics, constraining recogni…
- Integrated energy and thermal management of range-extended electric vehicles using causal global distillation and Koopman-based model predictive controlHaoming Gao, Chao Sun, Xupeng Song, Guodong Yang et al. · Energy · Oct 27, 2026
- A self-supervised and distillation-guided framework for robust cloud-layer identification in multi-channel Raman lidar networksWeijie Zou, Zhenping Yin, Tingxin Sun, Zhichao Bu et al. · Atmospheric measurement tec... · Oct 27, 2026
Abstract. Cloud-layer identification is a key prerequisite for automated ground-based lidar processing, but remains challenging in Raman lidar networks because of incomplete near-range overlap, weak high-level cloud echoes, day–night signal…
- Invisible Hands in the Perfume Capital: A Thematic Analysis of Womens Labor in Kannaujs Attar IndustryJyoti Yadav, Kanchan Kushwaha, Anju Lata, Nidhi Singh et al. · International Journal of Ma... · Oct 27, 2026
The traditional art of making perfumes in Kannauj district, Uttar Pradesh, is considered one ofIndias oldest and most culturally rich industries. Kannauj attars have survived for many years because of the deg-bhapka distillation process use…
- Difficulty Weighting and Semantic Centroid Regularization (SCR) for Raising the Performance Floor and Concentrating the Score Distribution of Small ModelsLong Huang, Zongshuai Cui, Jianzhong Zhang, Hao Wang et al. · Algorithms · Oct 27, 2026
Intelligent applications in the coal industry require models to read professional images such as safety tables and engineering drawings. Small multimodal models lack sufficient domain-specific knowledge, whereas large multimodal models are …
- Shapley-guided token pruning for real-time remote sensing scene classification on edge devicesChaoyu Li, Tianyu Zhou · Scientific Reports · Oct 27, 2026
Abstract Vision Transformers (ViTs) excel at remote sensing scene classification, but the quadratic cost of self-attention hinders real-time inference on edge platforms such as on-board satellite and UAV payloads. Token pruning accelerates …
- Learning When to Commit: Schedule Distillation for Fast Diffusion Language ModelsAidar Myrzakhan, Zhiqiang Shen · arXiv · Oct 26, 2026
Diffusion large language models (dLLMs) can generate multiple tokens in parallel, but often require many decoding rounds to complete a response. Efficient generation depends not only on which tokens are easy to predict, but also on how comm…
- An efficient hybrid weld seam detection network based on knowledge distillation and model pruningHuayi Cai, Liang Xie, Chaoqun Wu, Yun Cheng et al. · Measurement Science and Tec... · Oct 26, 2026
Abstract Accurate weld type recognition and stable keypoint localization are essential for automated weld seam inspection, yet real-time deployment is still challenged by the high computational cost of dense prediction models and visual dis…
- Ultrasound-based breast nodule malignancy diagnosis using CNN enhanced by knowledge distillationTao Hu, Rongrong Ru, Yingying Ding, Dong Ming Xu et al. · iScience · Oct 26, 2026
- $${\text {VLScene}}^{++}$$: Enhancing 3D Semantic Scene Completion via VLM Discrepancy-Aware Semantic Distillation and Sparse–Dense Hybrid AwarenessMeng Wang, Yujie Xue, Jijiang Liu, Ruihui Li et al. · International Journal of Co... · Oct 25, 2026
- CA-MobileNetV4: A Network Integrating CPCA and Knowledge Distillation for Efficient Gastrointestinal Image ClassificationBinghua Tao, Muzhou Hou, Jiaoju Wang, Shuijiao Chen · Sensing and Imaging · Oct 24, 2026
- Adaptive temperature distillation with curriculum learning for lightweight multilingual language teaching modelsSirui Wang, Shuai Wu, Haiyan Wang · Discover Artificial Intelli... · Oct 24, 2026
Knowledge distillation can compress large multilingual models into lightweight alternatives; however, existing approaches employ a uniform temperature setting that does not account for the varying characteristics of different languages. Thi…
- An adaptive transformer-based model with entropy-driven knowledge distillation for single-channel EEG REM sleep detectionJi-Jer Huang, S Chen, Irawan Dwiwahyono · Scientific Reports · Oct 24, 2026
Abstract Accurate detection of rapid eye movement (REM) sleep is critical for neurophysiological monitoring and early identification of sleep-related disorders. An entropy-driven adaptive knowledge distillation framework integrated with a l…
- Sustainability of TSFM Deployment and Pretraining: From Carbon Footprint Analysis to Mitigation via PruningCécile Luc, Nathanaël Lemonnier, Ievgen Redko · arXiv · Oct 23, 2026
Time Series Foundation Models (TSFMs) offer strong zero-shot forecasting capabilities, but their high pretraining cost raises a practical sustainability question: does this upfront investment pay off over a full deployment lifecycle, compar…
- On-Policy Self-Training: Data-Free Self-Improvement in Large Language ModelsLeon Sick, Lukas Hoyer, Yannick Strümpler, Christian Reisswig et al. · arXiv · Oct 23, 2026
On-policy self-distillation enables post-training foundation models without relying on stronger external teachers. However, existing approaches rely on ground-truth question-answer pairs, human-curated problem sets, or other supervised sign…
- Few-shot cross-domain fault diagnosis of motor couplings based on physics-informed acoustic feature distillationShuai Xu, Xianning Bi, Menglei Qi, Guangtong Ma · Applied Acoustics · Oct 23, 2026
To overcome the challenges of cross-domain interference, few-shot learning limitations, and edge deployment constraints in conventional deep learning, this paper introduces a novel fault diagnosis framework for motor couplings based on freq…
- A compact teacher-student knowledge-distillation framework for lumbar MRI segmentationYasemin Sarı · Medical Engineering & Physics · Oct 23, 2026
- Optimizing Entanglement Distillation Policies via Markov Decision Process FormulationJigyen Bhavsar, Rajni Bala, Siddhartha Santra · Quantum Science and Technology · Oct 23, 2026
Abstract Entanglement distillation is a fundamental operation in quantum information processing used to obtain higher-fidelity entangled pairs from a supply of less entangled quantum states using local operations aided by classical communic…
- TMDKD: tutor-guided multi-dimensional difficulty-aware knowledge distillationYu Wang, Minqi Li, Kaibing Zhang, Xiangjian He et al. · Multimedia Systems · Oct 23, 2026
- KD-APFNet: Knowledge Distillation and Adaptive Prototype Fusion for Cross-Load Few-Shot Bearing Fault DiagnosisCunli Song, Wenjing Yu, Weiguo Shi · Engineering Research Express · Oct 23, 2026
Abstract For bearing fault diagnosis under load-induced spectral drift and extremely scarce target-domain labels, this paper proposes the Knowledge Distillation and Adaptive Prototype Fusion Network (KD-APFNet). First, a lightweight time–fr…
- When to Follow the Teacher: Advantage-Based On-Policy DistillationMahdi Sabbaghi, George Pappas, Hamed Hassani, Adel Javanmard · arXiv · Oct 22, 2026
Online Policy Distillation (OPD) often outperforms offline methods such as Supervised Fine-Tuning (SFT) by training on trajectories sampled directly from the student. However, standard OPD trains the student to follow the teacher at every s…
- Knowledge distillation-based rapid identification of Ligustici Rhizoma et Radix decoction piece varietiesHu Haoyu, Shangzhou Jiang, Shengting Liang, Wentao Zhong et al. · Frontiers in Pharmacology · Oct 22, 2026
To address the commingling of different varieties of Ligusticum decoction pieces in the market and the difficulty of achieving rapid, efficient, and accurate on-site identification, this study aimed to develop a lightweight deep learning mo…
- New assessment of cyclohexane–ethanol vapor–liquid equilibrium using Peng–Robinson and UNIFAC modelsJONATHAN MELO DE AZEVEDO, Rogério N. C. de Siqueira · Chemical Papers · Oct 22, 2026
Abstract Vapor–liquid equilibria (VLE) are fundamental to distillation, one of the most widely employed separation techniques in chemical and pharmaceutical industries. The presence of asymmetrical molecules in shape and size, combined with…
- Towards lightweight coastal litter monitoring: A synergistic deep learning framework integrating efficient architectures with model compression and knowledge distillationXu Gao, Jianhua Zhu, He Wang, Zhen Tian et al. · Journal of Environmental Ma... · Oct 22, 2026
- The role of locality and weight sharing in convolutional neural networksFrancesco Mori, Melanie Weber · arXiv · Oct 21, 2026
The success of convolutional neural networks is often attributed to locality and weight sharing. However, how these two properties separately shape learning, and how their effects depend on the spatial correlations of natural images, remain…
- Asymmetric On-Policy Distillation Learns In-Context ExplorationSriyash Poddar, Yanda Bao, Jacob Krantz, Matthew Chang et al. · arXiv · Oct 21, 2026
Sequential decision-making under partial observability is difficult because agents must explore to infer hidden, task-relevant information before acting effectively. Learning such information-gathering behaviors tabula rasa via online reinf…