NeurIPS 2023: Our Favorite Papers on LLMs, Statistical Learning, and More

Summary

Researchers reviewed prominent machine learning papers presented at NeurIPS 2023 covering large language models and statistical learning theory. One investigation showed that claimed emergent abilities in models such as GPT-3 often result from nonlinear evaluation metrics rather than fundamental shifts in model capability. To reduce the computational burden of model adaptation, QLoRA enables 65-billion-parameter model fine-tuning on a single 48-gigabyte GPU via 4-bit NormalFloat quantization and paged optimization. Direct Preference Optimization eliminates complex reward modeling by casting reinforcement learning from human feedback into a preference classification task. Additional work resolved statistical anomalies like double descent using effective parameter counts and introduced stochastic gradient approximations for Gaussian processes.

Context

Researchers reviewed findings from NeurIPS 2023 addressing scalability, generalization, and computational constraints across large language models and statistical learning. Key challenges examined include high hardware demands for LLM customization, complex multi-stage alignment workflows in RLHF, the mathematical validity of emergent model capabilities, and computational bottlenecks in Gaussian process sampling.

Approach / What changed

The reviewed papers present algorithmic and theoretical frameworks: QLoRA combines 4-bit NormalFloat quantization with paged optimization for efficient fine-tuning, Direct Preference Optimization condenses RLHF into a direct classification objective, and pathwise conditioning with stochastic gradient descent approximates Gaussian process posteriors. Other studies assess metric sensitivity in LLM evaluation and introduce the DISCS benchmark for discrete sampling.

Takeaways

• Apparent emergent abilities in LLMs can disappear when switching from nonlinear evaluation metrics like accuracy to linear metrics like token edit distance or by increasing test sample resolution.
• QLoRA enables fine-tuning a 65B parameter model on a single 48GB GPU by combining a 4-bit NormalFloat data type, double quantization, and paged optimization to mitigate memory spikes.
• Direct Preference Optimization simplifies LLM alignment by reformulating the constrained reward maximization problem of RLHF into an equivalent human preference classification task.

Related reading

• Grab · Nov 4, 2025

  How we built a custom vision LLM to improve document processing at Grab

  Document processing for identity verification across Southeast Asia presents challenges due to varied layouts and non-Latin scripts. Traditional OCR and off-the-shelf vision models struggle with accuracy, high latency, or lack of regional language training data. Grab addressed this by creating synthetic regional datasets, using an automated labeling pipeline named Documint, and evaluating open-source multimodal architectures. After initial LoRA fine-tuning failed on complex scripts like Thai and Vietnamese, full-parameter fine-tuning of Qwen2-VL 2B yielded substantial gains. To optimize deployment costs and latency, the team constructed a custom 1B parameter model pairing a Qwen2-VL vision encoder with a Qwen2.5 0.5B language decoder, achieving performance within 3 percentage points of the 2B model at significantly lower latency.

  Jia Chen

• Apple · Aug 20, 2026

  Scaling Laws for Mixture Pretraining Under Data Constraints