Hyperparameter Search with Transformers and Ray Tune

Summary

Hugging Face Transformers models often require hyperparameter tuning to achieve high performance, but practitioners frequently rely on basic grid search or bypass tuning entirely. The Transformers 3.1 release integrates Ray Tune into the Hugging Face Trainer to provide advanced hyperparameter optimization without losing framework customizability. By invoking hyperparameter_search with the Ray backend, users can configure trial budgets, parallelize runs across multiple GPUs, and plug in search algorithms like HyperOptSearch alongside schedulers like ASHAScheduler. Experimental benchmarks on a BERT model using the RTE dataset show that Population-based Training reached 70.5% test accuracy in 48 GPU minutes, outperforming Grid Search at 65.4% test accuracy. The integration also supports experiment tracking tools like Weights and Biases out of the box.

Context

Machine learning models often require parameter tuning for optimal performance, yet practitioners frequently disregard hyperparameter search or restrict themselves to simplistic grid searches over small search spaces.

Approach / What changed

Hugging Face Transformers 3.1 integrated Ray Tune into the Trainer API, enabling hyperparameter optimization via trainer.hyperparameter_search with backend set to ray. Users can configure resources per trial, execute parallel searches across multiple GPUs, and customize search algorithms and schedulers such as HyperOptSearch and ASHAScheduler.

Takeaways

• In a benchmark fine-tuning BERT on the RTE dataset, Population-based Training achieved 70.5% test accuracy in 48 GPU minutes ($2.45), compared to 65.4% in 45 GPU minutes ($2.30) for Grid Search.
• By default, Ray Tune trials in the Hugging Face Trainer allocate 1 CPU and optionally 1 GPU per trial, with multi-GPU parallel searches supported via resources_per_trial.
• Custom search algorithms and schedulers, including HyperOptSearch and ASHAScheduler, can be plugged directly into Trainer.hyperparameter_search alongside tools like Weights and Biases.

Related reading

• Red Hat · Aug 24, 2026

  Run LoRA fine-tuning on Red Hat OpenShift AI with Ray

  Red Hat OpenShift AI 3.5 integrates Ray cluster runtime images with preinstalled Training Hub packages to support distributed fine-tuning algorithms including LoRA, SFT, OSFT, and GRPO. Using the CodeFlare software development kit within a Jupyter workbench, developers can provision an elastic single-GPU Ray cluster attached to shared ReadWriteMany persistent storage containing model weights and training datasets. The tutorial demonstrates fine-tuning the Qwen2.5-1.5B-Instruct model on the public sql-create-context dataset to produce direct, bare SQL output without explanatory conversational text. After training completes, the workbench loads and merges the resulting LoRA adapter checkpoints with the base model to evaluate output format correctness against sample database schema questions. The RayCluster automatically tears down resources once the job concludes, while the shared pipeline architecture allows switching between supported fine-tuning algorithms by modifying dataset inputs and configuration parameters.

  Fiona Waters

• huggingface.co · Nov 9, 2020

  Leveraging Pre-trained Language Model Checkpoints for Encoder-Decoder Models