Hyperparameter Search with Transformers and Ray Tune

Summary

Hugging Face Transformers models often require hyperparameter tuning to achieve high performance, but practitioners frequently rely on basic grid search or bypass tuning entirely. The Transformers 3.1 release integrates Ray Tune into the Hugging Face Trainer to provide advanced hyperparameter optimization without losing framework customizability. By invoking hyperparameter_search with the Ray backend, users can configure trial budgets, parallelize runs across multiple GPUs, and plug in search algorithms like HyperOptSearch alongside schedulers like ASHAScheduler. Experimental benchmarks on a BERT model using the RTE dataset show that Population-based Training reached 70.5% test accuracy in 48 GPU minutes, outperforming Grid Search at 65.4% test accuracy. The integration also supports experiment tracking tools like Weights and Biases out of the box.

Takeaways

1. 1

   In a benchmark fine-tuning BERT on the RTE dataset, Population-based Training achieved 70.5% test accuracy in 48 GPU minutes ($2.45), compared to 65.4% in 45 GPU minutes ($2.30) for Grid Search.

2. 2

   By default, Ray Tune trials in the Hugging Face Trainer allocate 1 CPU and optionally 1 GPU per trial, with multi-GPU parallel searches supported via resources_per_trial.

3. 3

   Custom search algorithms and schedulers, including HyperOptSearch and ASHAScheduler, can be plugged directly into Trainer.hyperparameter_search alongside tools like Weights and Biases.