# Hyperparameter Search with Transformers and Ray Tune

huggingface.co · system · Nov 2, 2020

**Type:** Tutorial

## Summary

Hugging Face Transformers models often require hyperparameter tuning to achieve high performance, but practitioners frequently rely on basic grid search or bypass tuning entirely. The Transformers 3.1 release integrates Ray Tune into the Hugging Face Trainer to provide advanced hyperparameter optimization without losing framework customizability. By invoking hyperparameter\_search with the Ray backend, users can configure trial budgets, parallelize runs across multiple GPUs, and plug in search algorithms like HyperOptSearch alongside schedulers like ASHAScheduler. Experimental benchmarks on a BERT model using the RTE dataset show that Population-based Training reached 70.5% test accuracy in 48 GPU minutes, outperforming Grid Search at 65.4% test accuracy. The integration also supports experiment tracking tools like Weights and Biases out of the box.

## Context

Machine learning models often require parameter tuning for optimal performance, yet practitioners frequently disregard hyperparameter search or restrict themselves to simplistic grid searches over small search spaces.

## Approach / What changed

Hugging Face Transformers 3.1 integrated Ray Tune into the Trainer API, enabling hyperparameter optimization via trainer.hyperparameter\_search with backend set to ray. Users can configure resources per trial, execute parallel searches across multiple GPUs, and customize search algorithms and schedulers such as HyperOptSearch and ASHAScheduler.

## Takeaways

- In a benchmark fine-tuning BERT on the RTE dataset, Population-based Training achieved 70.5% test accuracy in 48 GPU minutes ($2.45), compared to 65.4% in 45 GPU minutes ($2.30) for Grid Search.
- By default, Ray Tune trials in the Hugging Face Trainer allocate 1 CPU and optionally 1 GPU per trial, with multi-GPU parallel searches supported via resources\_per\_trial.
- Custom search algorithms and schedulers, including HyperOptSearch and ASHAScheduler, can be plugged directly into Trainer.hyperparameter\_search alongside tools like Weights and Biases.

**Tags:** [Machine Learning](https://yomu.fyi/topic/machine-learning), [Open Source](https://yomu.fyi/topic/open-source), [Python](https://yomu.fyi/topic/python)

[Read original post](https://huggingface.co/blog/ray-tune)