# Fine-Tune ViT for Image Classification with 🤗 Transformers

huggingface.co · Nate Raw · Feb 11, 2022

**Type:** Tutorial

## Summary

Vision Transformer models bring transformer architectures to computer vision by splitting images into grids of sub-image patches and projecting them into token sequences. To classify healthy and diseased leaves using the beans dataset, the Hugging Face datasets and transformers libraries enable streamlined data ingestion and model fine-tuning. Preprocessing relies on ViTImageProcessor paired with lazy on-the-fly dataset transforms to dynamically generate normalized pixel tensors. Training leverages ViTForImageClassification with the Trainer API, requiring remove\_unused\_columns set to False so raw image data is preserved for batch collation. Over four training epochs, fine-tuning the google/vit-base-patch16-224-in21k checkpoint achieves an evaluation accuracy of 98.5% alongside an evaluation loss of 0.0637.

## Context

Vision Transformers adapt transformer architectures to computer vision by tokenizing images into grids of linearly projected sub-image patches, enabling NLP-style pre-training and fine-tuning for image classification tasks.

## Approach / What changed

Load the beans dataset using Hugging Face datasets, configure a ViTImageProcessor from google/vit-base-patch16-224-in21k, apply transformations dynamically via ds.with\_transform, configure ViTForImageClassification with custom label mappings, and train using Trainer with custom collation, evaluation metrics, and TrainingArguments.

## Takeaways

- ViT tokenizes an image by splitting it into a grid of sub-image patches and projecting each patch linearly into a sequence of input tokens for the transformer.
- Using ds.with\_transform applies image processor transformations lazily upon indexing, avoiding the slower execution of mapping preprocessing eagerly across large datasets.
- Setting remove\_unused\_columns=False in TrainingArguments is required when raw image features must be kept in the batch to generate pixel\_values for the model.

**Tags:** [Machine Learning](https://yomu.fyi/topic/machine-learning), [Open Source](https://yomu.fyi/topic/open-source), [Python](https://yomu.fyi/topic/python)

[Read original post](https://huggingface.co/blog/fine-tune-vit)