Fine-Tune ViT for Image Classification with 🤗 Transformers

Summary

Vision Transformer models bring transformer architectures to computer vision by splitting images into grids of sub-image patches and projecting them into token sequences. To classify healthy and diseased leaves using the beans dataset, the Hugging Face datasets and transformers libraries enable streamlined data ingestion and model fine-tuning. Preprocessing relies on ViTImageProcessor paired with lazy on-the-fly dataset transforms to dynamically generate normalized pixel tensors. Training leverages ViTForImageClassification with the Trainer API, requiring remove_unused_columns set to False so raw image data is preserved for batch collation. Over four training epochs, fine-tuning the google/vit-base-patch16-224-in21k checkpoint achieves an evaluation accuracy of 98.5% alongside an evaluation loss of 0.0637.

Context

Vision Transformers adapt transformer architectures to computer vision by tokenizing images into grids of linearly projected sub-image patches, enabling NLP-style pre-training and fine-tuning for image classification tasks.

Approach / What changed

Load the beans dataset using Hugging Face datasets, configure a ViTImageProcessor from google/vit-base-patch16-224-in21k, apply transformations dynamically via ds.with_transform, configure ViTForImageClassification with custom label mappings, and train using Trainer with custom collation, evaluation metrics, and TrainingArguments.

Takeaways

• ViT tokenizes an image by splitting it into a grid of sub-image patches and projecting each patch linearly into a sequence of input tokens for the transformer.
• Using ds.with_transform applies image processor transformations lazily upon indexing, avoiding the slower execution of mapping preprocessing eagerly across large datasets.
• Setting remove_unused_columns=False in TrainingArguments is required when raw image features must be kept in the batch to generate pixel_values for the model.

Related reading

• huggingface.co · Mar 17, 2022

  Fine-Tune a Semantic Segmentation Model with a Custom Dataset

  Fine-tuning a semantic segmentation model requires domain-appropriate training data and an efficient pipeline. Existing autonomous driving datasets feature roadway imagery captured by cars, creating a distribution mismatch for sidewalk-based delivery robots. To resolve this discrepancy, a dedicated dataset of sidewalk imagery is loaded from the Hugging Face Hub, split into training and test sets, and augmented on-the-fly using SegformerImageProcessor and torchvision. The smallest SegFormer architecture, B0, is fine-tuned using Hugging Face's Trainer API with mean Intersection over Union evaluation metrics. The final pipeline pushes the fine-tuned model to the Hub and executes inference by upsampling output logits to original image dimensions.

  Tobias Cornille, Niels Rogge

• huggingface.co · Apr 25, 2022

  Supercharged Customer Service with Machine Learning

  Customer support teams often receive high volumes of messages that cannot all be answered manually. To prioritize urgent inquiries, support workflows can be modeled as a text classification task to identify the most unsatisfied customers. Using the Hugging Face ecosystem, an NLP pipeline is established by selecting the Amazon reviews multi dataset and fine-tuning a DeBERTa model for sentiment classification across five granular categories. Evaluation on test data shows that the model identifies roughly 95% of unsatisfied messages with an 11.7% false-positive rate on satisfied messages, potentially reducing human triage workload by 83%. For production deployment, performance can be optimized through hardware acceleration, lower precision arithmetic, open-source libraries like Optimum and ONNX Runtime, and inference servers.