Fine-Tune a Semantic Segmentation Model with a Custom Dataset

Summary

Fine-tuning a semantic segmentation model requires domain-appropriate training data and an efficient pipeline. Existing autonomous driving datasets feature roadway imagery captured by cars, creating a distribution mismatch for sidewalk-based delivery robots. To resolve this discrepancy, a dedicated dataset of sidewalk imagery is loaded from the Hugging Face Hub, split into training and test sets, and augmented on-the-fly using SegformerImageProcessor and torchvision. The smallest SegFormer architecture, B0, is fine-tuned using Hugging Face's Trainer API with mean Intersection over Union evaluation metrics. The final pipeline pushes the fine-tuned model to the Hub and executes inference by upsampling output logits to original image dimensions.

Context

Autonomous driving datasets like CityScapes and BDD100K are captured by cars on roads, causing a distribution mismatch for sidewalk-navigating delivery robots that need precise sidewalk and obstacle detection.

Approach / What changed

Using SegFormer-B0 with Hugging Face transformers, datasets, and evaluate libraries to fine-tune a semantic segmentation model on custom sidewalk imagery with on-the-fly transformations and push the resulting artifacts to the Hugging Face Hub.

Takeaways

• SegFormer employs a hierarchical Transformer encoder lacking positional encodings paired with a simple multi-layer perceptron decoder.
• Applying dataset transforms on-the-fly via set_transform avoids the disk space overhead of preprocessing entire image datasets in advance.
• SegFormer output logits have reduced spatial dimensions of height/4 and width/4, requiring bilinear interpolation to match original image dimensions before argmax classification.

Related reading

• huggingface.co · Feb 11, 2022

  Fine-Tune ViT for Image Classification with 🤗 Transformers

  Vision Transformer models bring transformer architectures to computer vision by splitting images into grids of sub-image patches and projecting them into token sequences. To classify healthy and diseased leaves using the beans dataset, the Hugging Face datasets and transformers libraries enable streamlined data ingestion and model fine-tuning. Preprocessing relies on ViTImageProcessor paired with lazy on-the-fly dataset transforms to dynamically generate normalized pixel tensors. Training leverages ViTForImageClassification with the Trainer API, requiring remove_unused_columns set to False so raw image data is preserved for batch collation. Over four training epochs, fine-tuning the google/vit-base-patch16-224-in21k checkpoint achieves an evaluation accuracy of 98.5% alongside an evaluation loss of 0.0637.

  Nate Raw

• huggingface.co · Apr 25, 2022

  Supercharged Customer Service with Machine Learning

  Customer support teams often receive high volumes of messages that cannot all be answered manually. To prioritize urgent inquiries, support workflows can be modeled as a text classification task to identify the most unsatisfied customers. Using the Hugging Face ecosystem, an NLP pipeline is established by selecting the Amazon reviews multi dataset and fine-tuning a DeBERTa model for sentiment classification across five granular categories. Evaluation on test data shows that the model identifies roughly 95% of unsatisfied messages with an 11.7% false-positive rate on satisfied messages, potentially reducing human triage workload by 83%. For production deployment, performance can be optimized through hardware acceleration, lower precision arithmetic, open-source libraries like Optimum and ONNX Runtime, and inference servers.