# Fine-Tune a Semantic Segmentation Model with a Custom Dataset

huggingface.co · Tobias Cornille, Niels Rogge · Mar 17, 2022

**Type:** Tutorial

## Summary

Fine-tuning a semantic segmentation model requires domain-appropriate training data and an efficient pipeline. Existing autonomous driving datasets feature roadway imagery captured by cars, creating a distribution mismatch for sidewalk-based delivery robots. To resolve this discrepancy, a dedicated dataset of sidewalk imagery is loaded from the Hugging Face Hub, split into training and test sets, and augmented on-the-fly using SegformerImageProcessor and torchvision. The smallest SegFormer architecture, B0, is fine-tuned using Hugging Face's Trainer API with mean Intersection over Union evaluation metrics. The final pipeline pushes the fine-tuned model to the Hub and executes inference by upsampling output logits to original image dimensions.

## Context

Autonomous driving datasets like CityScapes and BDD100K are captured by cars on roads, causing a distribution mismatch for sidewalk-navigating delivery robots that need precise sidewalk and obstacle detection.

## Approach / What changed

Using SegFormer-B0 with Hugging Face transformers, datasets, and evaluate libraries to fine-tune a semantic segmentation model on custom sidewalk imagery with on-the-fly transformations and push the resulting artifacts to the Hugging Face Hub.

## Takeaways

- SegFormer employs a hierarchical Transformer encoder lacking positional encodings paired with a simple multi-layer perceptron decoder.
- Applying dataset transforms on-the-fly via set\_transform avoids the disk space overhead of preprocessing entire image datasets in advance.
- SegFormer output logits have reduced spatial dimensions of height/4 and width/4, requiring bilinear interpolation to match original image dimensions before argmax classification.

**Tags:** [Machine Learning](https://yomu.fyi/topic/machine-learning), [Open Source](https://yomu.fyi/topic/open-source), [Python](https://yomu.fyi/topic/python)

[Read original post](https://huggingface.co/blog/fine-tune-segformer)