Fine tuning CLIP with Remote Sensing (Satellite) images and captions

Summary

Pre-trained vision-language models like CLIP generalize well across everyday imagery but struggle with remote sensing domain characteristics. To enable text-based search across satellite imagery, a distributed team fine-tuned the openai/clip-vit-base-patch32 model on TPU hardware using JAX and Flax during the Hugging Face Flax/JAX Community Week. The training pipeline incorporated satellite image and caption pairs from the RSICD, UCM, and Sydney datasets alongside extensive image transforms and backtranslation-based text augmentation. Contrastive optimization using Adam with a learning rate of 5e-6 achieved top-1 retrieval accuracy of 0.883 compared to 0.572 on the baseline model. An interactive demo uses a NMSLib index for text-to-image matching, image-to-image matching, and patch-level feature detection.

Context

Standard zero-shot CLIP models perform well on everyday images found across the Internet, but satellite and remote sensing imagery differs sufficiently from general domain images to degrade retrieval performance.

Approach / What changed

The authors fine-tuned openai/clip-vit-base-patch32 with JAX and Flax on TPUs using contrastive learning across RSICD, UCM, and Sydney satellite datasets. They applied Torchvision image augmentations and backtranslated captions via Marian MT models into multiple languages and back to English to mitigate overfitting. Learned embeddings were indexed using NMSLib for approximate nearest neighbor retrieval across full images and image patches.

Takeaways

• Fine-tuning CLIP on remote sensing data raised top-1 caption retrieval accuracy on a 30-category RSICD test subset from 0.572 to 0.883.
• Combining Torchvision image transforms with Marian MT backtranslation text augmentation progressively reduced training overfitting.
• Patch-based embedding matching allows querying fine-grained textual features within specific subregions of a single satellite image.

Related reading