Image search with 🤗 datasets

Summary

Hugging Face datasets expanded its capabilities by introducing an Image feature type, enabling image processing and integration with vector indexing tools. The library was applied to a sample of historical book embellishments extracted via OCR from the British Library. Images were loaded using the ImageFolder loader, enriched with filename metadata, and pushed to the Hugging Face Hub. A FAISS index and CLIP embeddings were used to retrieve images matching natural language queries, such as categories, specific objects, and boolean operators. While the retrieval system demonstrated reasonable semantic search results across various prompts, full public deployment was avoided due to CLIP model card restrictions and potential bias in the historical dataset.

Context

Historical book illustrations from the British Library had known source book metadata but lacked rich semantic descriptions of their visual content, limiting search capabilities.

Approach / What changed

Loaded the image collection using the datasets ImageFolder loader, extracted filename metadata into a new column, pushed the dataset to the Hugging Face Hub, and indexed CLIP embeddings with FAISS for text-to-image semantic retrieval.

Takeaways

• The datasets Image feature accepts absolute file paths, byte dictionaries, NumPy arrays, or PIL Image objects.
• The ImageFolder loader allows direct ingestion of directory structures containing images into a DatasetDict.
• Public deployment of the search demo was withheld due to CLIP out-of-scope deployment guidance and bias concerns in historical source books.

Related reading

• huggingface.co · Jun 23, 2022

  Getting Started With Embeddings

  Embeddings represent unstructured information such as text and images as numerical vectors in a shared semantic space. To demonstrate their utility, a simple semantic search engine is built over US Social Security Medicare frequently asked questions. The system generates 384-dimensional vector representations for thirteen FAQ entries by dispatching POST requests to the Hugging Face Inference API using the sentence-transformers/all-MiniLM-L6-v2 model. Incoming user queries are converted into matching vector representations and evaluated against stored dataset vectors using the util.semantic_search function from the Sentence Transformers library. By calculating cosine similarity scores, the system retrieves and ranks the five most semantically relevant questions without requiring custom keyword rules or massive labeled training sets.

  Omar Espejel

• huggingface.co · Feb 11, 2022

  Fine-Tune ViT for Image Classification with 🤗 Transformers

  Vision Transformer models bring transformer architectures to computer vision by splitting images into grids of sub-image patches and projecting them into token sequences. To classify healthy and diseased leaves using the beans dataset, the Hugging Face datasets and transformers libraries enable streamlined data ingestion and model fine-tuning. Preprocessing relies on ViTImageProcessor paired with lazy on-the-fly dataset transforms to dynamically generate normalized pixel tensors. Training leverages ViTForImageClassification with the Trainer API, requiring remove_unused_columns set to False so raw image data is preserved for batch collation. Over four training epochs, fine-tuning the google/vit-base-patch16-224-in21k checkpoint achieves an evaluation accuracy of 98.5% alongside an evaluation loss of 0.0637.