---
title: "Efficient Visual Representation Learning And Evaluation"
description: "Etsy uses image vector representations for visual search and similar-item recommendations, making low-latency, memory-aware models important for online query inference and downstream retrieval. The work compares EfficientNetB0, a compact CNN, with EfficientFormer-l3, which downsamples multiple blocks and applies attention only in its final stage; its representation uses the last block’s eight-head attention and average-pools the sequence. Fine-tuning adds an embedding and multiple classification heads, while multitask training supervises taxonomy, color, and review-photo labels; both models train on two 16GB P100 GPUs. Evaluation rebuilds a brute-force candidate index after each epoch and measures Recall@5 and Recall@10 on interaction and intra-item retrieval datasets, with early stopping available. EfficientFormer-l3 improves reported offline recall by about 5%, 17%, and 1.8% across three tasks, while experiments report lifts in CTR, purchase rate, click-recall@100, and clicks."
---

# Efficient Visual Representation Learning And Evaluation

[Etsy](https://yomu.fyi/company/etsy) · Eden Dolev · Jun 7, 2024

**Type:** Explainer

## Summary

Etsy uses image vector representations for visual search and similar-item recommendations, making low-latency, memory-aware models important for online query inference and downstream retrieval. The work compares EfficientNetB0, a compact CNN, with EfficientFormer-l3, which downsamples multiple blocks and applies attention only in its final stage; its representation uses the last block’s eight-head attention and average-pools the sequence. Fine-tuning adds an embedding and multiple classification heads, while multitask training supervises taxonomy, color, and review-photo labels; both models train on two 16GB P100 GPUs. Evaluation rebuilds a brute-force candidate index after each epoch and measures Recall@5 and Recall@10 on interaction and intra-item retrieval datasets, with early stopping available. EfficientFormer-l3 improves reported offline recall by about 5%, 17%, and 1.8% across three tasks, while experiments report lifts in CTR, purchase rate, click-recall@100, and clicks.

## Context

Etsy’s visually rich marketplace uses computer vision for visual search and visually similar recommendations. Training and serving expressive deep-learning representations is costly, while online image-query inference requires low latency and awareness of memory and retrieval constraints.

## Approach / What changed

The work uses EfficientNetB0 and EfficientFormer-l3 backbones, fine-tunes them with embedding and multitask classification heads, and evaluates representations through nearest-neighbor retrieval callbacks after each training epoch. It also applies efficient inference and retrieval designs for latency-sensitive downstream applications.

## Takeaways

- EfficientFormer-l3 applies attention only in its last stage and derives representations by average-pooling the last hidden state or attention-based sequence output.
- Multitask supervision covers top-level taxonomy, fine-grained taxonomy, primary color, and buyer review-photo categories; training uses two 16GB P100 GPUs.
- Compared with the production EfficientNetB0 variant, EfficientFormer-l3 representations produced a reported 0.65% CTR lift and similar purchase-rate lift in an initial visually similar ads experiment.

**Tags:** [Inference](https://yomu.fyi/topic/inference), [Machine Learning](https://yomu.fyi/topic/machine-learning), [Model Evaluation](https://yomu.fyi/topic/model-evaluation), [Recommendation Systems](https://yomu.fyi/topic/recommendation-systems)

- Source: [Etsy](https://www.etsy.com/codeascraft/efficient-visual-representation-learning-and-evaluation)
- Source URL: https://www.etsy.com/codeascraft/efficient-visual-representation-learning-and-evaluation
- Ingested by Yomu: 2026-09-09T16:02:59.750Z

[Read original post](https://www.etsy.com/codeascraft/efficient-visual-representation-learning-and-evaluation)