Efficient Visual Representation Learning And Evaluation

Summary

Etsy uses image vector representations for visual search and similar-item recommendations, making low-latency, memory-aware models important for online query inference and downstream retrieval. The work compares EfficientNetB0, a compact CNN, with EfficientFormer-l3, which downsamples multiple blocks and applies attention only in its final stage; its representation uses the last block’s eight-head attention and average-pools the sequence. Fine-tuning adds an embedding and multiple classification heads, while multitask training supervises taxonomy, color, and review-photo labels; both models train on two 16GB P100 GPUs. Evaluation rebuilds a brute-force candidate index after each epoch and measures Recall@5 and Recall@10 on interaction and intra-item retrieval datasets, with early stopping available. EfficientFormer-l3 improves reported offline recall by about 5%, 17%, and 1.8% across three tasks, while experiments report lifts in CTR, purchase rate, click-recall@100, and clicks.

Takeaways

1. 1

   EfficientFormer-l3 applies attention only in its last stage and derives representations by average-pooling the last hidden state or attention-based sequence output.

2. 2

   Multitask supervision covers top-level taxonomy, fine-grained taxonomy, primary color, and buyer review-photo categories; training uses two 16GB P100 GPUs.

3. 3

   Compared with the production EfficientNetB0 variant, EfficientFormer-l3 representations produced a reported 0.65% CTR lift and similar purchase-rate lift in an initial visually similar ads experiment.