---
title: "Model Evaluation"
description: "65 posts about Model Evaluation, summarised, each linking to the original."
---

# Model Evaluation
> 65 posts about Model Evaluation, summarised, each linking to the original.

## Articles

### [Intercom vs Zendesk: Two AI agents put to the test](https://yomu.fyi/post/intercom-vs-zendesk-two-ai-agents-put-to-the-test.md)
- Company: [Intercom](https://yomu.fyi/company/intercom.md)
- Author: Sagar Joglekar
- Published: Nov 14, 2024
- Type: Benchmark

The post compares Intercom’s Fin AI Agent with Zendesk’s AI agent, focusing on answer quality, complexity, and resolution performance. In a test using 48 fictional bed-and-breakfast help articles and 200 questions, both systems answered the same dataset, while four AI models judged responses and an Elo rating system selected winners. It reports that Fin produced better answers in 80% of cases, answered 96% of multi-source questions versus Zendesk’s 78%, and delivered responses averaging 120 words versus 50. The analysis also found no statistical difference in hallucination levels and estimated that Fin was 66% more likely to provide a complete resolution when both agents answered. The post notes limitations including simulated data, AI judging, and differing product features, then describes Fin’s integrations and 99¢-per-resolution pricing.


### [U.S. elections readiness](https://yomu.fyi/post/u-s-elections-readiness.md)
- Company: [Anthropic](https://yomu.fyi/company/anthropic.md)
- Published: Nov 5, 2024
- Type: Announcement

Anthropic outlines its preparations for the 2024 U.S. election cycle, the first in which generative AI tools are widely available. Since July 2023, it has updated its Usage Policy, strengthened misuse detection, and redirected users toward authoritative voting information. The policy prohibits campaigning, lobbying, election misinformation, targeting voting machines, and interference with vote counting or certification, while Claude's text-only output prevents it from generating image, audio, or video deepfakes. Controls include prompt modifications, API audits, possible account suspensions, cloud-provider coordination, red-teaming, Policy Vulnerability Testing, and automated evaluations of political parity, harmful-query refusals, misinformation prevention, and voter-profiling risks; a TurboVote pop-up handles voting-information requests because models lack real-time election data, while the system prompt states Claude's knowledge cutoff.


### [Machine Learning in Content Moderation at Etsy](https://yomu.fyi/post/machine-learning-in-content-moderation-at-etsy.md)
- Company: [Etsy](https://yomu.fyi/company/etsy.md)
- Author: David Azcona
- Published: Aug 6, 2024
- Type: Explainer

Etsy describes a supervised-learning system for detecting potentially violating listings at marketplace scale, alongside community reporting and other automated controls. Training data combines logged and annotated positives, hard negatives, and randomly sampled easy negatives, with multimodal textual and imagery signals split by time for progressive evaluation. The architecture uses BERT-based text encoding with ALBERT for reduced compute, EfficientNet image encoding, late concatenation of modality representations, a softmax output, and focal loss for class imbalance. Etsy evaluates candidates offline and in production through API-serving, canary releases, and champion-challenger A/B tests, while using proxy recall because full manual review is infeasible. Frequent retraining and monitoring support ongoing iteration; the methodology has identified and removed more than 100,000 violations, alongside other tools and services.


### [Efficient Visual Representation Learning And Evaluation](https://yomu.fyi/post/efficient-visual-representation-learning-and-evaluation.md)
- Company: [Etsy](https://yomu.fyi/company/etsy.md)
- Author: Eden Dolev
- Published: Jun 7, 2024
- Type: Explainer

Etsy uses image vector representations for visual search and similar-item recommendations, making low-latency, memory-aware models important for online query inference and downstream retrieval. The work compares EfficientNetB0, a compact CNN, with EfficientFormer-l3, which downsamples multiple blocks and applies attention only in its final stage; its representation uses the last block’s eight-head attention and average-pools the sequence. Fine-tuning adds an embedding and multiple classification heads, while multitask training supervises taxonomy, color, and review-photo labels; both models train on two 16GB P100 GPUs. Evaluation rebuilds a brute-force candidate index after each epoch and measures Recall@5 and Recall@10 on interaction and intra-item retrieval datasets, with early stopping available. EfficientFormer-l3 improves reported offline recall by about 5%, 17%, and 1.8% across three tasks, while experiments report lifts in CTR, purchase rate, click-recall@100, and clicks.


### [Charting a path to AI accountability](https://yomu.fyi/post/charting-a-path-to-ai-accountability.md)
- Company: [Anthropic](https://yomu.fyi/company/anthropic.md)
- Published: Jun 13, 2023
- Type: Explainer

Anthropic’s response to the NTIA’s Request for Comment on AI Accountability proposes processes and infrastructure for evaluating highly capable and general-purpose AI models. It recommends funding rigorous evaluations, requiring near-term disclosure of evaluation methods and results, and developing longer-term industry standards with agencies such as NIST. The framework would use capability-based risk thresholds, pre-registration of large training runs, confidential registries, technically literate and security-conscious third-party auditors, and external red teaming before advanced model release. It also calls for more interpretability research, acknowledges that regulations requiring interpretable models are currently infeasible, and seeks antitrust guidance for industry safety collaboration. Anthropic says these measures require collaboration among researchers, AI labs, regulators, auditors, and other stakeholders to build accountability and auditing mechanisms that support safe development and deployment.


[Newer posts](https://yomu.fyi/topic/model-evaluation/page/3.md)