Loading…
Topic
64 posts about Model Evaluation. Every summary links to the original.
See the idea
A benchmark can support two different verdicts: pass@1 on the first attempt, or pass@k with retries measured as cost per solved task. This comparison shows which reading matches your workload.
Compare assistive coding tools with agentic development environments to see where human control, automation, verification, and evaluation belong in the workflow.
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
Newest first
Anthropic outlines its preparations for the 2024 U.S. election cycle, the first in which generative AI tools are widely available. Since July 2023, it has updated its Usage Policy, strengthened misuse detection, and redirected users toward authoritative voting information. The policy prohibits campaigning, lobbying, election misinformation, targeting voting machines, and interference with vote counting or certification, while Claude's text-only output prevents it from generating image, audio, or video deepfakes. Controls include prompt modifications, API audits, possible account suspensions, cloud-provider coordination, red-teaming, Policy Vulnerability Testing, and automated evaluations of political parity, harmful-query refusals, misinformation prevention, and voter-profiling risks; a TurboVote pop-up handles voting-information requests because models lack real-time election data, while the system prompt states Claude's knowledge cutoff.
Etsy describes a supervised-learning system for detecting potentially violating listings at marketplace scale, alongside community reporting and other automated controls. Training data combines logged and annotated positives, hard negatives, and randomly sampled easy negatives, with multimodal textual and imagery signals split by time for progressive evaluation. The architecture uses BERT-based text encoding with ALBERT for reduced compute, EfficientNet image encoding, late concatenation of modality representations, a softmax output, and focal loss for class imbalance. Etsy evaluates candidates offline and in production through API-serving, canary releases, and champion-challenger A/B tests, while using proxy recall because full manual review is infeasible. Frequent retraining and monitoring support ongoing iteration; the methodology has identified and removed more than 100,000 violations, alongside other tools and services.
Etsy uses image vector representations for visual search and similar-item recommendations, making low-latency, memory-aware models important for online query inference and downstream retrieval. The work compares EfficientNetB0, a compact CNN, with EfficientFormer-l3, which downsamples multiple blocks and applies attention only in its final stage; its representation uses the last block’s eight-head attention and average-pools the sequence. Fine-tuning adds an embedding and multiple classification heads, while multitask training supervises taxonomy, color, and review-photo labels; both models train on two 16GB P100 GPUs. Evaluation rebuilds a brute-force candidate index after each epoch and measures Recall@5 and Recall@10 on interaction and intra-item retrieval datasets, with early stopping available. EfficientFormer-l3 improves reported offline recall by about 5%, 17%, and 1.8% across three tasks, while experiments report lifts in CTR, purchase rate, click-recall@100, and clicks.
Anthropic’s response to the NTIA’s Request for Comment on AI Accountability proposes processes and infrastructure for evaluating highly capable and general-purpose AI models. It recommends funding rigorous evaluations, requiring near-term disclosure of evaluation methods and results, and developing longer-term industry standards with agencies such as NIST. The framework would use capability-based risk thresholds, pre-registration of large training runs, confidential registries, technically literate and security-conscious third-party auditors, and external red teaming before advanced model release. It also calls for more interpretability research, acknowledges that regulations requiring interpretable models are currently infeasible, and seeks antitrust guidance for industry safety collaboration. Anthropic says these measures require collaboration among researchers, AI labs, regulators, auditors, and other stakeholders to build accountability and auditing mechanisms that support safe development and deployment.
The best engineering reads, once a week.
