Teaching Sidekick to say no: automated data curation with LLM judge consensus

Summary

Sidekick’s customer-segmentation skill was fine-tuned on tens of thousands of successful, de-identified production queries, so it generated zero-result queries instead of refusing impossible requests such as finding customers by occupation. A small Toloka dataset of roughly 600 standard queries and 602 refusal annotations became a seed for four frontier LLM judges, which were calibrated against human-labeled examples and required to agree on both verdict and reasoning. The pipeline used mutually exclusive categories to resolve conflicting labels, filtered disagreements for human review, and fed accepted examples from later production traffic back into fine-tuning. With automated curation, the segmentation pass rate rose from 0.762 to 0.798 versus naive merging, while manual validation measured 86.3% refusal accuracy and a 4.6% false-positive rate. The resulting flywheel is being applied to other Sidekick skill models and data-quality problems.

Context

Sidekick’s production training corpus contained only successful queries and no refusals. As a result, the customer-segmentation model could not recognize impossible requests and instead generated queries returning zero results, misleading merchants into believing that no customers matched. Directly merging the production data with a small refusal dataset also created contradictory labels and limited improvement.

Approach / What changed

A roughly 600-query standard dataset and 602 refusal annotations from Toloka were used to calibrate four LLM judges. The judges classified production queries into mutually exclusive categories and accepted label changes only when all four agreed on both the decision and reasoning. Disagreements were filtered for human review, while accepted examples entered a recurring production-to-training data flywheel.

Takeaways

• A four-model consensus gate improved the segmentation pass rate from 0.762 with naive merging to 0.798 when both strategies used the same refusal-aware baseline.
• Manual validation measured refusal accuracy at 86.3% with a 4.6% false-positive rate for the curated model.
• The curation taxonomy separated requests that needed more context, required missing capabilities, belonged to another skill, or were ambiguous; disagreements were excluded rather than arbitrated automatically.

Related reading