MosaicLeaks: Can your research agent keep a secret?

Summary

MosaicLeaks examines privacy leakage in deep-research agents that combine private enterprise documents with external web retrieval. Queries that appear harmless individually can reveal sensitive facts when an observer combines them over multiple hops, producing intent, answer, or full-information leakage. The benchmark contains 1,001 multi-hop chains interleaving local and web questions, and compares baseline behavior with task-only reinforcement learning and Privacy-Aware Deep Research (PA-DR). Task-only training raises strict chain success from 48.7% to 59.3% but increases answer/full-information leakage from 34.0% to 51.7%, whereas PA-DR reaches 58.7% success with 9.9% leakage by rewarding stage-specific decisions and penalizing query-level privacy risk. The evaluation is controlled, using synthetic enterprise documents, a fixed web corpus, three company contexts, and one multi-hop question-answering harness.

Takeaways

1. 1

   Task-only reinforcement learning raises strict chain success from 48.7% to 59.3%, but increases answer/full-information leakage from 34.0% to 51.7%.

2. 2

   PA-DR achieves 58.7% strict chain success and reduces answer/full-information leakage to 9.9%, while issuing more web queries than the base model.

3. 3

   Situational rewards reach comparable task performance with roughly 5–6 times fewer generated training samples than outcome-only reinforcement learning, and PA-DR retains that sample-efficiency benefit.