Context engineering case studies: Etsy-specific question answering

Summary

The post examines whether prompt engineering can provide truthful Etsy-specific answers for AI-assisted onboarding without fine-tuning an LLM, using Travel & Entertainment (T&E) policy questions and an external Etsy community-forum use case. For the internal pilot, T&E documents were indexed, embedding-based search retrieved relevant sections, and those sections were added to each question; a manually curated 40-question evaluation found approximately 86% correct or satisfactory answers, with 14% wrong or misleading. Explicit uncertainty instructions sometimes avoided hallucinations but also missed available answers, while asking for an explanation or chain-of-thought reasoning could improve results, though complex reasoning required careful setup. The experiments conclude that prompt engineering is a practical alternative to parameter-based fine-tuning, but requesting specific source snippets remains important for flagging potential hallucinations.

Takeaways

1. 1

   A manually curated evaluation of 40 T&E question-and-answer pairs judged approximately 86% of generated answers correct or satisfactory; 14% were wrong or misleading.

2. 2

   Embedding-based search retrieved Etsy-specific text sections for each query and appended them to the question, providing task-specific context without changing model weights.

3. 3

   Asking for specific source snippets helped flag potential hallucinations, while uncertainty instructions could prevent confident errors but also produce unhelpful admissions of ignorance.