Promptimus: Improving already good LLM prompts with zero manual engineering

Summary

Enterprise applications rely on mature large language model prompts that incorporate intricate business logic and regulatory constraints, making further optimization and cross-model migration challenging. Promptimus automates prompt refinement through a four-step iterative loop that evaluates candidate prompts against decomposed metric checkpoints, diagnoses failure bottlenecks, and generates targeted improvement strategies. The system operates either via full prompt rewrites in standard mode or through surgical, programmatic find-and-replace edits in edit mode to preserve structured taxonomies and schemas. Tested across multimodal classification and code generation tasks, the framework achieves performance gains with small development datasets of 20 to 50 samples. Amazon is making the model-agnostic optimization system available through Amazon Bedrock to support enterprise generative-AI model migration without manual engineering.

Context

Enterprise large language model applications rely on complex prompts encoding domain knowledge and regulatory rules that are difficult to refine without causing regressions. Furthermore, rapid model releases require recurring prompt migrations across providers, where prompts tuned for one model often underperform on another due to differing instruction-following behaviors.

Approach / What changed

Promptimus executes an automated four-step optimization loop using user-defined Python metric functions, initial prompt templates, and small datasets. A metric analyzer decomposes evaluation functions into intermediate validation checkpoints to pinpoint failure bottlenecks. Depending on prompt size and structure, the system operates in standard mode for full rewrites or edit mode for targeted programmatic find-and-replace modifications across exact, fuzzy, and similarity match levels.

Takeaways

• Promptimus decomposes user-defined Python metric functions into intermediate validation checkpoints to diagnose specific failure bottlenecks rather than relying on scalar scores.
• For large prompts containing intricate business logic, edit mode applies surgical find-and-replace modifications programmatically with a 97.3% matching success rate without making LLM calls for edit application.
• The framework operates effectively on small datasets of 20 to 50 samples, enabling automated prompt optimization and model migration across targets without requiring ground truth for all samples.

Related reading

• Dropbox · Feb 26, 2026

  Using LLMs to amplify human labeling and improve Dash search relevance

  Dropbox Dash uses large language models (LLMs) to amplify human labeling efforts for training its search relevance and ranking models. By validating and optimizing LLM evaluators against a small set of human-labeled internal data, Dropbox creates massive, high-quality training datasets offline for production rankers like XGBoost without incurring high latency or latency costs at query time.

  Ilya Yakovlev,Andrew Cheung,Binoy Dash,Simran Jumani,Dmitriy Meyerzon,Mark Breitenbach,Ishan Mishra,Kazuaki Okumura,Mike White,Kevin Altschuler,Facundo Agriel,Ishan Mishra,Eric Wang,Dmitriy Meyerzon,Dmitriy Meyerzon

• Dropbox · Jun 25, 2026

  How we used DSPy to turn AI evaluations into better responses in Dash chat

  Dropbox improved its Dash chat agent by establishing an automated optimization loop powered by DSPy and LLM-as-judge evaluations. Engineers first calibrated their LLM judges against human-annotated interaction traces, then used those judges to systematically optimize the agent's system prompts via offline counterfactual replay. This automated workflow reduced incomplete responses by 26% while decreasing overall token consumption.