Promptimus: Improving already good LLM prompts with zero manual engineering

Summary

Enterprise applications rely on mature large language model prompts that incorporate intricate business logic and regulatory constraints, making further optimization and cross-model migration challenging. Promptimus automates prompt refinement through a four-step iterative loop that evaluates candidate prompts against decomposed metric checkpoints, diagnoses failure bottlenecks, and generates targeted improvement strategies. The system operates either via full prompt rewrites in standard mode or through surgical, programmatic find-and-replace edits in edit mode to preserve structured taxonomies and schemas. Tested across multimodal classification and code generation tasks, the framework achieves performance gains with small development datasets of 20 to 50 samples. Amazon is making the model-agnostic optimization system available through Amazon Bedrock to support enterprise generative-AI model migration without manual engineering.

Takeaways

1. 1

   Promptimus decomposes user-defined Python metric functions into intermediate validation checkpoints to diagnose specific failure bottlenecks rather than relying on scalar scores.

2. 2

   For large prompts containing intricate business logic, edit mode applies surgical find-and-replace modifications programmatically with a 97.3% matching success rate without making LLM calls for edit application.

3. 3

   The framework operates effectively on small datasets of 20 to 50 samples, enabling automated prompt optimization and model migration across targets without requiring ground truth for all samples.