Loading post…
Together AIBenchmark
GLM-5.3 vs. GLM-5.3 Flash on DeepSWE: Cost, Coding, and Routing
Summary
The GLM-5.3 Flash, a distilled variant of the full GLM-5.3, achieves a pass@1 of 63.4% on the DeepSWE benchmark at $0.24 per rollout, 17 times cheaper than the full model’s 69.0% at $3.99. The pass@4 gap narrows to 2.6 points (87.6% versus 85.0%). Distillation removed consistency rather than capability: the Flash solves 93 of the 99 tasks the full model solves, but its ability to convert extra effort into a win is lost (46% win rate on flaky tasks versus 61%). A cascade that runs Flash first and escalates to the full model only when the answer is rejected solves 80.9% of tasks at $1.70 each, 57% lower cost than using the full model alone.
Context
The post benchmarks the full GLM-5.3 model against its smaller, distilled GLM-5.3 Flash variant on the DeepSWE software-engineering benchmark, analyzing cost, consistency, and routing strategies across 900 rollouts.
Approach / What changed
Use GLM-5.3 Flash as the first attempt and escalate to the full GLM-5.3 only when tests reject the answer, achieving 80.9% solve rate at $1.70 per task.
Takeaways
GLM-5.3 Flash costs 17x less per rollout than GLM-5.3 ($0.24 vs $3.99) and solves 264 tasks per $100 spent versus 17.
Distillation removed consistency rather than capability: the Flash solves 94% of tasks the full model solves, but on flaky tasks its pass rate is independent of run length (46% win rate vs 61%).
A cascade routing strategy (Flash first, full model on rejection) achieves 80.9% solve rate at $1.70 per task, a 57% cost reduction over using the full model alone.
FAQ
How much does GLM-5.3 Flash cost per rollout compared to GLM-5.3?
GLM-5.3 Flash costs $0.24 per rollout, 17 times cheaper than GLM-5.3 at $3.99.
What is the pass@4 gap between the two models?
At pass@4, GLM-5.3 achieves 87.6% and GLM-5.3 Flash achieves 85.0%, a gap of only 2.6 points.
What does the cascade routing strategy entail?
Run GLM-5.3 Flash first and escalate to GLM-5.3 only when the answer is rejected. This cascade solves 80.9% of DeepSWE tasks at $1.70 each.
Source and freshness
- Source
- together.ai
- Published
- Added to Yomu
Summaries are generated from the original by a language model. Check the original before relying on it.
Keep reading
Related reading
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
The best engineering reads, once a week
One useful email. Unsubscribe anytime.