DatabricksSudheesh Nair, Zach Richardson, Julia Powell, Tejas SundaresanProblem & solution
TinyFish builds infrastructure for enterprises that need agents to work across the live web and is developing its own specialized web-native model. The company uses Databricks AI Runtime to manage GPU compute for post-training with supervised fine-tuning and moving toward Direct Preference Optimization, MLflow for evaluation, and Model Serving for production deployment. TinyFish evaluates its model by measuring full agent task completion rather than individual responses, achieving a 77.6% success rate on its internal go/no-go benchmark where 1,691 of 2,178 tasks were completed successfully. Over a 15-day period, the web-native model handled 294,781 agent runs across 6.4 million tool steps at an estimated serving cost of $0.088 per run or $0.0040 per tool step. By consolidating training, evaluation, serving, and operational data on Databricks, TinyFish reduces infrastructure overhead and maintains a continuous development cycle from agent behavior to deployment.
TinyFish provides APIs for large-scale authenticated web agent workflows and is developing a specialized web-native model. The company needed to manage GPU compute for post-training, evaluate full agent task success, serve models in production, and track operational data without maintaining separate systems.
TinyFish consolidated its model development lifecycle on Databricks, using AI Runtime for managed GPU compute during post-training (supervised fine-tuning and DPO), MLflow for evaluation against full agent task success, and Databricks Model Serving for production deployment. Unity Gateway provides centralized cost tracking and governance for external foundation models.
TinyFish evaluates its web-native model by measuring full agent task completion rather than individual model responses, using a go/no-go benchmark that achieved a 77.6% success rate (1,691 of 2,178 tasks).
Over a 15-day period, TinyFish’s web-native model served 294,781 agent runs across 6.4 million tool steps at an estimated cost of 0.088 per run ( 0.0040 per tool step).
TinyFish uses Databricks AI Runtime for managed GPU infrastructure, MLflow for evaluation, and Model Serving for production, keeping training, evaluation, serving, and operational data in one environment to reduce infrastructure overhead.
TinyFish evaluates the full agent's task completion rather than individual model responses. It uses an internal go/no-go benchmark where a separate LLM judge reviews the original request, agent action history, and final result. A task passes only when the requested action is completed or the correct answer is returned with all requirements satisfied.
Over a 15-day period, the estimated model-serving cost was $0.088 per run or $0.0040 per tool step. These figures reflect serving costs for the web-native model only, not the full agent stack.
TinyFish started with supervised fine-tuning and is moving toward Direct Preference Optimization (DPO). These workloads run on Databricks-managed GPUs through AI Runtime.
Summaries are generated from the original by a language model. Check the original before relying on it.
See the idea
Amazon Bedrock is the managed reasoning tier in an agent architecture, not the whole system. This layered view shows the pipelines, governed tables, query layer, and guardrails around it.
Keep reading
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
One useful email. Unsubscribe anytime.
