TypeSafe AI scales Jev to trillions of tokens in three days on Modal

Summary

TypeSafe AI’s Jev is a System One Model that outputs structured decisions with probabilities rather than generating text, taking state and questions as input and returning typed values. It uses a custom architecture, a parallel sampler for maximum efficiency, and a training method called Reinforcement Learning for Calibrated Decisions (RLCD). Unlike autoregressive LLMs, Jev produces outputs in parallel, enabling low-latency responses suitable for workloads like large-scale automation, fast browser use, and generative user interfaces. Launched on a Thursday, Jev topped OpenRouter’s leaderboard for contexts between 1K and 10K tokens one week later, handling 15.1% of requests on that platform. To meet demand exceeding one trillion tokens in three days, TypeSafe used Modal’s serverless platform for bursty, high-concurrency inference without managing their own infrastructure, gaining control over networking, routing, and autoscaling needed for ultra-low-latency serving.

Takeaways

1. 1

   Jev is a System One Model that outputs structured decisions with probabilities in parallel, not autoregressive text generation.

2. 2

   Jev topped OpenRouter’s leaderboard for 1K-10K token contexts one week after launch, handling 15.1% of requests.

3. 3

   Modal served over one trillion tokens for Jev within three days of launch using serverless infrastructure with autoscaling.

FAQ