Loading…
Together AI
AI cloud platform for training, fine-tuning, and running open and custom large language models. Offers inference APIs, dedicated GPU clusters, and decentralized compute.
- 34 posts summarised
- latest
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
From their blog
Latest articles
Meta Muse Spark Open is live on Together AI
Meta Muse Spark Open, an open-weight model for agentic coding, is now available on Together AI as a Day 0 launch partner with a 1-million-token context window on serverless and dedicated infrastructure. The model is trained on real developer work including multi-file refactors, debugging sessions, and long-running tasks, leading to higher first-attempt accuracy and reliable tool calling. It features long-horizon autonomy with planning, observation, and self-correction, computer use for desktop and mobile, multimodal input accepting text, image, audio, and video, and intent commentary that states actions before execution for auditability. Early access customers Factory and Ollama tested the model pre-launch and are offering it to their users from day zero. Serverless pricing is $1.25 per million input tokens, $4.25 per million output tokens, and $0.15 per million cached input tokens.
Expanding our enterprise inference capacity with IBM Cloud and NVIDIA
Together AI is collaborating with IBM and NVIDIA to scale enterprise-grade AI inference using a dedicated cluster of NVIDIA B300 GPUs on IBM Cloud with NVIDIA Spectrum-X Ethernet networking. This is the first large-scale inference cluster of its kind on IBM Cloud, and Together AI is the first customer running on it. Together AI operates the inference layer, IBM provides the cloud, and NVIDIA delivers the silicon and networking. The company serves hundreds of trillions of tokens per month to over a million developers, and demand continues to climb. Enterprises choose open models to keep their data sovereign and achieve frontier-level performance at a fraction of closed-model cost.
Together Link: open models in the harness you already use. Start with one command today.
Together Link connects coding agents to over 40 production-tier open models on Together AI, cutting model spend by over 50% compared to premium closed models. It integrates with Claude Code, Claude Desktop, Codex, OpenCode, and Pi without requiring any workflow changes. Open models such as Kimi K3, GLM 5.3, DeepSeek V4.1 Flash, and GLM 5.3 Flash now handle both complex architectural tasks and everyday fixes at a fraction of the per-token cost. A per-session router reads each session’s first task and routes it to either a fast, low-cost model or a frontier model; when no Anthropic key is provided it routes between GLM 5.3 and GLM 5.3 Flash, and Together AI serves the largest share of OpenRouter tokens for these models. Billing uses the existing Together API key with pay-as-you-go or credit packs, and a per-session tracker shows what the session would have cost on Opus 5.5.
Canary rollouts: upgrade models in production without downtime
Deploying a new model checkpoint in production typically requires a tradeoff: a hard swap exposes all users to regressions at once, while a DIY staged cutover relies on manual traffic shifting. Together AI’s rollout system automates this by moving traffic between source and target deployments through configurable canary steps, such as 5%, 25%, 50%, and 100%. Each step runs health checks before shifting traffic, enforces a propagation wait for routing caches, and drains the source only after traffic has moved. Optional metric gates, like p95 latency, run after each step; if a gate trips, the rollout pauses automatically and waits for human action. In a real canary from Qwen2.5-7B to Qwen3.5-9B, a 137% p95 regression was caught at 10% traffic, allowing reversal with no failed requests.
How a global fintech scaled coding agent traffic with Dedicated Model Inference
A global fintech runs its coding assistant on GLM 5.2 through Together’s Dedicated Model Inference (DMI), but spiky engineering-hours traffic outpaced static capacity planning. DMI gives the customer’s teams direct control over endpoint provisioning, scaling, configuration, model changes, and performance data through APIs, UI, and CLI. The workload is peak-load and relatively low-TPS, with ISL p50/p90/p95 of roughly 81K/163K/178K tokens and RPS p50/p90/p95 of 3/6/7, so concurrency headroom mattered more than raw throughput; the deployment used dozens of B200s, multiple replicas, and 256K context. When a reshaped endpoint later reached near-100% prefill capacity, requests queued for one to three minutes and decode throughput fell to roughly 5 tokens per second, the team restored cache-session-aware routing and widened max-inflight-per-worker, pushing the change same-day with zero downtime.
Migrating from closed to open source models, Together
The post presents a five-stage playbook for moving production workloads from closed-source models to open-source models: discover, evaluate, adapt, decide, and take the system to production. It argues that managed services, model-agnostic harnesses, gateways, and tooling can reduce migration complexity, potentially making the effort weeks to months rather than months to years. Discovery combines workload definition, relevant benchmark filtering, cost-per-task and runtime analysis, playground testing, and narrowing candidates to two or three models. Evaluation separates accuracy from performance and recommends replaying real traffic against explicit goals for cost, speed, and quality, while adaptation may involve prompts, inference settings, context, and the surrounding harness. The excerpt frames migration as iterative rather than assuming a model passes unchanged.
Together AI expands fine-tuning service with more models, live metrics, and finer controls
Together AI is expanding its Fine-Tuning service to cover newer open-weight models and more of the workflow from dataset inspection through model selection. The update adds live training and evaluation metrics through the API, CLI, dashboard, and Python SDK, including curves that can be compared across in-flight jobs and underlying series that can be exported. Training controls now include Expert LoRA for mixture-of-experts layers, early stopping based on validation loss, and arbitrary effective batch sizes through gradient accumulation. The post reports that Expert LoRA recalled up to 89% of 200 invented facts versus 15% for attention-only adapters, while listed LoRA training prices fall by 30% to 70% for selected models, including the gpt-oss series.
Introducing preemptible compute: the same compute, half the price
Together GPU Clusters is introducing preemptible compute in public preview for Kubernetes clusters in all regions, offering interruptible GPU capacity at a flat 50% of the on-demand rate. Preemptible nodes use the same NVIDIA accelerated compute, can be reclaimed when unused capacity is needed, and receive a maximum five-minute drain window for checkpointing and graceful exit. Clusters retain the requested preemptible target and automatically refill as capacity returns, while usage is metered every one to two minutes and billed sub-hourly. The feature suits resumable, retryable, or requeueable work such as experiments, batch inference, and temporary bursts, but critical services, uncheckpointed multi-day runs, and strict-SLO serving should remain on standard nodes; users must schedule eligible pods with the together.ai/compute-class=preemptible label.
To Infinity and Beyond: ThunderKittens Now on NVIDIA Vera Rubin NVL72!
Together’s kernels team ported ThunderKittens to NVIDIA Vera Rubin NVL72 and investigated why existing Blackwell GEMMs underuse the platform. The work targets NVFP4 and FP8 GEMMs: although Rubin preserves Blackwell’s programming model, the original kernels reach only about 42.1% and 44.4% of roofline because tensor cores consume operands twice as fast as the old pipeline supplies them. The team integrates Rubin’s 64-byte K step, exclusive tensor memory extending to 576 columns, oversized shared memory up to 328 KiB, B-side collector buffers, and earlier A-release signaling into its NVFP4 kernel. The resulting kernel exceeds 22 PFLOPS and is described as competitive with cuBLAS and CuTe DSL, while the post also compares Rubin’s specifications with NVIDIA HGX B200.
The Open Source AI Stack
The post presents an open model AI stack for developers moving from closed-source models, focusing on ownership, control, and economics. It organizes the stack into independent layers: model, inference, gateways and routers, harness, and tools such as Skills and MCP. Models generate responses from prompts; large models generally handle ambiguity, multi-step reasoning, and long context better, while small models can be faster and cheaper for tightly specified work. The separation lets developers choose providers and models independently and experiment with newly released models without rebuilding their workflow, while the post uses Kimi K3 and GLM 5.3 Flash to illustrate differences in active parameters, capability, speed, and cost.
GLM-5.3 vs. GLM-5.3 Flash on DeepSWE: Cost, Coding, and Routing
The GLM-5.3 Flash, a distilled variant of the full GLM-5.3, achieves a pass@1 of 63.4% on the DeepSWE benchmark at $0.24 per rollout, 17 times cheaper than the full model’s 69.0% at $3.99. The pass@4 gap narrows to 2.6 points (87.6% versus 85.0%). Distillation removed consistency rather than capability: the Flash solves 93 of the 99 tasks the full model solves, but its ability to convert extra effort into a win is lost (46% win rate on flaky tasks versus 61%). A cascade that runs Flash first and escalates to the full model only when the answer is rejected solves 80.9% of tasks at $1.70 each, 57% lower cost than using the full model alone.
GLM-5.3 vs. GPT-5.6 Sol on DeepSWE: Cost, Coding, and Routing
GLM-5.3 and GPT-5.6 Sol were evaluated on 113 DeepSWE tasks with 904 rollouts, comparing accuracy, cost, and failure patterns. Sol edges ahead at pass@1 (72.7% vs 69.0%), but GLM-5.3 leads at pass@4 (87.6% vs 85.8%) at $3.99 per rollout, half Sol’s $8.37 cost and yielding 17 solves per $100 versus Sol’s 9. Their low per-task correlation of 0.43 makes a cascade effective: GLM-5.3 first, then Sol on test failure, achieving 85.9% solved at $6.61 per task. Sol runs faster (19 minutes, 61 steps vs 35 minutes, 124 steps) and achieves higher reliability (61 tasks solved all four tries vs 48), but it breaks passing tests in 20% of failures versus GLM-5.3’s 11%.
GLM-5.3 vs. Claude Fable 5 on DeepSWE: Cost, Coding, and Routing
GLM-5.3 and Claude Fable 5 were compared on the DeepSWE benchmark across 113 tasks with four trials each, totaling 904 rollouts. On first-attempt accuracy, Fable 5 scored 69.7% and GLM-5.3 scored 69.0%, a statistical tie inside error bars. GLM-5.3 led on pass@2 (81.1% vs. 77.1%) and pass@4 (87.6% vs. 84.1%), and cost $3.99 per rollout versus Fable’s $21.63, a 5.4x price gap. Per $100, GLM-5.3 solved 17 tasks while Fable solved 3. Both models regress test suites in only 11% of failures, and GLM-5.3 averages 35 minutes per rollout with 80k output tokens versus Fable’s 34 minutes and 114k tokens. The models are near-substitutes with 0.65 per-task correlation; GLM-5.3 won five of eight domains including JavaScript and concurrency, while Fable excelled only in Rust and serialization, making GLM-5.3 the practical default.
DeepSeek V4 Pro 0813 vs GPT-5.6 Sol on DeepSWE: Cost, Coding, and Routing
DeepSeek V4 Pro 0813 costs $0.24 per rollout and achieves 62.8% pass@1 on DeepSWE, while GPT-5.6 Sol reaches 72.7% pass@1 at $8.37 per rollout—a 35x price gap. At four attempts, DeepSeek V4 Pro 0813 surpasses Sol, with 88.5% pass@4 versus 85.8%. Sol is faster and more reliable, completing a rollout in 17 minutes and 53 steps, and breaking tests in only 11% of its failures. A cascade strategy—running DeepSeek V4 Pro 0813 first and escalating to Sol on test failure—solves 83.0% of tasks at $3.35 each, outperforming either model alone. Per $100, DeepSeek V4 Pro 0813 solves 260 tasks versus Sol’s 9, making it the value leader for budget-constrained workloads.
DeepSeek V4 Pro 0813 vs Claude Fable 5 on DeepSWE: Cost, Coding, and Routing
The DeepSWE benchmark comparison between DeepSeek V4 Pro 0813 and Claude Fable 5 reveals a trade-off between cost and accuracy. Claude Fable 5 leads on the first attempt with 69.7% pass@1 versus DeepSeek V4 Pro’s 62.8%, but costs 90 times more per rollout ($21.63 vs $0.24). Under repeated attempts, DeepSeek V4 Pro surpasses Fable, achieving 88.5% pass@4 compared to Fable’s 84.1%. The two models show unusually low per-task correlation (0.39), meaning they fail on different tasks. A cascade strategy that runs DeepSeek V4 Pro first and escalates to Claude Fable 5 only on failures solves 82.7% of tasks at $8.28 per task, outperforming either model used alone.
A/B test models in production
A/B testing model changes in production by sending fixed shares of live endpoint traffic to one control and up to 20 variants, allowing teams to compare user-facing outcomes rather than relying only on shadow-traffic checks. The endpoint router keeps variants out of the base traffic split, resamples requests that reach the control, and treats experiment percentages as absolute shares independent of replica counts or autoscaling. A 95/5 experiment can be updated to 80/20 and then 50/50 by resending the complete member list; an etag rejects stale concurrent updates. Measurement joins per-deployment platform metrics with product signals using the deployment ID in response metadata. To finish, a winning variant is promoted through a blue-green rollout before deleting the experiment, while deletion alone returns traffic to the control and can unwind a losing test.
DeepSeek-V4 Flash 0731 vs GPT-5.6 Luna on DeepSWE: Cost and Coding
On the DeepSWE coding benchmark, GPT-5.6 Luna achieves 67.2% pass@1 versus DeepSeek-V4 Flash’s 53.3%, a 14-point gap. DeepSeek costs $0.10 per task, roughly six times less than Luna’s $0.61, delivering 532 solves per $100 versus Luna’s 110. A cascade that runs DeepSeek first and escalates only on failure solves 78.9% of tasks at $0.385 each, beating Luna’s solo accuracy and cutting cost by 37%. DeepSeek fails more gracefully (9% broken tests vs 15%) but is slower and dramatically weaker on JavaScript (35% vs 60%). Luna dominates seven of eight task domains, while DeepSeek leads only on query and config languages, illustrating how inexpensive models can combine with flagships for superior price-performance.
Kimi K3: the complete developer guide
Kimi K3 is presented as Moonshot AI’s 2.8-trillion-parameter open-weights model for long-horizon coding, end-to-end knowledge work, and deep reasoning, served through Together AI’s OpenAI-compatible API. Its architecture combines Kimi Delta Attention (KDA), Attention Residuals, and Stable LatentMoE, which activates 16 of 896 experts per token—roughly 2%—and supports a 1M-token context. The implementation guide shows Python calls for configurable reasoning effort, reasoning-free instant mode, streaming, multi-image vision, strict JSON-schema output, and function tools, while noting automatic prefix caching. It also covers production-oriented details such as generous max_tokens for reasoning-heavy structured output, a 100 MB request-body limit for vision, tool-call history handling, and deployment from US infrastructure.
Autoscaling endpoints for LLM inference
Dedicated LLM inference deployments face a trade-off between paying for idle GPUs and suffering sharp p95 latency when replicas hit batching limits. The autoscaling loop converts observed load into desired replicas with ceil(N × observed/target), then applies scale-up and scale-down windows, replica bounds, and GPU placement. It presents inflight_requests as a leading, engine-agnostic default, while TTFT and end-to-end latency target responsiveness but trail demand, and GPU or token utilization prioritize efficiency with less latency headroom. Cold starts can take several minutes because replicas require GPU placement, weight downloads, VRAM loading, and warm-up, so policies should scale early; streaming is required for TTFT, decoding speed, and throughput-per-replica metrics, whereas end-to-end latency supports both streaming and non-streaming workloads.
Together AI announces strategic partnership with Moonshot AI to natively serve Kimi models
Together AI announces a strategic partnership with Moonshot AI to serve Kimi models as a launch platform, beginning with Kimi K3 and extending to future open-weights releases. Kimi K3 is described as a 2.8-trillion-parameter sparse Mixture-of-Experts model with native vision and a 1M-token context window, using Kimi Delta Attention, Attention Residuals, per-head Muon, and quantile-based expert load balancing. Together says developers can access the models through Serverless, Provisioned Throughput, and Dedicated Inference, with US-hosted infrastructure, zero data retention, autoscaling, and a 99% uptime SLA for Provisioned Throughput. The partnership also supports post-training through full-weight and LoRA reinforcement learning and supervised fine-tuning, with checkpoints deployable directly to inference; Kimi K3 is available at launch, while future Moonshot releases are planned for the same platform.
Topics they write about
All topics →Weekly digest
The best engineering reads, once a week.