How We Optimized the Qwen 3.6 Model for Our Junie Agent

Summary

Deploying the Junie coding agent locally on Apple MacBook M5 hardware requires overcoming steep performance bottlenecks during model execution. Engineers optimized the entire stack using Qwen3.6-27B rather than Qwen3.8-27B, which severely degrades without token-heavy reasoning enabled. At the agent layer, Junie appends requests directly into a rolling context and caches initial prompt prefixes to maximize KV-cache reuse across tasks. For model inference, the team disabled reasoning, selected 4-bit quantization, and patched MLX-VLM to execute self-attention prefill matrix operations using faster 8-bit instructions on the M5 processor. Combining these prefill improvements with simultaneous multi-token prediction and n-gram speculative decoding delivered up to a 2x generation speedup.

Context

Enabling local inference for the Junie coding agent on MacBook M5 hardware was constrained by slow prefill speeds, high token generation counts, and memory bottlenecks compared to discrete cloud GPUs.

Approach / What changed

Engineers integrated Qwen3.6-27B with 4-bit quantization, disabled reasoning, extended the agent's rolling context to preserve KV caches, and cached initial system prompt prefixes. They also patched MLX-VLM to use 8-bit matrix arithmetic for self-attention layers during prefill on the M5 and enabled both Multi-Token Prediction and n-gram speculative decoding.

Takeaways

• Patching MLX-VLM to run self-attention matrix operations in 8-bit precision on the M5 chip increased prefill speed by approximately 40%.
• Disabling reasoning mode in Qwen3.6-27B reduced token generation by 2x to 3x, yielding a 2x speedup on task execution with negligible impact on quality.
• Combining Multi-Token Prediction with a draft model alongside n-gram speculative decoding yielded up to a 2x speedup in generation.

Related reading

• Jetbrains · Aug 24, 2026

  Junie Can Now Run Entirely on Your Mac – No Credits, No Cloud

  JetBrains has introduced Junie Local, a feature that runs the Junie coding agent entirely on local hardware without cloud dependencies, subscriptions, or credit meters. Users execute a single command inside Junie to download a 4-bit quantized Qwen3.6-27B model, requiring an Apple M5 Mac with 64 GB of RAM. The team prioritized prefill throughput over pure generation speed by utilizing 8-bit arithmetic instructions on the M5 Neural Accelerator, which increased prefill throughput by approximately 40%. Performance optimizations also include KV-cache reuse, speculative decoding to double generation speed, and disabling reasoning to preserve performance. In JetBrains' evaluations, the local configuration performed comparably to cloud-based Sonnet 4.5 on everyday tasks while ensuring source code, diffs, and prompts never leave the local environment.

  Dmitry Savelev

• Meta · Aug 3, 2026

  GEM Training: How Meta Doubled the Efficiency of Its LLM-Scale Ads Foundation Model

  Meta's Generative Ads Recommendation Model (GEM) trains across thousands of GPUs using a hybrid architecture featuring trillions of sparse embedding parameters and billions of dense parameters. Standard training infrastructure built for large language models struggles with recommendation-specific workloads characterized by jagged user activity histories, asymmetric attention patterns, and numerical sensitivity. To overcome these bottlenecks, engineering teams introduced custom kernels like Jagged Flash Attention and Generalized Dot-Product Attention alongside MXFP8 mixed-precision execution. Pairing these kernel optimizations with topology-aware 5D parallelism doubled end-to-end training efficiency to 20–25% Model FLOPs Utilization while scaling total training FLOPs fourfold over twelve months.

• Grab · Jul 24, 2026

  Agent platform (Part 1): How we help Grab build and run AI agents at scale

  Grab scaled its internal AI tooling by transforming the infrastructure behind a single technical support bot into a standardized agent framework named LLM-Kit. The original Go-based bot revealed critical bottlenecks, including the lack of automated evaluation, painful provider migrations, fragmented observability, and excessive production scaffolding overhead. To address these frictions, engineers developed a FastAPI template pre-wired with LangGraph ReAct agents, remote Model Context Protocol servers, and built-in ROUGE, BLEU, and LLM-as-judge evaluation suites. The framework centralizes model access through an OpenAI-compatible GrabGPT Gateway and unifies observability across services, outbound calls, and agent steps using OpenTelemetry. This extracted scaffolding reduced day-one production setup time from two weeks to roughly an hour while supporting hundreds of production agents.

  Kendrick Tan

• Netflix · Jul 17, 2026

  In-House LLM Serving at Netflix

  Netflix established an in-house serving platform to run large language model inference directly inside existing production environments alongside traditional machine learning models. Built upon NVIDIA Triton Inference Server and vLLM, the unified architecture handles member-scale routing, candidate generation, feature fetching, and model execution over gRPC and OpenAI-compatible HTTP endpoints. The team shifted their primary engine from TensorRT-LLM to vLLM to support custom architectures, simpler debugging, and non-trivial constraint logic. Addressing production obstacles required patching Triton's frontend for guided decoding, pinning dependent library versions, rewriting logits processors in C++ for vLLM V1, and handling state machine resets during engine preemptions. The resulting platform unifies deployment flows while preserving operational stability across evolving model schemas.

  Netflix Technology Blog