Can we change an AI model’s architecture without retraining it from scratch?

Summary

The Transformer architecture’s memory cost scales with token count, making long-context AI agents expensive. In controlled comparisons of three state space models - Gated KalmaNet, Gated DeltaNet, and Mamba-2 - Gated KalmaNet performed best, offering variable-length inference without retraining. Priming identifies which attention layers can be replaced by state space models using a sliding-window attention proxy, then reuses pre-trained weights to convert them into hybrid architectures without retraining, reducing pretraining cost by over 99%. A 32-billion-parameter hybrid model matches the original Transformer’s reasoning quality while decoding up to twice as fast or reaching correct answers in two-thirds of wall-clock time. The team extended hybrid context to 256K tokens without additional training and released primed models at 8B and 32B scales along with training code, kernels, and a vLLM plugin under Apache 2.0.

Takeaways

1. 1

   Priming reduces the pretraining cost of hybrid architectures by over 99% compared to training from scratch.

2. 2

   A 32-billion-parameter hybrid model using GKA matches Transformer quality while decoding up to twice as fast or reaching answers in two-thirds of wall-clock time.

3. 3

   Under controlled comparisons, Gated KalmaNet outperforms Gated DeltaNet and Mamba-2, and offers variable-length inference without retraining.