---
title: "Can we change an AI model’s architecture without retraining it from scratch?"
description: "The Transformer architecture’s memory cost scales with token count, making long-context AI agents expensive. In controlled comparisons of three state space models - Gated KalmaNet, Gated DeltaNet, and Mamba-2 - Gated KalmaNet performed best, offering variable-length inference without retraining. Priming identifies which attention layers can be replaced by state space models using a sliding-window attention proxy, then reuses pre-trained weights to convert them into hybrid architectures without retraining, reducing pretraining cost by over 99%. A 32-billion-parameter hybrid model matches the original Transformer’s reasoning quality while decoding up to twice as fast or reaching correct answers in two-thirds of wall-clock time. The team extended hybrid context to 256K tokens without additional training and released primed models at 8B and 32B scales along with training code, kernels, and a vLLM plugin under Apache 2.0."
---

# Can we change an AI model’s architecture without retraining it from scratch?

[AWS](https://yomu.fyi/company/aws) · Luca Zancato · Oct 12, 2026

**Type:** Announcement

## Summary

The Transformer architecture’s memory cost scales with token count, making long-context AI agents expensive. In controlled comparisons of three state space models - Gated KalmaNet, Gated DeltaNet, and Mamba-2 - Gated KalmaNet performed best, offering variable-length inference without retraining. Priming identifies which attention layers can be replaced by state space models using a sliding-window attention proxy, then reuses pre-trained weights to convert them into hybrid architectures without retraining, reducing pretraining cost by over 99%. A 32-billion-parameter hybrid model matches the original Transformer’s reasoning quality while decoding up to twice as fast or reaching correct answers in two-thirds of wall-clock time. The team extended hybrid context to 256K tokens without additional training and released primed models at 8B and 32B scales along with training code, kernels, and a vLLM plugin under Apache 2.0.

## Context

The post from Amazon Web Services describes the high cost of training hybrid AI architectures that combine attention and state space models, and introduces ‘priming’ as a method to convert pre-trained Transformers into such hybrids without training from scratch.

## Approach / What changed

Priming proceeds in three stages: 1) layer selection using a sliding-window attention proxy to identify which attention layers can be replaced by state space models; 2) weight transfer and alignment, where SSM layers are initialized from the corresponding attention weights and a short distillation aligns each SSM to the behavior of the replaced layer; 3) end-to-end fine-tuning on next-token prediction. This avoids the need for pretraining from scratch.

## Takeaways

- Priming reduces the pretraining cost of hybrid architectures by over 99% compared to training from scratch.
- A 32-billion-parameter hybrid model using GKA matches Transformer quality while decoding up to twice as fast or reaching answers in two-thirds of wall-clock time.
- Under controlled comparisons, Gated KalmaNet outperforms Gated DeltaNet and Mamba-2, and offers variable-length inference without retraining.

**Tags:** [AWS](https://yomu.fyi/topic/aws), [LLMs](https://yomu.fyi/topic/llm), [Machine Learning](https://yomu.fyi/topic/machine-learning), [State Space Models](https://yomu.fyi/topic/state-space-models), [Transformer](https://yomu.fyi/topic/transformer)

- Source: [AWS](https://www.amazon.science/blog/can-we-change-an-ai-models-architecture-without-retraining-it-from-scratch)
- Source URL: https://www.amazon.science/blog/can-we-change-an-ai-models-architecture-without-retraining-it-from-scratch
- Ingested by Yomu: 2026-10-12T18:00:13.895Z

[Read original post](https://www.amazon.science/blog/can-we-change-an-ai-models-architecture-without-retraining-it-from-scratch)