Loading…
Topic
Transformer
3 posts about Transformer. Every summary links to the original.
- 3 posts
- 3 companies
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
Newest first
Posts
Can we change an AI model’s architecture without retraining it from scratch?
The Transformer architecture’s memory cost scales with token count, making long-context AI agents expensive. In controlled comparisons of three state space models - Gated KalmaNet, Gated DeltaNet, and Mamba-2 - Gated KalmaNet performed best, offering variable-length inference without retraining. Priming identifies which attention layers can be replaced by state space models using a sliding-window attention proxy, then reuses pre-trained weights to convert them into hybrid architectures without retraining, reducing pretraining cost by over 99%. A 32-billion-parameter hybrid model matches the original Transformer’s reasoning quality while decoding up to twice as fast or reaching correct answers in two-thirds of wall-clock time. The team extended hybrid context to 256K tokens without additional training and released primed models at 8B and 32B scales along with training code, kernels, and a vLLM plugin under Apache 2.0.
Putting a Transformer Inside Every Card Authorization
Ramp’s new card fraud model combines a small transformer pre-trained without fraud labels with an XGBoost stage to score each authorization in real time. The transformer reads a cardholder’s recent transactions and learns to predict the next transaction, producing embeddings and surprisal scores that capture how anomalous each field is. The model caught 32% more fraud dollars than its predecessor while flagging the same number of transactions, with average precision 12% higher than XGBoost alone. Most of the dollar gain comes from weighting XGBoost training examples by dollar amount, while the transformer improves ranking and catches more first fraudulent charges on a card. The transformer trains in under 90 minutes on four L40S GPUs and runs on CPU within the authorization latency budget after optimizations cut median model step time from 145 to 54 milliseconds.
Pruning LLMs Like a Physicist: Block Removal as an Ising Optimization Problem
Removing entire transformer blocks is a cheap way to speed up large language models, yielding predictable inference gains and stacking with quantization and other compression methods. The challenge is that block choices interact, making it a combinatorial problem rather than a simple ranking task. The authors reformulate block selection as a constrained binary optimization (CBO) problem, mapping it onto an Ising glass, a disordered spin system with all-to-all interactions and a fixed number of removed blocks. They note that mean-field methods, which treat each block independently, are inadequate for this interacting system. Block removal composes with quantization, low-rank compression, width pruning, and knowledge-distillation-based healing, fitting into a larger pipeline rather than competing with it. The paper is titled ‘LLM Compression by Block Removal with Constrained Binary Optimization.’
Related topics
All topics →Weekly digest
The best engineering reads, once a week.