---
title: "Transformers Model Integration Contracts"
description: "Transformers enabled PyTorch scaled dot-product attention for T5 on July 30, 2026, letting UMT5’s decoder attend to future tokens during training. The bug stayed in main for almost two months, showing why full integration tests cannot run on every pull request across more than 750 model types. The proposed Model Integration Contract combines a shrunk fixture from a real checkpoint’s configuration, a runnable usage path, and human-reviewed baselines, and holds when that path still produces the baselines. Fixtures preserve tokenizer and processor files and the original weights format, keep different layer kinds, and shrink a 1,227 MB UMT5 checkpoint to 66 MB or Llama 4 Scout from 108.6 billion parameters to 44.7 million. A five-month replay showed the UMT5 fixture flagged the regression under SDPA, while eager attention missed it; the MIC CI job reports failures without blocking pull requests."
---

# Transformers Model Integration Contracts

[Hugging Face](https://yomu.fyi/company/hugging-face) · Tarek Ziadé · Nov 3, 2026

**Type:** Problem & solution

## Summary

Transformers enabled PyTorch scaled dot-product attention for T5 on July 30, 2026, letting UMT5’s decoder attend to future tokens during training. The bug stayed in main for almost two months, showing why full integration tests cannot run on every pull request across more than 750 model types. The proposed Model Integration Contract combines a shrunk fixture from a real checkpoint’s configuration, a runnable usage path, and human-reviewed baselines, and holds when that path still produces the baselines. Fixtures preserve tokenizer and processor files and the original weights format, keep different layer kinds, and shrink a 1,227 MB UMT5 checkpoint to 66 MB or Llama 4 Scout from 108.6 billion parameters to 44.7 million. A five-month replay showed the UMT5 fixture flagged the regression under SDPA, while eager attention missed it; the MIC CI job reports failures without blocking pull requests.

## Context

Transformers supports over 750 model types. Its main test suite has 88,000+ CPU tests and a daily run adds 200,000+ GPU tests, covering pipelines, training, distributed execution, quantization and attention kernels. Full integration tests load real checkpoints on real hardware, so running them on every PR would require many GPUs all day. Scheduled runs can find a regression only after it has reached main. The UMT5 SDPA change was one such regression: it did not crash and stayed in main for almost two months.

## Approach / What changed

The proposed transformers-mic standard packages a Model Integration Contract with three parts: a fixture shrunk from a real checkpoint’s configuration while preserving layer kinds, tokenizer and processor files, and weights format; a runnable usage path; and human-reviewed baselines. CI runs the usage path on the fixture and compares outputs. Fixtures are published once and pinned, and a fleet dataset records revisions and package versions. On pull requests, MICs are selected from changed files. The tmic CLI drafts contracts and builds fixtures, with LLM-drafted usage code and baselines that require human review.

## Takeaways

- The UMT5 regression came from enabling SDPA for T5: the decoder could attend to future tokens, and the bug stayed in main for almost two months. A small CPU fixture flagged it on every replayed commit, but only when the contract used SDPA; the eager attention variant missed it.
- Fixtures shrink a real checkpoint’s configuration while keeping every layer kind and copying tokenizer and processor files byte for byte. The UMT5 fixture used 66 MB of weights versus 1,227 MB for the source checkpoint, and Llama 4 Scout shrank from 108.6 billion parameters to 44.7 million.
- In a weight-initialization study, random Whisper fixtures caught 18 of 24 injected faults and overfit fixtures caught all 24, but both missed an assisted-decoding bug that the real Whisper checkpoint exposed. Random weights are kept when they pass checks; overfitting is used for better sensitivity or readable answers.

## FAQ

### Why did the UMT5 bug pass a forward pass but fail the contract?

The bug did not crash a forward pass; it let UMT5's decoder attend to future tokens during training. The contract compared label log-probabilities and training loss against reviewed baselines, where the differences were 1.22 and about 0.20.

### What are the three parts of a Model Integration Contract?

A contract has a fixture, a usage path, and baselines. The fixture is a shrunk model published once and pinned; the usage path is a runnable Markdown example; the baselines are human-reviewed outputs recorded from that example. The contract holds if the usage path on the fixture still produces the baselines.

### Why keep the original weights file format for a fixture?

Keeping the original format ensures the conversion code that runs when a user loads the real checkpoint is still exercised. Otherwise the fixture would skip that code path.

**Tags:** [CI/CD](https://yomu.fyi/topic/ci-cd), [Model Evaluation](https://yomu.fyi/topic/model-evaluation), [Testing](https://yomu.fyi/topic/testing), [Transformers](https://yomu.fyi/topic/transformers)

- Source: [Hugging Face](https://huggingface.co/blog/transformers-mic)
- Source URL: https://huggingface.co/blog/transformers-mic
- Ingested by Yomu: 2026-11-03T09:00:14.676Z

[Read original post](https://huggingface.co/blog/transformers-mic)
