Introducing Decision Transformers on Hugging Face 🤗

Summary

Hugging Face has integrated the Decision Transformer architecture into its transformers library and Hub, expanding support for offline reinforcement learning. Unlike standard reinforcement learning methods that maximize returns by learning value functions or policies through active environment interaction, Decision Transformers model trajectories as conditional sequence problems. The model feeds past states, actions, and target returns-to-go over a context window into a GPT-2 autoregressive backbone to generate future actions. Evaluating pre-trained continuous control checkpoints requires standardizing observations with training set statistics and iteratively updating the target return based on received rewards. This sequence modeling paradigm enables practitioners to adjust agent behaviors and skill levels directly by specifying different target returns at runtime.

Context

Online deep reinforcement learning requires agents to learn via trial and error either in the real world or inside complex, costly simulators that risk being exploited. Offline reinforcement learning circumvents interactive training by learning policies from static datasets of prior demonstrations, but standard offline approaches still face challenges such as counterfactual query handling.

Approach / What changed

Hugging Face added Decision Transformers to the transformers library alongside pre-trained checkpoints for Gym environments like Hopper, Walker2D, and Halfcheetah. The architecture treats reinforcement learning as an autoregressive sequence modeling problem using a GPT-2 backbone conditioned on return-to-go, states, and actions to predict subsequent actions.

Takeaways

• Decision Transformer models trajectory sequences autoregressively with a GPT architecture instead of relying on conventional value function fitting or policy gradients.
• Inference requires passing a context window of tokens (states, actions, returns-to-go, and timesteps) and normalizing inputs using the mean and standard deviation from training.
• Conditioning the model on target returns enables direct control over agent performance at test time, such as modulating opponent bot difficulty in games.

Related reading

• huggingface.co · Jan 21, 2022

  Welcome Stable-baselines3 to the Hugging Face Hub 🤗

  Hugging Face announced an official integration with Stable-Baselines3, a popular PyTorch library for training and testing Deep Reinforcement Learning agents across diverse environments like Gym, Atari, MuJoco, and Procgen. The integration enables researchers and developers to host their saved reinforcement learning checkpoints on the Hugging Face Hub and download pre-trained community models. Interacting with the Hub requires installing the huggingface_hub and huggingface_sb3 packages, which supply helper methods for authentication, downloading, and uploading. Practitioners can retrieve checkpoint zip files using the load_from_hub function by providing the target repository identifier and filename before loading them into Stable-Baselines3 algorithms. Furthermore, users authenticated via CLI or notebook login can train policies such as PPO and publish their saved zip files to the Hub using push_to_hub.

  Thomas Simonini

• huggingface.co · Mar 9, 2021

  Hugging Face Reads, Feb. 2021 - Long-range Transformers

  Standard transformer models face severe memory and compute bottlenecks due to the quadratic scaling of self-attention with sequence length. To scale beyond short sequence limits, efficient architectures introduce varied mechanisms such as custom sparse attention patterns, compressed recurrence, low-rank projections, and kernel approximations. Longformer integrates local dilated windowed attention with task-specific global tokens, functioning as a drop-in replacement that enables standard pre-trained models to process long inputs without full retraining. Concurrently, Compressive Transformers extend temporal memory by compressing discarded activations, while models like Linformer and Performer reduce computational complexity through projection and kernel estimation. Progress in long-range modeling also reveals that incrementally training models from short to longer sequences accelerates convergence and improves downstream performance.