# Introducing Decision Transformers on Hugging Face 🤗

huggingface.co · Edward Beeching, Thomas Simonini · Mar 28, 2022

**Type:** Announcement

## Summary

Hugging Face has integrated the Decision Transformer architecture into its transformers library and Hub, expanding support for offline reinforcement learning. Unlike standard reinforcement learning methods that maximize returns by learning value functions or policies through active environment interaction, Decision Transformers model trajectories as conditional sequence problems. The model feeds past states, actions, and target returns-to-go over a context window into a GPT-2 autoregressive backbone to generate future actions. Evaluating pre-trained continuous control checkpoints requires standardizing observations with training set statistics and iteratively updating the target return based on received rewards. This sequence modeling paradigm enables practitioners to adjust agent behaviors and skill levels directly by specifying different target returns at runtime.

## Context

Online deep reinforcement learning requires agents to learn via trial and error either in the real world or inside complex, costly simulators that risk being exploited. Offline reinforcement learning circumvents interactive training by learning policies from static datasets of prior demonstrations, but standard offline approaches still face challenges such as counterfactual query handling.

## Approach / What changed

Hugging Face added Decision Transformers to the transformers library alongside pre-trained checkpoints for Gym environments like Hopper, Walker2D, and Halfcheetah. The architecture treats reinforcement learning as an autoregressive sequence modeling problem using a GPT-2 backbone conditioned on return-to-go, states, and actions to predict subsequent actions.

## Takeaways

- Decision Transformer models trajectory sequences autoregressively with a GPT architecture instead of relying on conventional value function fitting or policy gradients.
- Inference requires passing a context window of tokens (states, actions, returns-to-go, and timesteps) and normalizing inputs using the mean and standard deviation from training.
- Conditioning the model on target returns enables direct control over agent performance at test time, such as modulating opponent bot difficulty in games.

**Tags:** [Machine Learning](https://yomu.fyi/topic/machine-learning), [Open Source](https://yomu.fyi/topic/open-source), [Python](https://yomu.fyi/topic/python)

[Read original post](https://huggingface.co/blog/decision-transformers)