# Deep Q-Learning with Space Invaders

huggingface.co · Thomas Simonini · Jun 7, 2022

**Type:** Explainer

## Summary

Tabular Q-learning fails to scale to complex environments like Atari games due to massive observation spaces that make maintaining discrete state-action tables impractical. Deep Q-learning addresses this limitation by using a neural network to approximate Q-values for each possible action from input states. To process visual inputs effectively and capture temporal motion, raw game screens are grayscaled, resized to 84x84 pixels, and stacked in groups of four consecutive frames before passing through convolutional and fully connected layers. The training process stabilizes learning through experience replay buffers, fixed target networks that update periodically, and Double DQN architectures that decouple action selection from target evaluation. Agents can then be trained on environments like Space Invaders using frameworks such as RL-Zoo without tabular memory constraints.

## Context

Tabular Q-learning functions well only in environments with small, discrete state spaces like FrozenLake-v1 and Taxi-v3. In Atari environments like Space Invaders, the observation space of 210x160x3 with 256 pixel values yields an immense state space of 256^100800, making Q-tables inefficient and non-scalable.

## Approach / What changed

Deep Q-learning replaces the Q-table with a parametrized neural network that estimates Q-values. States are preprocessed by grayscaling, cropping, downsizing to 84x84, and stacking four consecutive frames to address temporal limitations. The network uses convolutional layers followed by fully connected layers, trained via gradient descent on batches from an experience replay buffer with fixed Q-targets and Double DQN to stabilize training.

## Takeaways

- Processing raw visual frames by downsizing to 84x84, converting to grayscale, and stacking four consecutive frames enables convolutional networks to capture temporal motion information.
- Deep Q-learning stabilizes neural network updates against non-linear instability and bootstrapping by using experience replay buffers and separate fixed-target networks updated every C steps.
- Double DQN addresses the overestimation of Q-values by decoupling action selection via the DQN network from target Q-value calculation via the target network.

**Tags:** [Machine Learning](https://yomu.fyi/topic/machine-learning)

[Read original post](https://huggingface.co/blog/deep-rl-dqn)