Deep Q-Learning with Space Invaders

Summary

Tabular Q-learning fails to scale to complex environments like Atari games due to massive observation spaces that make maintaining discrete state-action tables impractical. Deep Q-learning addresses this limitation by using a neural network to approximate Q-values for each possible action from input states. To process visual inputs effectively and capture temporal motion, raw game screens are grayscaled, resized to 84x84 pixels, and stacked in groups of four consecutive frames before passing through convolutional and fully connected layers. The training process stabilizes learning through experience replay buffers, fixed target networks that update periodically, and Double DQN architectures that decouple action selection from target evaluation. Agents can then be trained on environments like Space Invaders using frameworks such as RL-Zoo without tabular memory constraints.

Context

Tabular Q-learning functions well only in environments with small, discrete state spaces like FrozenLake-v1 and Taxi-v3. In Atari environments like Space Invaders, the observation space of 210x160x3 with 256 pixel values yields an immense state space of 256^100800, making Q-tables inefficient and non-scalable.

Approach / What changed

Deep Q-learning replaces the Q-table with a parametrized neural network that estimates Q-values. States are preprocessed by grayscaling, cropping, downsizing to 84x84, and stacking four consecutive frames to address temporal limitations. The network uses convolutional layers followed by fully connected layers, trained via gradient descent on batches from an experience replay buffer with fixed Q-targets and Double DQN to stabilize training.

Takeaways

• Processing raw visual frames by downsizing to 84x84, converting to grayscale, and stacking four consecutive frames enables convolutional networks to capture temporal motion information.
• Deep Q-learning stabilizes neural network updates against non-linear instability and bootstrapping by using experience replay buffers and separate fixed-target networks updated every C steps.
• Double DQN addresses the overestimation of Q-values by decoupling action selection via the DQN network from target Q-value calculation via the target network.

Related reading

• huggingface.co · May 20, 2022

  An Introduction to Q-Learning Part 2/2

  Q-Learning serves as a foundational off-policy reinforcement learning method for finding optimal action-value functions before advancing to deep reinforcement learning. The approach employs a tabular Q-function that updates state-action pairs at each step using temporal difference learning. Action selection balances exploration and exploitation via an epsilon-greedy strategy, where epsilon progressively decays as training proceeds. Updates compute a temporal difference target combining the immediate reward and the discounted maximum value of the subsequent state under a greedy policy. This discrepancy between the exploratory acting policy and the greedy updating policy classifies Q-Learning as an off-policy algorithm, enabling agents to learn optimal policies in environments like Frozen Lake and autonomous taxi navigation.

  Thomas Simonini

• huggingface.co · May 18, 2022

  An Introduction to Q-Learning Part 1

  Value-based reinforcement learning centers on finding optimal policies indirectly through value functions. In these methods, practitioners define policy behavior manually, such as using greedy or epsilon-greedy policies, rather than training the policy directly. Value estimation relies on two primary formulations: the state-value function, which outputs expected returns from a state, and the action-value function, which evaluates state-action pairs. Because summing all future rewards across an entire trajectory is computationally demanding, algorithms adopt the Bellman equation alongside learning strategies like Monte Carlo or Temporal Difference (TD) learning. Monte Carlo updates value functions only after complete episodes using actual returns, whereas one-step TD learning bootstraps updates at each individual step using immediate rewards and discounted estimates of subsequent states.