An Introduction to Q-Learning Part 1

Summary

Value-based reinforcement learning centers on finding optimal policies indirectly through value functions. In these methods, practitioners define policy behavior manually, such as using greedy or epsilon-greedy policies, rather than training the policy directly. Value estimation relies on two primary formulations: the state-value function, which outputs expected returns from a state, and the action-value function, which evaluates state-action pairs. Because summing all future rewards across an entire trajectory is computationally demanding, algorithms adopt the Bellman equation alongside learning strategies like Monte Carlo or Temporal Difference (TD) learning. Monte Carlo updates value functions only after complete episodes using actual returns, whereas one-step TD learning bootstraps updates at each individual step using immediate rewards and discounted estimates of subsequent states.

Context

Reinforcement learning agents require optimal policies to maximize cumulative rewards, but estimating expected returns directly by summing all future rewards across state trajectories is computationally tedious.

Approach / What changed

Value-based reinforcement learning trains value functions (state-value or action-value) combined with hand-defined policies like greedy or epsilon-greedy selection, updating values across episodes via Monte Carlo returns or per-step via Temporal Difference bootstrapping and the Bellman equation.

Takeaways

• Value-based methods train a value function rather than the policy directly, using hand-specified rules such as greedy or epsilon-greedy strategies to select actions.
• State-value functions compute expected returns from a given state, whereas action-value functions calculate expected returns for specific state-action pairs.
• Monte Carlo methods update value estimates only after complete episodes using actual returns, while one-step Temporal Difference learning bootstraps per step using immediate rewards and next-state value estimates.

Related reading

• huggingface.co · May 20, 2022

  An Introduction to Q-Learning Part 2/2

  Q-Learning serves as a foundational off-policy reinforcement learning method for finding optimal action-value functions before advancing to deep reinforcement learning. The approach employs a tabular Q-function that updates state-action pairs at each step using temporal difference learning. Action selection balances exploration and exploitation via an epsilon-greedy strategy, where epsilon progressively decays as training proceeds. Updates compute a temporal difference target combining the immediate reward and the discounted maximum value of the subsequent state under a greedy policy. This discrepancy between the exploratory acting policy and the greedy updating policy classifies Q-Learning as an off-policy algorithm, enabling agents to learn optimal policies in environments like Frozen Lake and autonomous taxi navigation.

  Thomas Simonini

• huggingface.co · Jun 7, 2022

  Deep Q-Learning with Space Invaders

  Tabular Q-learning fails to scale to complex environments like Atari games due to massive observation spaces that make maintaining discrete state-action tables impractical. Deep Q-learning addresses this limitation by using a neural network to approximate Q-values for each possible action from input states. To process visual inputs effectively and capture temporal motion, raw game screens are grayscaled, resized to 84x84 pixels, and stacked in groups of four consecutive frames before passing through convolutional and fully connected layers. The training process stabilizes learning through experience replay buffers, fixed target networks that update periodically, and Double DQN architectures that decouple action selection from target evaluation. Agents can then be trained on environments like Space Invaders using frameworks such as RL-Zoo without tabular memory constraints.