An Introduction to Q-Learning Part 2/2

Summary

Q-Learning serves as a foundational off-policy reinforcement learning method for finding optimal action-value functions before advancing to deep reinforcement learning. The approach employs a tabular Q-function that updates state-action pairs at each step using temporal difference learning. Action selection balances exploration and exploitation via an epsilon-greedy strategy, where epsilon progressively decays as training proceeds. Updates compute a temporal difference target combining the immediate reward and the discounted maximum value of the subsequent state under a greedy policy. This discrepancy between the exploratory acting policy and the greedy updating policy classifies Q-Learning as an off-policy algorithm, enabling agents to learn optimal policies in environments like Frozen Lake and autonomous taxi navigation.

Context

Learning reinforcement learning value-based methods requires transitioning from theoretical Monte Carlo and Temporal Difference concepts to practical tabular algorithms before advancing to Deep Q-Learning.

Approach / What changed

Train an action-value Q-function using a Q-table initialized to zero, select actions via an epsilon-greedy exploration strategy, and update state-action values after each transition using a TD target formed by the immediate reward and the discounted greedy next-state value.

Takeaways

• Q-Learning updates action-value estimates step by step using Temporal Difference learning rather than waiting for episode completion.
• The algorithm is off-policy because it acts using an epsilon-greedy policy while updating Q-values using a greedy policy over the next state.
• The epsilon-greedy mechanism manages exploration and exploitation by decaying the random exploration rate epsilon over the course of training.

Related reading

• huggingface.co · Jun 30, 2022

  Policy Gradient with PyTorch

  Policy gradient algorithms optimize reinforcement learning policies directly without learning intermediate action-value functions. Instead of assigning discrete Q-values to actions, these methods parameterize a stochastic policy and apply gradient ascent on an objective score function. This setup eliminates manual exploration tuning, resolves perceptual aliasing in identical states, and naturally accommodates continuous or high-dimensional action spaces. The Reinforce Monte Carlo policy gradient algorithm operationalizes this approach by collecting full episodic trajectories and adjusting parameters along the gradient of the log-action probabilities scaled by return. Practitioners implement this algorithm using PyTorch to evaluate agent robustness across environments like CartPole-v1, PixelCopter, and Pong.

  Thomas Simonini

• huggingface.co · May 18, 2022

  An Introduction to Q-Learning Part 1

  Value-based reinforcement learning centers on finding optimal policies indirectly through value functions. In these methods, practitioners define policy behavior manually, such as using greedy or epsilon-greedy policies, rather than training the policy directly. Value estimation relies on two primary formulations: the state-value function, which outputs expected returns from a state, and the action-value function, which evaluates state-action pairs. Because summing all future rewards across an entire trajectory is computationally demanding, algorithms adopt the Bellman equation alongside learning strategies like Monte Carlo or Temporal Difference (TD) learning. Monte Carlo updates value functions only after complete episodes using actual returns, whereas one-step TD learning bootstraps updates at each individual step using immediate rewards and discounted estimates of subsequent states.