An Introduction to Q-Learning Part 2/2

Summary

Q-Learning serves as a foundational off-policy reinforcement learning method for finding optimal action-value functions before advancing to deep reinforcement learning. The approach employs a tabular Q-function that updates state-action pairs at each step using temporal difference learning. Action selection balances exploration and exploitation via an epsilon-greedy strategy, where epsilon progressively decays as training proceeds. Updates compute a temporal difference target combining the immediate reward and the discounted maximum value of the subsequent state under a greedy policy. This discrepancy between the exploratory acting policy and the greedy updating policy classifies Q-Learning as an off-policy algorithm, enabling agents to learn optimal policies in environments like Frozen Lake and autonomous taxi navigation.

Takeaways

1. 1

   Q-Learning updates action-value estimates step by step using Temporal Difference learning rather than waiting for episode completion.

2. 2

   The algorithm is off-policy because it acts using an epsilon-greedy policy while updating Q-values using a greedy policy over the next state.

3. 3

   The epsilon-greedy mechanism manages exploration and exploitation by decaying the random exploration rate epsilon over the course of training.