# An Introduction to Q-Learning Part 2/2

huggingface.co · Thomas Simonini · May 20, 2022

**Type:** Tutorial

## Summary

Q-Learning serves as a foundational off-policy reinforcement learning method for finding optimal action-value functions before advancing to deep reinforcement learning. The approach employs a tabular Q-function that updates state-action pairs at each step using temporal difference learning. Action selection balances exploration and exploitation via an epsilon-greedy strategy, where epsilon progressively decays as training proceeds. Updates compute a temporal difference target combining the immediate reward and the discounted maximum value of the subsequent state under a greedy policy. This discrepancy between the exploratory acting policy and the greedy updating policy classifies Q-Learning as an off-policy algorithm, enabling agents to learn optimal policies in environments like Frozen Lake and autonomous taxi navigation.

## Context

Learning reinforcement learning value-based methods requires transitioning from theoretical Monte Carlo and Temporal Difference concepts to practical tabular algorithms before advancing to Deep Q-Learning.

## Approach / What changed

Train an action-value Q-function using a Q-table initialized to zero, select actions via an epsilon-greedy exploration strategy, and update state-action values after each transition using a TD target formed by the immediate reward and the discounted greedy next-state value.

## Takeaways

- Q-Learning updates action-value estimates step by step using Temporal Difference learning rather than waiting for episode completion.
- The algorithm is off-policy because it acts using an epsilon-greedy policy while updating Q-values using a greedy policy over the next state.
- The epsilon-greedy mechanism manages exploration and exploitation by decaying the random exploration rate epsilon over the course of training.

**Tags:** [Machine Learning](https://yomu.fyi/topic/machine-learning), [Open Source](https://yomu.fyi/topic/open-source)

[Read original post](https://huggingface.co/blog/deep-rl-q-part2)