# An Introduction to Q-Learning Part 1

huggingface.co · Thomas Simonini · May 18, 2022

**Type:** Explainer

## Summary

Value-based reinforcement learning centers on finding optimal policies indirectly through value functions. In these methods, practitioners define policy behavior manually, such as using greedy or epsilon-greedy policies, rather than training the policy directly. Value estimation relies on two primary formulations: the state-value function, which outputs expected returns from a state, and the action-value function, which evaluates state-action pairs. Because summing all future rewards across an entire trajectory is computationally demanding, algorithms adopt the Bellman equation alongside learning strategies like Monte Carlo or Temporal Difference (TD) learning. Monte Carlo updates value functions only after complete episodes using actual returns, whereas one-step TD learning bootstraps updates at each individual step using immediate rewards and discounted estimates of subsequent states.

## Context

Reinforcement learning agents require optimal policies to maximize cumulative rewards, but estimating expected returns directly by summing all future rewards across state trajectories is computationally tedious.

## Approach / What changed

Value-based reinforcement learning trains value functions (state-value or action-value) combined with hand-defined policies like greedy or epsilon-greedy selection, updating values across episodes via Monte Carlo returns or per-step via Temporal Difference bootstrapping and the Bellman equation.

## Takeaways

- Value-based methods train a value function rather than the policy directly, using hand-specified rules such as greedy or epsilon-greedy strategies to select actions.
- State-value functions compute expected returns from a given state, whereas action-value functions calculate expected returns for specific state-action pairs.
- Monte Carlo methods update value estimates only after complete episodes using actual returns, while one-step Temporal Difference learning bootstraps per step using immediate rewards and next-state value estimates.

**Tags:** [Machine Learning](https://yomu.fyi/topic/machine-learning)

[Read original post](https://huggingface.co/blog/deep-rl-q-part1)