An Introduction to Q-Learning Part 1

Summary

Value-based reinforcement learning centers on finding optimal policies indirectly through value functions. In these methods, practitioners define policy behavior manually, such as using greedy or epsilon-greedy policies, rather than training the policy directly. Value estimation relies on two primary formulations: the state-value function, which outputs expected returns from a state, and the action-value function, which evaluates state-action pairs. Because summing all future rewards across an entire trajectory is computationally demanding, algorithms adopt the Bellman equation alongside learning strategies like Monte Carlo or Temporal Difference (TD) learning. Monte Carlo updates value functions only after complete episodes using actual returns, whereas one-step TD learning bootstraps updates at each individual step using immediate rewards and discounted estimates of subsequent states.

Takeaways

1. 1

   Value-based methods train a value function rather than the policy directly, using hand-specified rules such as greedy or epsilon-greedy strategies to select actions.

2. 2

   State-value functions compute expected returns from a given state, whereas action-value functions calculate expected returns for specific state-action pairs.

3. 3

   Monte Carlo methods update value estimates only after complete episodes using actual returns, while one-step Temporal Difference learning bootstraps per step using immediate rewards and next-state value estimates.