Policy Gradient with PyTorch

Summary

Policy gradient algorithms optimize reinforcement learning policies directly without learning intermediate action-value functions. Instead of assigning discrete Q-values to actions, these methods parameterize a stochastic policy and apply gradient ascent on an objective score function. This setup eliminates manual exploration tuning, resolves perceptual aliasing in identical states, and naturally accommodates continuous or high-dimensional action spaces. The Reinforce Monte Carlo policy gradient algorithm operationalizes this approach by collecting full episodic trajectories and adjusting parameters along the gradient of the log-action probabilities scaled by return. Practitioners implement this algorithm using PyTorch to evaluate agent robustness across environments like CartPole-v1, PixelCopter, and Pong.

Context

Value-based reinforcement learning methods rely on estimating action values to derive policies, which requires manual exploration-exploitation tuning, struggles with perceptual aliasing, and scales poorly in continuous or high-dimensional action spaces.

Approach / What changed

Using the Reinforce Monte Carlo policy gradient algorithm implemented in PyTorch, the agent directly estimates policy weights via gradient ascent on the expected return of full collected episodes without calculating action values.

Takeaways

• Policy gradient methods learn stochastic action distributions directly, avoiding the need for manual exploration-exploitation strategies.
• Direct policy optimization mitigates perceptual aliasing, allowing agents to choose differing actions across seemingly identical states.
• Reinforce updates policy parameters by weighting the direction of steepest log-probability increase by the episode return, but can suffer from high variance and local optima.

Related reading

• huggingface.co · May 20, 2022

  An Introduction to Q-Learning Part 2/2

  Q-Learning serves as a foundational off-policy reinforcement learning method for finding optimal action-value functions before advancing to deep reinforcement learning. The approach employs a tabular Q-function that updates state-action pairs at each step using temporal difference learning. Action selection balances exploration and exploitation via an epsilon-greedy strategy, where epsilon progressively decays as training proceeds. Updates compute a temporal difference target combining the immediate reward and the discounted maximum value of the subsequent state under a greedy policy. This discrepancy between the exploratory acting policy and the greedy updating policy classifies Q-Learning as an off-policy algorithm, enabling agents to learn optimal policies in environments like Frozen Lake and autonomous taxi navigation.

  Thomas Simonini

• huggingface.co · Jun 7, 2022

  Deep Q-Learning with Space Invaders

  Tabular Q-learning fails to scale to complex environments like Atari games due to massive observation spaces that make maintaining discrete state-action tables impractical. Deep Q-learning addresses this limitation by using a neural network to approximate Q-values for each possible action from input states. To process visual inputs effectively and capture temporal motion, raw game screens are grayscaled, resized to 84x84 pixels, and stacked in groups of four consecutive frames before passing through convolutional and fully connected layers. The training process stabilizes learning through experience replay buffers, fixed target networks that update periodically, and Double DQN architectures that decouple action selection from target evaluation. Agents can then be trained on environments like Space Invaders using frameworks such as RL-Zoo without tabular memory constraints.