# Policy Gradient with PyTorch

huggingface.co · Thomas Simonini · Jun 30, 2022

**Type:** Tutorial

## Summary

Policy gradient algorithms optimize reinforcement learning policies directly without learning intermediate action-value functions. Instead of assigning discrete Q-values to actions, these methods parameterize a stochastic policy and apply gradient ascent on an objective score function. This setup eliminates manual exploration tuning, resolves perceptual aliasing in identical states, and naturally accommodates continuous or high-dimensional action spaces. The Reinforce Monte Carlo policy gradient algorithm operationalizes this approach by collecting full episodic trajectories and adjusting parameters along the gradient of the log-action probabilities scaled by return. Practitioners implement this algorithm using PyTorch to evaluate agent robustness across environments like CartPole-v1, PixelCopter, and Pong.

## Context

Value-based reinforcement learning methods rely on estimating action values to derive policies, which requires manual exploration-exploitation tuning, struggles with perceptual aliasing, and scales poorly in continuous or high-dimensional action spaces.

## Approach / What changed

Using the Reinforce Monte Carlo policy gradient algorithm implemented in PyTorch, the agent directly estimates policy weights via gradient ascent on the expected return of full collected episodes without calculating action values.

## Takeaways

- Policy gradient methods learn stochastic action distributions directly, avoiding the need for manual exploration-exploitation strategies.
- Direct policy optimization mitigates perceptual aliasing, allowing agents to choose differing actions across seemingly identical states.
- Reinforce updates policy parameters by weighting the direction of steepest log-probability increase by the episode return, but can suffer from high variance and local optima.

**Tags:** [Machine Learning](https://yomu.fyi/topic/machine-learning), [Python](https://yomu.fyi/topic/python)

[Read original post](https://huggingface.co/blog/deep-rl-pg)