---
title: "Training a coding model to paint watercolours with TRL and OpenEnv"
description: "The project reproduces an earlier idea: training a language model to write JavaScript that paints watercolours with p5.brush, using TRL and OpenEnv. Its open pipeline includes a hand-rated reference pool, an RL environment, training scripts, trained models, and Hugging Face Jobs, Spaces, Inference Providers, and Hub artifacts. The environment renders sketches in headless Chromium, restricts the model to ten library methods, rejects cheats through a gate, and combines gate, length, pairwise-judge, and HPSv3 rewards. Three reward mixes were trained: judge-led (0.60/0.30), hps-led (0.30/0.60), and hps-only (0.00/0.90); the first two ran to step 110, while validation stopped at step 60. The post reports that both pairwise-judge runs learned and identifies the hand-rated pool as the bottleneck."
---

# Training a coding model to paint watercolours with TRL and OpenEnv

[Hugging Face](https://yomu.fyi/company/hugging-face) · Sergio Paniego · Sep 3, 2026

**Type:** Explainer

## Summary

The project reproduces an earlier idea: training a language model to write JavaScript that paints watercolours with p5.brush, using TRL and OpenEnv. Its open pipeline includes a hand-rated reference pool, an RL environment, training scripts, trained models, and Hugging Face Jobs, Spaces, Inference Providers, and Hub artifacts. The environment renders sketches in headless Chromium, restricts the model to ten library methods, rejects cheats through a gate, and combines gate, length, pairwise-judge, and HPSv3 rewards. Three reward mixes were trained: judge-led (0.60/0.30), hps-led (0.30/0.60), and hps-only (0.00/0.90); the first two ran to step 110, while validation stopped at step 60. The post reports that both pairwise-judge runs learned and identifies the hand-rated pool as the bottleneck.

## Context

The work aims to reproduce Surya Narreddi's watercolour-model recipe in the open, while teaching a smaller model to combine code-generated painting with a person's artistic preferences. The reward has no single correct answer and instead relies on aesthetic preference encoded by a reference pool and model judges.

## Approach / What changed

The implementation wraps the painting library, restrictive system prompt, headless Chromium renderer, and anti-cheating gate in an OpenEnv RL environment. It trains with TRL on Hugging Face Jobs, uses a pairwise Qwen3-VL judge and HPSv3 for reward components, and publishes the pool, environments, scripts, models, rollouts, metrics, and gallery.

## Takeaways

- The pairwise judge compares each candidate with four randomly selected references in both presentation orders, scoring the share of comparisons the candidate wins.
- The reward mixes include a 0.05 gate term, 0.05 length term, and model-based components whose weights vary between the pairwise judge and HPSv3.
- The author reports that both longer runs using the pairwise judge learned, while the reference pool remains the project's unresolved bottleneck.

**Tags:** [OpenEnv](https://yomu.fyi/topic/openenv), [p5.js](https://yomu.fyi/topic/p5-js), [Reinforcement Learning](https://yomu.fyi/topic/reinforcement-learning), [TRL](https://yomu.fyi/topic/trl)

- Source: [Hugging Face](https://huggingface.co/blog/train-to-paint-with-code)
- Source URL: https://huggingface.co/blog/train-to-paint-with-code
- Ingested by Yomu: 2026-09-03T11:17:42.080Z

[Read original post](https://huggingface.co/blog/train-to-paint-with-code)