# Training CodeParrot 🦜 from Scratch

huggingface.co · Leandro von Werra · Dec 8, 2021

**Type:** Tutorial

## Summary

Training code generation models from scratch requires careful data filtering, tokenization, and compute management. To build CodeParrot, a 1.5-billion-parameter Python generation model, engineers extracted 20 million Python files from BigQuery and cleaned the resulting 180 GB corpus down to 50 GB after discovering extreme duplication. A custom GPT-2 tokenizer was trained over streamed samples, and the GPT-2 large architecture was initialized with layer-scaled, full-precision attention mechanisms. The training pipeline leveraged Hugging Face Accelerate alongside gradient checkpointing and a streaming iterable dataset designed to yield fixed-length concatenated token sequences. CodeParrot was trained on roughly 25 to 30 billion tokens and evaluated on coding tasks using the pass@k metric.

## Context

Building a dedicated Python code completion model required training a large model from scratch. Raw GitHub dumps contained extensive duplicates that severely degraded model performance, and handling tens of gigabytes of training data introduced storage and memory constraints during tokenization and training.

## Approach / What changed

A 180 GB Python dataset was extracted from BigQuery and cleaned down to 50 GB using Codex heuristics and deduplication. A GPT-2 tokenizer was trained over streamed data, and a 1.5B parameter GPT-2 large architecture was configured with attention scaling and full-precision attention computation. Training was scaled using Hugging Face Accelerate, gradient checkpointing, and an iterable dataset that concatenated and chunked samples to fixed context lengths.

## Takeaways

- Severe duplication in the raw 180 GB dataset impacted model performance, with 0.1% of unique files representing 15% of all files and 10% making up 66%.
- Deduplicating and applying Codex cleaning heuristics reduced the raw 20-million-file dataset from 180 GB down to 50 GB.
- Training stability and memory efficiency were managed using layer-indexed attention scaling, full-precision attention computation, and gradient checkpointing.

**Tags:** [LLMs](https://yomu.fyi/topic/llm), [Machine Learning](https://yomu.fyi/topic/machine-learning), [Open Source](https://yomu.fyi/topic/open-source), [Python](https://yomu.fyi/topic/python)

[Read original post](https://huggingface.co/blog/codeparrot)