Training CodeParrot 🦜 from Scratch

Summary

Training code generation models from scratch requires careful data filtering, tokenization, and compute management. To build CodeParrot, a 1.5-billion-parameter Python generation model, engineers extracted 20 million Python files from BigQuery and cleaned the resulting 180 GB corpus down to 50 GB after discovering extreme duplication. A custom GPT-2 tokenizer was trained over streamed samples, and the GPT-2 large architecture was initialized with layer-scaled, full-precision attention mechanisms. The training pipeline leveraged Hugging Face Accelerate alongside gradient checkpointing and a streaming iterable dataset designed to yield fixed-length concatenated token sequences. CodeParrot was trained on roughly 25 to 30 billion tokens and evaluated on coding tasks using the pass@k metric.

Context

Building a dedicated Python code completion model required training a large model from scratch. Raw GitHub dumps contained extensive duplicates that severely degraded model performance, and handling tens of gigabytes of training data introduced storage and memory constraints during tokenization and training.

Approach / What changed

A 180 GB Python dataset was extracted from BigQuery and cleaned down to 50 GB using Codex heuristics and deduplication. A GPT-2 tokenizer was trained over streamed data, and a 1.5B parameter GPT-2 large architecture was configured with attention scaling and full-precision attention computation. Training was scaled using Hugging Face Accelerate, gradient checkpointing, and an iterable dataset that concatenated and chunked samples to fixed context lengths.

Takeaways

• Severe duplication in the raw 180 GB dataset impacted model performance, with 0.1% of unique files representing 15% of all files and 10% making up 66%.
• Deduplicating and applying Codex cleaning heuristics reduced the raw 20-million-file dataset from 180 GB down to 50 GB.
• Training stability and memory efficiency were managed using layer-indexed attention scaling, full-precision attention computation, and gradient checkpointing.

Related reading

• huggingface.co · Apr 16, 2021

  Introducing 🤗 Accelerate

  🤗 Accelerate enables PyTorch developers to execute raw training loops across CPUs, multi-GPU setups, and TPUs with minimal boilerplate modifications. Standard distributed training typically demands manual device assignments, DistributedSampler configurations, and divergent code paths that break single-device portability. By wrapping models, optimizers, and dataloaders through a unified prepare call, Accelerate automatically manages hardware placement, mixed precision operations, and sampler batch slicing without requiring custom distributed samplers. Distributed evaluation is streamlined using a gather utility that consolidates prediction tensors across active worker processes. Furthermore, a dedicated command-line interface provides an interactive questionnaire to store runtime configuration defaults and orchestrate local or AWS SageMaker runs.

  Sylvain Gugger

• huggingface.co · Apr 28, 2022

  Opinion Classification with Kili and HuggingFace AutoTrain

  Understanding mobile application user feedback often requires expensive and labor-intensive manual analysis. To streamline this process, an active learning pipeline was constructed using Kili for data annotation alongside HuggingFace AutoTrain and standard transformers for text classification. The workflow utilized approximately 40,000 Medium reviews collected from the Google Play Store, classifying content into categories such as Subscription, Content, Interface, and User Experience. Hyperparameter searches with Ray backend were compared against AutoTrain, which automated training within 30 minutes. Subsequent sentiment analysis of the categorized reviews revealed that while overall sentiment remained positive, subscription and interface categories contained predominantly negative feedback, especially in application version 4.5.