# How to train a new language model from scratch using Transformers and Tokenizers

huggingface.co · Julien Chaumond · Feb 14, 2020

**Type:** Tutorial

## Summary

Recent library enhancements simplify pretraining custom language models from raw text corpora. A small 84-million-parameter RoBERTa architecture comprising six layers, twelve attention heads, and a hidden dimension of 768 was pretrained on a 3 GB Esperanto text dataset. Training began by fitting a 52,000-vocabulary byte-level Byte-pair encoding tokenizer, which natively handled Esperanto diacritics and reduced average sequence lengths by approximately 30 percent compared to GPT-2 tokenization. The model was trained using masked language modeling, evaluated using fill-mask pipelines, and subsequently fine-tuned for part-of-speech tagging via standard token classification scripts. Final model artifacts and configurations were uploaded for community reuse via pre-trained model interfaces.

## Context

Recent updates to the transformers and tokenizers libraries aimed to make pretraining language models from scratch more accessible, using the low-resource, grammatically regular language Esperanto as a demonstration target.

## Approach / What changed

The process concatenated OSCAR and Leipzig Corpora Collection texts into a 3 GB corpus, trained a 52,000-vocabulary byte-level BPE tokenizer, and trained an 84M-parameter RoBERTa model on masked language modeling using run\_language\_modeling.py. The resulting model was evaluated with fill-mask pipelines and fine-tuned for part-of-speech tagging using run\_ner.py with CoNLL-2003 formatted data.

## Takeaways

- Training a language-specific byte-level BPE tokenizer natively encodes accented characters and reduced average sequence lengths by approximately 30% compared to a pretrained GPT-2 tokenizer.
- Masked language model pretraining for an 84M-parameter RoBERTa model can be run from scratch by setting the model name or path parameter to None in the language modeling script.
- Part-of-speech tagging formatted in CoNLL-2003 format can be trained directly as a token classification task using the standard run\_ner.py script.

**Tags:** [LLMs](https://yomu.fyi/topic/llm), [Machine Learning](https://yomu.fyi/topic/machine-learning), [Open Source](https://yomu.fyi/topic/open-source), [Python](https://yomu.fyi/topic/python)

[Read original post](https://huggingface.co/blog/how-to-train)