How to train a new language model from scratch using Transformers and Tokenizers

Summary

Recent library enhancements simplify pretraining custom language models from raw text corpora. A small 84-million-parameter RoBERTa architecture comprising six layers, twelve attention heads, and a hidden dimension of 768 was pretrained on a 3 GB Esperanto text dataset. Training began by fitting a 52,000-vocabulary byte-level Byte-pair encoding tokenizer, which natively handled Esperanto diacritics and reduced average sequence lengths by approximately 30 percent compared to GPT-2 tokenization. The model was trained using masked language modeling, evaluated using fill-mask pipelines, and subsequently fine-tuned for part-of-speech tagging via standard token classification scripts. Final model artifacts and configurations were uploaded for community reuse via pre-trained model interfaces.

Takeaways

1. 1

   Training a language-specific byte-level BPE tokenizer natively encodes accented characters and reduced average sequence lengths by approximately 30% compared to a pretrained GPT-2 tokenizer.

2. 2

   Masked language model pretraining for an 84M-parameter RoBERTa model can be run from scratch by setting the model name or path parameter to None in the language modeling script.

3. 3

   Part-of-speech tagging formatted in CoNLL-2003 format can be trained directly as a token classification task using the standard run_ner.py script.