How to train a new language model from scratch using Transformers and Tokenizers

Summary

Recent library enhancements simplify pretraining custom language models from raw text corpora. A small 84-million-parameter RoBERTa architecture comprising six layers, twelve attention heads, and a hidden dimension of 768 was pretrained on a 3 GB Esperanto text dataset. Training began by fitting a 52,000-vocabulary byte-level Byte-pair encoding tokenizer, which natively handled Esperanto diacritics and reduced average sequence lengths by approximately 30 percent compared to GPT-2 tokenization. The model was trained using masked language modeling, evaluated using fill-mask pipelines, and subsequently fine-tuned for part-of-speech tagging via standard token classification scripts. Final model artifacts and configurations were uploaded for community reuse via pre-trained model interfaces.

Context

Recent updates to the transformers and tokenizers libraries aimed to make pretraining language models from scratch more accessible, using the low-resource, grammatically regular language Esperanto as a demonstration target.

Approach / What changed

The process concatenated OSCAR and Leipzig Corpora Collection texts into a 3 GB corpus, trained a 52,000-vocabulary byte-level BPE tokenizer, and trained an 84M-parameter RoBERTa model on masked language modeling using run_language_modeling.py. The resulting model was evaluated with fill-mask pipelines and fine-tuned for part-of-speech tagging using run_ner.py with CoNLL-2003 formatted data.

Takeaways

• Training a language-specific byte-level BPE tokenizer natively encodes accented characters and reduced average sequence lengths by approximately 30% compared to a pretrained GPT-2 tokenizer.
• Masked language model pretraining for an 84M-parameter RoBERTa model can be run from scratch by setting the model name or path parameter to None in the language modeling script.
• Part-of-speech tagging formatted in CoNLL-2003 format can be trained directly as a token classification task using the standard run_ner.py script.

Related reading

• huggingface.co · Apr 25, 2022

  Introducing Hugging Face for Education 🤗

  Hugging Face established an initiative to teach machine learning to 5 million people by the end of 2023. As machine learning expands across software development and impacts non-technical users, adapting workforce skills and assisting educators with ethical considerations become critical challenges. The educational framework provides browser-based model testing widgets, documentation on model biases, and free courses covering Natural Language Processing, Deep Reinforcement Learning, and interactive demo creation. Instructors receive free infrastructure to create Hub classrooms, alongside a multilingual toolkit spanning topics like Transformers and Gradio. Through ongoing global tours and free compute events, the organization supports collaborative machine learning education across diverse experience levels.

  Violette

• huggingface.co · Nov 9, 2020

  Leveraging Pre-trained Language Model Checkpoints for Encoder-Decoder Models

  Pre-training sequence-to-sequence transformer models incurs massive computational costs, limiting development primarily to large institutions. To mitigate these expenses, encoder-decoder architectures can be warm-started using existing pre-trained checkpoints from encoder-only or decoder-only models like BERT and GPT-2. This walkthrough details the methodology and implementation of warm-starting sequence-to-sequence architectures using Hugging Face Transformers. By utilizing the EncoderDecoderModel framework alongside Seq2SeqTrainer, practitioners can construct and fine-tune models such as BERT2BERT on datasets like CNN/DailyMail. The resulting fully trained BERT2BERT model achieves a ROUGE-2 score of 18.22 on the full evaluation set, matching competitive sequence generation baselines at a fraction of standard pre-training costs.