# BERT 101 - State Of The Art NLP Model Explained

huggingface.co · Britney Muller · Mar 2, 2022

**Type:** Explainer

## Summary

Developed in 2018 by Google AI Language, Bidirectional Encoder Representations from Transformers addresses the historical challenge of machines lacking contextual understanding of human language. The model relies on an encoder-only Transformer architecture pre-trained on a 3.3-billion-word corpus consisting of Wikipedia and Google BooksCorpus. Training simultaneously combines masked language modeling, which hides 15% of tokenized words to enforce bidirectional context learning, with next sentence prediction across balanced sentence pairs. Pre-trained on Cloud TPUs over four days, BERT unifies solutions for more than eleven common NLP tasks and can be fine-tuned on task-specific annotated data within minutes. Unmasking experiments demonstrate that the model can also inherit distinct societal and gender biases from its underlying training corpora when predicting professions.

## Context

Computers historically lacked contextual understanding of human language, and traditional natural language processing solved tasks using separate, individual models for each specific use case.

## Approach / What changed

BERT utilizes an encoder-only Transformer architecture pre-trained in an unsupervised manner on 3.3 billion words using Masked Language Modeling and Next Sentence Prediction, followed by supervised fine-tuning on annotated task data.

## Takeaways

- BERT trains simultaneously on Masked Language Modeling (predicting 15% hidden tokens bidirectionally) and Next Sentence Prediction (evaluating 50% correct and 50% random sentence pairs).
- Pre-training original models took 4 days using 4 Cloud TPUs for BERTbase and 16 for BERTlarge, whereas fine-tuning on downstream tasks takes 1 to 25 minutes on a single TPU.
- DistilBERT provides a compressed alternative that runs 60% faster than BERT while preserving over 95% of its performance for resource-constrained environments.

**Tags:** [LLMs](https://yomu.fyi/topic/llm), [Machine Learning](https://yomu.fyi/topic/machine-learning), [Open Source](https://yomu.fyi/topic/open-source), [Search](https://yomu.fyi/topic/search)

[Read original post](https://huggingface.co/blog/bert-101)