BERT 101 - State Of The Art NLP Model Explained

Summary

Developed in 2018 by Google AI Language, Bidirectional Encoder Representations from Transformers addresses the historical challenge of machines lacking contextual understanding of human language. The model relies on an encoder-only Transformer architecture pre-trained on a 3.3-billion-word corpus consisting of Wikipedia and Google BooksCorpus. Training simultaneously combines masked language modeling, which hides 15% of tokenized words to enforce bidirectional context learning, with next sentence prediction across balanced sentence pairs. Pre-trained on Cloud TPUs over four days, BERT unifies solutions for more than eleven common NLP tasks and can be fine-tuned on task-specific annotated data within minutes. Unmasking experiments demonstrate that the model can also inherit distinct societal and gender biases from its underlying training corpora when predicting professions.

Context

Computers historically lacked contextual understanding of human language, and traditional natural language processing solved tasks using separate, individual models for each specific use case.

Approach / What changed

BERT utilizes an encoder-only Transformer architecture pre-trained in an unsupervised manner on 3.3 billion words using Masked Language Modeling and Next Sentence Prediction, followed by supervised fine-tuning on annotated task data.

Takeaways

• BERT trains simultaneously on Masked Language Modeling (predicting 15% hidden tokens bidirectionally) and Next Sentence Prediction (evaluating 50% correct and 50% random sentence pairs).
• Pre-training original models took 4 days using 4 Cloud TPUs for BERTbase and 16 for BERTlarge, whereas fine-tuning on downstream tasks takes 1 to 25 minutes on a single TPU.
• DistilBERT provides a compressed alternative that runs 60% faster than BERT while preserving over 95% of its performance for resource-constrained environments.

Related reading

• huggingface.co · Oct 20, 2021

  The Age of Machine Learning As Code Has Arrived

  Recent findings from the 2021 State of AI Report and Kaggle State of Machine Learning and Data Science Survey indicate that machine learning is expanding into critical infrastructure while Transformers become general-purpose architectures across text, vision, and audio. In response, organizations face questions about scaling infrastructure, team composition, and engineering maturity. Rather than treating machine learning as isolated sandbox experiments or hiring solely data scientists, teams benefit from adopting established software engineering and DevOps principles like versioning, testing, automation, and continuous deployment. Furthermore, the rise of pre-trained Transformer architectures enables practitioners to fine-tune existing off-the-shelf models rather than training from scratch, reducing compute costs and training duration. Tools from platforms such as Hugging Face streamline model deployment, latency optimization, and infrastructure abstraction.

  Julien Simon

• huggingface.co · Dec 15, 2021

  Perceiver IO: a scalable, fully-attentional model that works on any modality

  Standard Transformer architectures scale poorly in compute and memory because pairwise dot-product self-attention depends quadratically on input size. Perceiver IO addresses this constraint by computing self-attention across a small set of latent variables rather than directly on high-dimensional inputs. Inputs and outputs interact with the model via cross-attention operations, decoupling compute and memory costs from input and output dimensions. Integrated into Hugging Face Transformers via the PerceiverModel class, the architecture supports diverse data types using optional preprocessors, decoders, and postprocessors. Experiments demonstrate competitive performance across text, multimodal video classification, 3D point cloud classification on ModelNet40, and StarCraft II reinforcement learning in AlphaStar.