In October 2018, Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova at Google published “BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding.” BERT stands for Bidirectional Encoder Representations from Transformers, and it changed how the field approached language tasks.
Earlier language models read text in one direction, left to right. BERT’s key idea, as the paper states, was to pre-train deep bidirectional representations from unlabeled text by jointly conditioning on both left and right context in every layer. The model learns general language understanding once on huge amounts of plain text, then gets fine-tuned with a small amount of extra task-specific data.
The payoff was immediate and broad. The paper reports that a single pre-trained BERT model, fine-tuned with minimal added architecture, achieved state-of-the-art results across eleven natural language processing benchmarks at the time.
For business readers, BERT marked the moment “pre-train once, adapt many times” became the dominant recipe for language AI. The same approach underpins the search, classification, and question-answering systems that became common in the years that followed.