Ming-Wei Chang offers an overview of a new language representation model called BERT (Bidirectional Encoder Representations from Transformers). Unlike recent language representation models, BERT is designed to pretrain deep bidirectional representations by jointly conditioning on both left and right context in all layers. As a result, the pretrained BERT representations can be fine-tuned with just one additional output layer to create models for a wide range of tasks, such as question answering and language inference, without substantial task-specific architecture modifications.
BERT is conceptually simple and empirically powerful. It obtains new, state-of-the-art results on 11 natural language processing tasks, including pushing the GLUE benchmark to 80.4% (7.6% absolute improvement), MultiNLI accuracy to 86.7 (5.6% absolute improvement), and the SQuAD v1.1 question-answering test F1 to 93.2 (1.5% absolute improvement), outperforming human performance by 2.0%.
Ming-Wei Chang is a research scientist at Google AI Language. He enjoys developing interesting machine learning algorithms for practical problems, especially in the field of natural language processing. He has published more than 35 papers at top-tier conferences and won an outstanding paper award at ACL 2015 for his work on question answering over knowledge bases. He also won several international machine learning competitions on topics like entity linking, power load forecast prediction, and sequential data classification. His recent paper, “BERT: Pretraining of Deep Bidirectional Transformers for Language Understanding“—cowritten with his colleagues in Google AI Language—demonstrates the power of language model pretraining and details the new state of the art for over 11 natural language processing tasks.
Comments on this page are now closed.
©2019, O'Reilly Media, Inc. • (800) 889-8969 or (707) 827-7019 • Monday-Friday 7:30am-5pm PT • All trademarks and registered trademarks appearing on oreilly.com are the property of their respective owners. • firstname.lastname@example.org