October 28–31, 2019
Please log in

Natural language processing using transformer architectures

Aurélien Geron (Kiwisoft)
11:00am11:40am Thursday, October 31, 2019
Location: Grand Ballroom C/D
Average rating: ****.
(4.50, 6 ratings)


New to TensorFlow

Whether you need to automatically judge the sentiment of a user review, summarize long documents, translate text, or build a chatbot, you need the best language model available. In 2018, pretty much every NLP benchmark was crushed by novel transformer-based architectures, replacing long-standing architectures based on recurrent neural networks. In short, if you’re into NLP, you need transformers.

But to use transformers, you need to know what they are, what transformer-based architectures look like, and how you can implement them in your projects.

Aurélien Géron dives into recurrent neural networks and their limits, the invention of the transformer, attention mechanisms, the transformer architecture, subword tokenization using SentencePiece, self-supervised pretraining—learning from huge corpora, one-size-fits-all language models, BERT and GPT 2, and how to use these language models in your projects using TensorFlow.

What you'll learn

  • Understand transformers and modern language models and how they can tackle complex NLP tasks
  • Identify what tools to use and what the code looks like
Photo of Aurélien Geron

Aurélien Geron


Aurélien Géron is a machine learning consultant at Kiwisoft and author of the best-selling O’Reilly book Hands-on Machine Learning with Scikit-Learn, Keras, and TensorFlow. Previously, he led YouTube’s video classification team, was a founder and CTO of Wifirst, and was a consultant in a variety of domains: finance (JPMorgan and Société Générale), defense (Canada’s DOD), and healthcare (blood transfusion). He also published a few technical books (on C++, WiFi, and internet architectures), and he’s a lecturer at the Dauphine University in Paris. He lives in Singapore with his wife and three children.

  • O'Reilly
  • TensorFlow
  • Google Cloud
  • IBM
  • Databricks
  • Tensor Networks
  • VMware
  • Amazon Web Services
  • One Convergence
  • Quantiphi
  • Lambda Labs
  • Tech Mahindra
  • cnvrg.io
  • Determined AI
  • Inferencery
  • Manceps, Inc.
  • PerceptiLabs
  • Valohai

Contact us


For conference registration information and customer service


For more information on community discounts and trade opportunities with O’Reilly conferences


For information on exhibiting or sponsoring a conference


For media/analyst press inquires