Presented By
O’Reilly + Cloudera
Make Data Work
29 April–2 May 2019
London, UK
Please log in

The unreasonable effectiveness of transfer learning on NLP

David Low (
14:0514:45 Wednesday, 1 May 2019
Data Science, Machine Learning & AI
Location: Capital Suite 17
Average rating: ***..
(3.57, 7 ratings)

Who is this presentation for?

  • Data scientists, ML engineers, researchers, and NLP practitioners



Prerequisite knowledge

  • Basic knowledge of machine learning, deep learning, and natural language processing concepts

What you'll learn

  • Understand how transfer learning allows data scientists to build accurate models without much data
  • Learn how to apply transfer learning to natural language processing tasks


Transfer learning has been proven to be a tremendous success in computer vision—a result of the ImageNet competition. In the past few months, there have been several breakthroughs in natural language processing with transfer learning, namely ELMo, OpenAI Transformer, and ULMFit. Pretrained models derived from these techniques have been proven in achieving state-of-the-art results on a wide range of NLP problems. The use of pretrained models has come a long way since the introduction of word2vec and GloVe, and these two approaches are considered shallow in comparison.

David Low demonstrates how to use transfer learning on an NLP application with SOTA accuracy. David starts with an introduction to transfer learning followed by explanations on why pretrained models are handy for tackling machine learning problems with limited data as well as how they could be used as fixed feature extractor for downstream tasks and applications.

David then walks you through fine-tuning a transfer learning model to achieve state-of-the-art accuracy (92%) on a real-world sentiment classification problem using the Amazon Reviews dataset. In comparison to a FastText-based model trained on the full dataset (3.6 million samples), it takes just 1,000 samples of training data to produce a model that achieves similar performance.

Photo of David Low

David Low

David Low is the cofounder and chief data scientist at, a company building an AI-powered chatbot to disrupt and shape the booming conversational commerce space with deep natural language processing. He represented Singapore and the National University of Singapore (NUS) in the 2016 Data Science Games held in France, and clinched the top spot among Asian and American teams. David has been invited as a guest lecturer by NUS to conduct master classes on applied machine learning and deep learning topics. Throughout his career, David has engaged in data science projects across manufacturing, telco, ecommerce, and the insurance industry, including sales forecast modeling and influencer detection, which won him awards in several competitions and was featured on the IDA website and the NUS publication. Previously, he was a data scientist at the Infocomm Development Authority (IDA) of Singapore and was involved in research collaborations with Carnegie Mellon University (CMU) and Massachusetts Institute of Technology (MIT) on projects funded by the National Research Foundation and SMART. He competes on Kaggle and holds a top 0.2% worldwide ranking.