Natural language understanding at scale with spaCy and Spark NLP

Description

Natural language processing is a key component in many data science systems that must understand or reason about text. Common use cases include question answering, paraphrasing or summarization, sentiment analysis, natural language BI, language modeling, and disambiguation. Building such systems usually requires combining three types of software libraries: NLP annotation frameworks, machine learning frameworks, and deep learning frameworks.

David Talby and Claudiu Branzan lead a hands-on tutorial for scalable NLP using spaCy for building annotation pipelines, Spark NLP for training distributed custom natural language machine-learned pipelines, and Spark ML and TensorFlow for using deep learning to build and apply word embeddings. You’ll spend about half your time coding as you work through three sections, each with an end-to-end working codebase that you are then asked to change and improve.

Outline

Using spaCy to build an NLP annotations pipeline that can understand text structure, grammar, and sentiment and perform entity recognition

• Built-in spaCy annotators
• Debugging and visualizing results
• Creating custom pipelines
• Practical trade-offs for large-scale projects, as well as for balancing performance versus accuracy

Using TensorFlow to build domain-specific machine-learned annotators and then integrating them into an existing NLP pipeline

• Feature engineering and optimization
• Measurement
• Practical considerations when working on problems that require understanding text beyond keyword matching and one-hot encoding

Using Spark ML and TensorFlow to apply deep learning to expand and update ontologies

• Comparison of word2vec and doc2vec
• When each is useful
• How to apply them to increase the accuracy of classification or information retrieval problems
• Current trade-offs in integrating spaCy and Spark when engineering distributed, large-scale NLP pipelines