Spark NLP in action: Intelligent, high-accuracy fact extraction from long financial documents

Description

Answering questions accurately based on information from financial documents, which can be a hundred or more pages long, is a challenge even for human domain experts. While traditional rule-based or expression-matching techniques work for simple fields in templated documents, it is harder to infer facts based on implied statements, on the absence of certain statements, or on the combination of other facts. Also, often the most interesting questions are not factual (e.g., Who is the CFO of this company?) but fuzzy (e.g., Are there suspect equity transactions disclosed here?). Answering such questions at a very high level of accuracy requires state-of-the-art deep learning techniques applied to NLP.

Spark NLP, John Snow Labs’s NLP Library for Apache Spark, is an open source library that natively extends Spark ML to provide natural language understanding capabilities with performance and scale that was not possible to date and provides advanced NLP algorithms like named entity recognition, fact extraction, spell checking, sentiment analysis, assertion status detection, and others. It enables training domain-specific machine learning and deep learning NLP models at a performance and scale that are one to two orders of magnitude better than existing alternatives.

David Talby, Saif Addin Ellafi, and Paul Parau explain how Spark NLP was used to augment the Recognos smart data extraction platform in order to automatically infer fuzzy, implied, and complex facts from long financial documents, covering the technical challenges, the architecture of the full solution, and lessons learned that you can directly apply to your next data extraction project.