It is not enough to simply collect big data. It must be processed and delivered to end users in an efficient and useful manner, turning big data into “smart” data. The Bitvore for Munis system examines news articles collected from a large number of sources and applies a hierarchical processing technique that identifies the location, business sector, and any triggering material events (“signals”) that indicate that an article will be of interest to decision makers in the fixed-income securities market. Up until now, these people have had to waste hours each day sifting through the noise generated by technologies such as Google Alerts or running hand-crafted internet searches.
Bitvore has developed an automated analysis system using its highly configurable business intelligence platform that can associate a specific news article with the organization (an “obligor”) that is responsible for repaying a municipal bond. It is possible for a user to provide a list of these obligors and then only receive email alerts giving them a list of articles that is clearly material to the specific obligor. This allows the user who is typically a highly compensated financial analyst to focus their efforts on important news rather than spend a lot of time filtering through noise. Only 0.1% of the articles collected are found to be relevant as part of an alert.
Alan Chaney offers an overview of the basic architecture of the platform on which the product is built and and draws on three years’ experience working with real customers to explain the design choices made. An overview will be given of the wide range of AI and NLP techniques used to identify that a story is important, including some of the refinements used to solve real-world customer issues. The system is so successful that it’s now replacing less efficient and less effective outsourced manual systems.
Alan Chaney is chief architect and vice president of engineering at Bitvore Corp. Alan started down the technology path years before he knew anything about software. A guy who’s always been fascinated by how things work, he fed his curiosity by disassembling everything from clocks to motorcycles. When he learned that software offered the same sort of intellectual stimulation (but without the solder burns), he quickly changed his focus from rewiring objects to writing code. During his career as an academic and entrepreneur, Alan dedicated himself to reinventing everything from networked storage to streaming media, always focused on doing things better, faster, and more elegantly than previously imagined.
©2017, O'Reilly Media, Inc. • (800) 889-8969 or (707) 827-7019 • Monday-Friday 7:30am-5pm PT • All trademarks and registered trademarks appearing on oreilly.com are the property of their respective owners. • firstname.lastname@example.org
Apache Hadoop, Hadoop, Apache Spark, Spark, and Apache are either registered trademarks or trademarks of the Apache Software Foundation in the United States and/or other countries, and are used with permission. The Apache Software Foundation has no affiliation with and does not endorse, or review the materials provided at this event, which is managed by O'Reilly Media and/or Cloudera.