Big Fish Games is a leading producer and distributor of casual and midcore games. The company has created a mobile game platform that supports real-time data analytics and other business processes. The platform processes on average 5,000 events per second—a number that is continually growing. While the Hadoop ecosystem includes platforms and software library frameworks that can support this goal, not all components are appropriate within a given architecture, and other components require performance tuning and customization.
Stephen Devine and Kalah Brown explain how they used Spark, Flume, and Kafka to build a live-streaming data pipeline, focusing on their decision to move from Flume to Kafka. Along the way, they cover Spark Streaming Check Pointing, its limitations, and how they overcame them by building a custom fault tolerance feature in their Spark consumer. Stephen and Kalah also address how they removed performance bottlenecks by using Scala futures to asynchronously write to HDFS and share how to more efficiently tune Spark Streaming by discussing which of its properties and configurations they found most impactful for improving performance.
Stephen Devine is a Seattle-based data engineer at Big Fish Games, where he wrangles events sent from millions of mobile phones through Kafka into Hive. Previously, he did similar things for Xbox One Live Services using proprietary Microsoft technology and worked on several releases of Internet Explorer at Microsoft.
Kalah Brown is a senior Hadoop engineer at Big Fish Games, where she is responsible for the technical leadership and development of big data solutions. Previously, Kalah was a consultant in the greater Seattle area and worked with numerous companies, including Disney, Starbucks, the Bill and Melinda Gates Foundation, Microsoft, and Premera Blue Cross. She has 17 years of experience in software development, data warehousing, and business intelligence.
©2017, O'Reilly Media, Inc. • (800) 889-8969 or (707) 827-7019 • Monday-Friday 7:30am-5pm PT • All trademarks and registered trademarks appearing on oreilly.com are the property of their respective owners. • firstname.lastname@example.org