Petascale Processing On An Open Source Budget: An Introduction to QFS

Beyond Hadoop Great America Ballroom J

Managing Big Data is not for the faint of heart. Data driven organizations, especially those operating petascale workloads, know that managing and making sense of vast quantities of data requires an intrepid and innovative engineering team, and that the need to address data volume and cost efficiency is paramount. Few companies understand this imperative better than Quantcast.

The company, which ingests more than 40TB of new data and processes in excess of 20PB of data each day, pioneered the Quantcast File System (QFS) and released it to open source in 2012. This best of breed file system offers a high performance, more efficient alternative to HDFS for managing, storing and processing massive data clusters. Leveraging QFS as its primary data store in tandem with Apache Hadoop, Quantcast can now better support the performance demands of its multi-petabyte workload, driving greater cost efficiency and consuming only half the hard drive space.

In this session, Quantcast VP of Research and Development Jim Kelly will offer a comprehensive overview of the QFS platform, its genesis, and explore when it makes sense to leverage QFS, HDFS or another comparable file system.

Photo of Jim Kelly

Jim Kelly

Quantcast

Jim Kelly leads Quantcast’s R&D team, which works on both adding computing capacity through cluster software innovations and using it up through new analytic and modeling products. Having been at Quantcast for six years, he has seen its data volumes and processing challenges grow from zero to petabytes and led technical and organizational changes that have kept Quantcast a step ahead. Previously he held engineering leadership roles at Oracle, Kana, and Scopus Technology (acquired by Siebel). Jim holds a PhD in physics from Princeton University.

Sponsors

Sponsorship Opportunities

For information on exhibition and sponsorship opportunities at the conference, contact Susan Stewart at sstewart@oreilly.com

Media Partner Opportunities

For information on trade opportunities with O'Reilly conferences contact Kathy Yu at mediapartners
@oreilly.com

Press and Media

For media-related inquiries, contact Maureen Jennings at maureen@oreilly.com

Contact Us

View a complete list of Strata contacts