Present Tense: The Challenges and Trade-offs in Building a Web-scale Real-time Analytics System

Location: Mission City M
Average rating: ****.
(4.18, 17 ratings)

The massive increase in the amount of data generated from the sensors that surround us, in our cell phones, our computers, our hospitals, our power network, and the enormous pressure to produce results in real-time are beyond the capabilities of existing databases. To meet these emerging needs, we have built a platform for web-scale OLAP on sensor network data. In this talk, I will describe the challenges faced, the many false starts, and the approaches we ultimately took in building our storage and query infrastructure capable of real-time update and query performance over hundreds of terabytes of multidimensional data.

Photo of Benjamin Black

Benjamin Black


Mr. Black is a recognized expert in large-scale technology
infrastructure design, implementation,and operation. Previously, he
was an engineering director in the ECN group at Microsoft Windows
Live, where he helped build one of the largest content delivery
networks in the world. As the manager of website engineering at
Amazon, he lead the design and implementation of a new, layer 3
network for Amazon’s production network. As principal engineer for
Amazon infrastructure, he co-authored the first documents on what
later became EC2. As senior manager for information security, he owned
and successfully delivered Sarbanes-Oxley and PCI DSS compliance for
Amazon infrastructure, including the critical payments processing
systems. His work in the Internap R&D team resulted in his being
awarded 3 patents in Internet routing and traffic optimization. Mr.
Black is also co-author on several IETF RFCs on MPLS control protocols
and IPv6 network operations.

Comments on this page are now closed.


Todd Hoff
02/15/2011 2:48am PST

Any idea when the video will be up?

Picture of Alex Pinkin
Alex Pinkin
02/08/2011 2:56pm PST

I also thought this was one of the best developer oriented talks at the conference. A must see for anyone doing “home grown” OLAP.

Picture of Benoit Sigoure
Benoit Sigoure
02/03/2011 3:21pm PST

I have to agree with Adam’s comment. This was an excellent talk, with a great story behind it, and interesting pointers and food for thought.

Adam Browning
02/03/2011 8:29am PST

This was easily one of the best presentations that I’ve seen at Strata. Mr. Black was an engaging speaker and the presentation entertaining and informative. As an added bonus, the information was presented at a level useful to developers, instead of the more common extremely high-level management/media focus that many talks had.


  • Thomson Reuters
  • EMC Data Computing Division
  • EnterpriseDB
  • Microsoft
  • Gnip
  • Rackspace Hosting
  • IBM
  • Windows Azure MarketPlace DataMarket
  • Amazon Mechanical Turk
  • Amazon Web Services
  • Aster Data
  • Cloudera
  • Clustrix
  • DataStax, Inc. (formerly Riptano, Inc.)
  • Digital Reasoning Systems
  • Heritage Provider Network
  • Impetus
  • Jaspersoft
  • Karmasphere
  • LinkedIn
  • MarkLogic
  • Pentaho
  • Pervasive
  • Revolution Analytics
  • Splunk
  • Urban Mapping
  • Wolfram|Alpha
  • Esri
  • ParAccel
  • Tableau Software

Sponsorship Opportunities

For information on exhibition and sponsorship opportunities at the conference, contact Susan Young at

Download the Strata Sponsor/Exhibitor Prospectus

Contact Us

View a complete list of Strata Contacts