Presented By O'Reilly and Cloudera
Make Data Work
March 28–29, 2016: Training
March 29–31, 2016: Conference
San Jose, CA

Scaling your business with a messaging platform on the Zeta Architecture

Jim Scott (NVIDIA)
4:20pm–5:00pm Thursday, 03/31/2016
IoT and Real-time

Location: 210 D/H
Tags: real-time
Average rating: ****.
(4.67, 3 ratings)

Prerequisite knowledge

Attendees should have a general understanding of software engineering.


Building scalable-application platforms is not easy. Enabling servers to communicate with other servers is difficult as well. RPC can be used to manage communication between tiers of a scalable application, but often some level of sharding of communication occurs. While this can be an effective method, it brings with it a certain amount of management overhead. Every time a server is added or taken away, some type of rebalancing must occur. (Another option is to utilize a registry.)

The intent of the Zeta Architecture is to support elastic expansion and contraction of services in the different tiers of your stack to optimize resource utilization across the data center. Manual sharding of communications between applications is NOT an option. The best way to support communication between these dynamically scalable applications is to communicate via a messaging platform that can easily handle trillions of events per day. After all, if the messaging platform can’t handle the scale, then it will not suffice as a communication channel between applications.

Jim Scott covers the benefits of this model and demonstrates its effectiveness by walking attendees through an example of data-center monitoring. In addition, Jim discusses the pros and cons of alternative methods, like using a registry to track servers that are alive and taking requests.

Photo of Jim Scott

Jim Scott


Jim Scott is the head of developer relations, data science, at NVIDIA. He’s passionate about building combined big data and blockchain solutions. Over his career, Jim has held positions running operations, engineering, architecture, and QA teams in the financial services, regulatory, digital advertising, IoT, manufacturing, healthcare, chemicals, and geographical management systems industries. Jim has built systems that handle more than 50 billion transactions per day, and his work with high-throughput computing at Dow was a precursor to more standardized big data concepts like Hadoop. Jim is also the cofounder of the Chicago Hadoop Users Group (CHUG).