Building scalable-application platforms is not easy. Enabling servers to communicate with other servers is difficult as well. RPC can be used to manage communication between tiers of a scalable application, but often some level of sharding of communication occurs. While this can be an effective method, it brings with it a certain amount of management overhead. Every time a server is added or taken away, some type of rebalancing must occur. (Another option is to utilize a registry.)
The intent of the Zeta Architecture is to support elastic expansion and contraction of services in the different tiers of your stack to optimize resource utilization across the data center. Manual sharding of communications between applications is NOT an option. The best way to support communication between these dynamically scalable applications is to communicate via a messaging platform that can easily handle trillions of events per day. After all, if the messaging platform can’t handle the scale, then it will not suffice as a communication channel between applications.
Jim Scott covers the benefits of this model and demonstrates its effectiveness by walking attendees through an example of data-center monitoring. In addition, Jim discusses the pros and cons of alternative methods, like using a registry to track servers that are alive and taking requests.
Jim Scott is the Head of Developer Relations, Data Science, at NVIDIA. He is passionate about building combined big data and blockchain solutions. Over his career, Jim has held positions running operations, engineering, architecture, and QA teams in the financial services, regulatory, digital advertising, IoT, manufacturing, healthcare, chemicals, and geographical management systems industries. Jim has built systems that handle more than 50 billion transactions per day, and his work with high-throughput computing at Dow Chemical was a precursor to more standardized big data concepts like Hadoop. Jim is also the cofounder of the Chicago Hadoop Users Group (CHUG).
©2016, O'Reilly Media, Inc. • (800) 889-8969 or (707) 827-7019 • Monday-Friday 7:30am-5pm PT • All trademarks and registered trademarks appearing on oreilly.com are the property of their respective owners. • email@example.com
Apache Hadoop, Hadoop, Apache Spark, Spark, and Apache are either registered trademarks or trademarks of the Apache Software Foundation in the United States and/or other countries, and are used with permission. The Apache Software Foundation has no affiliation with and does not endorse, or review the materials provided at this event, which is managed by O'Reilly Media and/or Cloudera.