Most data centers (and many cloud deployments) are statically partitioned into siloed clusters dedicated to running individual datacenter-scale applications, including web services, databases, and batch/stream processing. This static partitioning model limits overall cluster utilization, decreases flexibility, and poses operational challenges. There is an increasing need to integrate big data applications like Apache Hadoop and Apache Spark with other data center services like Apache Cassandra or Apache Kafka, ideally colocating the data with the services that need it.
Adam Bordelon and Mohit Soni demonstrate how projects like Apache Myriad (incubating) can install Hadoop on Mesosphere DC/OS alongside other data center-scale applications, enabling efficient resource sharing and isolation across a variety of distributed applications while sharing the same cluster resources and hence breaking silos. The multitenancy strategy improves overall cluster utilization and operational efficiency while allowing cluster operators to run multiple isolated data services on the same hardware.
Adam Bordelon is a distributed systems engineer at Mesosphere and an Apache Mesos committer. Before joining Mesosphere, Adam was lead developer on the Hadoop core team at MapR Technologies, developed distributed systems for personalized recommendations at Amazon, and rearchitected the LabVIEW compiler at National Instruments. He holds a master’s degree from Rice University, where he built a tool to analyze supercomputer performance data for bottlenecks and anomalies.
Mohit Soni is a distributed applications engineer at Mesosphere building the Datacenter Operating System (DCOS). Previously Mohit worked as an engineer on the Platform team at eBay, where he focused on maximizing efficiency, increasing agility, and reducing cost. Mohit has presented at DockerCon 2014, Hadoop India Summit 2011, and BarCamp 2010. You can follow him as mohitsoni on GitHub and Twitter.
©2016, O'Reilly Media, Inc. • (800) 889-8969 or (707) 827-7019 • Monday-Friday 7:30am-5pm PT • All trademarks and registered trademarks appearing on oreilly.com are the property of their respective owners. • firstname.lastname@example.org
Apache Hadoop, Hadoop, Apache Spark, Spark, and Apache are either registered trademarks or trademarks of the Apache Software Foundation in the United States and/or other countries, and are used with permission. The Apache Software Foundation has no affiliation with and does not endorse, or review the materials provided at this event, which is managed by O'Reilly Media and/or Cloudera.