SmartHub, the analytics division of StarHub, and NTT DATA, a global IT innovator in Japan with committers to Hadoop and Spark, have embarked on a partnership to design next-generation architecture to power the data products that will help generate new insights. YongLiang Xu and Masatake Iwasaki explain how deep learning and other analytics models can coexist on the same platform to address opportunities and challenges in initiatives such as smart cities.
Deep learning is the next key-enabler to transform data into actionable analytics products. However, big data platforms using technologies such as Hadoop and Spark remain the backbone for analytic applications. Therefore integrating big data platforms with deep learning technologies like TensorFlow is crucial to support the development of cutting-edge data analytics products.
YongLiang and Masatake present a reference architecture that incorporates distributed deep learning with an existing big data platform through frameworks such as Intel BigDL and TensorFlowOnSpark. This architecture creates an environment in which deep learning workloads can coexist with other existing analytics workloads and continue to leverage the same real-time data pipeline and monitoring frameworks within the platform.
YongLiang Xu is the lead data architect for SmartHub, the analytics division of StarHub, where he is responsible for transforming and architecting the next generation of big data architecture. His work includes reengineering SmartHub’s big data platform for real-time processing to support real-time machine learning and experimenting with new Apache projects and optimizing the big data platform for streamlined and seamless performance. Previously, YongLiang was a software engineer at DSO National Laboratories, Singapore, where he developed solutions based on big data technologies.
Masatake Iwasaki is a software engineer at NTT DATA, where he works on OSS professional services, including consulting, system integration, and technical support of open source software
such as Hadoop, Spark, and Storm for enterprise systems. He is also a committer for Apache Hadoop and Apache HTrace (incubating).
©2017, O'Reilly Media, Inc. • (800) 889-8969 or (707) 827-7019 • Monday-Friday 7:30am-5pm PT • All trademarks and registered trademarks appearing on oreilly.com are the property of their respective owners. • firstname.lastname@example.org