The big data platforms of many organizations are underpinned by a technology that will soon celebrate its 45th birthday: SQL. This industry stalwart is applied in a multitude of critical points in business data flows; the results that these processes generate may significantly influence business and financial decision making. However, the SQL ecosystem has been overlooked and ignored by more recent innovations in the field of software engineering best practices, such as fine-grained automated testing and code quality metrics. This exposes organizations to poor application maintainability, high bug rates, and ultimately corporate risk.
Elliot West and Jay Green share approaches for applying software engineering best practices to SQL-based data applications to improve maintainability and data quality. They first define the relevance of such approaches and demonstrate how automated testing can be applied to Hive SQL using HiveRunner, a JUnit based testing framework. They then consider how best to structure Hive queries to yield meaningful test scenarios that are maintainable and performant. Elliot and Jay conclude by demonstrating how test coverage reports can highlight areas of risk in SQL code bases and weaknesses in the testing process and offer an overview of Mutant Swarm, an open source mutation testing tool for SQL languages developed by Hotels.com that can deliver insights similar to those produced by Java-focused tools such as Jacoco and PIT.
Elliot West is a principal engineer at Hotels.com in London, where he designs tooling and platforms in the big data space. Previously, Elliot worked on Last.fm’s data team, developing services for managing large volumes of music metadata.
Jay Green is a final-year student in computer science at King’s College London. She joined the big data platform team at Hotels.com for her industrial placement year, where she’s worked with Apache Hive, modularization techniques for SQL, and mutation testing tools.
For exhibition and sponsorship opportunities, email strataconf@oreilly.com
For information on trade opportunities with O'Reilly conferences, email partners@oreilly.com
View a complete list of Strata Data Conference contacts
©2019, O’Reilly UK Ltd • (800) 889-8969 or (707) 827-7019 • Monday-Friday 7:30am-5pm PT • All trademarks and registered trademarks appearing on oreilly.com are the property of their respective owners. • confreg@oreilly.com