The glue: Building the connectors and tools to manage big data warehouses

For most companies, data analysis means collecting the data, building a data pipeline to clean and transform the data into a usable form, and only then looking for insights. Without good tools to automate the data pipeline, data flow management can become a tedious and brittle process.

In this talk we highlight some useful tools that we built in-house:

• Scheduling of nightly jobs and anomaly detection to catch errors in the data or in the data transformation code
• A backfill tool to retroactively update historical data to reflect new changes in the code
• A dependency management tool to capture dependency restrictions in the data pipeline and schedule jobs to run in the optimal order
• A data versioning tool that “remembers” which query generated the data. Without this, there is no way to tell apart data generated from two different versions of the code that may have very different logic, leading to faulty conclusions