At SignalFx, we deal with high-volume high-resolution data from our users. This requires a high performance ingest pipeline. Over time we’ve found that we needed to adapt architectural principles from specialized fields such as HPC to get beyond performance plateaus encountered with more generic approaches. Some key examples include:
This presentation will provide examples of putting these principles into practice and the before/after results we’ve experienced in the performance of our own services. We believe these lessons will be useful to anyone building services that have to consume large amounts of data. :) Which is probably everyone.
Rajiv Kurian is a software engineer with over five years experience building high performance distributed systems like databases, networking protocols and image processing. At SignalFx, Rajiv works on improving the performance of the ingest pipeline.