Skip to main content

Adaptive Adversaries: Building Systems to Fight Fraud and Cyber Intruders

Ari Gesher (Kairos Aerospace)
Average rating: ***..
(3.25, 4 ratings)

Statistical machine learning techniques tend to fail when faced with an adaptive adversary attempting to evade detection in the data. Humans do an excellent job of correctly spotting adaptive adversaries given a good way to digest the data. On the other hand, humans are glacially slow and error-prone when it comes to moving through very large volumes of data, a task best left to the machines.

Fighting complex fraud and cyber-security threats requires a symbiosis between the computers and teams of human analysts. The computers use algorithmic analysis, heuristics, and/or statistical characterization to find interesting ‘simple’ patterns in the data. These candidate events are then queued for in-depth human analysis in rich, expressive, interactive analysis environments.

In this talk, we’ll take a look at case studies of three different systems, using a partnership of automation and human analysis on large scale data to find the clandestine human behavior that these datasets hold, including a discussion of the backend systems architecture and a demo of the interactive analysis environment.

The backend systems architecture is a mix of open source technologies, like Cassandra, Lucene, and Hadoop, and some new components that bind them all together.

The interactive analysis environment allows seamless pivoting between semantic, geospatial, and temporal analysis with a powerful GUI interface that’s usable by non-data scientists.

The systems are real systems currently in use by commercial banks, pharmaceutical companies, and governments.

Photo of Ari Gesher

Ari Gesher

Director, Software Engineering, Kairos Aerospace

Ari Gesher is the founding director of software engineering at Kairos Aerospace, a startup building and operating the next-generation of airborne and spaceborne sensors for monitoring oil and gas infrastructure. Ari also serves as consulting architect for Jupiter, a company productizing high-quality datasets that describe the long-term effects of climate change. Previously, he was a very early engineer at Palantir Technologies and later served as Palantir’s engineering ambassador to the tech community at large; before Palantir, he was the maintainer of the SourceForge.net open source archive. Ari is the coauthor of The Architecture of Privacy, which explains how to responsibly hold data about people while preserving their privacy to the greatest extent possible. Ari is a frequent speaker on various topics, including the need for modern, high-leverage engineers to work on substantive problems, human-computer symbiosis as system design aesthetic, the limits of automated decision making, and privacy architectures for a world where everything is recorded.