The ubiquity of sequential decision problems throughout Computer Science makes Deep Reinforcement Learning one of the most exciting developments of modern AI. However, realizing the potential of such general frameworks in real applications has proven to be much more challenging. I use my work over the last few years on building and deploying an RL-based relational query optimizer, a core component of almost every database system, as an exemplary application that highlights some of the under-appreciated challenges in Deep RL practice. RL algorithms today: (1) do not fully exploit the structure of software simulators by collecting data episodically instead of strategically rewinding, fast-forwarding, skipping, (2) are very sensitive to policy parametrization especially in cases where there are hierarchical or discontinuous policy structures, and (3) struggle in “over-actuated” problems where the action space has significant redundancy. For all three challenges, I present experimental results illustrating phenomena in practice, our algorithmic solutions, and highlight the ways in which the same phenomena appear in other RL domains such as robotics.
Sanjay Krishnan is an Assistant Professor of Computer Science at the University of Chicago. His research focuses on applications of machine learning and control theory to computer and cyber-physical systems problems. Sanjay completed his PhD and Masters Degree at UC Berkeley in Computer Science in 2018. Sanjay’s work has received a number of awards including the 2016 SIGMOD Best Demonstration award, 2015 IEEE GHTC Best Paper award, and Sage Scholar award.
Help us make this conference the best it can be for you. Have questions you'd like this speaker to address? Suggestions for issues that deserve extra attention? Feedback that you'd like to share with the speaker and other attendees?
Join the conversation here (requires login)
©2019, O'Reilly Media, Inc. • (800) 889-8969 or (707) 827-7019 • Monday-Friday 7:30am-5pm PT • All trademarks and registered trademarks appearing on oreilly.com are the property of their respective owners. • email@example.com