Recurrent neural networks (RNN) and related models represent the state of the art in language modeling. Trained on sufficiently large corpora, RNNs can learn to generate convincing text that obeys rules of syntax and even matches parentheses. What is especially remarkable is that these models can synthesize text from diverse inputs: text in another language (for translation), images and video (for captioning), and scores and categories for creation of personalized product reviews.
Josh Patterson and David Kale offer a thorough introduction to context-dependent text generation using conditional RNNs. Traditional RNN-based language models are utilized in an unsupervised fashion, taking as their inputs only previous tokens in the sequence. In contrast, conditional RNNs (also called context-dependent or concatenative RNNs) augment the input sequence with auxiliary information, enabling the RNN to model rich and complex interactions between the input and output spaces. Josh and David present an overview of conditional RNN architectures, both simple (generative concatenative networks) and elaborate (attention models), and discuss diverse potential applications, ranging from automatic generation of personalized user content to video captioning.
They then demonstrate a real-world example—an interactive Twitter bot that dynamically generates text responses based on user requests—before concluding with a rigorous debate about the implications of such technology, including both the potential business impact (could RNNs be used to generate fake Yelp reviews?) and the philosophical question of whether it represents machine creativity.
Josh and David will provide a fully open source implementation using the Deep Learning for Java (DL4J) library and make the training dataset freely available.
Josh Patterson is CEO of Patterson Consulting, a solution integrator at the intersection of big data and applied machine learning. In this role, he brings his unique perspective blending a decade of big data experience and wide-ranging deep learning experience to Fortune 500 projects. At the Tennessee Valley Authority (TVA), Josh drove the integration of Apache Hadoop for large-scale data storage and processing of smart grid phasor measurement unit (PMU) data. Post-TVA, Josh was a principal solutions architect for a young Hadoop startup named Cloudera (CLDR), as employee 34. After leaving Cloudera, Josh co-founded the Deeplearning4j project and co-wrote Deep Learning: A Practitioner’s Approach (O’Reilly Media). Josh was also the VP of Field Engineering for Skymind. Josh also co-wrote the upcoming Oreilly book “Kubeflow Operations”
David Kale is a deep learning engineer at Skymind and a PhD candidate in computer science at the University of Southern California, where he is advised by Greg Ver Steeg of the USC Information Sciences Institute. His research uses machine learning to extract insights from digital data in high-impact domains, such as healthcare, and he collaborates with researchers from Stanford Center for Biomedical Informatics Research and the YerevaNN Research Lab. Recently, David pioneered the application of deep learning to modern electronic health records data. At Skymind, he works with clients and partners to develop and deploy deep learning solutions for real world problems. David co-organizes the Machine Learning for Healthcare Conference (MLHC) and has served as a judge in several XPRIZE competitions, including the upcoming IBM Watson AI XPRIZE. He is the recipient of the Alfred E. Mann Innovation in Engineering Fellowship.
©2016, O'Reilly Media, Inc. • (800) 889-8969 or (707) 827-7019 • Monday-Friday 7:30am-5pm PT • All trademarks and registered trademarks appearing on oreilly.com are the property of their respective owners. • firstname.lastname@example.org
Apache Hadoop, Hadoop, Apache Spark, Spark, and Apache are either registered trademarks or trademarks of the Apache Software Foundation in the United States and/or other countries, and are used with permission. The Apache Software Foundation has no affiliation with and does not endorse, or review the materials provided at this event, which is managed by O'Reilly Media and/or Cloudera.