word2vec and friends

Description

Word embeddings have received a lot of attention ever since Tomas Mikolov published word2vec in 2013 and showed that the embeddings that a neural network learned by “reading” a large corpus of text preserved semantic relations between words. As a result, this type of embedding began to be studied in more detail and applied to more serious NLP and IR tasks, such as summarization and query expansion. More recently, researchers and practitioners alike have come to appreciate the power of this type of approach, creating a burgeoning cottage industry centered around applying Mikolov’s original approach to different areas.

Bruno Gonçalves explores word2vec and its variations, discussing the main concepts and algorithms behind the neural network architecture used in word2vec and the word2vec reference implementation in TensorFlow, and shares some of the applications word embeddings have found in various areas. Bruno starts with an intuitive overview of the main concepts and algorithms underlying the neural network architecture used in word2vec. Bruno then presents a bird’s-eye view of the emerging field of “anything”-2vec methods (dna2vec, node2vec, etc.) that use variations of the word2vec neural network architecture.

Outline:

Neural network architecture and algorithms underlying word2vec

• Basic intuition
• Skip-gram
• Softmax
• Cross-entropy
• BackProp
• Online sources for pretrained embeddings

Properties and applications of word embeddings

• Visualization
• Analogies

A brief overview of TensorFlow

• Installation
• Computational graph
• Simple example (linear fitting)

A detailed discussion of TensorFlow’s reference implementation

word2vec variations and their applications

• Linguistic Change
• node2vec
• dna2vec