Recurrent Neural Networks

Objectives

  • Define notation for building sequence models
  • Describe the architecture of a basic RNN
  • Identify the main components of an LSTM
  • Implement backpropagation through time for a basic RNN and an LSTM
  • Give examples of several types of RNN
  • Build a character-level text generation model using an RNN
  • Store text data for processing using an RNN
  • Sample novel sequences in an RNN
  • Explain the vanishing/exploding gradient problem in RNNs
  • Apply gradient clipping as a solution for exploding gradients
  • Describe the architecture of a GRU
  • Use a bidirectional RNN to take information from two points of a sequence
  • Stack multiple RNNs on top of each other to create a deep RNN
  • Use the flexible Functional API to create complex models
  • Generate your own jazz music with deep learning
  • Apply an LSTM to a music generation task

Natural Language Processing & Word Embeddings

Objectives

  • Explain how word embeddings capture relationships between words
  • Load pre-trained word vectors
  • Measure similarity between word vectors using cosine similarity
  • Use word embeddings to solve word analogy problems such as Man is to Woman as King is to ______.
  • Reduce bias in word embeddings
  • Create an embedding layer in Keras with pre-trained word vectors
  • Describe how negative sampling learns word vectors more efficiently than other methods
  • Explain the advantages and disadvantages of the GloVe algorithm
  • Build a sentiment classifier using word embeddings
  • Build and train a more sophisticated classifier using an LSTM

Sequence Models and Attention Mechanism

Objectives

  • Describe a basic sequence-to-sequence model
  • Compare and contrast several different algorithms for language translation
  • Optimize beam search and analyze it for errors
  • Use beam search to identify likely translations
  • Apply BLEU score to machine-translated text
  • Implement an attention model
  • Train a trigger word detection model and make predictions
  • Synthesize and process audio recordings to create train/dev datasets
  • Structure a speech recognition project

Transformer Networks

  • Create positional encodings to capture sequential relationships in data
  • Calculate scaled dot-product self-attention with word embeddings
  • Implement masked multi-head attention
  • Build and train a Transformer model
  • Fine-tune a pre-trained transformer model for Named Entity Recognition
  • Fine-tune a pre-trained transformer model for Question Answering
  • Implement a QA model in TensorFlow and PyTorch
  • Fine-tune a pre-trained transformer model to a custom dataset
  • Perform extractive Question Answering