nlp course, nlp, tutorials,

NLP course Week 3 - Context

Ben Ben Follow Apr 14, 2020 · 1 min read
Share this

More notebooks from my upcoming course at Harvard Extension, here is week 2, focusing on going from tokens to vectors.

Week 3 notebook.

In this notebook, we build on the previous two weeks by considering one of the main sources of information in natural language: Sequence. Our previous methods essentially turn text into bags of words. But we know, intuitively, that the order of words changes the meaning.

Think of this example:

“Dog bit man”

“Man bit dog”

The word count vector for these would be exactly the same. But we know, on reading it, these aren’t the same. For the first, we’d call animal cotrol. The second, well, maybe PETA?

The topics covered here are:

  • Recurrent Neural Networks
    • With one-hot word encoding
    • With embeddings
  • Word Embeddings
    • Extracted from topic models (NMF and LDA)
    • Pre-trained (GloVe)

This is a pretty code-heavy notebook, a lot of it adapted from a really great example implementation of LSTM in PyTorch. I tried to document it extensively and the plan in the course is to walk, in more detail, through it. The implementation is pretty simple, but I think it covers, at a high level, the main considerations of designing one of these models.

Something that I found was that this notebook can be run really easily on Google Collaboratory with a GPU runtime. To compare, on my local machine it took ~30 minutes to run for five epochs. On the Collab GPU, it took <1 minute. Definitely will be recommending running it on Collab to the class.

Let me know what you think!

More to come.

Written by Ben Follow
I am the type of person who appreciates the telling of a great story. As a data scientist, I am interested in using AI to understand, explore and communicate across borders and boundaries. My focus is on Natural Language Processing and the amazing ways it can help us understand each other and our world. The material on this blog is meant to share my experiences and understandings of complex technologies in a simple way, focused on application.