Learn Natural Language Processing

Key Takeaways

I just made you a free NLP curriculum hello world it's Suraj and lately I've been getting a lot of questions on how to learn natural language processing or NLP so I personally designed a free up-to-date curriculum to help anyone learn it as efficiently as possible if you don't know NLP is a field of artificial intelligence that deals with helping computers understand interpret and manipulate human language I've designed this curriculum for absolute beginners with the goal of helping them either start a start up do consulting work or find full time work doing NLP after completing it NLP is endlessly fascinating with topics like sentiment analysis text generation dialogue systems there's so much you can do with text data in 2019 it's truly incredible memes that write themselves the prerequisites for this curriculum are knowledge of basic Python syntax it also helps to familiarize yourself with basic linear algebra calculus and probability luckily for you I've got really dense cheat sheets that give you an overview of each of these subjects before you start the curriculum and if you really want to learn more about them I've linked to my friend grants Anderson aka 3 blue one Browns YouTube channel playlists that cover them in detail if I was a beginner I'd give myself two months to take this curriculum but if you want to take longer please do the beauty of self-directed learning is that you can take as long as you'd like find a study buddy in our slack channel to have discussions with along your journey and keep you motivated to complete it to give you a sense of the speed of innovation in NLP just a few days ago a team at Google brain released Excel net a pre trained model that outperforms the previous state of the art released just a few months ago called Bert by a large margin on 20 tasks researchers on reddit lamented that they had literally just finished integrating Bert into their pipelines when something new just broke the state of the art such is the world of machine learning it's dank a F and honestly I wouldn't have it any other way there are seven modules in my curriculum an intro to NLP terminology and pre processing techniques language models and lexicons neural nets and word embeddings sequence modeling dialogue systems transfer learning and finally modern NLP research topics the goal is to help you learn pre deep learning NLP deep learning NLP techniques get you very comfortable with transfer learning then give you a taste of deep reinforcement learning at the end the most important part of this curriculum is the transfer learning bit if you really don't have time to learn anything else and you just want to build something ASAP go straight to the transfer learning portion each of these weeks has a series of video lectures reading assignments and a project to ensure you've understood the material remember most learning occurs by doing not by just passively absorbing information so the projects are important I've set pi torch to be the library of choice for this curriculum since there are so many clean pi torch NLP examples on github thanks Facebook you abusive lover so let's start with week 1 NLP terminology and pre-processing even though deep learning makes up the majority of NLP state-of-the-art algorithms it's important to start our journey before ml and I found a course that does this perfectly it's called speech and language processing by dan jurafsky at stanford i found the video lectures in this youtube playlist watch the first few up until 2.5 they cover important fundamental NLP terminology terms like pragmatics syntax and morphology there's a whole theory of language that linguists have devoted their lives to studying before statistical analysis came along and we should pay homage to it by gaining some domain knowledge of language theory before beginning to model it for specific tasks later on and let's not forget pre-processing the video lectures cover all the different pre-processing techniques to perform when working with text data which is usually very messy limitation stop word removal tokenization rarely do we find a corpus that is perfectly ready to be fed into a model we have to clean it up first the reading assignments that are already a part of the course from the professor's book of the same title are sufficient for this week read the first two chapters and the Associated slides the project for the first week is to perform three different pre-processing techniques of your choice on a paragraph of text you pull from the web or create yourself with the NLT K or natural language toolkit library before we start using PI torch we can use NLT Kay to clean up text data and this first week will help you master that skill now onto the second week language models and lexicons don't worry deep learning is coming but there are still decades of research in NLP that have nothing to do with deep neural networks that researchers have been using that we should learn about it's important to understand our history in order to figure out where we want to go next a statistical language model is a probability distribution over a sequence of words if we can create a language model from some text corpus we can use it to perform tasks like text classification or text generation in this style of that corpus it captures the essence of it the video lectures for this week come from the University of Washington's courts on NLP lectures 2 through 6 cover language models text classifiers hidden Markov models and parsing techniques the associated readings for each lecture are really good read 4 through 10 skip number 5 though since that one talks about neural nets and we're holding off on that until later let's also fit a technique called latent darish lay allocation into here by reading this tutorial blogpost it's a technique that lets you generate the topic from a corpus it's called topic modeling as for the project use PI torch to build a popular statistical technique called a hidden Markov model I have an example app for you to look at for help on this one but use your hmm to classify texts from a data set accordingly week 3 is dedicated to neural networks and word in weddings all modern NLP techniques use neural networks in some way or form so it's very important to understand how this statistical architecture works we can use these networks to create embeddings these are mathematical representations of words sentences and sometimes even hold documents and these embeddings allow us to compare texts visualize it improve the accuracy of newer models and prevent warfare kidding about that last one kind of Stanford has the perfect course on how to get started with this it's called NLP with deep learning video lectures 1 through 5 from word embeddings up until optimization strategies fit this topic nicely and it's already got a list of suggested readings for each video lecture as well as notes definitely read those as for the project The Associated assignments are pretty nice go through and complete all three but just the coding parts and do them all in PI torch these assignments will have you visualize and implement word to Veck models and then create a dependency parser all in pi torch now we're going to dedicate two weeks to the next module just because there is so much involved with sequence modeling whether it be an audio waveform of your voice or a sentence we can think of this type of data as sequential since it consists of related data points that occur sequentially like words or waves when we feed a sequence let's say of words into a model and output a new sequence say a new sentence of words we can consider that sequence to sequence modeling so many tasks in NLP can be formulated as such machine translation text summarization question answering and the encoder/decoder model that takes in and outputs a sequence of words has been the go-to model for this type of modeling there are many variants of sequence to sequence models out there and all sorts of techniques like how to apply attention to certain layers with papers like attention is all you need and all sorts of variants of recurrent neural networks like LS TM & GRU networks we more of this Coursera course on NLP covers all sorts of sequence modeling techniques particularly machine translation for the reading lectures check out Chapter ten of the deep learning book by Ian Goodfellow on sequence modeling also I wanted to fit deep semantic similarity models into here as well they are able to model how similar to text strings are and have a wide variety of use cases in popular tasks this blog post will bring you up to speed pretty fast on this now the bulk of the work from this week will come from the project using the popular movie lens data set that contains movie dialogues in PI torch build two different models a text summarizer and a question answering system yes deep learning requires GPUs it's okay if you don't have one just do it in a web browser with Google collab with their free compute this is a crucial project the reason it's important to implement recurrent sequence models using PI torch for these tasks is so that you have a deep appreciation for the state of the art today in sequence modeling which actually has nothing to do with recurrent Network variants today it's all about the transformer model and yes we'll get to transformers but first we need to move on to the next week dialog systems like Alexa Google home and Siri this is the study of how humans interact with machines and all the different ways we can architect these types of dialogue based systems the last week of the same Coursera course called dialogue systems has the perfect set of video lectures for this covering different dialogue system designs that help classify user intent and send the appropriate response or follow-up question to the user this is an active area of research google recently demoed its duplex product that was able to automate a phone call with a human in a surprisingly realistic way I'm looking for something I'm a third don't give me one what mm-hmm what for reading assignments chapter 24 of the speech and language processing book will do great and there's one project here create a dialogue system using PI torch that enables someone to check the weather that means it has to take context into account each question depends on the previous response text now on to week six transfer learning this is the most important week in fact if you really wanted to get started with NLP like make something impressive in just a few hours just start here recent models like Bert are all free for anyone to download and use in their app as a starting point for training on a specific task like sentiment analysis or text generation I have a great video on how Bert works for both theoretically and in practice in the form of an app I also have a video on GPT to these models are generally all variants of what's called the transformer network the architecture that's now used by all the major research labs instead of recurrent networks for sequence learning that's huge and it's not yet a part of any major curriculum because it's so new I've got some great reading assignments for you on this front Sebastian Reuter has a great blog post on why NLP just had its imagenet moment and what that means and two other blogs that describe in technical detail how the transformer Network works in the context of all of these different pre-trained models I've also got this giant list of high torch pre-trained examples your project for this week is to pick two of these models and use them both for sentiment analysis for product reviews which will involve transfer learning get used to the idea that you have to retrain an existing giant model on some data specific to your task this process is much faster than doing it from scratch now on to the last week modern NLP research so first let's just get this vision language hybrid model thing out of the way by that I mean anything having to do with translating between an image and text image captioning visual question-answering or even text to image this involves combining sequence models with convolutional nets and module 6 of this EDX Microsoft course will give you some great video lectures on this now on to deep reinforcement learning this is pretty experimental first check out this hilarious yet educational blog post by Goldberg called an adversarial review of adversarial generation of natural language he's basically saying that you can't just throw some hot new technique like adversarial networks at a problem that requires a deep understanding of language theory and expect it to beat the state of the art with that being said there is interesting work being done applying deep rl2 language this Carnegie Mellon lecture will give you a great primer and so will module 5 of that same EDX course reinforcement learning applied to static text data is really interesting to me because it's usually applied to scenarios where time is a crucial element like games real-world supply chains or sensor data there's a policy gradient model that's used for text summarization I found implemented in tensorflow the last project is to merely re-implement that int I torch this will enable you to more fully grasp this concept of combining the function approximation capability of deep neural networks with reinforcement learnings mathematical framing of a time-based continuous environment to more efficiently solve an objective function and that's it after finishing this curriculum I want to see you do one of three things using your NLP knowledge start a startup start consulting or find full-time work knowledge and action are inextricably connected so act use your knowledge to act in this world for the betterment of your own life in the lives of others in this way AI technology will help you find happiness wealth meaningfulness and inner peace I hope you found this video useful and if you did please click the red subscribe button good luck wizards I'm rooting for you [Music]