The Reformer

Key Takeaways

hey nice computer oh thanks I bought 40 GPUs to do some text processing hmm well now you just need one hello world it's Suraj and Google just released a new language model called the reformer that's able to process text sequences that are up to 1 million words long using only one accelerator and 16 gigs of memory what that means for you is that it's now easier than ever before to build a surprisingly impressive app or project that can generate text classify text categorize text or summarize text with a low cost computer in this episode I'll help you understand what makes the reformer architecture unique compared to its predecessors and will repurpose Google's simple Python demo to ingest an entire biology textbook and help us answer questions we have about its topics automatically to truly understand the Reformers architecture we've got to wind the clock back to seven years ago it was 2013 the year the movie sharknado was released it's so bad that it's good up until that point the field of natural language processing consisted of rule-based & heuristic techniques that worked well enough but when a model called word Tyvek was released that year it changed how developers thought about natural language processing more Tyvek was a neural network that was able to literally turn words into vectors aka numerical representations and this numerical format allowed us to apply mathematical operations to these words to compute results we could say create an expression using vectors like King - man plus woman and it would output Yas Queen without the Yas yeah we could visualize these vectors and see how similar words cluster together or use them to help make other types of models more robust this technique was a new lens with which to see the world of data this way of learning representations rather than applying rules to raw text then in 2015 a model called fast text extended the word Tyvek model by treating each word vector as the sum of a series of character level vectors doing cell allowed us to present rare words words that didn't show up that often in the training data more robustly since characters are often shared between words two years later in 2017 Google release in architecture called the transformer in a paper titled attention is all you need in the paper Google showed how they achieved state-of-the-art results in automatic language translation using a concept called an attention mechanism this attention mechanism allowed their model to not just ingest all the input data during the training process but instead pay attention to the most relevant parts of the input the parts that matter most to make the most accurate predictions this in turn helped reduce the amount of training time necessary by a large margin so by early 2018 there were some powerful open source models available for anyone to use on their own text processing projects but to get really great results you still needed lots of computing power and lots and lots and lots of data that is until Google released Bert or bi-directional encoder representations from transformers they modified the original transformer architecture to take the full context of a word into account when learning from it Bert's bi-directional nature learned a words context from both ends of a sentence ie bi-directionally they trained Bert on an absolutely massive amount of text data including all of Wikipedia and 800 million words from books and then open sourced it for anyone to use as a starting point for their own project that moment the release of Bert might just have been the most significant contribution to the field of natural language processing of all time similar to how convolutional networks changed computer vision forever in 2012 achieving state of the art accuracy during the image net competition because Bert was already trained on so much text data developers could leverage its knowledge to quickly and easily train NLP apps for a huge range of tasks without requiring huge data sets tasks like question answering machine translation text summarization sentiment analysis text cat zatia dialogue systems and common-sense reasoning all became very accessible to build and if you didn't want to use Bert you could just use the slew of Bert variations that were released by other companies throughout the rest of the year like Ernie Elmo I love these Sesame Street naming conventions roberta excel ned GPT to transformer x and my personal favorite bio bert bio Bert was Bert retrained on biomedical domain knowledge from experts and its variations included an automatic diagnostic app just give it your symptoms and it will tell you a preliminary diagnosis including possible treatment options so now it's 2020 and google has released the reformer architecture they've re-engineered the transformer architecture to be much more efficient in terms of speed and storage or in more computer science e terms its space and time complexity are much better to do this they used two techniques locality-sensitive hashing and reversible layers let's dive into why they used both of them let's say we have a sequence of words in the form of a book and we want to learn from it we wouldn't want to use a feed-forward neural network since they aren't meant for sequential data where all the inputs are related in some way we'd want to use a recurrent Network but while recurrent Nets are great they're sequential processing nature is not well-suited for the parallel processing nature of GPUs and TP use we could maybe use convolutional networks instead but they tend to get computationally expensive for long sequences sorry yawn let's just discard recurrence and convolutions entirely maybe all you really need is attention if we're reading a mystery novel and we're trying to remember the most relevant characters we pay attention to certain words more than others namely character names this internal mechanism we have allows us to interpret text data faster since we focus on the most relevant parts we can mathematically represent this as an attention mechanism think of it as a vector of importance weights in order to predict a given element like a word in a sentence we estimate you an attention vector how much it's correlated with or attends to other elements then we take the sum of their values weighing them by the attention vector as a proper prediction of the target attention can be thought of as a clever matrix math trick that makes training time faster but the attention mechanism used by the transformer architecture took way too many words into account when trying to understand the relationships between them and thus the memory footprint reflected that so in the reformer paper they use a hash function to match similar input vectors together a hash function takes a group of characters also called keys as input and map's each of them to a smaller length ID called a hash value the hash represents the original inputs too many words might impose big watch me hash it I like to hash it I like to hash it so when feeding the network input text each vector from layer 1 represents a given word the hash function will then give similar vectors the same hash value and once these hashes are given the input sequence is reordered to bring together similar vectors based on their hash and chunked to enable parallel processing it's only then that we apply attention to these much smaller chunks which makes the training process much faster they call this technique locality sensitive hashing because sensitivity is so 20/20 more seriously in their own words a hashing scheme that assigns each vector X to a hash H of X is called locality sensitive if nearby vectors get the same hash with high probability and distant ones do not okay so that was a first solution the other one was an idea introduced in 2017 called reversible layers the way neural networks are trained is by using a technique called back propagation which LaRon Sanderson made an excellent video on it's basically a way of propagating the lessons learned during the training process to the entire network this process however requires storing what are called activations in so the more activations the higher the memory footprint activations help a network learn the optimal function from the data to make predictions instead of storing all the activations in memory the reformer recomputes the input of each layer on demand during the backpropagation process two sets of activations are kept for each layer one captures changes to the first layer and the other captures the last layer thus using subtraction the network is able to recompute the necessary input values during back propagation without having to store it all in memory SiC right but what better way to understand the reformer and by playing with the code ourselves let's take a look at our text generation collab demo first we'll need to install jax for hardware acceleration and tensorflow for deep learning then we'll load a biology textbook from the Gutenberg library into it instead of the example they used okay boomer then we tokenize it using the sentence piece processor function in order to prevent overfitting we'll add random padding to each token so the model does not just memorize the data set completely it'll be able to add some novelty to its generated predictions clearly in this example there are a lot of hyper parameters that we can configure here and no doubt the values of what these numbers should be will sprout multiple papers this year then we can use the Trax library which they decided to wrap this model into to setup a trainer object once trained we can sample from it giving it a seed sentence and it'll fill in the rest I'm pretty excited to see what NLP researchers come up with this year the primitives have been defined attention mechanisms sequence learning and gradient based optimization now the next phase is to tune and dial up these models a millionfold to produce amazing results the winner of last week's coding challenge is bintang a swamp bintang submitted this aircraft design repository which isn't exactly a healthcare app but it's still cool so good job in tang i've got links to some great learning resources for you in the video description so hit that subscribe button like the video and happy learning