Build an AI Writer - Machine Learning for Hackers #8

Key Takeaways

My dear Megan Tang, this is our love story. Human or machine? Human. Why? I found some drops of soilent on it. Hello world. Welcome to Serology. In today's episode, we're going to build an AI writer. That is an app that can write a short story about an image just by looking at it. Sorry, Stephen King. You're out of the game. Isn't it weird how just by stringing together an exact combination of words, we can produce something of profound beauty? When we read these stories, our brains are somehow encoding them into thoughts. When we encode a sentence to a thought, the more semantically similar it is to an existing thought, the more we'll be able to relate to it. So, how do we get an AI to write a story if it doesn't have any experience living life in the real world? Well, we're going to build an AI writer in Python using the deep learning library lasagna. And we've got a lot of code to go over, so I'll explain as we go. Let's get Pythonic. At the highest level, we could code this app in just three lines of code. It's a little ridiculous. We import the generate class, then call the load all function, which will initialize all of our machine learning model. Then call the story function with the generated models and image location as the parameter. That's it. It'll output a story. But let's dive a little deeper. The load function is just boilerplate initialization. So let's take a closer look at the story function where the real magic happens. We'll start off by loading an image into memory. This will be the image that we want to tell a story about. We'll use the load image function to load it and have the parameter set to the location of the image on our machine. The load image function uses the scientific computing library numpy to get the bite representation of the image and then resize it so it's smaller. while preserving its aspect ratio. Once we've loaded our image, it's time to input the image into a deep convolutional neural network to retrieve its features. In a previous episode, we talked about how convolutional neural nets were great for image recognition since they roughly mimic the human visual cortex. This CNN is pre-trained. We initialize it in the build connet function which is called in the boilerplate load all method. Once we specified all the layers, we load up our pre-trained synapse weights file called BG19. This file was trained on a huge data set of labeled images. So it will be able to recognize the objects in a novel image. Once we input our image into our CNN, it'll return an array of features for us. These features are the highest level features in our neural net, the layer right before the output layer, the most abstract representation of the image, its content. Once we have our features, we'll want to encode the image features into a multimodal neural language model. So what is this? Well, it's based off a paper called unifying visual semantic embeddings. In our code, we're using a pre-trained model that will input a joint image sentence embedding into a multimodal vector space. It used an LSTM to encode the sentence and a CNN to encode the image. Then a decoder neural language model generates a novel description from the image. Since our model is pre-trained, when we embed our image into this multimodal space, our features are updated to include the weights of the joint space. Then we compute the nearest neighbors. To do this, first we retrieve the array of scores, that is a list of all novel sentences generated from the novel image, which we then sort in order of closeness. Then we'll want to print out the nearest captions. Now that we have a set of captioned sentences, we'll want to compute a set of skip thought vectors for each sentence. Skip thought vectors are a vector representation of a sentence. This is another implementation of the encoder decoder model. The encoder and decoder are both recurrent neural networks. We take an input sentence and encode it into a skip thought vector by inputting it into the encoding recurrent neural net. Since we are modeling a sequence of words, we use gated recurrent units or GRUs at each neuron. GRUs consists of two gates, an update gate and a reset gate. The gating units modulate the flow of data inside the unit. And unlike LSTM cells, there are no separate memory cells. LSTM cells control the amount of memory content that is seen or used by other units in the network. GRU cells don't. They expose its full content without any control. So GRUs have a less complex structure and are thus more computationally efficient. We're starting to see these be used more and more. They're relatively new. So when we feed the sentences into the RNN, it'll create an abstraction, the vector representation or skip thought vector. Sentences that share semantic and syntactic properties will be mapped to either the same or similar skip thought vectors. The function returns these vectors as a numpy array which we can then modify via the styles shift function. We'll take our thought vectors and modify them to match the style of stories using a pre-trained recurrent neural network. The RNN was trained on a data set of romance novels where each passage was mapped to a thought vector. So we're essentially computing a function that looks like this for a style shift. F ofx is a book passage thought vector. X is an image caption, C is a caption style vector, and B is a book style vector. We remove the caption style from the caption and replace it with the book style to create a book passage vector. Once we have our book passage styled vector, we can generate the story by running the decoder function on it. The decoder is another recurrent neural network that given a vector representation of a sentence can predict the previous and the next sentence. We'll run the decoder on our passage vector and that will generate our story based on the image for us. Let's take a look at what it says about this picture. Let's read the first few sentences. She was taking the man out of her mouth and she gave him a gentle shake of her head. Oh my god, I can't wait to see what happened in the past 24 hours. I had never met a woman before. This thing is a pro. For a small chunk of code, there's a lot of machine learning going on here. We use a convolutional neural net to compute image features, an LSTM recurrent neural net to encode our image into joint space and retrieve the sentence captions, a GRU recurrent neural net to calculate the skip vectors of those sentences, and after style shifting, an RNN to decode our passive vector to a story. That's four neural nets. You can run this on your local machine since the necessary models are pre-trained. For more info, check out the links below. And I just signed up for Patreon. So, if you guys find my videos useful, I'd really appreciate your support to help me continue doing this full-time. Please subscribe for more ML videos. And for now, I've got to go fix a null pointer exception. So thanks for watching.