Predicting Protein Structures using Deep Learning with Jonathan King

Key Takeaways

everybody um like I said my name is Jonathan and I'm gonna be telling you a little bit about protein structure prediction oh wait uh maybe I was using the wrong first slide there but you know if only there was a way that we could easily translate languages you know maybe something like Google Translate oh that's great so if you don't speak Chinese like I don't you can just use Google Translate and you can figure out that I'm actually going to be talking to you today about how to predict protein structure using language translation techniques so I'm I'm really excited about this and there's been a lot of development and language translation techniques and I'll try to talk about how that is really but first let's go over some biology so oh I should should really say that my project is actually about how Google Translate is going to write my thesis so anyways back to the biology so here's a really cool diagram of what a cell might look like in the cartoon but it really goes to demonstrate how densely packed cells on and other than these yellow items here which are DNA and the green which are some extra still your components everything else in this picture is a protein and it's doing really important work to keep the cell alive and the function that protein play is actually entirely dependent on what their 3d shape is and just again to a little bit more details about this proteins are linear chains of amino acids so here we see a chain of amino acids here's one is another and each amino acid is differentiated by the kind of side chains s attached to it so this great part of the chain we would call the backbone and all these colored parts here are the amino acid side chains and they're about 20 different amino acids and each of them with the unique side chain will fall together in 3d space to create a unique shape for that which determines its complete functioning now another way to look at the protein structure is by drawing it like this so here are these feel it shapes represent the backbone of the protein while all these stick figures kind of hanging off the side of the backbone are the side chains that I talked about now the bad thing about proteins is if they ever stop working you know that that could lead to a lot of disease or medical issues and in fact a lot of drugs are designed to be able to interact with proteins by binding to their side chains and a protein structure is so important for this because if you know the shape of the protein you can design a drug that will be able to bind to the right spot and change the behavior of the protein now unfortunately even though protein structures are super important they're extremely expensive and in fact this the red part of this graph shows the total number of protein structures over time and the blue part of the graph shows the total number of protein sequences you can see there's far fewer protein sequences available more our protein structures and a large part of this has to do with the fact for the experiment that can produce what a protein structure looks like and costs tens of thousands of dollars and sometimes it doesn't even work I went to go update these numbers just based on this month alone and so as of March 20 there were 170 million protein sequences but only a hundred and fifty five thousand protein structures and so because of this issue for the past several decades people have been trying to take protein structure in from sorry protein sequence information and use that to predict what proteins might actually look like to fill in this gap there's a couple ways that people have been working other it's the newest perks and most like iconic way that you could try to figure out what a cookie looks like by using something called molecular dynamic Anna market dynamics you take a protein sequence and you basically simulate it and the protein vibrates and jostles around until you the physics involved will maybe show you long what the right structure is but this is really computationally expensive in the concoct takes weeks of computation for a single protein and at the end you not even sure if it's correct another set of methods are called fragment or template based assembly and in these kinds of methods you say well I don't know what this protein looks like but I know what a lot of its similar proteins do look like so you can try to compare it with ones that you know that are similar but if there's any small difference between them your prediction might be wrong if all it takes is like a single mutation on the protein for it to completely unfolding so these these methods in particular are or accounted for so this brings me to the obvious next question and life we're here today when this is you know what about deep learning what can be she football condition learning play in solving this problem I'll talk a little bit about that so if you've been paying attention to some of the news about biology or proteins and machine learning you may have seen this graphic this comes from Google when they participated in the critical assessment of protein structure prediction context so in this contest people meet every two years and they try to make predictions for proteins and in 2019 deepmind had a team that participated in this contest and so it was their first time participating and they use the complete I like complete machine learning method and they basically blew everyone out of the water they beat everybody by it in extreme marketing and everyone was really surprised and frankly a little and this method works by predicting these potentials which kind of you can interpret as energy and it kind of makes the protein fold on its own another important method that I'll be talking about is one that uses something called the current neural network the recurrent neural networks take their input as sequence elements one at a time so here's the protein sequence coming in at the bottom and for each protein sequence element these represents acids they will produce a few angles and from those angles you can build out the amino acid however I recurrent neural networks on the curiously slow to Train in addition this method that was published I used a lot of extra information which may be helpful and also when they participated in this contest at first remember the way they trained it to use a loss called distance based root mean square distance when they optimized for this value it represents how good a structure is they actually caused the model to make predictions for angles that were so unrealistic that they got rejected by the contest automatically so even though the predictions from this model looked pretty good in reality there was something wrong with being so this brings me to kind of address you know where can where can I fit in somewhat what is my research project trying to address well the first major shortcoming which is true for most of the methods that I just described is that they do not predict the size shape happens for simplicity so they're only predicting the backbone happens and maybe this is just because the problem is really hard and people haven't developed solutions to address this yet but I would argue that a model that does not predict the sidechain atom is that severe disadvantage and this is because like I was saying earlier I'm sizing atoms are what make the protein they're what awesome thing to help fold up onto itself I may these interactions the assigned atoms are what dictate what the protein darns is I can design medicines and so if you're ignoring this information it might be a detriment to the model and also we won't be able to do things that are important like the size drugs the second shortcoming that I'm try to address is the fact that the current neural networks I don't think have the right inductive bias for this task so it's true that the method that Muhammad or actually that I just mentioned is method work militantly well for getting good structures but because recurrent neural networks operate on a sequence element one piece at a time they're not able to really remember two things that are very far apart so I think it's really important for a proteins because you'll have amino acids that are very far apart in the sequence but they'll fold up together in the 3d space and I think that that deserves a special model architected which I'll talk about in a little bit okay so now that I've covered a little bit of the basics online here I'll go into a little more detail so first thought I'll tell you a little bit about I've made an all out of predictive model so one that predicts both the side chain that backbone atoms those protein I'd then I'll talk a little bit about what it takes to make a training set for this information and then finally I'll go over just some preliminary results that I have so far okay so first of all you know what does this have to do with language translation so that's the primary inspiration for this project I would say and so basically I think since 2014 the 2017 language translation tools in huge milestone and I have to note if you've ever played with Google Translate before 20 2014 it was not super great but ever since then people develop machine learning methods that really make it really it has to do with deep learning and being able to transform see one type in the sequence another and so I thought you know if I could frame this problem of predicting protein structure as English translation might be use all methods that they're using make this so successful so in order to do this I'm treating the input my input language as a sequence of amino acids so here's one character and the output language after it goes through a learning model is a sequence of angles so here I have symbols representing all the movie called original angles this amino acid now you can see they dictate how the amino acid is shaped they even describe the positions of all the sidechain of X so we translate the language from the input to the output one amino acid at a time and then we can use this and continue to predict the entire protein structure and angles we can use a straightforward algorithm to build one at a time the Cartesian coordinates represent the protein and then we have actual 3d structure of the protein and in order to train the model we compare this prediction with the true protein structure prediction and we use a loss function I'll talk about a little bit more later in order to improve the models prediction writing the great thing about the learning new set all of these operations even the ones that take the angles and turn them into corniest all of these operations are completely differentiable so so we can use machine learning to optimize to this whole prediction setup so I'll go into a little more detail about the exact model that I need them so hopefully if you're not interested in deep learning lately I have since done 17 you've heard about this model called the transformer and it's really changed the field 35 language translation for natural language processing and perceived sequence sequence prediction in general basically this is a transformer model has two parts and in a decoder and rather than a recurrent neural network which operates by looking at a sequence one piece at a time the transformer using something that pet called attention and it looks at the entire sequence at the exact same time and this isn't the main difference and so if we were to show you an example of like what this will look like if we were using my images but here's here's a sentence and for part of a sentence the animal didn't cross the street because it here's the same sentence repeated here and all these lines that connect the two poems those are what we would call attention wait so if you wine is darker that means that if stronger linked between those keywords so what this is telling us is that we'll have a transformer model which will actually could dick these attention weights and what it's predicting is that I think that the word it if strongly related to the whole Orange Street no because they're they're nouns and you know maybe it's related to the UH and it and so this is why an intention liner does it tries to predict what things are going to interact and this isn't perfect this is exactly what we would want for it what can you structure we want to know which amino acids are going to interact we want to know which ones are close in 3d space which ones are far apart and so that's why I was using this model however I wasn't I did made I did make one maintain to the transformer model and I would love feedback on that if you guys happen but I decided to throw away the entire second half of the transformer problem now I'm doing some reading and I think that when you translate from one language to another you don't know the difference you send sentence lengths like for instance if you you same sentence and English ain't translated into Spanish you know the translation might mean the same thing but it will have a different number of words and that's one of the main reasons why people have an encoder and a decoder since that a decoder can take the information and it can figure out when it's not predicted however for proteins this is not the case and we've you know that there's exactly the same number of words and they can put languages are in the output and so I that's why I've just thrown away the decoder and I'll see how far I can get back okay so the next step is the training set to do to do this okay so unfortunately I don't need to start from scratch there is this great thing called Coquina which you can see on github it's made by the previous author I mentioned Mohamed authority-- and protein F is based on that contest I was talking about yeah so this is what it looks like so here is almond access this time so at this point we have the deadline for the contest and here is the actual meeting so what Muhammad does is he says okay everything that happened all the structures that we know that are available before the meeting when we're going to make the training set for that me and they're going and then he takes out a couple pieces the training set can cause that the Foundation's back and then on the structures that they comparing with each other at the actual contest those would be the test structures so the reason this is really helpful is that you know one of it's really hard to design a really good machine learning data set know what's the test set going to be like are you going to randomly select items from your training set so to be the best set and in this case because there's a contest every two years we if you just copy exactly what the contest is doing then an honest is is handled for you you know every two years the the organizers say come up with a list of really really difficult tokens and they make a test set so Muhammad went back to every task competition that's happened for a while and he made these training sets for them the only problem is that they do not include any information about hygiene habits so I went through the big list of side chains which are shown here Palmer code again he's kind of like shows the complexity of all the different side chains you can have you know they have a different number of atoms here each little to let early significant pipe they take on different shapes and so basically I made a data set that's based off of cookina but includes all these information so if you have a protein whose length is L there can be at most 13 atoms for that protein and each atom has an X Y Z coordinate that's part of the input high and the output language if you remember our angles so if you go and measure all these angles you can get up to 12 different angles for every asset and then actually to go one step further gone and the whole thing is about trying to predict angles first right so I I got to the meeting some new search papers have said you know instead of trying to predict the angle in radians which is funny like PI to PI that kind of thing you can actually predict angles as sine or cosine and the reason this is like important is because for a machine learning model it doesn't really understand that I and maybe define those angles are exactly the same right you have to use some sort of fancy loss function to account for that but a trick from the signal processing community is to actually just use these these items these are different agencies period these are the sine and cosines represent the x and x and y coordinates on the unit circle but actually you predict these you can get better performance than just predicting radians in case you never have to deal okay so I made its dataset if we have to go into all the proteins you know here's like an example you go and measure the angles right you know if you you know make it a sine or if you swap sine and cosine you know like I didn't hereby make it'd make a tiny mistake that could make a big difference and so that's what I've been working on for a while okay moving on so there's the now we have a model that can predict all the atoms of the 13 we've had data set which is based off of an existing data set but includes all the information we need and finally we'll talk about you know what it's been like to Train this it's been really hard it's I have not had really great results yes this is apparently a difficult problem and hopefully I'm not the only person because this time so here is an actual prediction I'm sorry in the actual structure of a protein it's a very simple protein with just the helix I just tried to train on this one often delays and I didn't get something that looked very great these top three lines here represent the loss I was talking about the RMS speed which manager is the structural similarity between them they're really hot you know this dotted line I is around two and if they were at two that would be good but they're not so I've been working out ways to try to fix this a long time okay so this is just another thing that I learned that might be interesting to anyone we'll see okay thank you for the thank you for the support in the chat there have been a pity okay sorry just to continue on if you are working with transformers this next piece of information maybe you know maybe don't but I didn't know this until I took the class that told me that it was ok so this is the picture of the transformer model okay they're in here we Steve something called add in more so added or need to take the value from this part of the graph you add it to the output of this part of the graph and then you do a layer of Momo's each so like you're writing it out input plus the feed-forward layer and then you take a layer like that okay so this is what's in the paper but did you know that this is not what you should probably do what you should actually do is move the arrow so that you add after you do the layer mode so this is the X plus this is called a residual connection and residual connections are supposed to help your model train by having like a shortcut from the output of the layer all the way into the back the beginning of the way right and unless you have it written like this the gradient information will have more trouble propagating through the layer okay so just a tip and that's something I learned and I hope my model string okay um so the next thing I'll talk about that I that I found out recently is that the idea using transformers was you know maybe it was good but in actuality it wasn't working that well so I was thinking you know an attention layer the good thing is that it can figure out if it a new asset over here is interacting with the mean last of the year but it doesn't compare anymore acids that are right next to each other very well if it won't give a word and the following words are very similar then you know you it won't do that comparison as well as a recurrent building so one thing that decided to do was add something called a sequence convolution okay so a sequence convolution is kind of depicted on the left here so you give your sequence in blue bigger dispute of just integers you can take a filter the convolutional filter or a window and you can pass it over the sequence and at each point in time you compute the convolution so this one the output is one fun because you have one one times one plus negative 1 and plus zero times sum all these values together you get the output of the table tonight okay so my my deal is if I maybe if I add convolution sequence compilations did this modified transformer you know maybe I can pick up information by these does alpha helix get now maybe you can recognize that these amino acids are close to each other and it can you know figure out that they're supposed to be an optical okay so that was the idea I'll straighten you what happened so to do inspect this I use the weights and biases look forth and happy that shows I made this recently and so you when you do a project on lipids biases we can kind of make a little write-up for yourself and you can you know do an experiment and share it with people trying to get out okay so regarding my see let's see David sorry my ideas on okay oh yeah you can you can you can okay great let me I'll just finish this up in the hall make sure I have time for questions at the end thanks everyone okay so I wanted to know if adding a convolution to my transform Otto but help and so this is what the that's the experimenting here so I added convolution with windows of size 3 and size 11 and I also ran a model that had no solution and so this blue line that you see here on the top is when a model is training out any convolution lands the green and purple are ready to add a convolution who has a filter size 3 and the orange and yellow are competition that has it so precise of life this is great is showing that if I add convolution layers maybe I can and so here here are some really basic predictions I'm the red is the is the predicted helix and the blue is the actual one you can see they all would perfectly fine except for this one in the lower-left foreign which happens to be the one without a convolution leg except maybe you know maybe that's showing something here's a couple more examples of some proteins that were predicted but it's not it's not exactly clear from the pictures know just how good this is getting okay I'll go on a little bit so it's great to know that adding the sequence convolution can help to transform but I got to thinking you know I've given that word that this has to do with language translation you know one of the biggest parts of the transformer model or language translation models are you know what an embedding way and so an embedding layer is something that takes a high dimensional version of a word so you can take a word Apple and you know it's it's word number seven thousand out of forty thousand and so it's a huge vector it's a high dimensionality and you pass it through an embedding layer and you get a smaller vector that represents that word and this is like really critical to help machine learning models that work with languages understand know what we mean in those words and so this is something that I have in the transformer model but you know do we know let me know that this is important like Princeton there's only twenty different amino acids there's not like a hundred thousand amino acids so do I even need to add embedded laggards oh and so that's the that's the exact sort of thing I tried to answer with this next experiment so here is another big chart on the right side you can see that Iran may be 12 different models with different convolution settings and I also turned the embedding layer on and so great cool thing about weights and biases is that you can pull up to start and you can group all the runs by whether or not they were using a penny there's no this is another chart that shows the performance over time and we see that the orange chart which is the user Betty is higher than B blue which is sorry that orange represents no embedding and the blue represents using a bit so there's a significant improvement you added and embedding and I thought about that was great okay move on here on so then I was thinking you know even though this is telling me that we have many more sorry but the embedding is really helpful it is it just the fact that there's so many grandmother's nians models like for instance this is this correlation the importance is telling me that the biggest difference between these is the fact that though the models that have embedding layers just have more parameters and so maybe that's the reason why them so I did one more experiment so in this experiment I tested the new models the purple model has an embedding where and the brown and the green models do not have any plan one of the difference between the brown and the green is that I tried to increase the number of parameters in the green so that would have the exact same number of parameters but this shows that even though when you try to increase the number of fragments to view the same embedding light it's still really important okay so that's that's one other thing I learned in my recent research project and weights and biases has allowing individual ions these predictions they're still not as good as I would hope but you know they're I hope that they're on their way and another cool thing I'm looking forward to using in way to bias and existing all the weights and biases dot molecule where you can visualize all the different okay so that was a little bit of divergence but I'm you know in in the future I'll be hoping that publish the side chain meant with this data set that I'm using so that people eventually contributed I think one other really important thing that might help these predictions is to incorporate some mystic behavior like some physics perhaps or maybe three deconvolution monel that birds can help the model make better predictions of the time so it I thank you so much for for your patience and for listening to me as I describe this used to be a money problem and you know thank you so thank thanks to my research group here's my research group thank you to Nick and Levani for their help humbling biases and some Mohammed I'll cry she and whose work is brilliant inspiring and thank you again for listening feel free to follow me on Twitter or github answer any questions I have some questions that I pulled out for you Jonathan someone asked is a notebook available publicly and I'm curious how you created the graphics yeah the notebook is available publicly and you can find like a permanent link on Twitter or we can you can share it in the chat and it was the question how did I create the graphics yeah yeah um so these graphics can be made automatically who would wait to Bisons which is really nice and these graphics are 3d objects and I use a software called primal to visualize the predictions and then you pie mold will export a 3d object file which you can then save weights and biases cool someone else asked why do we need a sequence model uh well I think a sequence model makes good sense for this problem because the input is all like all the information you have it's the protein C so at some part you have to have the sequence inscrutably the output is three dimensional and so maybe we can use something that's good right there but but definitely could beginning and then someone else asked what graphs be better than doing our none our contributions or combined approach would that be more helpful he thinks it might just might be framed as a graph problem so do you have some thoughts on that yeah that's it that's a really great idea people had a lot of success using fraps for a small molecule I know that you can take a drug turn it into what looks like a graph and trying to predict properties about that drug and you can also do this with proteins but that's a it's a much bigger graph and so you wouldn't take I think a lot of a lot of work like to get this data into that format but I don't I don't see why us as it like are you dealing but 3d protein side chains how do you deal with them how are they represented yes I am I am dealing with the 3d fourteen side chains but I I work with them by but when I take the true structure I measure all the angles that the side chain hands and I measure where it is in 3d space and then when I make the verdict you might try to recreate those correct angles and there is an algorithm that lets you take angles and turn them into 3d structures and but that's not any machine learning in that and then the last question if you don't use a decoder what are your sanity to find out that you in coatings are correct yeah that's a good good question um well in my case the encoding is actually the entire answer so write in an encoder decoder model if you make any coding which represents the output but it's not yet the output in my case i when i encode it i am directly could be seen yeah and I I haven't thought about looking at the layers in between but I might be a good idea to dig like that cool those are all the questions we had thank you so much great thanks everybody