Google DeepMind's AlphaFold 2 explained! (Protein folding, AlphaFold 1, a glimpse into AlphaFold 2)

Key Takeaways

it will change everything deep minds ai makes gigantic leap in solving protein structures so i guess all of you heard about this uh so far or most of you and uh basically uh what happened is that deepmind uh released published their results uh on this year's protein folding challenge and their method alpha fold actually the second iteration of their algorithm alpha fold kind of ruined the the whole like like beat everybody uh and got the best scores and we can see so this is the the blog that they uh published so far they still haven't published the paper uh the paper will come probably in maybe maybe even a year or something because the the last paper for the alpha fold one came out in december 2019 where the competition happened in 2018 so we'll have to wait some time to get to that uh to get the paper now basically you can see the uh the scores throughout the years so that's a biennial competition called casp and it happens so it happens every every two years and you can see here that uh in 2018 their method alpha fault one also dominated the competition but this year they crossed the 90 on this global distance test metric which pretty much tells you how good the uh the alignment between the experimentally found uh protein structure and the computationally found protein structures are and i'll i'll first so the idea of this video will be to first uh explain you what the problem is so what the what the protein uh folding problem is uh give you a bit about like history and background of how proteins work and everything and then my idea is to go into the alpha fold one paper because that's what we have so far and i'll explain how it works how the deep learning method works so there'll be a deep in-depth uh explanation of the paper and i'll give some uh assumptions of what will happen with the alpha fault too what the method is they don't only have one chart so i'll try and deduce something out of it so uh basically so what why this is interesting uh is that uh 90 is consider is considered as something that's that's like if you if you get to 90 or above you're pretty much uh comparable to experimental methods the reason being they are also not perfect and they they do have their like measuring uncertainties uh so here we can see let me zoom in a bit here we can see uh like what the what the problem is basically the green thing is the pseudo ground truth so that means uh uh somebody uh found that one using some of the experimental methods and i'll i'll tell you a bit more about those and the the the blue one so the blue one is a computational prediction that deep minds alpha fall to uh made and you can see it's pretty pretty close uh the results are really really good and you can see 90.7 93.3 and these are just the names of the proteins t1 0 4 9 whatever so let me uh kind of step back a little bit here and tell you something about like how the problem came to be so we've been trying to solve this problem in biology for 50 years already and so this is kind of really big big thing that happened so uh yeah the hype is real this time first of all there's this guy called uh christian anfinsen and uh you can see in 1972 he gave this noble uh uh nobel prize lecture so he basically got his nobel prize because he uh he showed that the function of the protein is pretty much uniquely determined by its 3d structure which is really interesting so here is a really cool example of hemoglobin a small molecule with an iron atom in the center that binds to oxygen so basically like the structure is made so that this protein the hemoglobin has a small pocket where this molecule that has a like a iron atom which binds to like the oxygen and thus transports oxygen throughout our body like just that molecule fits so nicely in this hemoglobin so that's a really plastic example of how the structure uh determines the the function that's a pretty obvious one so there's this cool example of this molecule that has this y shape and the molecule attaches to bacteria or viruses and thus tags those cells for destruction in the immune system so let me give you just a brief example of how the proteins are are created so um i like to think of this as like a as like kitchen where the ribosome is basically this macromolecule acts as a cook we have the messenger rna which is pretty much the recep and then we have amino acids which are the ingredients which are transported by this thing called the transport rna so basically the messenger rna contains this linear sequence of amino acids and it gets translated so it gets so the receipt gets read by this ribosome and we form like a chains of of proteins which slowly start folding because different different parts different amino acids have different charges and so they attract or or repel and thus we get that 3d structure so that's basically how how the how the how the protein is formed and now we're trying to figure out this 3d structure computationally now the thing is how do we how do people uh find uh the 3d structures now that we know that it's really important for the like the uh for figuring out the function of the proteins how do we find the structure uh of the protein so there are three methods that are used uh extensively in uh different labyrinth uh in different labs and those are mainly x-ray diffraction uh nuclear magnetic resonance and this electron microscopy and as you can see the x-ray diffraction was so far uh had the the highest amount of proteins were like protein structures were determined using this method although it does has its own flaws it cannot solve so depending on the size of the protein it has its constraints so you can solve everything now these two are getting more popular lately but they are still super expensive and really slow methods uh to to obtain the the 3d structure so that's why the computational uh biology and this these computational methods for figuring out the structure were so important so if we take a look at this chart here we can see that the number of proteins we sequenced successfully sequenced uh is around 200 million now in 2020. so it's been rising like so those are the linear sequences that are encoded in the dna so but we still don't have the 3d structure for all of those and if we take a look at this chart we can see there is this thing called a protein database which is really important because that's the data set that's been used in the alpha fold uh elf fold paper and we'll get to it in a couple of minutes but basically here you can see that um like uh we we have so we have a lot of sequences being uh kind of ingested into this data set but the the the number of structures is actually a lot smaller so here 2020 we can see only 12 000 structures are are it are there but we only but we have like much more uh sequences so so that there was a motivation for why we wanna um why we wanna accelerate the pace of uh finding three structures the experimental one is slow it's really expensive so yeah we we need we need an alternative so again why we care about uh figuring out the structure the 3d structure of the protein is because as as you can see on deepmind's blog an error in the genetic recipe may result in a malformed protein which could result in disease or death for an organism many diseases therefore are fundamentally linked to proteins so diseases like uh diabetes uh like dementia parkinson's alzheimer's cancer some of the diseases that are known to be like plaguing the like the human kind for for for decades or more uh can probably be solved by just us knowing the 3d structures so that's why it's super important to so this is that's why this problem is super important aside from that also we could develop much better materials if we if we knew how to construct and understand the the shapes of different proteins and we could create like plastic eating enzymes uh and thus reduce the pollution like on the on the like global level uh which is something we do care about obviously so that was the background story it was a bit longer uh i assume most of my like uh audiences like machine learning uh with has machine learning background so i felt i need to really give you a nice overview what the problem is and everything that goes inside and like the give you some numbers and methods etc so let's start exploring the alpha fold one now as you can see here uh this is an animation that deep mind provided which basically shows uh how during their optimization procedure which will uh get you in a sec uh like falls the proteins so it starts from a semi like unfolded uh protein uh protein strand which then slowly uh takes its 3d shape in space okay let's overview the half a fold one paper you can see so the title is alpha fold improved protein structure prediction using potentials from deep learning uh they had like a bunch of folks uh working on this project as is as it's pretty common for for huge projects like this one um so they started in 2016 so it's already been like a four year effort of many smart people uh so i already mentioned it's deep mind they're based in london and okay so the paper is pretty complicated there is a lot of text and a lot of domain knowledge so i'll just try and extract the deep learning side of it and hopefully that will give you like a good understanding of how the method works um let's start with this chart so uh basically the the on this uh chart here they just show that they are better than other groups so many other uh research groups were participating in this in 2018 cusp 13 uh challenge that that was the name of the challenge um and uh basically the green lines represent uh the like the groups uh those other groups and the blue line is deepmind and so on the um basically on the x-axis we have something called tm score so there are a bunch of metrics for uh determining how good the structure prediction is so one is tm uh there are other scores like we saw the global distance test score uh there are some like uh idd or something yeah there are a lot of different scores and uh basically so fm domain domain is uh some like fancy name for for protein pretty much um so here in this free modeling domain uh we have uh we had 43 sequences which we needed to figure the structure for and you can see that on average uh the the deepmind had a lot higher tm score than others so that's what this thing is and now i should probably uh tell you what what the difference between free modeling and template modeling is so template modeling is when you're trying to predict the structure for a sequence uh that has similar sequences whose structure we already know now uh the thing is once you know uh some some similar sequence uh you can kind of uh assume that the structure of your of your sequence will be similar to that one so you just need to find where the like the sequence differs and tweak uh kind of tweak the the structure so starting from that maybe that structure so that's a template modeling so i'm simplifying a little bit here but like that's that's the rough notion again here on this chart they just displayed uh five different proteins and they again showed that like the blue dots that deepmind is uh crushing it i mean the so these two were bragging charts so now now let's get to to the actual uh the actual method and this is how it looks like we can see some um neural network here of course and that's actually uh resnet with dilated convolutions and so basically before i get into those details they're they're pretty much um let me see three parts of this pipeline so the first one is preparing uh the features that we'll need in order to predict the structure then the second part is the distance and uh torsion angle predictions and i'll tell you what those are in in a minute so there's the second part of pipeline and then we have the gradient uh descent uh or the non-linear optimization part of the pipeline so in stage one they uh they somehow find a way to go from 1d amino acid sequence that represents the protein to 2d uh to the map and the 2d map actually contains co-variations between our targeting sequence and similar sequence from this uh huge data set of sequences and so this is the this thing called multi-sequence alignment so you basically find a group of sequences like thousand sequences which which are similar to your target sequence which you're trying to predict the structure for and you encode the covariations in these maps so if i take one uh point on this 2d map and this is to the volume so this will be like like 500 or something channels or more there are some details down in the paper so one thing they encode is like the one hot encoding of the amino amino acid so basically uh that means that if we take this point so that's let me zoom in so oops so basically uh let's say this is like uh row five and let's say this is like row i don't know 40. um they'll basically uh encode five as as a one they'll basically find what the fifth amino acid is and because we have only 21 amino acids in human in human uh genome we'll we'll we'll encode this let's say this is because because there are 21 we can represent each one with the letter from the alphabet uh luckily we have 26 letters so let's say this one is i don't know the fifth one is amino acid a uh this uh 40th is maybe i don't know like s and they just encode this one as as a one hot let's say this one will be like zeros and then on some spot like 13 i don't know they'll have one et cetera so what they will do is they will concatenate this one hot vector with one hot vector that uh corresponds to amino acid a and there will be a part of the features they are using there is a much more uh like domain domain specific things they're including also they're including these uh multi-sequence alignment things and i'll probably explain those a bit a bit more i'll go into detail about those because that's interesting a bit later so that's that's how they get the this uh 3d volume uh and now you can treat this as a simple computer vision problem and they pretty much use this uh uh like classic uh deep learning model like resnet with the dilated convolutions uh and treat this as a sequel like an image to image uh translation problem so uh the network itself is as you can see has kernels a site of size 64 by 64. so they're just actually randomly sliding across this volume and they end up getting the distance predictions okay so what is this distance map and how do we get one so in order to understand that i have to make a small tangent here and explain how the amino acids connect and uh which distances and torsion angles are we actually measuring so uh so these are all the 21 amino acids we have in in the human in the human genome and basically so as you can see this part is always always the same for every single amino acid no matter uh the properties this group always this part always remains the same so uh this thing here is called something called like i think alpha uh alpha carbon atom this one is called beta carbon atom and that one is really important so uh let's say for the sake of argument that we have a sequence like r h k which is pretty much these three amino acids here so what will happen is uh once they start going through the ribosome they'll start connecting and so these parts will form the backbone so these parts and the groups which is this part that goes off from the from the beta carbon atom those will be like for example some will be positively charged maybe this one uh will be negatively charged so what will happen is that they'll start attracting and that's the thing that co that causes the uh the protein to fold so imagine if you had like a protein that had 140 amino acids so what what could happen is that in this complex um like interaction uh maybe the first maybe the the the the the first amino acid would be like uh in the 3d structure would be really close to maybe 127th amino acid so that means we have a long range dependencies uh in this in this structure so now uh when we are so i mentioned distance map so what we are actually uh looking for is uh distance between beta carbon atoms between different amino acids so basically um that's that's what that's what we're looking at and then we have uh torsion angles so uh phi and psi angles which are um basically in that 3d structure two angles that we care about let me see if i can find a nice visualization of that and you can see it here hopefully let me zoom in so this is the protein chain and so one of the two dihedral angles inside here without getting into uh too much explanation is two angles that we care about so for the sake of argument just think there are two angles we care about there is third but that one is almost always 180 so deep deepmind kind of hardcoded that one to be 180 and they're just considering these two so okay let me get back to the onenote so now that we know uh this information about how we calculate distance and phi and psi angles let's go back to the deep learning pipeline so uh basically you can see here uh we have we we have um amino acids here and we we have the same sequence here and so this particular point maybe corresponds uh to uh finding the so i'll actually use a different drawing which will probably be a bit easier to understand okay let me see if i can nicely explain the distance maps um so we have the experimentally found the pseudo ground truth uh distance map for this particular amino acid sequence and we have the predicted uh distance map uh over here so basically what they showed here is they took a specific amino acid from the sequence like uh in particular they took 29th amino acid and that's uh depicted on these charts by these uh red uh stripes and um basically uh you can see that the distance between 29th uh carbon uh uh beta carbon and um with itself is obviously zero which is this colored with yellow and over the whole main diagonal is going to be yellow for that specific for that particular reason because um the distance between a beta carbon and itself is always going to be zero so that's why we see this pattern along the main diagonal and what it did here is uh they took they took uh the neighborhood uh for from the 29th amino acid and they just extracted all of the 41 distances and that's the chart on the right and now the thing is uh these things here are not scalars they're actually probability distributions so if we take maybe i don't know 40th probability distribution 40th amino acid and its corresponding probability distribution we end up with something like this and you can see on the x-axis we have the distance and on the y-axis we have the probability so this particular um like uh position has uh we can see that the distribution is somewhere like the peak is pretty much around like four angstroms where angstrom is 0.1 um nanometer so this is how they uh symbolize the angstrom and the red bar is the ground truth value so the red bar says i don't know like five and we predicted four which is pretty good if we take a look at because 29 is missing obviously because we know that is going to have a spike on the zero uh distance basically if we take a look at the surrounding probability distributions so those are those correspond to this part here um basically distributions are really narrow and uh really precise so you can see that the peak the mode of the distribution corresponds perfectly to the red bars here also really nice as we go further apart further away uh we can see that um the red bars don't correspond always with the like with the mode of the distribution so say let me take an example where we have some problems like here like this one is a good example because uh we predicted that the distance is here so that's whatever this value is but the red bar is here so yeah um by the way these black lines um is the kind of uh length distance uh when we consider the the the two carbon um beta carbon atoms to be in contact so if we are so this one is at eighth angstrom so if you fall below eight angstrom we consider those two beta carbon atoms to be in contact otherwise they are they are not uh in contact obviously so those are basically uh so the map is basically the like the inter residue distances for the amino sequence and we want to predict obviously uh as close as possible to the ground truth so let me now go back to our initial deploying pipeline if i can get there okay we're here so we we do a similar thing for for the uh for torsion angles and once we have those maps uh what we now do is we uh form a geometric model of the of our sequence meaning uh we basically take uh like we have a geometric model and we have two arguments we have the phi and we have psi so the torsion angles and those gives us give us the like 3d coordinates of those beta carbon atoms in 3d space so what i mean by that so if we have us like a sequence of length 100 we'll have 200 parameters here and uh basically for one specific configuration of these 2l torsion angles we get a we get a specific configuration in 3d space like i don't know like we will have some thing going on in space and uh so what this part does is we are optimizing so we are changing these two so that uh the 3d coordinates or the corresponding 3d structure gets as close as possible to uh the to the gray one so that this one is the native experimentally found structure so what happens is by tweaking these phi and psi angles we are also tweaking you can you can imagine like we are tweaking the distance map for this particular configuration so for a given phi and psi we have a given distance map uh which which uh which we are trying to uh to uh to match with this one so the two parts of the pipeline are pretty much independent the second pipeline so the second part of pipeline just figures out uh figures out the depth map and the torsion maps and then the the the third part uh of the pipeline uh just tries and tweaks this differentiable g model so that we uh minimize the distance and we can see on this chart here that the tm score as we are it did i'm losing my cursor my god uh the team score is going uh uh down as we are progressing with our optimization steps and uh sorry that the the rmsd that's the uh one one of the metrics that they use to figure out whether the 3d structures are are like uh getting closer together and the tm score on the other hand is going upwards what's else interesting so they just took a snapshot here and we can we can see how the how it's progressing and slowly getting into the final shape which is really close to the gray one here now the interesting thing is uh the and i didn't mention this because those details were probably not as relevant as now uh so the the protein has uh something called the the primary the secondary the tertiary and the coronary like uh structure the primary structure is just the amino sequence the secondary structure are those things that form spontaneously like the this thing it's called alpha helix and then we have uh these things here which are called beta sheets and uh then those are forming in these intricate like shapes which are the tertiary structure of the protein and finally the the quarter tertiary quad coronary let me actually use uh this shot it will be easier to understand what i'm talking about so we have the primary structure here uh then we have the secondary structures which form so alpha helixes and these pleity or beta sheets then they form in this thing called like the tertiary structure and finally multiple proteins can interact and that's the the coordinate structure i hope i'm pronouncing pronouncing that well so that's about the the shapes and aside from the the distance map i already showed you um uh this pipeline is also predicting uh these secondary shapes so the the the blue one is the uh the the the alpha helix and the red one is the beta the beta beta beta sheet so this is the ground truth the suit of ground truth and these here are taken is a snapshot taken from the last optimization step and we can see that the modes uh pretty much uh we are pretty pretty uh well correlated with the ground truth and thus the final 3d shape is also also coincides with the experimentally found shape there was there was uh that was that was crazy now there is one more question how do we initialize the the files and the size initially uh and what they did is they they use those uh predicted torsion angles which are not depicted here they only show the distance map and they sample from those distributions to get the initial like uh uh configuration you can see it here this is the initial one and uh it's getting crowded here um so basically that's how they initialize this they randomly sample from that distribution and then they do the optimization but now uh what they what they do they do one more thing that's they they keep a pool of um a final structures so they keep like a 20 uh final structures so uh whatever they get at the end of these optimization procedures uh they minimize the potential and they just keep those uh in this pool and then they also sample from deadpool so they take one of these minimal configurations and they randomly add noise to files and size so they kind of uh get uh experimentally showed that they are getting those are giving them better initializations than just randomly sampling from those torsion angle maps so that's one more detail i wanted to mention and now we have the whole pipeline hopefully uh in place let me go once more uh like the high level thing we have the features related to amino acids we do uh basically convolutions uh over uh over those uh input maps we get the distance maps we get the torsion angles we get these uh secondary structure predictions and once we have those we we just freeze it and then we have an independent step and that's the gradient descent optimization uh they used lbfgs actually forgot to mention that detail and they basically are tweaking the files and size so that the geometric model the 3d coordinates are getting in a configuration which will produce distance maps similar to the one that we got that we got here so that's a high level overview of how it all fits together okay that was the overview of the alpha fold one uh hopefully that was useful i know there was a lot of details um let that sink in um so regarding the alpha fault two uh that's the main topic today but uh we don't know much so this is what we what we got from from deepmind so we trained this system on public available data consisting of 170 000 protein structures from the protein data bank so we already mentioned this one uh together with large bases blah blah blah and so they they used um 128 tpuv3 cores for for maybe a couple of weeks to train this thing and um basically this is this is the the diagram they gave us uh and the paper so uh they are preparing a paper uh to submit for a peer-reviewed uh journal and let me let me let me see what i can deduce from from this chart alone so basically um what i expect here and they mean they mention spatial graphs in the blog so i won't get into any assumptions aside from from from this one once the paper comes out i'll i'll i'll cover it but for now let me just um try and see what we can deduce here so basically what i expect to happen is because they were using um like uh only 40 46 64 by 64 uh comp maps here is that so basically you have the amino sequence that's a bad depiction of amino sequence and basically until now they could only model uh they could model the long range like throughout the sequence they could only take chunks so so it's basically a form of local attention so um they could maybe model uh like uh amino acids that like close together if we are on the uh main diagonal here so that's so if we if if the com map is currently maybe here that's when we model that's when we modeled uh like the like uh local local attention of of of close by uh amino acids if the uh if the comp map is maybe here then what happens is the following we can model maybe this group and this group and all the interactions between them like this one corresponds to this one and this one etc so it becomes a mess really quick but uh basically what i expect they will do is they will use transformers and they'll just have so their other use they'll probably use some some efficient transformer a lot of them came out this year like uh i know informer performer reformer fill in the blank uh and basically uh they'll um they won't have to to create these inductive biases that we currently have and that's the like the intrinsic property of convolutional neural networks um yeah that was that was it about the the the second paper i won't get into any more details i just want to uh tell you about some really cool repercussions uh that the system will have uh in the like uh cavite 19 pandemic so basically they already helped us find structures of certain proteins uh that we that we that are present in in the in the source code of two virus one of those is orf-3a protein and what they can do is they can accelerate the pace of experimental methods they can they can predict the structure and then they can help the experimentation labs figure out the structure much faster than by doing it from scratch so yeah that was that was that was it uh hopefully uh you found this video insightful uh if you did uh go ahead and subscribe to my channel and click that bell icon to get notified when i upload a new video and until next time keep learning deep