Blaine Mooers - Why develop a snippet library for Jupyter in your subject domain? | JupyterCon 2020

Key Takeaways

hi i'm blaine moores i'm an associate professor of biochemistry and molecular biology at the university of oklahoma health sciences center in oklahoma city i'm developing a library of code fragments our snippets to ease the use of pymol a powerful powerful molecular graphics program for viewing the molecular structures of proteins and nucleic acids i'll introduce pymol and the nature of snippets and then i'll demonstrate the use of the snippet library in a jupiter notebook using jupiter lab this slide shows the workflow in my laboratory we use x-ray crystallography to determine the atomic structures of proteins and nucleic acids from the rna editing system and trypanosomes we also work with several colleagues on proteins important in cancer such as the one shown in the lower left we purify protein we grow crystals this is a large crystal probably 0.7 millimeters in length it's mounted in a cryoloop and put in a cold stream at minus 173 degrees celsius this enables the crystal to continue to refract for a long time in uh when we're shooting it with x-rays x-rays some x-rays are diffracted and we wind up getting these collecting these diffraction spots using a image detector shown here the spots are collected as the crystal is rotated we have to do this for on average 180 degrees we then process the data and take a inverse fourier transform of the fraction data to obtain an electron density map we fit the model of the protein to that map one contour level is being shown as this chicken wire mesh so looking at the proteins on the top and bottom and then in the middle you're looking at a drug molecule that is binding to the atp binding site of this protein tyrosine kinase in the lower right we have compared two structures with different drugs bound and we're examining how steric effects may inhibit the binding of drugs to a mutant of this protein so figures like this that are made in pymol are often used to illustrate scientific papers about structures pineal is also popular for the purpose of making cover artwork here are some examples of lecture structures made in pymol on the covers of several leading scientific journals primo's popularity is reflected in this graph this is a result of a poll that's been running for about a decade at the rosetta design group website there are actually about 25 monthly graphics programs that have been nominated imo has been sitting at 38 for several years pymol was developed by warren delano who's shown here at an open source software conference in 2005. he started work on pi mall in the early 1990s while he was undergraduate student in computer science and molecular biophysics at you know university at the time he was working in the laboratory of axel brunner a leading protein crystallographer worked on the development of the crystallographic software known as explore which has since evolved into the program known as cns which still isn't in use today here under his phd in protein engineering with robert wells at the university of california san francisco in his spare time he continued to develop pymol took a postdoc position with robert wells and likewise continued to develop a pinewall in his spare time in 2001 he released piemol in 2002 he left his postdoc position to start a company called delano scientific to further development of pymol unfortunately died in 2009 his widow made arrangements with the schrodinger incorporated to distribute uh pymol and to continue his maintenance shortager incorporated develops proprietary drug design software in addition arrangements were made to set up a warren delano pima open source fellowship a link to which is at the bottom i won one of these fellowships in 2018 to develop a library of snippets for the pinewall macro length this has been developed for about 18 leading text editors support for snippets has become widely available in text editors over the past this is an animation the use of a tab trigger that's called three maps that's mapped to 12 lines of code that generate three contours of electron density the text editor has been used as visual studio code the nice thing about visual studio code is that it provides a auto completion of the snippet name and it also provides a preview of the snippet along with a description of what the snippet is as soon after entering the appropriate snippet name you hit tab tab leads to that action leads to the insertion of the code so the preview enable allows you to uh ascertain whether or not that's the right snippet and then uh you can see these highlighted areas uh the cursor will advance to the first so-called tab stop this is some coding that's hidden not shown to the viewer that controls the movements of them so while for dgr is highlighted we can start typing over that default value and we can replace it with a different name or we can just hit tab again to advance to the next tab stop so the tab stops that have identical parameter values are mirrored so you can see that four are highlighted and we're going to change glycan to sugar in this example this ensures that all four sites are edited so one problem with the editing of code templates or snippets is a failure to edit all the sites that need to be changed that can lead to bugs and time wasted on debugging okay and then it will advance to the next tab stop this is the gui for pie mall the area the viewport in the lower left is allows one to interact with the molecule by using the mouse you can change the orientation by using the computer mouse or trackpad multiple models can be uploaded each model receives a shows up as a tab our entry in this internal gui each of these entries has five pull-down menus available and so in this case a selective a and under a i selected preset and then under preset i've selected publication so a series of parameter settings have been set up or to give a rendering of the atomic coordinates in this fashion the default display normally would be just a display of the structure as sticks with the sticks representing bonds between atoms so the the raw data are just atomic coordinates of the atom and these uh parameter settings around allow the generation of these fancy renderings so in this case we have a fancy alpha helix in which the interior of alpha helix has been colored gray there are additional cascading pull down menus across the top and we'll return to one of these later in addition and then below this window is a command history window you can copy commands out of this window and paste them on at the pimol prompt the pi mode prompt will take commands are both in written in the pi mole macro language as well as in python primal is written in c due to the need for speed in the in the manipulation of the molecule but it's wrapped in python to ease its use by users and and then that in addition there's a pimol macro language which is has a much simpler syntax than python and much easier for non-programmers to use these are the default like the representations of pymol upper left we have lines in which the bonds connecting atoms represented as points are shown the atoms are colored uh by their element type so carbon is colored green although it can be colored any color whereas oxygens are red nitrogens are blue sulfurs are yellow so in this case we've the stick representation has the lines replaced with thicker lines and the carbons here have been colored yellow here the carbons have been colored cyan here they've been covered magenta spheres represent our representation in which the van der waals radius a particular element is used to generate the sphere so these spheres are centered on the atomic positions of every atom if you roll a ball that has the diameter of a water molecule across this van der waals surface you'll generate a molecular surface which is quite smooth here's a molecular surface generated as a mesh and then here's a the van der waals spheres represented as dots another kind of representation is that it helps to visualize secondary structure elements is to simply connect the c alpha carbon atoms that are at the center of every amino acid and so we're just have one atom represented here for each amino acid and this is highlighting two beta strands then on the lower right we have a what's called a cartoon and pymol or a ribbon diagram normally it shows two beta strands as returning to the pull down menu so now i want to return to the setting pull down menu we're going to go to edit all but we obtain this menu we can enter on this line the start of the name of a parameter to bring it up they're actually you have access to over 600 parameters primal has about 500 commands and over 600 parameters this vast array of commands and parameters allow you to exhibit exquisite control over the appearance of the molecule and this is a source of the popularity of pi mole however setting these values is only temporary there's of course and if you have to select a set a large number of them you're not going to have a record of your work so it's prudent to instead use a script file to store such values it's quite common to have to set several dozen parameters to generate a publication quality figure so here's an example of a script file written in the pymal macro language this is for a snippet that's involved in generating grayscale this is actually quite a long snippet several hundred lines long it's involved in applying grayscale to a figure for the purpose of producing the figure four uh it may be a low budget book chapter such an option is not available directly in primal so the commands are shown orange the parameters are shown in blue and the primer values are shown in red this image shows an ambient occlusion being applied to a short double-stranded rna helix so on the left result of entering that ao tap trigger in the middle we've entered the ao bw tab trigger and we've gotten the effect on the left applied in grayscale and then on the right is a variant in which the carbons have been colored black these capital letters above are actually mapped to python functions so these can be applied in interactive session you don't have to script file necessarily to use these they're available at this website however if you have loaded these pimol shortcuts out and they're always present when you start up a pimol you can include ao in a pml uh script and it'll be applied the problem with the primal macro language even though the syntax is pretty simple uh the commands are so numerous it's hard to remember their names so many of us suffer from the forgetting curve our memory of the commands falls with our time since our last primal session many of us use pine mall on an episodic basis when we're preparing images for papers and grant applications perhaps every several months so our probability of recall is going to be quite low herbin ebbinghaus is a person who quantified the forgetting curve and he published his work in 1885. so the primary reason for developing the snippet libraries overcome this poor recall of the commands amongst people who are infrequent users of pimol which is the vast majority of users pymol has over 100 000 users most of whom are not computer programmers the second reason is to help students develop the habit of using scripts so generally i suggest that students start off by using the shortcuts and that shortcut file and then they can migrate to writing scripts the snippet library will speed up the assembly of the pml's scripts because of the tab stops this can reduce the time spent debugging in addition the snippet library has unfamiliar highlights unfamiliar features of primo so it can help users learn about the dark corners of primal in addition the snippet libraries you've already seen with ambient occlusion can extend what can be done in primal then finally i think pymol can help with two fears you've heard of fomo if you're missing out well there's fog called the fear of the command line and the fear of writing scripts foss these are two widespread fears amongst molecular biologists so what about running pymol and jupiter an ideal world we would have the viewport available to us in a output cell where we could interactively or we could issue commands from [Music] a code cell and have output show up in the output however this is not possible right now there are two packages available that allow interactive viewing of macromolecules in jupiter one's called ngl view and the other one's uh bio 3d view another option is to have pymal open and the jupiter notebook open and then you're to launch pimol as a remote procedure call server then you can send commands from pymall excuse me from the jupiter notebook to pymal and there's a number of these receivers have been developed however i find that they often break with the frequent updates of primo so primal is updated about once every four months another option is simply to live with working with the static views in the jupiter notebook and to run pi mall via the pimol api which has just become available so if you have installed a kernel that is mapped to the python interpreter inside of pymol you can import the cmd module from pymol which provides access to over five to about 500 commands so in order to run pymol and a jupiter notebook i use the i'm going to use jupiter lab jupiter labs snippets multi snippets extension for jupiter lab included this snippet library called pymol pi snips which has the pi pymal code rewritten in python because at present you can't send the pymol macro language code from jupiter notebook to now included snippets in this library that provide for the imports that are needed so we need this command module and then we need these two imports for mypython in order to load up an image from disk and then i've provided a path where the images will be stored so for this test case we're going to apply a script that will make that will display trna molecule shown down here with the bases replaced with rectangular blocks purines have are the long rectangular blocks and the pyrimidines are shown as a square rectangular block this script is a wrapper around the dssr program so the great thing about the multi-menu extension is that you could it's very easy to create menus simply by creating folders within the jupiter library and the name of the menu is the name of the folder and then you have subfolders in this folder which provides sub-menus and then so under nucleic acids we actually have about eight uh snippets we're going to select the first snippet and this is the corresponding code in single quotes you can see the pml code you can see the cmd dot do so the do function will convert the pml code we'll send the pml code with pymol so after doing that we get the png file written to our disk but we need to load that so under imports we'll import ipython display to get the code that we need so we need this function load.png file so here's the corresponding png file as just a side note uh i made the file originally bought with the 160 by 160 pixels but to fit this notebook well i had to reduce the size to 500 by 500 however the disadvantage of this approach is that uh of that library is that there aren't any tab stops nor tab triggers so to compensate for that i made a second library and in this library the code has been replacing a comment and the corresponding metadata required for the tab stops is shown here so the idea is you can use your eye to guide yourself manually through these and to ensure that you edit all the sites that you need to of course when you're done with this you could always delete this comment then below i've also provided a description and a source so my recommendations for the use of snippets in jupiter are for the developers to add support for tab triggers with auto completion like we saw in visual studio code i highly recommend the use of camel case and the writing of tab trigger names it is the standard of course in python to use snake case with underscores between the names and our words and lower case the problem with that is that underscores are hard to reach on the keyboard it's more ergonomic to use camel case and since what you may be using entering snippets quite often it would be beneficial to stick with camel the menu labels should also include these tab trigger names the port for tabs stops should be provided that and these tab stops should have the capability of being mirrored such that change at one is propagated to the others and finally the snippets should be stored in a simple format so some of the existing extensions store snippets in json files that's not very user friendly so more user-friendly formats can be found in multi-steps for vim are you snippets for evacs so in summary snippet libraries can ease the building of platinum scripts the tab triggers and tab stops are needed in jupiter and the snippets can support literary programming with pymol in jupiter i like to acknowledge thomas holder he's the lead developer of pimol schwander incorporated he administered my fellowship which i have to submit for quarterly reports and he handled that i'd like to thank miranda brown a rotation suit in my lab who wrote the documentation found on the github site for this project also like to thank the ohsc python workshop who to whom i presented some of this work and finally i'd like to thank the fellowship and the sources of funding or support of my salary you