Nucleus: TensorFlow toolkit for Genomics (TensorFlow Dev Summit 2018)

Key Takeaways

[Music] hello everyone my name is Cory MacLean and I'm an engineer on the genomics team in Google brain and today I'm excited to tell you about nucleus which is a library we've released today to make it easy to bring genomics data to tensorflow so genomics is a study of the structure and function of genomes in every cell in your body you have two copies of the genome one from each parent and this is strings of DNA which is a four-letter alphabet and about three billion letters in the genome so here is a picture of a snapshot and on chromosome one 150,000 letters what we can see is there's a number of known things about this area one there are functional elements like the genes depicted in that second row biological measurements allow us to analyze what are different things that are active in cells so on that third row we can see the amount of gene expression across different tissue types is quantified there and at the bottom through sequencing many people we can identify places where there's variation across individuals there's a many different computational and algorithmic challenges in developing that image this range is from on the experimental generation data generation side can we better take the output of these physical measurements to get accurate DNA readings or reduce noise in the experiments that quantify this expression can we take the DNA sequence and interpret where our functional elements like these genes or predict how active are they in different tissue types and can we identify places where individuals vary compared to our reference and how is that different in small variants versus say and cancer and how do those changes influence human traits so one thing that is really exciting for us is there are many opportunities for deep learning in genomics a lot of that is driven by the increase in the amount of data available this graph shows the dramatic reduction in cost to sequence a million bases of DNA over the past decade but also there's a lot of structure in these datasets that is often complex and difficult to represent with relatively simple models but this may just display convolutional structure so we can use techniques from it was classification as well as sequence models and there have been a number of proven successes of applying deep learning to problems in genomics such as deep variant which is a tool our group developed to identify small variants using convolutional neural networks so our goals in genomics are multifaceted one is to make it easy to apply tensor flow to problems in genomics and do this by creating libraries to make it easy to work with genomics data we're also interested in developing tools and pushing the boundaries on some of the scientific questions using those things that we've built and then want to make all of that publicly available as tools that can be used by the community so today I'll focus on the first part of making it easy to bring genomics data to tensorflow so what is a major problem one major difficulty is that there are many different types of data that are generated for genomics research we can see here on the right subset of different types used and these different file formats have varying amounts of support and I in general no uniform aps we also have some concerns about efficiency and language support where we would like to be able to express some manipulations in Python but it needs effective ways to efficiently go through this data such that native Python wouldn't make that possible so to address these challenges we developed which is a C++ and Python library for reading and writing genomic data to make it easy to bring to tons of phone models and then feed through the TF data API that Derek talked about earlier today for training models for your particular tasks of interest in this release we support the reading of many of the most common data formats in genomics and provide a unified API across the different data types so we're able to iterate through the different records of these different types and be able to query on specific regions of the genome to access the data there the way that we develop this uses protocol buffers under the hood so that we can implement all of the general parsing in C++ and then make those available to other languages like Python and for those of you familiar with genomics we end up using HTS Lib which is a canonical parser for the AI throughput sequencing formats like the aligned reads and variants and then wrap that to generate the protocol buffers and then use cliff on top of this to make the data available to pythons and finally we use some of the tensorflow core libraries so that we can write out these data as TF records so that they can be ingested by the TF data API so the data types that we currently support are the following ranging from general genome annotation to reference genomes and different sequence reads or whether they're direct off the sequencer or mat as well as genetic variants so to give an example of the reading API it's quite straightforward so this is kind of a toy example but is essentially similar to what is used for deep variant when we want to train a model to identify actual genome variations based on mapped sequence reads and a reference genome so we have three different data types that we need we import the different reader types and then say in this region that we're interested in we can issue queries to each of the different reader types and then have iterables of these protocol buffers has output which we can then manipulate and turn into tensorflow examples on the writing side it's similarly straightforward so if we have a list of variants for the VC of the common VCF format we'll have an Associated header which provides metadata about this and then open a writer with that header and then just loop through the variants and write them and note that we support writing to block gzip format no which is common for the subsequent indexing by other tools however we can also write directly to TF records and here provide some convenience methods to write out charted data which we found helps avoiding certain hotspots in the genome using a very similar API finally we have been working with the Google Cloud team which has some tools for analyzing variant data and so they have developed a tool called variant transforms which allows you to load the VCF Marron files to bigquery using Apache Beam and then you can do structured queries over that data and so we're working now to integrate here to have nucleus under the hood providing that generation of the variants and to learn more about that tool you can go to the link below so to summarize we have developed nucleus which is a C++ and Python library to make it easy to bring genomics data to tensorflow your models of interest for genomic problems and we have the ability to interoperate with cloud genomics and are being integrated into the variant transforms at the moment and this ended up being the foundation of our CNN based variant caller which is also available open source at the link below so with that I would like to thank you all for your attention today [Applause] [Music] [Music]