Inside TensorFlow: Summaries and TensorBoard

Key Takeaways

hi everyone my name is Nick I'm engineer on the tensor board team and I'm here today to talk about tensor board and summaries so first off just an outline of what I'll be talking about first I'll give an overview of tensor board what it is and how it works just mostly sort of as background then I'll talk for a bit about the TF dot summary api's I am particularly sort of how they've evolved from TF 1 point X 2 TF 2.0 and then finally I'll talk a little bit about the summary data format blog directories map files some best practices and tips so let's go ahead and get started so tensor board hopefully most of you have heard of tensor board if you haven't it's the visualization toolkit for tensor flow that's a picture of the the web UI on the right typically you run this from the command line as the tensor board command prints out a URL you view it in your browser and from there on you have a bunch of different controls and visualizations and the sort of key selling point of tensor board is that it provides cool visualizations out-of-the-box without a lot of extra work you basically can just run it on your data and get a bunch of different kinds of sort of tools and and different sort of analyses you can do so let's dive into the the sort of parts of tensor board from the user perspective a little bit first off there's multiple dashboards we have this sort of tab set up with dashboards across the top the screenshot shows the scalars dashboard which is kind of the default one but there's also dashboards for images histograms graphs a whole bunch more you know more being added every every month almost and one thing that many of the dashboards have in common is this ability to sort of slice and dice your data by run and by tag and a run you can think of that as a sign of a run of your tensorflow program or your tensor flow job and a tag it corresponds to like a specific named metric or a sort of piece of summary data so here the the runs we have a trained and evolved run on the lower left corner and the run selector and then we have different tags including the cross memory tag is the one being visualized and one more thing I'll mention is that one thing a lot of tents report emphasizes is seeing how your data changes over time so most of the data takes the form of a time series and in this case with the skyways dashboard the time series is sort of as a step count across the x-axis so we might ask you know what's going on behind the scenes to make this all come together and so here's our sort of architecture diagram for tensor board we'll start over on the left with your tensor flow job it writes data to disk using the TF summary API and we'll talk both about the the summary API and the sort of event file format a little more later then the center component is sensor board itself we have a background thread that loads event file data added because the event file data itself isn't efficient for querying we sort of construct a subsample of the data in memory that we can query more efficiently and then the rest of our tensor board is a web server that has sort of a plug-in architecture so each dashboard on the front end is house that as a back-end it has a specific plugin back-end so for example the scalar sasch board talks to a scalars back-end images to an image back-end and this allows the backends to do sort of pre-processing or otherwise structure the data in an appropriate way for the front-end to display and then each plug-in has a front-end dashboard component which are all sort of compiled together by tensor board and served as a single single page and index.html and that page communicates back and forth with the backends through standard you know HTTP requests and then finally hopefully we have our happy user on the other end seeing your data analyzing it getting useful insights and I'll talk a little more about just some details about the front-end the front-end is built on the polymer web component framework where you define custom elements so the entirety of tensor board is one large custom element a TF tensor board but that's just the top from there on each plug in front end is like each dashboard is its own front-end component for example there's a TF scalar dashboard and then all the way down to shared components for sort of more basic UI elements so we can think of this as like a button or a selector or like a card element or a collapsible pane and these components are shared across many of the dashboards and that's one of the key ways in which tensor board achieves what is hopefully a somewhat uniform will can feel from dashboard to dashboard the actual logic for these components is implemented in JavaScript some of that's actually typescript that we compile to JavaScript especially the more complicated visualizations typescript helps sort of build them up as libraries without having to worry about some of the pitfalls you might get writing them in pure JavaScript and then actual visualizations are sort of a mix of different implementations many of them use plottable which is a wrapper library over the d3 of the standard javascript visualization library some of them use native d3 and then for some of the more complex visualizations there are libraries that do some of the heavy lifting so the graph visualization for example uses a directed graph library to do it layout the projector uses a sort of WebGL wrapper library to do the 3d visualizations and the recent we introduced what if tool plugin uses the facets library from a pair folks so we sort of bring a whole bunch of different visualization technologies together under one to sort of tensor board umbrella this is how you can think about the front-end so now that we have sort of a overview of tensor board itself I'll talk about how your data actually gets to tensor board so how do you unlock all of this functionality and the spoiler announcement to that is the the TF summary API so to summarize the summary API you can think of it as structured logging for your model the goal is really to make it easy to instrument your model so to allow you to log metrics weights details about predictions input data performance metrics pretty much anything that you might want to instrument and you can walk these all save them to disk for later analysis and you won't necessarily always be calling the summary API directly some frameworks call the summary API for you so for examples estimator has the summary saber hook caris has a tensor board callback which takes care of some of the sort of nitty-gritty but underlying that is still the summary API so most data gets to tensor board in this way there are some exceptions some dashboards have different sort of data flows the debugger is a good example of this the debugger - where it integrates with TF debug it has sort of a separate back-channel that it uses to communicate information it doesn't use the summary API but many of the commonly used dashboards do and so the summary API is actually has sort of there's several variations and when talking about the variations it's useful to think of the API as having two basic halves on one half we have sort of the instrumentation surface so these logging these are like walking ops that you place in your model code they're pretty familiar to people who have used the summary API things like scalar histogram image and then the other half of the summary API is about writing that log data to disk and sort of creating a specially formatted log file which sensor board can can read and extract the data from and so just to sort of give a sense of how those relate to the different versions there's there's four variations of the summary API from TF 1.1 point x2 2.0 and the sort of two key dimensions on which they vary are kind of the instrumentation side in the writing side and we'll go into this in more detail but first off let's start with the sort of most familiar summary api from TF one point X so just sort of as a review again if you've used a summary API before this will look familiar but this is kind of a code sample of using the summary pin1 point X the instrumentation ops like scalar actually output some reproduce directly then those are merged together by emerge all op that generates a combined at proto output the combined output you can fetch using sessions of run and then that output you can write to a file writer for a particular log directory using this sort of add summary call that takes the summary proto itself and also a step so this is in a nutshell that sort of flow for TF one point X summary writing there's some limitations to this which I'll describe in two parts the first set of limitations is house to do with the kinds of data types that we can support so in TF one point X there's a fixed set of data types and adding new ones is a little involved it requires changes to tensor flow in terms of you would need a new prototype finishin field you'd need a new op definition a new kernel and like a new Python API symbol and this is sort of a barrier to extensibility for adding new data types to support new tensor board plugins it's led people to do sort of creative workarounds for example like rendering a map table plot in your training code and then logging it as an image summary and the prompt here is what if we instead had sort of a single opera set of ops that could generalize across data formats and this brings us to our first sort of variation which is the tensor board summary API where we try to make this extensible to new data types and the tensor board summery API the mechanism here is that we use the tensor itself as a generic data container which can correspond to for example we can represent a histogram and image scaler itself we can represent these all in certain formats as tensors and what this lets us do is use a shared tensor summary API with some metadata that we can use to describe the tensor format for our sort of one place to society data and so the tents our Board of summary the principle it takes is actually that you can re implemented summary ops and AP is as Python logic to call tensorflow ops for sort of pre-processing and then a call to tensor summary and this is a win in the sense that you no longer need individual C++ kernels and proto fields for each individual data type so the tensor board plugins today actually do this they have for a while they have their own summary ops defined in tensor board and the result of this has been that for new tensor board plugins where this is the only option there's been quite a bit of uptake for example the PR curve plug-in has a PR cur summary and that's the main route people use but for existing data types there isn't really much reason to stop using TF dot summary and so for those it makes sense you know that's that's been what people have used but then TF dot summary it still has some other limitations and so that's what we're going to look at next so the second set of limitations in TF dot summary is sort of around this requirement that the summary data flows through the graph itself so merge all uses the hidden graph collection essentially to achieve the effect to the user as though your summary ops have side effects of writing data a kind of like a conventional the way you use the standard logging API but because it's using a graphic election it's not really safe for use inside control flow and functions and also with eager execution it's very cumbersome to use you would sort of have to keep track of outputs by hand or in some way wait to send them to the writer and these limitations also apply to tensor board dot summary ops themselves because they don't really change anything about the writing structure and these limitations have sort of led to the prompt of what about if summary recording was an actual side-effect of op execution and so this brings us to TF contribs amuri which has new writing logic that achieves this and so here's a code sample for TF contribs summary which looks pretty different from the original TF summary it works with eager execution but as sort of the the change we have to make is now we create a writer up front via create file writer it's still tied to a specific log directory which we'll talk more about later you set the writer as sort of the default writer in the context you enable summary recording and then the individual instrumentation ops will actually write directly to the writer when they run so this gives you the sort of standard usage pattern of logging api that you would expect and it's compatible with eager execution and also with graph execution so some details to how this works with contribs summaries the writer is backed in the tensor flow of runtime by a C++ resource called summary writer interface that essentially encapsulates the actual writing logic which makes it possible in principle to have different implementations of this the default writer has sort of conceived of by the Python code that's executing is just a handle to that resource stored in the context and then instrumentation ops like scalar an image now or C flops they have side effects and they achieve this by taking the default writer handle as input along with the data they're supposed to write and then actual op kernel implements the writing using the C++ resource object and with this model the Python writer objects mostly manage this state they don't quite completely aligned because the C++ resource could actually be shared across Python objects which is a little bit different stole from the tensorflow 2.0 paradigm where we want our sort of pythons day to reflect runtime state one-to-one and this was just one example of a few things that were changing with TF 2.0 and with TOF 2.0 we had this opportunity to sort of stitch some of these features together and make sort of one unified for new TF dot summary API and so here we sort of are completing our filling out of the space of possibilities where we have the TF dot summary API bringing together features from really all three of the existing API is in in 2.0 so TF summary in TF 2.0 it really represents like I said this unification of three different ap is the instrumentation ops are actually provided by tensor board and they use this generic tensor data format that's the same format as sensor board summary which lets them extend to multiple different kinds of data types we borrowed the sort of implementation of the writing logic from TF contribs amuri and some of the api's but with slightly adjusted semantics and some places mostly just so that we aligned with the sort of state management and TF 2.0 and then there's actually a not trivial amount of just sort of glue and and circular import fixes to get the two halves of both tensor board and the original TF writing summary writing api to talk to each other I'll go into a little bit more detail about that so the dependency structure of your TF dot summary and TF 2.0 is a little bit complicated it the the actual Python module contains API symbols from both sensor board and tensorflow fuse together but because the censor board symbols also depend on tensorflow sort of this creates this complicated dependency relationship and the way we sort of linearize this dependency relationship is that tensorflow dot summary or TF dot summary and the original tensorflow code exposes the writing api's like create file writer and the sort of actual underlying writing logic then we have what I kind of call it shim module intensive board that merges those symbols via wild card import with regular imports the instrumentation API is like scaler an image that are now defined in tensor board and this produces sort of the combined namespace but now it's it's a tensor board module so then tensor flow in it's sort of top level in a pie where it's assembling the API together imports the tensor board module and sets that as the new TF dot summary and this does mean that the API surface depends sort of directly on tensor bored but tensorflow already has a pip dependency on tensor board so this isn't really a change in that respect but the IPI surface is now sort of being constructed through through multiple components where the summary component is provided by a tensor board and what that gives us is a single module that combines both sets of symbols so that you know the Delta four users are smaller but we can sort of have the code live in the appropriate places for now and so these are some code samples for the TF 2.0 summary API the first one shows it under eager execution it should look fairly similar to contribute as default you can call the instrumentation obsolete so you no longer need to enable summary writing which makes it a little more streamlined and I should say that the ops they write when executed sort of in theory but there's actually some buffering in the writer so usually you want to make sure to flush the writer to ensure that it is actually written to disk and this actually this example shows an explicit flush in eager it will do flushing for you and you exit the as default context but it's good if you sort of care about making sure like for example after every iteration of the loop that you have data you know persist into disk it's good to flush the writer and then this is an example with the Atia function decorator again you create the writer upfront one important thing to note here is that the writer has to you have to maintain a reference to it as long as you have a function that uses the writer and this has to do with the sort of difference between when the function is traced and executed it's a limitation that hopefully we can sort of improve this a little bit but for now at least that's wine Kavya so the best way to sort of handle that is you set the writer as default in the function itself and then call the instrumentation ops that you need and these write again when executed meaning when the function is actually called so we can see that's happening down with them my phone call and then you can flush the right area again and then here we have an example with sort of legacy graph execution since there's so folks who use that in 2.0 this is a little bit more verbose but again you create the writer you set it as default you've sort of constructed your graph and then in this case you need to explicitly initialize the writer if you're running in it up we have sine of a compatibility shim that lets you run all of the v-2 summary ops so that you don't have to keep track of them manually and then again flushing the writers so these are the sort of this is how you'd use it and legacy graph execution so how do you get to TF 2.0 the contribs summary API is close enough to the 2.0 summary API that we do some odd actually mostly we auto migrate this in the the TF upgraded v-2 migration script but TF dot summary in 1.0 is sufficiently different on the writing side that we can't really do a safe Auto migration so here's the kind of three bullet version of how to migrate by hand the first thing is that the writer now needs to be sort of present created and set via as default before using the summary ops and this is a limitation that you know where it's a little bit tricky we're hoping to sort of relax this a little bit so it's possible to set it writer later but for now you want to have the default writer already present otherwise the summary ops basically just become no ops if there's no writer since they have nowhere to write to then each op takes its own step argument now this is because there's it's there's no sort of later step where you add the summary to the writer that's where the step was previously provided and there's also no global step in TF 2.0 so there isn't really a good default variable to use so for now steps are being passed explicitly and I'll talk about this a little more later on the next slide and then the function signatures for the instrumentation ops like scalar and image have changed slightly the most obvious thing being that they no longer return an output because they arrived via side-effect but also there's slight differences and the the keyword arguments that won't affect most people but it's something good to know about and these details will all be in the external migration guide soon and so the other changes and this is some of the other stuff I was mentioning one change is with graph writing since there's no sort of default global graph in 2.0 there's no direct instrumentation up to write the graph instead the approach here is there's a set of sort of tracing style API so enabled and disabled tracing and what those do is they they record the graphs of executing TF functions so functions that execute while the tracing is enabled to the summary writer and this sort of better reflects the TF 2.0 understanding of graphs is something that are associated with functions as they execute then this is sort kind of what I was alluding to it's still a little bit tricky to use default writers with graph mode since it's not always the case that you know which writer you want to use as you're assembling the graph so we're working on making that a little bit more a little bit more user friendly and setting the step for each op is also definitely boilerplate so that's another area where we're looking to make it possible to set the step you know maybe in one place or somehow in the context to avoid the need to pass it into to ops individually and then the event file binary representation has changed this only affects you this doesn't affect tensor board in that tensor board already supports this format but if you were parsing event files in any manual way you might notice this change and I'll talk a little bit more about that change in the next section and finally as mentioned the writers now have sort of a one-to-one mapping to the underlying resource and event files so there's no more sharing of sort of writer resources okay and then the last section will be about the summary data format so this is log directories event files how your data is actually sort of persisted so first off what is a log directory the Tencent board command expects a required - Josh Wagner flag in fact your first introduction to tensor board may have been trying to run it and then it spits out an error that you didn't pass the log dirt flag so the log directory flag is the location that tensor board expects to read data from and this is often sort of the primary output directory for a tensor flow program frameworks again like estimator and caris have different knobs for where output goes but often people will put it all sort of under one root directory and that's often what people use this log directory but unser board has sort of this flexible interpretation where it really all it cares about is that it's a directory entry containing summary data and when I say directory tree I really do mean a tree because the data can be arbitrarily deep tensor board will traverse the entire tree looking for summary data and you might think that could sort of be a problem sometimes especially if there's hundreds thousands of event files and it's true yeah log directories can be pretty large and so tensor board tries to sort of take advantage of structure in the log directory by mapping sub directories of the log dirt to this notion of runs which we talked about a little bit in the early section about the tons of board UI so again these are runs like run of a program not definitely they're not individual session dot run calls and when tunts our board loads of Ryan the sort of definition it uses is that it's any directory and the log der that has at least one event file in it and in this case we mean only correct children so the directory has to contain an actual event file and an event file is just defined as a file that has the name has the string t of events in the name which is just the standard naming convention used by summary writers so as an example of this we have this log directory structure which has sort of a root directory logs it has two experiments sub directories in it the first experiment contains an event file so that makes that itself a run it also contains two sort of sub directories train and eval with event files so those two also become runs they're sort of visually they look like sub runs but they're all considered independent runs for tensor board at least you know in the current interpretation and then an experiment - that doesn't contain an event file directly so it's not a run but it has a train sort of subdirectory under it so tensor board looks at this log directory and traverses it and finds four different runs and this traversal step happens continuously tensor board will pull a log directory for new data and this is to facilitate using tensor boards sort of as a way to monitor the progress of a running job or even potentially a job that hasn't started yet you might start tensor board and your job at the same time so there may not this directory may not even exist yet and we may expect that different runs will be created as it as it proceeds so we need to continuously check for new new directories being created new data being appended in the case where you know that your data is not changing like you're just viewing old data you can disable this using the reload interval flag and you can also adjust the the interval at which it pulls so when it's traversing the wide directory it does this in two passes the first pass is finding new runs so it searches the directory tree for new directories with TOF event files in them this can be very expensive if your tree is deeply nested and especially if it's on a remote filesystem and especially if the remote filesystem is on a different continent which I've seen some times so a key here is that you know walking the whole directory tree can be can be pretty slow we have some optimizations for this so for example on a Google Cloud storage we take it rather than walking each directory individually we have this sort of iterative globbing approach which we basically use to find all directories at a given depth at the same time which takes advantage of the fact that GCS doesn't actually really have directories there's sort of an illusion and there's other sort of filesystem optimizations like this that we would like to make as well but that's just one example and then the second pass after it's found all the new runs is that it reads new event file data from each run and it goes through the runs essentially in series there's sort of a limiting factor in Python itself for paralyzing this but again something that we could be working that we are interested in working on improving and then when you actually have the set of event files for a run pencil board iterates over them basically and directory listing order you might have noticed on the previous slide with example logger that the event files all have like a fixed prefix and then a timestamp and so what this means is that the directory order is essentially creation order if you have n files and so in each event file we read records from it sequentially until we get to the end of the file and then at that point tensor board checks to see if there's a subsequent file already created if so it continues to that one otherwise it says ok I'm done with this run and then it goes to the next run and after it finishes all the runs it waits for the reload interval and then it starts a new reload cycle and this reload resumes the read from the same offset in every file per run that it stopped in and an important sort of thing to point out here is that tensor board won't ever revisit an earlier file within a run so if it if it finishes reading a file and continues to a next later one it won't ever go back to check if that the previous file contains new data and this is based on the assumption the last file is the only active one the only one being actively written to and it's important to avoid sort of checking all event files which can be sometimes there's thousands of event files in a single run directory and so that's a sort of mechanism for avoiding wasted rechecking but this assumption definitely doesn't always hold their cases when a single program is using multiple active writers within a run and in that case it can seem like the data is being skipped because you proceed to a new file and then data added to the original file no longer appears intense or bored and luckily it's fairly straightforward to work around this you just restart tense or bored and it will always pick up all the data that existed at the time that's it started but we're working on a better fix for this so that we can still detect and read when there's data added to files other than last one well this is something that is has bitten people before so just a heads up and then so the actual event file format this is based on TF record which is a sort of standard tensorflow format it's the same as TFI OTF record writer and it's a pretty simple format enough that it fits on the left side of this slide it's basically just a bunch of binary strings prefixed by their length with CRC use for data integrity and one particular thing that I'll note is that because it's there's no sort of specific length for each string there's no real way to seek ahead in the file you basically have to read it sequentially and there's also no built-in compression of any kind and tensor board it's it's possible in theory to have the whole file will be compressed and some board doesn't support this yet but it's something that could help save space when there's a lot of redundant like a lot of redundant strings within the event file and then each individual records who TF record is sort of the framing structure for the event file each individual record is a serialized event protocol buffer and this sort of simplified schema for the protocol buffer is shown on the Left we have a wall time in a step which are used to construct the time series and then we have a few different ways to store data but the primary one is a summary sub message the main exception is graph data gets stored in the graph def separately and then we can look at the summary sub message which is itself basically a list of values sub messages that's where the actual interesting part is and each one of these contains a tag again that's from sort of our overview of the UI that's like the name or ID of the summary as shown in tensor board we have metadata which is used to describe the more generic tensor formats and then specific type fields including ones for the original TF one point X sort of specific fields for each type and then the tensor field which can be used with the new sort of new tensor style instrumentation ops to hold a sort of general forms of data and then in terms of loading the summary data into memory so I mentioned this briefly in the sort of architecture stage but tensor board has to load the summary data into memory because like I said there's no real indexing or random access in the event file you can think of them like they're just like Rob blogs and so tensor board loads it into memory and and creates sort of its own indexes of data bye bye run plugin and tag which support the different kinds of visualization queries that plugins need and tensile board also does down sampling in order to avoid running out of memory since the log directory may contain far more data then it could reasonably fit in tensor boards Ram and to do the down sampling it uses a rose water sampling algorithm it's essentially just an algorithm for uniform sampling when you don't know the size of your sequence in advance which is the case when we're sort of consuming you know data from an active job and because of this it has sort of a random aspect which can be surprising to users where you might not understand like why is you know this step being taken and not this one and there's like a gap between steps this can be tuned with a samples per plugin flag so that Tunes the size of the reservoir and basically if you make the reservoir larger than your total number of steps you always see all of your data so that gives you a certain amount of control over how that sampling works and just to sort of review some of this section there's some best practices for at least in the current tensor board you know how to get sort of the best performance one of the basic ones is just to avoid having enormous log directories in terms of the number of files sub directories quantity of data this doesn't actually mean that the sort of overall log directory has to be small it just means that tensor board itself will run better if you can launch it at a directory that just contains relevant data for what you want to examine so sort of a hierarchical structure where you can pick sort of an experiment subdirectory or a group a few experiments together works really well here I mentioned the reload interval you can set it to zero to disable reload so for unchanging data this helps avoid sort of extra overhead and that's especially useful on a remote file system case in that case it's also useful if you can run tensor board close to your data or sort of download a subset of it that you need so that it doesn't all have to be fetched over the network and for now due to the sort of way that superseded event files aren't reread it's best to avoid multiple active writers pointed at the same same directory this is something again that we're actively working on improving this but at least for now that can lead to sort of this appearance that some data get skipped and then in general stay tuned for logged or performance improvements we're hoping to make a number of these improvements soon and that pretty much rounds out the content that I have for today so I'd like to thank everybody for attending and if you want to find out more about tensor board we have a new sort of sub-site on intensive Florida org slash tensor board you can also find us on github and people interested in contributing are welcome to join the tensor board special interest group thank you all very much [Applause] the potential use cases for the multiple summary writers sometimes this is just sort of a matter of code structure or sort of if you're using a library that itself creates a summary writer it's not always straightforward to ensure that the you can use the same writer I mean because it's different so P F one point X had this file writer cache which was sort of the best practice then was to use this sort of shared cache to ensure you only had one writer per directory and it was to work around this problem I think it's better to work around it on the tensor board side and have some ideas for how to do that so hopefully that part will be out of date soon like within a mantra too yeah so I think a lot of this depends on yeah I think that the event file format itself could be a lot more it could be a lot better tailored to what tensor board actually needs and I mean some of the things I mentioned would just be like even if we had an index into the event file that could potentially help sort of we could we could potentially Perl its reads or we could sort of scan ahead and do sort of like smarter sampling like rather than reading all the data and then down sampling it we could just like pick different offsets and sample from there we're sort of we're constrained right now mostly by this being kind of the legacy format but I think it would be pretty interesting to explore new formats something particularly when you have different data types mixed in something sort of columnar could be kind of useful where you can like read-only images if you sort of need to read images or sort of otherwise avoid the phenomenon where so this happens sometimes where like one one particular event file contains like lots and lots of large images or Groff Duff's and this sort of blocks the reading of like a lot of small scalar data and that's obviously not really you know it doesn't really make sense but again it's a sort of limitation of having the the data only be accessible sequentially yeah so graph is a graph visualization which was actually originally created by big picture it's a visualization of the the actual tensor flow graph so like the computation graph with ops and edges connecting them and sort of for the data flow it's pretty cool it's it's really a good way to sort of if you want to sort of visually understand what's going on it works best when the code has sort of uses like scoping to kind of delineate different parts of the model if it's just sort of a giant soup of ops it's a little hard to like understand the higher order structure and there's actually some cool work done for TF 2.0 which which isn't in the presentation about for Karis showing sort of the Karis conceptual graph using Karis layers to give you sort of a better view into the kind of high-level structure of the model like you'd expect to see you know in a diagram written by a human but the graph board can still be useful for sort of understanding exactly what ops are happening and sometimes it's useful for debugging cases if some some sort of part of the graph is behaving weirdly you know maybe you didn't realize that you know you actually have like an edge between two different ops that was unexpected are there any plans to add more basically like that sort of higher order structure annotation so I'm imagining for instance like you have resume having like the whole model and then a block and then a sub block and sort of there's like five layers of structural taps that would be nice building yeah I think this is this is an interesting question I think so right now the main tool you have for that structure is just sort of named scoping but it only really works the the part of the graph is all being defined in the same place anyway I think it would be really nice to have the graph visualization support more kind of strategies for human friendly annotation and organization I think the recent work that we've done on this has been the curious conceptual graph would just find it over there but I think having that work for not just Karis layers but sort of more more general like model decomposition approaches would be really nice like often I find that the problem isn't that it's not capturing enough it's actually that it's capturing too much so for instance like you'll have like I publish but then there's a bunch of like loss and regularization they're ones of being a bunch of tensors clutter and so actually even the ability to like to filter stuff out so there is there's like a feature in the graph dashboard where you can sort of remove nodes from the main graph but I believe the only way that has to kind of be done by hand it does like a certain amount of like automatic extraction of things that are sort of less important out of the graph but so maybe that's a place we can look into having like you've a sort of a smarter procedure for doing that or a way to sort of tag like hey this section of like I don't actually want to see any of these or like this sort of should be factored out in some way things [Applause]