W&B Deep Learning Salon - Reproducibility Challenge

Key Takeaways

all right i believe we are live it'll take a second or two for it to actually start up on the weights and biases youtube page so give us just one second for that to kick off all right we are going all right welcome everyone to the weights and biases deep learning salon i am your host charles fry what we'll be doing this time is hearing from a bunch of weights and biases community members who uh participated in uh the reproducibility challenge so the reproducibility challenge started off with papers with code uh and uh we've decided to join that help uh help out with it actually rather than me telling you anything about it i'm gonna hand it over to um our reproducibility challenge expert at weights and biases daganta mishra de ganta go ahead hello everyone my name is uh i'm a machine learning engineer and as charles stated i'm glad that i can take the title of vp series challenge experts but the experts will be doing the talking after me the people who participated in the challenge uh when i joined the team last year uh coming from an academic background with pc challenge uh is something that i really really help with high importance because it's super important to make uh research uh in deep learning more transparent and reproducible and we decided at weights and biases that we'll be supporting the papers with code initiative uh for you please recharge uh and we provided our participants with the resources uh to ensure that we could reproduce as many papers from top ai conferences from 2020 um and we did that with a fair share of success we had so many interesting uh reports that we uh we gathered from our participants uh who we produced super exciting papers ranging from genetic algorithms to attention models uh to to many other variants like pruning and lottery ticket hypothesis that you'll hear in a short while and uh before i start like before i give the mic to the people who participated in the challenge i just want to give a super quick teaser trailer that we'll be also doing the spring 2021 edition of the republic challenge starting off in april 1st and i really really encourage everyone uh to consider participating because it's going to be super awesome and we upscaling our efforts in helping you all out to make this a huge success with that i like to give it to the participants all right yeah i think uh first up we have the uh team that worked uh from the fast ai community that worked on uh the reformer paper one of uh one of the most exciting of the many papers that came out this uh this past year on making transformers more efficient uh so arto if you'd like to uh uh take it take it away hey everyone my name is arto and i will be presenting uh our entry to reproducibility challenge for reform as a efficient transformer paper uh from google research uh authored by nikki takitayev lukas kaiser and anselm levskaya and it's done by fast ai community represented here by my teammates morgan and halvar so since their release of attention is all you need paper transformers have become by far most popular architecture for nlp and now is successfully applied to region time series and audio data and key mechanics in transformers is attention which is used to produce contextualized token embeddings and for each token keys queries and values are computed and then every key is compared to every query to produce the attention scores and this results in uh quadratic uh complexity of the computation and this is the main limiting factor to applying transformers to the tasks requiring attention in one context so making transformers more memory and time efficient is an active area of research right now and one can refer to long-range arena benchmark for comparison of the current uh techniques uh so uh the one we were reproducing is reformer uh which is uh designed to allow training on uh input sequences of uh lens of over 64 000 tokens on a single accelerator and to achieve this uh five key techniques were used so the lsh attention mechanism is an approximation to full attention using locality sensitive hashing it utilizes the sparsity like natural sparsity of attention matrix so all the key query pairs are hashed using random projections bucketized and in such a way that similar query pairs with will end up in the same bucket uh then buckets are sorted uh the keys and queries are sorted by buckets and the attention scores are computed within each bucket only and in to one neighboring bucket this allows to reduce the time and memory complexity of the computation from order n square to order n log n and this is a randomized algorithm and its robustness might be improved by doing multiple rounds of such hashing the next technique is reversible layers it's a variant of residual layers allowing to avoid storing intermediate activations during the forward pass in cost of recomputing those at backward paths and this enables to train very deep models under constrained memory requirements and also there are some supplementary techniques so for lsh to work properly we need keys and queries produced from a single token to end up in the one bucket and to ensure this keys and queries are computed using shared projection also to avoid feed-forward computation becoming memory bottleneck of the system uh it might be computed in chunks and when combined with reversible layers this allows to control memory consumption by feed forward layers also to reduce memory used for storing positional encodings for very long sequences excel positional encoding technique might be used which uses factorization to enable very long sequences so the central claims of this paper are following the reformer performs on par with traditional transformer models while being much more memory efficient and much faster on longer sequences and to verify these claims is the main goal of the reproducibility challenge and the original implementation of reformer was done in tracks which is based on jack's framework and our community emerged from fast ai so we had additional goals to re-implement the model in pytorch and train it adapted to train using fast eye training look so the uh memory and time complexity related claims uh may be verified by direct experiments and here you can see memory profiles for variants of six layer deep models and as one can see there are this memory allocation jumps associated with big [Music] attention matrix computations for full attention and also memory is accumulated during the forward pass and only released during the backward pass using lsh attention allows to avoid these memory peaks by doing it in and logan memory complexity instead of quadratic one and uh using reversible layers allows to make memory allocation constant in regard to number of uh layers in the model and here it's shown in orange and the resulting reformer model which is shown in red here has by far lower memory consumption as compared to the classic transformer and this will be even uh more strong when the context lens grows also notice that the memory consumption of reformer grows linearly linearly with a number of hashing rounds and this results in some accuracy memory trade-off so the next plot shows the computation time for lsh attention and it is constant with respect to sequence lens and grows linearly with number of hashing rounds while uh the full attention computation grows approximately exponentially at our observation so uh now to the next part to verify the performance related claims uh there are three experiments same groups of experiments one on synthetic task text compression experiments on nvk8 and translation experiments on wmt14 so the synthetic task may be formulated as follows given a twin sequence of this form uh where a word is a sequence of 511 random integers the task is to predict the second half of the sequence in outer regressive manner and this is this may be simply done by copying from the first part to the second part the task is trivial to transformer with global attention but it is infeasible if attention is local so our experiments uh verified that uh lsh indeed uh attends globally and we were able to achieve uh high accuracy on this task with four hessian rounds also we observed unstable convergence when using shared qk projection on this task and here you can see a bunch of runs which only differ in random seed while like classic transformer converges very stably on this task and next to the ablation studies which are done on text compression using nvk8 dataset we compared each of the modifications with baseline model and here you can see validation bits per character for uh shared qk transformer and for reversible transformer on this task and for both experiments uh we observed worse performance as compared to baseline also we did uh experiments with encoder recorder architecture uh comparing classic and reversible transformers on wmt14 and the results are consistent with those observed on nvk8 and there are a number of possible explanations for the discrepancy of results we observed with the paper one the most obvious is it might be that classic model is just superior to the modified ones but also as you saw in our experiments uh the training did not fully converge so maybe given enough time uh the models will end up with the similar final score um and also we put substantial effort uh to obtaining a strong baseline and we had uh restricted time for doing supplementary experiments on the modified models and also we used the hyperparameters from the paper and original source code there are a number of possible numerical differences in the framework or subtle implementation details which result in some impact of on the performance of the models also as shown before the random factor might play some role here and it might be just due to some random variation also we compared performance of lsh attention with a different number of hashing rounds and as uh climbed in the paper the lsh attention with eight hashing rounds performed very similarly to the fully tension and in our experiments four hashing rounds were enough to reach some similar results but with somewhat slower convergence so that's all with experiments done and i also wanted to highlight some tools which made our uh reproducibility experience much smoother in nbdev is a tool for library development using jupyter notebooks it allows automatic code to export to by files documentation auto generation uh writing tests in the notebook and uh using them for continuous integration with github as well as running locally so just a great thing to have in your toolkit and also uh ways and biases were really useful for fine-tuning hyper parameters and debugging some subtle things when you can uh like examine uh all the training dynamics and to gradients uh during training uh that's that might be really useful uh also it's really useful when working uh in distributed teams such as ours and it's really a pure joy to watch a bunch of runs uh progressing simultaneously when you're done with your implementation and only running experiments is what's left to do so to summarize we were able to validate the memory and time complexity reduction is reformer and it indeed allows to train models on unprecedentedly long sequences uh also the performance figures were not as good as in the paper in our experiments and results from other papers uh also shown that the reformer uh did not perform uh just as good as a classic transformer and uh this may be attributed to the fact that uh the reformer implementation is technically elaborated and might be difficult to reproduce correctly so on that thank you for attention and we can proceed to questions uh great so thanks so much for sharing your work arto and uh yeah the other folks from the team also want to uh uh pop up on video for just a few questions the thing i wanted to start out with was um asking um so you so you mentioned that you had like a pretty big team uh or that you that weight spices were really helpful for your team and i noticed that you had quite a large team of people on this project so could you talk a little bit just about um you know how you spread work around the team like like what different people were working on how you coordinated as what appears to be like kind of a worldwide uh and large team of of ml engineers uh yeah i could i could maybe take that um yeah it was actually it was actually a challenging thing um part of us that we we mentioned in our report as well um kind of structuring it in a way that everyone could work on something and didn't have to ping you know someone else to ask what they should be working on um so we used initially it was all in discord and it was just a lot of messages and pin pin messages and like task lists we moved to trello which worked better but also requires like a lot of maintenance and really requires like a kind of a champion to you know make sure everyone's moving their cards um along the board um so in terms of like task management i don't think i know maybe you might disagree or maybe not like i think we'd like to try github projects maybe i don't think trello was the right solution for us um but yeah we need basically we were trying to find a way to um have make it so that like team members who weren't you know able to commit like a lot of time could jump in at any point see a task that needed to be done and go run with it run with it without like too much input from anyone else um so open to hear any potential solutions and i'd not have anything else to add yeah yeah definitely like having this task management tools integrated into your workflow flawlessly is really important for such team as ours we had a bunch of guys who uh like wanted to participate but sometimes it was difficult to effectively allocate the tasks and anyway i think it was a great learning experience for everyone involved so but there is definitely a place to improve in terms of this project management yes i also think um as you mentioned nbdev was helpful in in this respect uh so having like a proper development process with uh sort of uh gift branching pull requests and uh sort of having a yeah like a better development process also helped a lot uh in in the first part of the project was a bit chaotic and we might possibly have a repo that we had to leave behind and move to a new it happens definitely uh it's interesting yeah how uh you know how much that coordination problem rears its head once you have a group of people uh dagonta had a question uh de ganta if you uh want to speak up yeah um first of all uh awesome presentation i have been following the space of transformers and obviously it has been in the trends lately uh in literally every domain not just nlp but in cv as well so just a general question more from an academic perspective you stated that uh the results you obtained from uh running the experiments with performer weren't as uh good as the classic transform model still so do you have any plans of uh extending some more investigations into why and maybe do you have any future ideas around taking inspiration from what performer did and maybe coming up with something of your own i think that would be really a cherry on the cake too to actually be able to manage uh well yeah uh actually there are like a couple of directions it may progress uh first off we like after finishing this reproducibility challenge we created a repo to implement more modern techniques related to improving the efficiency of transformer transformers in different directions also like this reversible layers might be really a cool trick as they are kind of more efficient for particularly transformer architectures as compared to a regular gradient checkpoint for example and it would be great if we could make it work just as good as the regular residual layers although uh they are not quite mathematically equivalent so it it is possible that uh there would be some difference in implementation needed to make it work just as good so yeah and there is really a lot of new research published all like almost uh we can say every week in respect of the efficient attention and i think it's really fascinating because now transformers are while while they apply it to tasks other than nlp and sometimes really very long context is like necessary to achieve great performance um i wanted to talk a little bit more about this like gap between the results that you found and the results reported in the paper it seems very plausible to me i think all the explanations you put out there were very plausible so if you had to pick one explanation to go with is there one that really stands out in your mind is the most plausible or do you think that it's too uncertain well i don't know how others would say but i would go with like some subtle implementation details actually we had some fun time like fun weekend with morgan trying to debug really a minor scene which is it doesn't matter mathematically but you need to fix it to really make a model and the performance difference was really consistent while it just does not follow from the mathematics and like it's really hard to find this uh information of why does it happens also there is like implement a variant of implementation of efficient attention which is not very time efficient but memory efficient which is done in chunks and when done in by torch uh it works uh while being just exactly mathematically mathematically equivalent to regular attention but it works uh just worse out of the box and i'm pretty sure it's possible to make it work just as well as full attention but it requires some fiddling with this numerical scenes and i will i would uh most probably attribute the difference with the differences we observed to these factors yes and and also uh the reformer is uh sort of a much more finicky architecture than the transformer itself so i think also it's much harder to tune it properly so i guess if you spent more time in compute on proper tuning we might have been able to to squeeze extra performance out of it that's interesting i think um yeah that kind of points often there's a there's like a subtle phenomenon over the course of years usually where architectures and techniques that are really robust to implementation noise uh architectures and ideas that find themselves in a very smooth part of the landscape of ideas um end up uh being a lot more successful than than ones that even though they work very well uh the if the if the landscape around them is very sharp kind of like um kind of like bad local minima or something um i also noticed this one one of the things about the reformer model that seemed particularly kind of strange to me and i noticed it was also in the reviews on open review is this like shared key query component of it could you talk a little bit more about like sort of why that's necessary and whether that's maybe an important part of this performance gap so first uh probably two more simple simply current question why is it necessary uh well actually as uh it was shown uh for locality sensitive fashion to work you want to both quiz keys and queries for the same token to end up in the same bucket because attention is only computed within the bucket and this is uh like this necessities shared computation also like [Music] probably multiple recent papers shown that in some way transformers uh might be and particularly attention might be viewed as uh like uh lookup tables here so we do lookups into the data uh like not like uh directly in sql way but uh more like to some uh stored representation of the data and uh basically this uh can like this view on the attention can give you an intuition why uh shared qk should work uh on the other hand uh we like when we share a qk obviously is by far the highest attention score would be for the token to return to itself that is why uh in reformer the self self-attention is uh uh masked so the token is not allowed to attend to itself uh but but only when it can not attend to anywhere else right and uh as transformer architecture is done this makes like this uh disallowing self-attention might make uh a flow of positional information a bit more difficult through the models so that's uh my idea about that and probably that's why we observe such like uh strange results in regards to like effect of random seed uh on that uh synthetic experiment we showed because that synthetic experiment uh particularly requires uh this uh good attention to positional information we just need to copy uh a bit from particular position into the output and uh for shared uk that turned out to be uh like more difficult that's interesting and yeah that's a that's a neat connection to make between that that result that you had on the synthetic task and the and the issues that you saw on the more realistic tasks and all right so um i could talk about this experiment all day but unfortunately we've got to move on to our well unfortunately we've got to close this off but fortunately it's to move on to our next experiment so thanks arto halvar morgan and everybody else in the fast ai community who worked on that project thanks thanks so much for your work thank you for inviting us and thanks everyone for your attention uh all right so now let us switch over to um we have varun and rajat who worked on the uh reproducing uh utku evchi's paper on the wriggle method uh varun and rajat are you are you here um yeah i'm we are here and we can start if you like yeah yeah go ahead all right um thank you so much let me start just let me know if you can see my screen then i'll begin yup go for it all right thank you so um hi everyone i'm varun and i'll be present presenting our recent reproducibility work i'd like to first thank charles and the wnb team for hosting us today i'm also joined by my co-author rajat and yeah yeah yeah hi everybody i am rajest [Music] and both of us graduated last august from the indian institute of technology madras so uh let's begin a very prominent characteristic of deep networks today is the incredible over parameterization routinely deep networks have parameter counts for exceeding data points for instance on the right we have three models trained on imagenet each of their parameter counts is clearly greater than a million which is the size of imagenet this trend continues to grow today especially with newer models such as transformers especially the one we just saw transformers from the previous team compressing deep networks is therefore an important topic for both training and deployment sparse networks are a compelling choice to achieve this unlike a dense network most of the sparse networks weights are set to zero such networks can be computationally and memory efficient they can make it possible to train larger models with limited hardware typically sparse networks are obtained via pruning a process where you iteratively remove small magnitude weights unfortunately pruning almost always degrades performance in light of this i'd like to introduce the lottery ticket hypothesis which i could say is arguably one of the most interesting yet accessible works in recent past learning so the idea is to identify sparse networks or we call them tickets that can be trained to attain performance similar to the original dense network so you don't lose performance the trick lies in how these networks are initialized to do so we first train a dense network and prune it we now take this past network and re-initialize it with the original untrained dense weights unfortunately this whole process is costlier than a dense network and sort of nullifies the computational benefit of sparse networks so we want to do something better than this um wriggle short for rigging the lottery tries to do something entirely different they try to directly train a sparse network from scratch instead of pruning a dense network and wriggle achieves this by two steps the first every certain number of steps they prune a number of weights and they grow the exact number of weights back more importantly regal does not alter the number of non-zero weights in each layer so if you remove two weights in a layer you always add back to weights but you may add it at a different place this ensures that the floating point operations of the network are always constant which is something that we desire so let's talk briefly about the connection updates and uh more specifically wriggle prunes the smallest magnitude weights which is typical of pruning and for growing we grow the largest gradient magnitude the growth strategy is exactly where regal differs from other methods for instance set grows weights back randomly and snfs uses the largest momentum where momentum is your accumulated gradient snfs in addition to this also tends to redistribute layer wise opacity throughout training which results in non-constant flops as a result of the growth strategies these three methods have very different gradient requirements for instance set does not require dense gradients throughout training you can get away by supplying only sparse gradients regal requires access to dense gradients every case steps or so when you perform this topology update snfs on the other hand requires dense gradients throughout training so the only benefit you get from snfs is this sparse forward pass you don't really get any benefit through the backward pass let's now move on to the main claims regal makes rigor makes two claims the first is that it outperforms existing direct pass techniques which is basically set in snfs compared to the same parameter and flop count the second claim is the more interesting one it matches or exceeds pruning when trained longer now this is important because previously all these direct pass techniques were always inferior to pruning and regal manages all of this while maintaining constant training flops this is also where we come in we reproduce regulf from scratch in pitoch and we open source our code on github so um let me go over the claim validity i'll give you the results right away uh like there's no suspense uh the claims are true which is nice um but it's not true for parameter count alone on this graph you can clearly see that snfs outperforms regal for the same parameter count you also need to consider flop count when considering flop count regal performs the best trading of parameter count and flop count now the more interesting claim is does it perform better than pruning when trained longer and the answer is yes you can see that when we train regal twice as long to match the computational cost of iterative pruning we can achieve performance that is equal or slightly greater so um regal so far we've seen a lot of good stuff so let's see a few caveats the first interesting caveat is that the initialization matters more than the method uh what do i mean by initialization so we start off with this past network and i still haven't given you a recipe of how do you how to get that past network initialization is exactly that the authors use two initializations the first is called random where the sparsity of each layer is set constant it's equal to the global sparsity except for the first layer which they keep dense the other initialization strategy is called the erdos kernel or erk for short and here the sparsity is non-linear it tends to allocate more density or less sparsity for layers with smaller capacities so you'll see all these shortcut layers which have nearly full density all right let's see what the implication of these initialization methods are on both data sets we make comparisons on cipher 10 and c500 switching to erk initialization for any method always boosts performance and the surprising aspect is that this performance boost is often greater than swapping methods so for instance if i were to swap between sct and wriggle i may get a performance gain of say a 0.2 accuracy percentage or 0.3 accuracy percentage but if i swap from random to erk i'm going to get a one percent increase this sort of gives you this sort of puts in perspective whether you should be focusing your research on new algorithms or you should be focusing on new initialization strategies the second caveat is so far we have spoken about all these nice flop benefits while maintaining accuracy but the truth is you have no real benefits on current hardware so why is that the reason is cuda does not efficiently support sparse operations at this point cuda does support certain kind of pass operations which are blocks pass that is they have a certain structure structure to them but in our case we do not have any structure so cuda doesn't really offer us any speedups this is in fact an ongoing research direction so we might see things change in a few years from now but the more the more positive news is that there are a class of processes such as the graph code which support this really really efficiently on a graph core having a sparse network would give you the exact theoretical benefits that we have stated so far these processes are completely suited for fine grained parallelism so they can perform tasks that require really different kind of workloads they won't suffer any performance degradation like a gpu and in fact there's a regal implementation on this on graph code and you can find it in graph core examples this is one thing that i really liked about regal because they have actually gone ahead and tried to test their algorithm on a processor which supports um sparse implementation the third caveat is quite actually related to the first um so what do we actually care about do we care about flop count or do we care about parameter count i think a couple of slides ago i said that erk is better than random when you're considering a fixed parameter account why don't we invert that question what happens if you consider a fixed flop count instead we get a very surprising result that is erk is worse than random on a per flop basis now if you put together caveat 1 and 2 it means that before you come up with a new method like say regular set snfs or a new one or a novel initialization scheme you need to clearly define what your goals are if you care only about flop count like say you want computational benefits on alone you don't really care about storage benefits then maybe you should be doing something different all right i'll briefly describe a few additional findings we found in our reproducibility work the first is that regal is quite robust to its choice of hyper parameters what do i mean by robust i mean that the two parameters regal requires which is this alpha and delta t where alpha is your pruning rate and delta t is the update frequency the frequency at which you will perform these topology updates can be set largely independent of both sparsity and learning rate which is nice because now you don't have to tune n plus two parameters you can set these two parameters independent of everything else and just tune your uh remaining parameters this does not add a burden on a practitioner uh regal can benefit from radius distribution so uh recall that um i mentioned that in snfs you have this concept of redistribution where you move sparsity across layers that is sparsity for a layer does not remain fixed throughout training so we asked ourselves the question what if you incorporate that integral so it turns out that it can't bridge the performance gap between erk and random but it considerably shoots up the computational cost and probably isn't the way to go about bridging the gap between erk and random you can find more details about these experiments on our paper so um let me now present a one slide summary of our rc 2020 journey so our initial impression was that each random safartan usually took us uh two three hours at least from the tensorflow repository that utku and authors had open sourced so we expected that we'd probably take around two three weeks to wrap up in reality with all the configurations considered the hyper parameters it took us a couple of months so uh that was a good thing to remember so um a brief timeline we spent i think the entirety of october getting our implementation correct there were always points where we figured out that we had missed something especially pertaining to sparsity of the gradient and momentum buffers we then spent a couple of months experimenting on sephard 10 and safar 100 as well as conducting these extension experiments and the main challenge in our work was the volume the sheer volume of experiments so we spent a good amount of time thinking how best to represent these experiments we did not directly just want to write down a bunch of metrics and um here are a few goof ups so uh there was this time when i think we had to throw out a week's worth of experimentation simply because we forgot to consider random seeds and turns out that random seed was pretty important in this whole work and we had to average across multiple random seeds flop counting was tricky we also found that snfs was costlier than what wriggle paper the wriggle paper reported which is also very different which is quite unusual you don't usually see a baseline being reported uh better than it is on a paper and in terms of structure so as i mentioned before the main challenge in this reproduction was the sheer volume of experiments we had to adopt certain tools to manage our runs better first we found hydra to be super useful particularly for composable configurations as as you can see on the right experimenting with different methods just involved swapping yaml files so we could experiment with different data sets masking strategies optimizers we also found hydra's multi-run format and support for hyper parameter tuning especially packages such as optuna to be really useful hydra is a pretty new and extensible library so i would highly recommend checking it out if you haven't used it before second we started making use of technology from i think make files come from the 80s and 90s but turns out they're much more useful than bash files for storing experiment runs storing the commands used to run those experiments so that you can quickly replicate experiments or make minor changes in case something goes wrong or you want to run variants of these experiments finally w and b was really useful in terms of its dashboard that allowed us to easily group and visualize experiments for each method we had to run three or four seeds as you can see on the right these numbers three six five and um there were a bunch of um such methods so for cifar10 i suppose we ran around 150 methods uh different configurations of these masking strategies all of that and wnb centralized dashboard allowed us to easily visualize these and initially we were using tensorboard but the moment we started performing more experiments it became super unwieldy so uh we switched to wnb where all the runs show up in a single place and it was a much nicer alternative the api support also helped us plot so every time we made changes to our runs we didn't have to go and manually replot we just probably re-ran a python script which did our plotting and here are a few takeaways so the first is reproducibility can be as fun as working on new ideas and is certainly important for the ml community authors aren't adversaries and will often be interested and invested in your reproducibility efforts they want to see that their paper actually works when used in different contexts different scenarios and reproducibility need not stop at replication indeed you can discover some overlooked findings or you can even spawn new ideas we'd like to thank um utko and the other regular authors for their responsive communication throughout we had a pretty extensive mail thread with the original authors we'd also like to credit tim detmers we built our code base by extending his open sourced snfs code you can find our code base on github you can read our report on archive and you can also find our entire set of training experiments on wnb so thank you so much for your time and we'd be happy to take any questions great thanks uh thanks for sharing that work with us uh baroon and uh and rajat it was also i love the um the tool stack that you had there with hydra weights biases and make files yeah we didn't expect the make files to show up like make files were an ad probably in december when we started um rap we started putting all our experiments together in one um bash file or something and it got really long yeah i actually used a make file for my phd dissertation it was make thesis to set it up so i appreciate the makefile um so a couple of things one uh like on the more sciency side the you mentioned that you implemented redistribution uh so actually if you could go back to that slide where you showed comparing random and then random with redistribution and erk yeah let me in fact show you a bigger figure that might help so right let me just share it see i've just blown up the figure yeah all right so yeah it looks to me a little bit like maybe it's not a perfect lineup but it does seem like the redistribution methods which are in green and red in this chart are basically learning the erdos-renyi kernel is that accurate or am i missing something so um it's far more extreme than the erdos-renyi kernel and we had the same intuition we actually suspected this um we actually wanted to show this result we were biased towards showing this result that redistribution learns an erk like kernel because that makes that consolidates a lot of things but the even more surprising result which we have is if you take this distribution and re-initialize a network from scratch it doesn't learn it performs really really bad it performs worse than static in fact i could pull up those results so we went that's interesting that's wild yeah that's really wild and unfortunately we uh we tried a bunch of things but we weren't really able to explain this and um so uh we had a very different phenomenon like the famous thing about lottery tickets right is that you can if you take it reinitialize it it works is that and so you're doing the same type of reinitialization right where you keep the values of the weights the same and you and like apply the topology at the beginning uh we tried both in fact we tried the lottery initialization and uh fine tuning like initialization that is you keep the weights you just resume training again and both religion the fine tuning initialization works but a lottery like initialization does not work so uh let me just share this real quick if i can sure yeah so uh one of the reasons this uh big table is not on the slide was because we didn't want to push off too many numbers on to viewers so so if you go to the last section right re-initialization with regal sm you will notice that the numbers are like 90.3 and 90.2 which is far lower than what you were getting before 92 point something and similarly on this 500 column we were getting 67 ish and previously we were getting 72s so if you were to take this trained initialization and perform a lottery like reunit it actually performs worse which we found kind of surprising possible explanations are this extreme distribution if you see this distribution it's kind of extreme like it goes it's higher than erk like wherever erk goes dense it is dense and wherever erk is passed it is more sparse than erkin so maybe this distribution is too extreme for you to reinitialize and train successfully maybe if you do some sort of a warm-up before reinitializing it might help so there are these lottery ticket variants where you do a warm-up and then you re-initialize with those weights maybe that could help again this is just uh some i'm hazarding a guess so yeah yeah no that does make sense i've i have seen those results about how important it is to sort of warm up before the lottery uh for for like resnets in particular and uh i think one more surprising aspect was that regardless so i could start with random and do redistribution or i could start with erk and do redistribution turns out you end up with a very similar distribution regardless of where you start you could start with erk or start with a random you end up with nearly the same distribution and the computational cost just shoots up this distribution the computational cost is not due to redistribution here the computational cost is actually due to the nature of this um density itself interesting and snfs also has a distribution of this kind uh which is what i alluded by saying when i earlier said that snfs is much costlier than it seems i see interesting um on the topic actually there is i was going to ask another question but i believe de ganta actually has one that he wanted to ask taganta yeah yeah i have three questions and this is a space that i really follow uh quite extensively not work on but i do read uh literature on this um first question is uh if it is purely on deployment or level considering the difference between google and snfs as you stated snfs is essentially uh fast and forward pass and doesn't do anything on backward cost so if it's purely on deployment which only concerns with the inference time or influence latency uh which one do you think forms actually better is it regular-based networks or sms i would recommend going with just iterative pruning in that case so if you like uh hover back to any of our tables where we have these results you will see that the inference cost of pruning is proportional to the density like 0.11 is very similar to 0.1 snf is the distribution the nature of the distribution makes a forward pass also costly so for the same density you would see like 0.38 so um if you're concerned only about deployment i guess iterative pruning is still your best bet today um my second follow-up question is uh this whole space looks pretty much similar to even or maybe let's say synchronous and parallel uh to the space of neural architecture search especially when you have latency constraints or you're trying to find optimal width uh so this is paper by uh amazon which was at cbpr2020 called as adaptive and optimum network width search with latency constraints and just to summarize that paper basically tries to find uh the uh adequate number of channels for each uh layer uh based on a latency constraint provided uh and there's an interesting graph in that paper which kind of conflicts the density graph that you should so in that graph apparently it shows that the latter layers like the layers towards the end of the network uh requires uh to be more dense than the earlier layers in fact more dense even more dense than the baseline uh density uh while the earlier is and it's like linearly increasing but uh in your density graph that you should uh the trend isn't looking similar uh of that sort so um just wanted to understand why uh would there be a conflict uh in that case uh what the aos shows when they did a neural architecture search and what uh wriggle or any other pruning method shows so i'm not sure how to really reconcile these two facts but what i do know is that the fully connected layer of both these resonate 50s and the wide resonant 22 are set to one so the density of those fully connected layers are set to one these are really huge like fully connected layers in vgg for instance i think it hosts around 70 of the parameters and it's the last layer so um while it might seem that erk and sparse gradient are actually like dipping density as you go deeper for the fully connected layers they tend to allocate full density that is you do not remove any parameters from those layers so maybe that can be used to reconcile maybe we should try architectures where there is no fully connected layer and maybe then compare because these fully connected layers tend to host a much higher number of parameters so that could be a possible explanation and my last question is more on your implementation which is uh just a general question i haven't yet come to a realization but why did you say that uh makefire was more useful than a bash script in your case so let me let me put it this way so for instance let's say a single run of regal on safar 10 that would involve running at you had to run for densities i think it was point five point two point one point zero zero five and two initializations then we also had these redistribution strategies so it's like two into two into five we're already at um a 20 and for if you were to do it in a batch loop you could so you could technically do it through hydras you could do it through hydra's multi-run this syntax which is kind of neat but it sort of constrains you to a single device and if you have a slum cluster to operate with you maybe want to send some of these runs to different nodes so uh we didn't really play around with hydra's plug-in for slurm clusters but what we did find useful was sort of wrapping it up in a bash script or make file and if you were to do it in a bash script you would maybe have to loop over these variables or supply them using command line and i feel that make files could do the same thing but slightly more elegantly and it was far more readable for the other person make sense thank you one question i had a little bit you mentioned that there is actually a lot of subtlety to this like trade-off between flops versus parameters like where you're getting your efficiency and like my mental model there a little bit is like if you're designing for mobile devices maybe what you care most about is reducing parameters uh whereas if you're like going for some super low latency setting then maybe what you care about is flops so like you know i guess is that a good way of thinking about this trade-off here or what have you learned well thinking about that trade-off in the context of this uh ex this set of experiments um i i think that's an excellent way to think of these trade-offs so um in my mind at least there are two two communities in ml which use compressed networks one is uh they try to train larger models on smaller devices in fact the lynn former reformer fall under these categories there's also this community of people who want to use ml for edge devices or ml to transmit data ml2 maybe even represent data especially with these implicit kind of functions that we're seeing so maybe they'd be cons concerned more about parameter storage as opposed to computational cost so they'd be okay actually using a dense computational cost but they want to save on the they want to compress the number of bits transmitted yeah question uh just curiosity especially um so two questions one is uh in an optimal setting what do you care about most uh is it flops is it latency throughput or parameters now i understand that you distinguish between parameters and flops but flops is not always representative of latency or throughput some people might care about how many images it passes in in a second which essentially is the throughput uh that is the first question and the second question is uh is there any alignment between uh the current state of the research in pruning with the state of research and quantization because that is also something very relevant to making new electrics being able to run on much lighter compute devices so that's a really interesting question and a really good one so i'll actually answer the second part first so um i was briefly going through quantization literature recently so i may have missed things so like please get back to me and let me know if i miss certain things but the way i see it i think there were three or four main approaches one was range based linear based k-means or bloom bloomer filters and there are also these papers which use new lossy compressors to quantize um but most of them do pruning and then quantization these quantization parameters may be trained uh may be tuned they're not really trained and there's also this work where you do quantization of our training but besides that i don't really see works where quantization is done during training you do it as a fine tuning step to maybe tune some of these quantization parameters but this is a work where you're doing pruning as you train even iterative pruning for instance you're doing pruning as you train you don't do it post training so uh maybe once we start to see works where quantization is done during training that could change things you could use it in combination with pruning and get even more storage benefits now uh the first question which is what do you really care about and are flops representation representative of throughput um i don't think the flops calculated here are completely representative of throughput for one we assumed there were efficient implementations of sparse convolutions available so if i were um working with matrices of so-and-so sparsity i get the complete speed up in comparison to a dense model to put this in perspective suppose i had i had a layer which was 90 percent sparse you would expect a speed up of around 10x whereas um goose powers and cuda spars and these blocks pass matrices today will give you only around i think three to four x so these flop counts are not really representative of the throughput that you can achieve on a real device but that is also where i find something like graph code exciting you should you can probably try out these algorithms on graph code and benchmark the throughput there that would be a good playing field all right we don't have we're basically out of time but i did want to touch a little bit on what you just brought up with the graph core and their chips and their their their the things that they make possible because i think that's a really exciting direction of research so it seems to me that that like basically what's possible with that kind of a chip is just like much much greater than what's possible with gpus and tpus so do you