Eco-aware GNN Recommenders

Key Takeaways

[Music] Welcome to Data Skeptic, a podcast exploring the methods, use cases, and consequences of recommener systems. Welcome to the first guest interview for our recommener system season. Interestingly enough, we're going to ease out of the graphs and networks topic just a bit. Our focus today is the recent paper ecoaware graph neural networks for sustainable recommendations. Now in the machine learning process, training is usually the expensive part. Of course, inference costs money too, but you can usually inference on cheaper hardware or sometimes just with the CPU. The training on these massively parallel GPUs is most where your costs get sunk. While of course you do that once, well, do you ever really? There's usually a bunch of attempts before you get your final production model. The bigger the model, the more energy it requires. And there's an environmental cost against that that we can pretty well measure in CO2. Tools like the one we're going to discuss called code carbon can help you measure that. But there's a question being begged along the way. Do we really need those models, at least in all cases? Maybe it's nice to have some large language models, but are we bringing bazookas to gunfights? In their research, Antonio and his collaborators explored a variety of different models and the trade-off between the energy consumed and the accuracy of those models. It seems natural to assume that the less energy you use, the less good of a model you'll get. But we'll see if that story is a little bit more complicated than that. There's one particular model we're going to talk a lot about called light GCN. GCN standing for graph convolutional neural network. So, we've got a lot of ground to cover in this one. Let's jump right into the interview. >> Hi everyone, I'm Antonio Pogifificato and I'm a second year PhD student from the University La Sapiens of Rome and I'm also collaborating as an applied scientist visitor with Amazon research. My PhD focuses in different fields. My main topics are recommenered systems, supervised learning with a focus on noisy labels and the environmental impact of these deep learning algorithms. So these topics are the main ones. >> And what first got you interested in recommener systems specifically? >> I work mostly on sequential recommendation systems. So finding some patterns and studying some ways in order to improve what is the state-ofthe-art in sequential recommendation systems like this field of research is well known since mostly 10 years but every day practically new architecture are coming and this is also the focus of my study. So studying possibilities to improve our architectures and way of also of evaluating these recommended systems. >> And could you share some details for listeners who don't know? I assume everyone has the sense of what a recommener system is. What is a sequential recommener system? >> So what a sequential recommendation system does is based on the previous interactions of a certain user, the goal is to predict the next interaction of the user. For example, if we have a streaming platform and the user watched Narcos and then Breaking Bad, the sequential recommendation system based on a chronologically reordered sequence of items that in this case are Narcos and Breaking Bad as the goal wants to predict the next item the user will go to interact with. For example, El Chapo. El Chapo could be a possible choice for the user to interact with. This is the goal of sequential recommendation system. So we have a sequence that is describing the interest of the users that can be for example TV series or podcast. And then what a sequential recommener system wants to do is to predict the next item the user will go to interact with. >> Well, let's assume there's like a green field application. A company has just started and they have this problem or an organization or website of some kind. How do they pick an appropriate architecture? >> There are different choices we have to do in this case. First of all, this is based on the amount of resources this company has because there are for example different approaches in the literature quite some of them are quite well known for being lightweight while others are more computationally expensive. For example, like the easiest strategy that the company is able to select is we have a huge collection of data and this describes a sort of table and then obviously some items are missing because not all the users interact with all the items. So this table like has a lot of missing values. One possibility to have is to define a a similarity between the users and then based on the on this similarity to predict the next item. For example, if we have some items in common between me and you watch the both Breaking Bad, then one possibility for our recommendation algorith could be to suggest to both of us uh like better cold soul. This is one simple strategy for sure is the easiest one. Then there are also some more difficult ones but this is the easiest. Obviously neural networks have become grown in popularity over the last I don't know decade maybe two decades at this point. Uh how has the emergence of neural networks played a role in the evolution of the popular architectures? >> Practically sequential recommendation system since seven or eight years is totally based on neural networks. One like approach that is central to sequential recommendation system is based on the attention mechanism. Why? Because like the attention mechanism can be easily move transformed and adapted to sequential recommendation system because it's able to define a mask. So what is a mask? A mask is a sequence of values on which we want to focus. So for example, if for a certain user we have a sequence and we want to focus on some items of of this sequence, we can use an attentionbased sequential recommendation system algorithm and select define a mask and for this sequence of the specific user, we can then focus only on the items that we want to like discuss and we want to study. And for this reason like the majority of the approaches for sequential recommended systems in the literature are based on the attention mechanism like for example the gastas or ber for wreck that are let's say variation of architectures that are let's say focuses on natural language processing but they can be easily adapted to recommendation simply because if we think natural language processing is a sequence of text so our sentences are sequence quences of words and what we have in recommendation systems instead is a sequences of numbers because what we can think is that each movie for example if we are in a streaming platform as an ID so our sequence of interactions is no more than a sequence of numbers so for this reason there is a strict parallelism between sequential recommendation and let's say all the approaches that are focusing on NLP >> well almost every recommener system paper I read that proposes some methodology benchmarks it and they're going for accuracy or you know some of these common metrics that uh you want to maximize but in your paper ecoaware graph neural networks for sustainable recommendations it seems maybe optimizing for accuracy or something like this isn't always the the only thing to consider what brought you to this realization >> the literature is full of papers that are let's say focused on maximizing some metrics which can be like retrieval ones or the accuracy or some other metrics that are common in the field of in data science in general or deep learning if we want to be more specific and there are some papers but still few of them that are instead studying what is the impact like the environmental impact of some recommendation systems in deep learning but literally there are few of them there is one seminar work from I think three or four or probably four years ago that was studying the the environmental impact of some transform based architecture. So still working in lateral language processing. But what I noticed instead is that like in the recommendation field there were like few papers that were studying what is the environmental impact of the recommendation stage. So the goal of this paper was to look to some well-known approaches for recommendation system that were focused on graphs and analyze their environmental impact. So the cost of training them and then using them for inference like if it is a company that is using a recommended systems that is based on graph but also analyzing what is the impact on of using this algorithm. I imagine every situation's a little different, but broadly speaking, do you have a sense of where the cost is? You'd mentioned it can be training and inference. Are they equal or what part of the pie does each get? >> Yeah, usually let's say the majority of the cost is given by the training because as you can imagine during training in general, machine learning model runs for a lot of epochs. It has to train. So it has to learn something that can be a strategy to recommend items to a certain user or other tasks. So for this reason training is usually more expensive than inference. And this is also the case for example when working with LLMs most of the cost of the LLMs derives from training them. So for this reason we were expecting that the training cost was higher than the inference one. But instead what we also were looking in our paper was like how each of the components is influencing this cost. For example what we did is that we use different architecture which are quite known in the literature. They are usually the most used ones. So each researcher is using them if it if it is in the field and like they are based on some parameters which are common and they are shared between all these architectures. And what we did was to test and to analyze how these parameters are affecting the environmental impact of all the training step of a certain architecture. >> I'm curious if you have any thoughts on maybe impact. Could we say that the cost of training is you know on par with taking a flight or something like this? Can we frame the you know the actual cost in terms listeners might understand? >> Let me do an example of this. I usually define this as impact. But there is a terminology which is more appropriate. There is CO2 equivalent. >> How can we measure this CO2 equivalent? This CO2 equivalent is mostly based on two components. The first one is based on the hardware we use. So the amount of energy we are consuming in order to train our model and this is like measure in amount of hour. It depends on different like parameters like amount of hours that we use for our training the GPU so the graphical processing unit that we are using and a lot of hardware characteristics and there is another parameter that is like depending on the country in which we are because like as we know each country uses different sources of energy they can be renewable ones or not and there are some information that define how each of the sources contribute to the final energy request. So what I define as environmental impact or let's say in a more accurate term CO2 equivalent is the product of the component let's say that is defined by the country. So what is the use of the renewable sources and not renewable ones that we are using times the hardware requests we are having. So which which kind of hardware we are using, how many hours we are using for train all the models and this kind of information. So our study is based on this kind of measure that fortunately now well known also from some other papers and from research and there are also some really really useful libraries that provide these tools in order to measure it. So they are able to detect automatically which region we are from our IP address and it already has the information on the amount of sources we are using and then they are also able to measure the training time as well the hardware we are using. So these libraries these frameworks they are super useful and I really encourage all the people to use them. >> Could you describe code carbon and how you used it in the work? Yeah, this is exactly the definition of code carbon because code carbon is one of these frameworks probably is the most common. It's not only the most common but it's also like the decent one to be used because you use like one line of code to start the experiment with literally defining a few parameters for this experiment. Then this tracker will start will automatically select the region where we are from our IP address and then it will measure the amount of time we will spend for our training and some other information related to our training like the hardware that we are using and then once we stop our training code carbon automatically stop also what we can define as a tracking. So what code carbon does is exactly this one. So tracking our experiments based on some information from our hardware and from our country. >> If I want to train a model locally on my computer, I can set all that up, kick it off to run. I can see that maybe my GPU goes to 100. I can feel it get warm. I know a lot of energy is being spent training this model. But how do I measure the consumption? >> Uh you can measure the consumption by using an hardware. uh your GPU is on a certain hardware and your hardware is like generating some energy is consuming some energy. So what code carbon does is measure the amount of energy that your laptop is consuming not only your laptop. This can also work on your process because you can also select to only track your process. For example, if like four processes and only on one of them you are performing some experiments and you want to track only this like this process this can be done and this will this process will be say highlighted and then you can check the CPU power the GPU power the hardware you are using and all this kind of information. It's always suggested to track the entire machine because the use of an another GPU for example on the same machine could obviously have an impact on your process. So if you want to target one single process, you can imagine that probably can be there can be an influence between one process and another one. Well, sounds like a useful tool then for gathering some data. Delete Me makes it easy, quick, and safe to remove your personal information online. At a time when surveillance and data breaches are common enough to make everyone vulnerable, your personal information isn't just floating around online. It's being bought and sold. Data brokers are compiling everything from your home address to your social security number, even details about your family members, and selling it to the highest bidder. This puts you at risk for identity theft, fishing attempts, and harassment. As someone with an active online presence, privacy is really important to me. I've seen firsthand how exposed we all are. And that's why I trust Delete Me to help protect my information by removing it from hundreds of data broker websites. Take control of your data and keep your private life private by signing up for Delete Me now at a special discount for our listeners. The only way to get that 20% off is to text data to 64,000. That's data d a ta to 6400 0. Message and data rates may apply. Isolated agents have limitations. Connected ones have possibilities. Agency AG ntcy is the open-source collective transforming how AI systems interact through their revolutionary collaboration layer. Their mission, build the internet of agents where discovery, connection, and cross- framework cooperation become standard. Developers can now leverage standardized protocols and modular components to compose and scale multi- aent workflows with unprecedented ease. Join Crew AI, Lang Chain, Lambda Index, Browser Base, Cisco, and dozens more who are already contributing to this movement. Build with other engineers who care about highquality multi-agent software. Visit agency ag ntcy.org. That's where you can add your support at agency.org. >> Can you talk about the experimental parameters you were exploring? >> We were exploring different parameters that are let's say the standard ones in gross neural networkbased recommendation systems. For example, the standard way is the embedding sides. So the embedding side because at a certain point when we have graph based approaches, we define a graph. But then in order to train a graph neural network, we have to embed this amount of knowledge because you can imagine this graph can start being super huge quite easily. Like already with probably 100 users, this graph can be super super big. So what we should do is to extract some of the most relevant information. How this is done? By embedding that this information in a lower dimensional space. So the embedding size is one of the information that is quite common and is super useful. Why? Because we can expect that usually the bigger the embedding size, the more we the information we are gathering up to a certain point obviously. But this also means that probably like we are focusing on a bigger graph. So what we were expecting in terms of environmental impact that let's say is also confirmed by the result the experimental results that we had is that the bigger the embedding size the higher the computational complexity and this means also the higher the environmental impact. But this is also related to the fact that the bigger the bedding size, the bigger the training time. And from different studies in in the research field, what they discovered is that the training of a network of a neural network in general is pretty colinear with a certain coefficient to the environmental impact. So what we can expect when training a model is that by increasing the training time then also the environmental impact the related environmental impact will be higher. So can you or would it be possible to come up with a little heruristic or some regression that could predict or are there cases when I don't know the training could suddenly be unexpectedly higher or lower cost. There is also another studying that I was doing to get in collaboration with some other colleagues from Sapiensa University of Rome that is doing the following. So assume you have a different like a pool of neural networks that you can use for a certain task. So for example a classification one you want to select uh like the best classifier for images of dogs or cats and then you can select between for example multiple models with different configurations like a learning rate a batch size and so on. But what you want to do is for example to find the model that is the one which is the most environmental friendly. So this study what does is that automatically like is giving method that is able to select the most environmentally friendly network and configuration of parameters based on the task you have. So what this work does is that you only specify the task you want to do and the data set and then like our predictor our approach returns as output the neural network that is the best one for your task obviously you can always select. So for example, if you have this task and you want an I accuracy because you have strict requirements, you can like put to us an importance of the accuracy of 99%. So you literally want to focus on the accuracy except for the this 1% that is like the environmental friendness and then you can also use our still our predictor. So this predictor is still working but by default our our approach what we'll output is like the most environmentally friendly one. >> Well I think the most environmentally friendly recommener would be the worst recommener right just do no training and then flip a coin and uh output the results. >> Yeah we tried also this approach recommendation systems. We tried this like this approach on three different tasks that are recommendation system, computer vision and natural language processing. But like the results were pretty impressive because the best recommener in terms of environmental friendness is not always like the worst one. Why is not always the worst one? Because for example, you can use two architectures and one takes like three hours to train and one takes one hour to train. Assume also that the one that takes three hours has three times the number of parame hidden parameters than the network that only takes an hour but the accuracy is still pretty comparable or for example the ranking magic we want to we want to analyze is pretty comparable then at this point what we will do is to use the architecture that is like taking one hour and there's much less parameters but then the performance can be comparable. So this means that in this case like probably there could be a small gap in terms of relevance metric or in general recommendation metrics but like we were saving a lot of energy. >> I believe you tested a number of different approaches like GCN being one of the best performers. Could you talk a little bit about the scoreboard? What did you test and uh what did well on average? >> The scoreboard is exactly based on CO2 equivalent in kilograms. So is the amount of CO2 that we are injecting in our environment let's say in poor words and this amount can be like weighted and so for this reason we are using kilowatts as measure it was a bit expected that light GCN was the best performing one in some situations because the name of this network is light GCN. So light means something practically what this network does is to aggregate some information but like literally with few parameters. So there are few hidden parameters and each iteration of the training takes few seconds to train because the parameters that are shared between one iteration and the next one are literally a few of them. So for this reason this is the let's say the the most lightweight and what we were expecting is that this was the best one. For example, we also tested some more recent approaches in this paper but they like contrasty learning based ones and in this case we were expecting these approaches to be a bit more computationally expensive. But this is also this is also depending on the task we are defining. Obviously when performing for example an approach that is like the contrastive learning one you are you can expect that this network will be a bit computationally more expensive because like each iteration you have to do some like computations probably the graph also the computational graph will be much bigger and this is kind of kind of expected. So the results with like GCN uh were interesting but what is missing now is a small step but quite important is like a way for an architecture to detect when it is possible like to for example stop the training. So for a recommended system at a certain point you can like find a way in order to stop this training because like this network probably reached already some good performance or for some other like reasons we we can define and then we stop our training so that we are saving some resources. We are also saving some time because a lot of time this is also matters of time because probably if we are a company we don't have two years for training our architecture and usually companies have like data collection that are like of terabytes of memory. So saving go two or three days of training is not only saving a lot of energy. So being environmentally friendly but it's also saving some time and money. So early halting is an interesting way maybe an algorithm could be improved to be more eco-friendly. What about something like a regularization technique where you mentioned something like this earlier? We want to get an accuracy of at least 90% but a secondary concern is to be eco-friendly. >> Yeah, this is a good approach for sure and this is also what we also did in part with the other work. you assume like you have two target metrics let's say because one is the recommendation one and the other one is based on the CO2 equivalent and then what we want to do is either find a tradeoff or like focus more on the accuracy or focus a bit more on the CO2 equivalent but there are also some approaches which are pretty common in the literature that are named early exit. So what you do is that based on usually some math or some theory of graphs, what you do is that you you start your training, you set a certain number of epochs, but you define an early exit. Well, I know light GCN that I'd mentioned earlier performed very well from an eco perspective. Is there a tradeoff though? Would I uh if I were just to adopt that because I need to pick one and move along, am I giving up on accuracy in a lot of the data sets you experimented with? Yeah, these are only a few of the huge amount of graph neural network based approaches for recommendation. What we try to insert in in our contribution was also a way to provide some suggestions to the user because then you read the paper and if there is no suggestion, yeah, it is a research. So, it's something that for sure could be helpful on the data sets that we were testing. We try different architectures and from our let's say point of view the suggestion we provide is to use this kind of architecture. For example, what we did in this paper is that in most of the data sets you can use let's say a simple but also very important approach like GCN and achieve a good trade-off. Obviously, it is possible also that there are some specific configurations where using more complex architectures like SIMG GCL that was introduced also in our paper by setting some configuration can perform in a better way. But if you want to follow our suggestion and not only focus on the metrics that are for recommendation but also being environmentally friendly, these are our guidelines. When you look across data sets, do you see a consistent pattern? Does method A always have less consumption than method B reliably or are there some data sets like they say no no free lunch theorem there just going to be one where an algorithm does exceptionally well or anything like that? How much variance is there? >> We were expecting something better from a certain point of view. So we were expecting like when we read a recommendation data set paper there are at least three data sets because you try to benchmark your approach on data sets that can be different between them. For example they can vary a lot between the number of users, the number of items and also which is what is quite important is the sparity of the data set. What can we define the sparity? Imagine we have a sequence of sequences because each user has a sequence of items but this is repeated for all the users. So what we have is a sequence of sequences of items. If for each of the user like each of the user was looking only to few items this data set probably would be more sparse otherwise it will be more dense. So for our paper what we notic it is that the results were let's say pretty consistent across all the data sets and this is something that probably can be also investigated with some other data sets see to only to check if this consistency still is able to remain but yeah let's say that for our paper the results were remaining pretty consistent for example the movie lens data set is a data set which is like consisting of reviews of films and this is quite dense in the sense that each of the user has a certain number of interactions so of review which is pretty high while instead the Amazon one the Amazon beauty one which is still super super common in the in the literature um has a number of interactions between the users in their atoms that is a bit lower but let's say the insights that we can derive from both the data sets are pretty the same So this is something unexpected but that we like it. >> Any surprises in there? >> Yeah, there were some surprises that we were also brought in the paper. For example, what we discovered is that there were some situation in which by increasing the value of the embedding size, the network was performing better from an environmental point of view. From what I was saying before, this is something totally unexpected. So for example, if we move to an embedding size from 16 of 32, what we expect is that the number of parameters will be a bit higher and this means that probably the impact that our nectar will have will be also higher. But for example, what we notice it is that there is one situation if I remember correctly in the Amazon beauty data set in the NGCF approach where we move from an embedding size of 32 to another one of 64 and the environmental impact of the training with an embedding size of 64 is smaller than the one with the size of 32. >> And this is pretty strange and this also deserves like a more careful investigation. We were trying to investigate a bit, but still there are no clear answers to this question. >> So what do you think a practitioner should do then when trying to pick an embedding size? It seems like, you know, in the most part it's going to be linear, right? If I double it, I'm going to maybe not double my cost, but have some collinear growth in cost. Uh but maybe not in in every case. How do you approach it? probably I'm slightly modifying your question in order to answer to you why because what I think that a practitioner should do is to for sure find the embedding sides that best fits to its approach but it should be also like take into account that finding the best configuration of an architecture usually takes a lot of time for hyperparameter tuning and this is something we want to avoid in the sense for sure hyperparameter tuning changes the final results because changing the value of the parameters could be for example to overfeitting or other problems that are well well known in the machine learning literature but increasing the number of experiments in the hyperparameter tuning is wasting a lot of resources. So what is my suggestion that we have also from our paper is to still keep using that like some parameters that fit for the approach. So that are giving results that are promising but in the same time trying to reduce the amount of experiments if they are not needed. So this is like our suggestion. Could it be that some of the cost is because the particular code implementations weren't well optimized? You know, in the early days of software, you want to make it bug-free. Maybe later you can make worry about it being efficient. Could the scoreboard change rapidly if a very clever developer got under the hood? Yeah, this could for sure change, but this is also the reason why we try to be as close to the reality as possible. We try to use one of the most well-known libraries that are like targeted for recommendation system. There are different of them in the literature but this is one like one of the most famous ones and this is like supported by different researchers in the community is like there are a lot of contributions daily on GitHub and this for sure could impact in the sense that there could be implementations that are not well optimized but I think that this also depends on for example if you use some frameworks like PyTorch or TensorFlow depending on the use There always could be like some modules that are not well optimized. This usually does not happen because like PyTorch, TensorFlow are from giant companies and there are a lot of developers that are probably the best ones in doing this. But this for sure can influence our results. Also, if you think for example of using an neural network in general without using the early stopping, you can probably train your model for 5,000 epochs, but they are not necessary. So, for sure you will consume a lot of energy and a lot of resources in general, but probably what could happen is that at a certain point you start overfeit because probably 5,000 debugs are not necessary. They probably 2,000 were enough to train your architecture. So not only you are wasting a lot of energies but you're also introducing some errors in your architecture. >> So I think this topic of being eco-aware of machine learning in lots of context is good specifically we're going to talk about what not niche but what subset within machine learning of eco-friendly algorithms. Yeah, there are different topics. Let's say for example, we were discussing before about sequential based recommendation systems. But another way of approaching the problem of recommendation systems is by using graph neural networks. Graph neural networks are simply based on a structure which is a graph based one. So you have some nodes and some edges and you can think to a graph like the Facebook network. So I'm friend of you and then you are friend of some other people and we are super easily generating a graph and this can be also easily moved and easily also modified in order to account to the recommendation problem because like I am a user or uh and you are another user and we have some items in common or and this can easily generate a graph. So you can define for example a bipartite graph where in one side of the graph you have all the users another side of the graph you have all the items and then you connect users and items. This is generally a bipartite graph because from one side of the graph there is all the users on the other one there are all the items and then there are the connections between between them and one way of approaching to the recommendation problem is by using this graph and using some graph neural networkbased approaches. In the machine learning community, a lot of focus goes into common benchmark data sets and a scoreboard around them, usually focused on a metric like F1 score or accuracy. Would you like to see that process evolve in the future to include something eco-friendly? >> Absolutely. I think that is quite important and also in one of our contributions what we did is also to provide a new data set. This data set is composed by uh different nean network based approaches for different tasks like as I was saying before computer vision natural language processing with different configurations for the networks and so different type of parameters and the results but together with the results there are also results in terms of environmental impact so in terms of CO2 equivalent I think we this data set is public and we released did to the community because I think this could be pretty pretty useful and I think that this can be what we are trying to do is also to generate a framework that will automatically augment this data set. Why? Because if you are a practitioner and you are testing some data using some models, what I think it could be useful is to use a framework that automatically save your experiment without like stressing you, taking your time in order to use some libraries, downloading them that could take a lot of time and then save these results like for example in a centralized platform like a drive or something like this and then extend this data set. So this is also one of our contributions and the aim is data set soon as possible as big as possible. >> Well, in technology there's often a focus on growth over efficiency especially if you're making a lot of money. Do you have a sense of where industry is at on the adoption scale here? Are they embracing these ideas and considering them? Yeah, I think that since a couple of years this topic is pretty like pretty well known bodies in the research community but the research community a lot of times is communicated and this is strict in contact with also the industry community. So there is a huge interconnection between them. For example, there are already some companies which has some sectors that are based on optimizing the resources or trying also some ways in order to reduce the impact of their training or also or of their evaluation because we can imagine that there are some big companies and if you evaluate their model and you use in inference their model like thousand or millions of times daily also this has a computational cost that is easily translated ated into some environmental impact. So I think there is still a lot of research to do but this problem now is starting also like being quite common well speaking into when speaking into conferences. So usually there are some tracks that are based on for example green topics and how to adapt like the methods that we have in order to be as green as possible. So I think that this topic still need a bit more attention but like so far the effort that we did let's say started a clear path a clear road that then will be explored >> in terms of the impact or even maybe the urgency I don't know that machine learning is the industry we have to wag our finger at maybe it is but what is yeah I guess the impact and urgency you think with which machine learning practitioners should be considering this maybe on a daily basis >> I think that the machine learning community is increasing also because of this boom of AI. We can practically hear about AI in each TV show or we can read it in the newspapers and so on. So I think that now what the community has to do is not like all the people studying this problem like not focusing the effort and the knowledge of all the people in order to derive some algorithm that are efficient but like following simple suggestions for example trying to avoid experiments that are useless. First of all, they're running some mathematical conditions. For example, assume you have an experiment you want to run and you drive before a mathematical condition that is labeled to say to us, okay, this experiment is totally useless because you will not have results that are scientifically sound before you start your training. This is a an easy way first of all to check if your algorithm is good or not. That is something we should do independently of the environmental impact. and then you will also save some resources. So what I think we should do that so far is not that common is to try to think a bit more before starting our experiments because it's always easy to run an experiment. You define a list of values and use whip between them. But does it make sense to use all of them? This is the question that we should have like we should pose. So does it make sense? Can we like avoid testing some of them because they are not useful? And if this is done by a lot of researchers researchers daily, we can save a lot of resources and also do some experiments which are more useful. So not wasting time in experiments that are also useless. >> And what's next for you? >> We are so far trying to build up on these two papers. the one on graph neural networks for recommendation and also the other one that I was describing before that is finding a way in order to derive the most green architecture and we are trying to build on both of these papers in order to first of all finding a way in order to automatically increase this knowledge base so that also the prediction of this strategy will be always more accurate and then also So trying to see how these approaches generalize to other tasks. For example, so far we were testing on recommendation approaches as well as computer vision. But is there a way in order to study this problem without being task specific? So for example, you provide some a strategy that works independently of the task you are doing. For example, if you want to do retrieval element generation, if you want to do semantic segmentation, you can still keep in using this strategy to save some energy and you you don't need to have information about the task. This is my next step, but I think it will take a bit of time. But yeah, this is something we I'm studying together also with some colleagues from the university. >> And is there anywhere listeners can follow you online? Yeah, they can follow my ex account and I also have a website where I usually put uh my publication as well as the code related to this publication. So I also invite you to try to find bugs if there are some of them so that I can prove my code and the code is simply uh Google sites with my name and surname. So Antonio Brificato at Google sites. If you find some bugs, I'm available if you want to reach me out in order to discuss about some problems of the code as well as new opportunities, new ideas or ways to improve my code base. Sounds good. We'll have links in the show notes for listeners who want to follow up. Antonio, thank you so much for taking the time to come on and share your work. >> Thanks to you, Kai. It was a pleasure.