Interpretable Real Estate Recommendations

Key Takeaways

[Music] Welcome to Data Skeptic, a podcast exploring the methods, use cases, and consequences of recommener systems. Welcome to another episode of Data Skeptic Recommener Systems. Today on the podcast, our focus is the paper Z-rex, human interpretable graph neural network explanations for real estate recommendations. This obviously clicked on a number of my keyword searches of the archive. I can't get away from graphs and networks. I guess you know Kunel had an interesting observation I hadn't have thought of that as CO occurred and houses were bought and sold and now in a maybe I guess we call it a postcoid era the real estate landscape has changed. Certain markets have emerged some have evolved in other ways and people who are searching for properties and want to get good recommendations that match their genuine interest in those properties so they don't have to sort through as much. Well, those users would almost certainly prefer more interpretable explanations. Telling someone, "We gave this property score a 9.72 for you is not as informative as, hey, we verified it had these three key features, and although it's a little bit outside of your price range, you get X, Y, and Z to make up for it." Or something along those lines. We talk about the use of graph neural networks, a couple different frameworks and approaches, and even some good industry insights on how you roll these things out. All that and more coming up right now. My name is Kunal Mukharji. Currently I am a postdoctoral research associate at Virginia Tech. So for my post-doctoral studies I am currently working on adversarial manipulation of intrusion detection system focused on graph detection system because the core domain that I work primarily is called provenence domain. All our secure computers that we have have logs and these logs capture the system provenence. So how the data and the access control flows through the system as a user interact with different processes files. So once we get this logs how do you work on it? Back in 2018 2019 people moved away from using the logs as a textbased data source and converted it into graphs. When I joined my PhD in 2019 that transition was made and there were some successful papers. Then the work became now that we have graphs can we learn better intrusion detection models and now we are at the stage we have many different intrusion detection systems but we are not confident that will they actually survive a zero day attack. My PhD work would be on creating this kind of adversarial attacks for graph based in detection systems. And the last thirst is explanability. Can we explain the detection to a security analyst? Because a researcher versus a actual security research like a security analyst in a company like let's say not or casper sky they are not really interested in saying oh um this is malicious or this is benign but they're more interested that okay when I say it is malicious based on what artifact or what evidence am I making this detection for graph explanation there was uh like there have been some explanation methods like gn explainer EG explainers but they are kind of targeted towards your how do I say vanilla graph neural networks like there have been works saying that oh you can extrapolate or you can expand upon this theoretical concept but there wasn't much work so in my post-doal studies I want to create explanation methods for this graph detection system that is focused towards the cyber security domain >> I think I got to have you back in the future to talk about intrusion detection that's is a topic I'm fascinated with and I feel like we're still in the early days of deployments. But today, yeah, let's get into the paper I invited you primarily on for which is titled Z-Rex human interpretable GNN explanations for real estate recommendations. To kick off, can you just give me some background on how you got involved in the project? >> Once the COVID ended, right, like 2020 2022, there was this technological boom, right? Like people were getting hired, but the campuses were not on-site anymore. They were virtual. So people had this freedom to go and stay to stay near their family or move towards places where they want to live. Right? A lot of people started moving from places where you have a state tax like Illinois towards Texas where you don't have state tax. You have Dallas which was like Chicago. Now it started expanding. So it started expanding. There were new regions that started popping up but people were not aware of it. So people who are coming from let's say California and they don't have ground intelligence like let's say family members or friends they do not know about this new regions that are out there. So what do they do? Usually either you go by recommendation by mouth or you go to a place and you search right but if you don't know the name of the location how do you search houses near that in Dallas there is a place called like frisco prosper that have suddenly boomed in the last 2 three years and it has boomed so much that the city had to um declared an ordinance that you cannot build more houses because the water level is decreasing. Oh my. >> Yeah. So, so it is boobing like crazy. But, but the thing is people from other places are coming, but they're only looking at houses near the like let's say a radius of 50 mi from Dallas, Prosper, Frisco, they do not fall in that radius. So, the old recommendation models were kind of skipping through it. So, they were just showing this houses. During the whiteboarding session, a question came, can we do some kind of a novelty or diversity based search so that when a user is looking for houses, we will recommended new regions to look at. And I give you a similar substitute of houses like same bedroom, bathroom, but a little bit farther down the road, people are usually okay driving that extra 20 m if they get a better suited home for a less price. This was the kickoff of the project that can we um recommend new regions. This project was supposed to be a three uh 3 month long or 12 week long project which I was able to deliver in half the time or 3/4 of the time. So we had like one month in hand to try something right to try something new. So I kind of proposed that to my manager like um and they were really kind of I would say supportive and really kind like encouraging to try new things. So I said we can go two ways. one I showed you a graph based recommendation for similar region but I think as a customer if I'm let's say coming from a different region and you give me a house in let's say frisk or prosper and I don't have the ground intelligence I might be suspicious so how do you stop the suspicion because you would see an Amazon if you search for let's say a product and Amazon gives you some other unrelated products you are not really like you don't see it just like breeze through them. >> And over time you get this uh like a behavior where you're going to breeze through them naturally. So user span is very limited and getting the confidence of the user is more important in the beginning because if you lose it once the user might just considered is useless even after like it becomes relevant. So, so I said to my manager that can we create some kind of an explanation system so that we can give the user an explanation why we recommend it something like x number of bedrooms bathroom so the user might be like oh okay so this houses have this much bedroom bathroom but I am getting in a smaller price or let's say better school district so something like that so that's why this project started the paper that you are seeing now that can we create some kind of an human interpretable explanation ations for the region recommendations that we have we are doing. >> So then does the Z-Rex project would you say it's strictly an interpretability effort or are there other aspects to the work? >> It was an interpretability effort but it was focused towards humans because I want to draw a distinction that one of my papers in the providence domain is on similar kind of human interpretable explanations. Okay. So the distinction I'm drawing here is there are two kinds of people that work in this domain or any domain per se. One who are creating the base models such as the recommendation model or the intrusion detection model. They are the model developers. For the model developers an explanation might be different. So model developer might uh might be kind of concerned with which let's say neurons are getting affected. The second category of users which have become prevalent in this day and age are the analyst or the consultants. So an analyst or a consultant does not really care about how the model is created or what's happening inside the model. For them model is a black box. Now for them it is more important to say that oh my blacksbox saw this let's say a graph pattern and this graph pattern has is observed in other recommendation places like for in detection places you can say oh this graph pattern is associated with a malicious graph porting that idea to recommendation system a similar idea happens that where you are creating this kind of human interpretable explanation explanations. So these explanations are different than what the explanation model developer wants. Circling back to your answer, yes, it was an interpretable effort, but our end users were analysts. [Music] Delete Me makes it easy, quick, and safe to remove your personal data online. At a time when surveillance and data breaches are common enough to make everyone vulnerable, your private details have become big business. Data brokers make a profit by selling your personal information to anyone willing to pay. Think about it. Your home address, contact information, and other sensitive details are all up for grabs. This exposure increases your risk of fishing attempts and other security concerns. As someone with an active online presence, privacy is really important to me. I've been using Delete Me for over a year now. And knowing that they're continuously working to remove my information from hundreds of data broker websites gives me confidence that I've taken this proactive step to protect my privacy. Take control of your data and keep your private life private by signing up for Delete Me now at a special discount for our listeners. Today, get 20% off your Delete Me plan by texting data to 64,000. That's D A TA to 64000000. Message and data rates may apply. The future of AI isn't about single agents working alone. It's about networks of specialized agents working together seamlessly. But there's a problem. Building multi-agent systems today is like trying to connect devices before the internet existed. No common language, no shared protocols, just isolated systems that can't talk to each other. That's where Agency comes in. That's a gny. Agency is the open-source foundation for the internet of agents, providing the essential infrastructure that lets AI agents discover each other, verify identity, communicate securely, and collaborate across any platform. Imagine specialized agents that can find each other, understand capabilities, and work together to solve complex problems regardless of who built them or where they're deployed. This isn't just a vision. It's happening now with industry leaders like Crew AI, Lang Chain, Lambda Index, and Cisco already building on agency's open standard. Ready to build a future of multi-agent systems? Join the collective that's defining how agents will collaborate in our AI powered future. Visit agency.org. org today. That's a g nnty.org. [Music] So along the way there's a lot of approaches one could take. Uh the a priority algorithm comes to mind or using like an xg boostbased approach to deep learning approaches. What are the trends? Where are people going in the methodologies they use to do recommener systems? The main um how I say a surprise that I had from coming from academic to industry is industry the preference towards a model is based on external factors rather than the internal capacity of the model. neural network might be a might be best suited for recommendation but the latency of getting a prediction or the amount of data you need to keep the model updated is colossal compared to your as you mentioned XG boost model or let's say histograms. So in industry people tend to stay with simpler models uh more frequently rather than using some kind of a complicated model. So that's why like histogram, XG boost model, CAT boost models are used more prevalently and all the kind of the recommendation answers that I'm giving you is based on that exposure I had for the three months and what I have learned by just googling myself or looking things up. >> Could you expand on what uh how the interpretability is developed? So like or or maybe what as a user would I see? So I um I'm getting some predictions and then what can you add on top of that that helps better inform me >> when you have a recommendation data right so in recommendation data is usually user item the interaction between user and item and the associated features of user item the interactions taking this idea putting it into the real estate recommendation domain in real estate recommendation there are two kind of items items. First kind of items are cities. So cities where you have like Poria, Neapville, like Springfield, Dallas. Each city has multiple neighborhoods and then this neighborhoods have listing but these neighborhoods are kind of a sorry multiple clusters of neighborhood are associated with one city. So we can kind of remove that. So let's say a city has a listing. Yeah. Real estate recommendation data have three different types. So users, listings and cities. Between users listing and cities, two kinds of items, users and cities have different interactions with users. A user usually goes and looks at the listing. Now the listings belongs to a city. There is this kind of a tripartite graph formation that happens and this is distinct from other recommendation where you have a bipartite graph such as let's say user and item. Here we have tripartite graph and this has a different uh kind of implication for the modeling task. This zerx is specific for a graph neural network based recommendation. And the reasoning is because when we were developing this um explanation framework, we saw that there was a paper from the Amazon research labs called page link that worked on these problems are called like link prediction problems where you're trying to predict links between two different nodes and they try to create this explanation. We try to kind of expand that idea and kind of say that what if there were different number of attributes. Next is the types of interactions are different between users and city. Let's say you have one kind of listing but between a user and listings you have four different kinds like you can view a listing you can save a listing you can favorite a listing and finally you can tour a listing and as I mentioned this four different types they are in a decreasing orders. A listing can have multiple views but not all of those views convert to let's say saves. Not all the saves convert. Now when you are trying to recommend a user another kind of let's say region you want to recommend the user a region where it contains different kinds of listings that the user will have preference in looking at. There are two aspects like two that comes in explanation right. one how they relate to the user's preference and second what are the values of the features that are there. Okay, these were the two kind of concerning dimensions that we had to look at when developing this explanation task. >> So at face value machine learning people will look at like accuracy or recall or F1 score or things like that. Of course interpretability is going to be more human readable. Could you give an example of what one interpretability example might be? Like if I got some recommendations, what can you append to that? >> In explanability, there are usually four metrics that one look at like the one of the primary metric that they look at is fidelity. So fidelity means there are two kinds. Fidelity plus and fidelity minus. So fidelity plus says when we remove an important subgraph, the prediction should change. Fidelity minus says if we remove something unimportant from the graph your prediction should not change. That is fidelity plus fidelity minus. Well, could you walk me through what happens in the presence of an irrelevant feature like maybe uh you know for some reason Zillow knows if you have yellow doorork knobs or not. Doorork knobs easy to replace shouldn't affect the sale of the house really you know but maybe we gave it to the model anyways. what would go on when we go to interpret a model with the presence of a feature like that >> for Zillow like there are more than I think 40 50 features that are out there and I'm not counting the language based features such as descriptions because for this work description features were not used I guess I can say that so when you have irrelevant feature even before let's say we come into recommendation we have this with the recommendation stage like even like the base model so in the base model. First we try to actually when we get this feature we go through the basically the normalization process and looking into outlier values and we kind of do some kind of a feature selection there as well to see that what features are relevant for the recommendation task that has a practical side to it. So the practical side is if you have let's say 30 features that means that you need that much amount of memory right to train a neural network or whatnot. turning down the features you kind of kind of make sure your model does not overfit like your basic classical machine learning advantages you are ensuring that your model stays up to date you have better regularization etc. Now we come into let's say the interaction stage. So in the interaction stage u for the feature pertubation we actually let's say zero out the feature and we see that does this change the uh what's called the represent like the node representation right when we zero out a feature and we see how much change this is tolerant to that's why like we kind of use a thresholding value. So, we've talked about attribute perturbation. I think that's insightful. Like taking out the doororknob's color probably wouldn't change your similarity, but taking out the bedroom count is going to have a big impact on what's similar. Can we also look at or what can we look at in the graph itself? >> There are a concept of co-click cities. So this co-click city is where you as an user clicked on a city where like users with similar preference also clicked on that city. So we basically create a subgraph of all the clicked cities based on similar users that have clicked on your city. >> Those are my peers then essentially right people like me. >> Exactly. So this does two things. One, it decreases the subgraph search space of important subgraph search space. Second, it creates a kind of a similarity subgraph uh kind of a network where something in the network is important to you. So this kind of is better than your random pertubations because in random pertubations one you're randomly pertubing and two let's say you pertube one edge in the graph. Okay. So you were done in a big O of E time. Now you put up two edges in total. Now you are in E² time. Right? Now three edges, four edges. So it it becomes a factorial like it it is not even now in a big O of N square. It becomes basically like exponential time, right? Like a network where you are doing this click cities. You are making the problem tractable making it search smaller and also you are creating it creating it in a datadriven way like suddenly we won't choose a part of the graph that is let's say clicked by some user which did not click to you like who is not your peer this is what the structural pertubation does and this is a tested thing in the real estate domain so colexities is important now okay so answering your question back so that's why the structural pertubation happen like is there so that we can effectively say let's say this part of the graph is important to you and in this part of the graphs these are the selected users and the selected users looked at this kind of houses which have similar kind of features as the important features that you are looking at and based on that we will give you your um recommend uh explanation for the recommendation. Well, I know this is really uh probably best described as an the R&D phase of the project that you participated in. Maybe in some time we'll see the fruits of this labor in the Zillow product line in some way. With that in mind, uh would you be comfortable speculating what might we see or just share some of those top important features so people get a sense of what the system's able to figure out? So as you can see on page number seven right um on uh figure six I tried to kind of show what are the some of the important um features that were listed um based on the NDCG metric. So NDCG metric is a metric that not only cares about the item that's recommended but also the position of the item where it was recommended. Would you mind defining NDCG before we dig in? >> So NDCG full form is normalized discounted cumulative gain. So NDCG basically says that when you are giving some kind of a recommendation, the position of the recommendation matters. So a relevant item down the list will be given a penalty. Now an irrelevant out item up the list will again be penalized. So it kind of ensures that you are giving relevant relevant items but also at the beginning of the list. So that's why you have NDCG at K. So K is the length of the list. So here one thing we need to keep in mind is we are looking at region recommendation. So when we say region recommendation like these are for cities not for listings. So that's why you don't see bedrooms, bathrooms but rather you see latitude, longitude, year built, is it on a vacant land um but square feet are being important. Now for listings what I have seen right. So for listings importance it becomes bedrooms, bathrooms is it uh a pits are allowed or not does it contain let's say c um cfacing view or not. Um another thing to note is that this data work that we data that we worked on was from Seattle. So in Seattle what I understood was that don't like lot of houses don't have centralized heating. So and most of them don't have centralized cooling. So that's why those features while important for let's say Chicago or Dallas like where centralized cooling is very important like it gets very hot down here. >> Oh yeah >> those might not show up in the listings there. What the similar people look at kind of impacts the recommendation that we are kind of giving. So that's why in the Seattle area people care about what's is on a vacant land or not. Does it have um cardboard port heating or not? And then that there are some unimportant features. So in the unimportant features you can see there is does it have cooling or not as I mentioned because the houses do not have you know heating cooling because unimportant features similar as fireplace. Okay. And then um bedrooms um pool like those are kind of a region specific like Seattle specific features that are unimportant but they become very important for places like uh Chicago, California where let's say cooling or having a pool is kind of a good thing or or even let's say fireplace. >> Sure. >> So so you can see how the features kind of are important by the regions but yeah these are kind of the features. Yeah, in some city whether or not you have a deed parking spot is quite relevant, but in other places there's ample parking and people don't care so much. Very I guess geographically insensitive in that way or robustly geographic perhaps is a better way to put it. >> Yeah. And I think that is a characteristics of the real estate domain again because real estate of different places go like boom differently and as mentioned like I mentioned Dallas as a place where different things are growing but there are other places where let's say Fort Worth or let's say some Phoenix Arizona like let's say the very pre-established locations new loc like new cities are not forming but in places like let's say Dallas California Austin um new regions are opening up but this new regions need to be evaluated or like recommended. When I explain the typeight nature of the graph like the interaction graph that is the reason we use graph neural network because in the graph neural network um based on my previous experience in the anomaly detection domain we saw that not only the features impact the recommendation but also the structure of the graph. So we are able to kind of get best of both worlds. We are getting the attributes, we are getting the structures, you mix it, your interactions are better, your recommendations are better. And second, discoverability of new regions are also better because graph network are able to discover new kind of pathways to this new regions that your traditional um xg boost or even decision tree or any kind of cat boost model will not be able to do. That is some that's a power of graph neural network. >> Well, I know this project was part of an internship, so there's a cap on exactly how much you can do. Um, if you had infinite time, where where else might you take the effort >> for this paper? If you go to like let's say page number 10, you will see I explain the ZGNN architecture. It's basically a graph neural network but the net is a simple convolutional network like like your what you call a GCN. If I had infinite amount of time, I would like to um kind of explore different kinds of network such as GAT or even uh transformer best network because I think we need to first make sure we exhaust the capability of the network so that we are getting good like the best recommendation that there are and once we have done that then I would like to circle back into the what's called feature pertubation and structural pertubation Here structural pertubation is kind of datadriven to the most part but the feature pertubation I think we can do a better job at it in the sense some features like we can do some kind of thresholding value to select for which fe which kind of users which features become important. I think that is something um I would like to look at whether I think that would make this um effort better and something that I kind of got through my experience as a PhD candidate and what not. We need to go back to the data. So here um when we go back to the data in the interactions we have different kinds of interactions but we consider all the interactions in the similar lens when we are learning it in the form of graph neural network. I think some kind of a weighted system can help us distinguish between let's say touring and searched and that can later help or define what is let's say a hard negative or soft negative when we are doing this modeling task because using the hard and soft negative help us kind of make a precision recommendation for be sensitive to users which they are. So yeah, these are the some kind of directions I would definitely push it in and sorry and I think this should have been my first answer is do a human verification. I think that is very critical and also um that experimentation should be designed correctly like how do you like this case study or human like human explan sorry human verification of the explanation is by itself a project that is kind of very interesting and intriguing. >> What's next for you? So like currently as mentioned that I am uh currently going like working as a post-doal associate but I am also like I am looking at what's called research positions in the industry so that I can make a transition and also I want to stay in this domain of basically explanations and human centric machine learning because I think this is an area that I see as genai boom we are kind of relying a lot on generative AI And how do I say um blackbox plus machine learning techniques like we need to make a conscious effort where we work and we bring in the human in the loop back not as a human who to do leveling or um reward function but actually a human that can say when a machine is going out of bound or basically creating these guardrails. >> And is there anywhere listeners can follow you online? >> Yeah, absolutely. I actually am very active in LinkedIn or Twitter. So LinkedIn it's I think Kunal Mukharji I don't know like there are some kind of codes >> we'll have in the show notes. Yeah >> X it is Kunmuk K U N M U KH and uh I keep my website updated as well. So kunmuk.com and uh please feel free to anyone to reach out with potential opportunity or collaboration. >> We'll have links to all the above in the show notes for listeners to follow up. Quinnal, thank you so much for taking the time to come on and share your work. >> Yeah, thank you so much. It was a really, I would say, nice uh interview and I really enjoyed it. Same here. Good talking with you.