Offensive vs Defensive Data Science with Deep Varma - #25

Key Takeaways

[Music] hello and welcome to another episode of twiml talk the podcast where I interview interesting people doing interesting things in machine learning and artificial intelligence I'm your host Sam charington last week was a big week for the podcast I announced the first anniversary of the show this week I want to start the show by thanking everyone who's participated in our first anniversary contest we asked you to comment on the show notes page or post an iTunes review and wow did you deliver your stories have been personal thoughtful and downright encouraging I've got a couple that I'd like to share and really it's so hard just to pick a couple of these but first Andrew posted on the show notes page I've used this podcast to maintain a pulse on current ML and AI big thank you for both helping me in my day-to-day but also getting me interested in ml in general when I first started listening heck I've been listening to this podcast when I was a student then when I was an intern and then when I was an entry-level analyst and then when I was promoted to analyst and now as a senior analyst where I am now is in no small part thanks to this podcast wow that's a ton of ground to have covered in just a year Andrew Cong congrats we are so proud to have been a small part of your success this next one has a bit of a backstory Bill b123 posted a five-star review on iTunes titled sold on the Deep interview format Bill said new interview style format was not initially good I gave it two stars initially later raised it to four stars now I'm sold great podcast with tons of insights and learning keep up the great work Sam now I remember when Bill's first review hit the decision to switch to the interview format was really tough for me but I knew it was necessary for my efforts to be sustainable Bill's initial twar review really really hurt I think at the time him or another user said the interviews were just a bit too fluffy and that I was like the Tim Ferris of AI I wasn't really sure what to do with that cuz I kind of like Tim Ferris is podcast but I really took that to heart and I was pumped when Bill raised his review to four stars and now I'm super excited to see that I've earned Bill's fifth star thanks so much Bill starting from nothing I never imagined this podcast would begin to blossom into such an awesome community of users I say begin because we still have so much ground to cover and we are truly just getting started for those who have not yet had a chance to enter the contest please visit visit twiml ai.com birthday for more info don't forget first prize gets a bronze pass to the oilly AI conference this June which is an $1,800 value second prize gets a Google home powered by AI of course and everyone that participates gets a couple of twiml laptop stickers the contest ends June 1st and winners will be announced on the 2nd if you've posted a review on iTunes to enter the contest please reach out to us at Team twiml a.com to let us know who you are all right this week on the show my guest is deep Vara vice president of data engineering at real estate startup Trulia deep has run data engineering teams in Silicon Valley for well over a decade and he's now responsible for the engineering efforts supporting truly as Big Data technology platform which encompasses everything from data acquisition and management to data science and algorithms in the show we discuss all of that with an emphasis on trulia's data engineering Pipeline and their personalization platform as well as how they use computer vision deep learning and natural language generation to deliver their product along the way deep offers great insights into what he calls offensive versus defensive data science and the difference between datadriven decision making versus products another great interview and I'm sure you'll enjoy it and now on to the [Music] show all right hey everyone I am on the line with deep Vara deep is Vice President of data engineering with Trulia uh and I'm excited to have deep join us uh deep how are you doing today oh I'm doing great it's you know not that hot this California San Francisco been hot for last few days but seems like the fog is coming back so I'm definitely doing amazing how are you Sam nice nice well I'm doing very well and I'm really looking forward to our conversation and to learning a little bit more about how you guys use data at Trulia uh why don't we get started by having you talk a little bit about your background and uh how you got into working with data yeah know I I think it's you asked me a great question because you know when I go and I speak in some of the schools to help undergrads or the grads you know those who are doing the management and one of the guys I think few months back in Berkeley asked me the similar kind of a question that the why data right and I think Sam it goes back to my reasoning my mindset from the childhood where I was always looking into you know the reasons why this is this why this is this and when I get into my masters uh on comp computer science you know I I still remember you know there were old databases which some of like you may know and you may not know is dbas or the fox base those were the early versions or the manifestations of the structured databases coming in and I was always being very interested and then entering into my first job in IBM you know it's where we are working on those XML direct Exchange how we are going to have the data exchange between one entity and other entity how the web dis web services Discovery locator going to come into the picture so early on the foundation was where I've been from the get-go from my own personal desire to look into the answers as well as exposure to the early Technologies get me into the databases and you know entering into then you know my journey where exploring why and what I realized is at the end of the day we are always surrounded by data and the data doesn't mean that it has to be a textual data the data is how we interact with each other when we are making phone calls to the people when I'm searching for something and that's I think it's 2001 2002 time frame was started getting into my DNA that you know my my God you know every day whenever I interact with anyone anything I do it's is the data and this is where I got into Yahoo so then that was a step you know where getting into Yahoo helping advertisers and Publishers you know try to render good quality ads to the consumers this is all again the platform is you know how you understand your consumers better and then going into my startups and you know looking into the data again where we are looking into you know how the data floating from one system to other system what predictions we can do so in nutshell I will say it is this is how I got into the data and I think is for me my behavior sometimes I'm at a point Sam I will tell you when I go home my wife have to remind me honey you're back home don't think from a data point of view just think you know you're back home that reasoning is not going to work here that's funny that's funny I uh I did my some of my grad school work on queuing Theory and my wife is so tired of me analyzing lines and queuing scenarios and Banks and grocery stores and trying to tell her which line she should be in so I definitely relate to that yeah so you were in Yahoo back in the Glory Days oh you trust me you know those were the Glory Days and I still miss those days because you know Yahoo was the center of Attraction and the talent was huge there so I worked there for four years and you know unfortunately Yahoo is no longer Yahoo but it was amazing absolutely were you involved in um uh did you use sop or were you involved in kind of the development and advancement of Hadoop at that time period yes soly on so when I went in we were trying to so Yahoo bought this company Overture and we were trying to integrate Yahoo's like platform backend platform to get the Search keywords so this is where hadu pipelines were integral part of the data flow between the systems that how we get the data from our Pasadena based company and then into our system so yeah I I was not deeply like I was not part of the Hadoop ecosystem but I was one of the consumers of the Hadoop system to get the data floating around okay okay and then now at at Trulia um tell me a little bit about your role it sounds like at least from LinkedIn that you've got a pretty broad set of responsibilities spanning everything from kind of your data platform you know I'm sure there's some Hadoop ecosystem something in there there somewhere um to you know the data science and the applications that run on top of it is that right that's that's fair and let me walk you through first why Trulia I think that's to me is the biggest piece which inspired me to join Trulia and do you mind Sam if I ask you do do you rent a home or do you you have your own home uh I own yes awesome I'm pretty sure you're going to relate to this story so this is way back in 1999 when we decided to buy our first home you know it is me and my wife you know the the data was there but the data was in storage like I have to go into police stations I have to go into counties I have to go into those areas to collect the data we take this data then me and my wife sit together we go through the listings we look into the neighborhood we used to maintain our Excel sheet oh let's look into this listing and it it was an cumbersome process and it was an emotional Journey for us to go through this exercise and it took us months to buy our first home and we did it finally right and that inspired me to join Trulia because you know when you think it's how can we use this data when I when I was in my early conversations with Trulia that was the biggest thing for me is you know how I'm going to come and join Trulia and make an impact to build Trulia as more of a data-driven product company and first thing in my mind was the use case of me 1999 buying a home that the can I make for millions of consumer that Journey much more meaningful much more enjoyable by using this data so that's how my journey began with uh Trulia now just to go a little bit more detail into my role in Trulia I think it's it's you know Trulia is you know our number one goal and this is where our Founders they saw a huge opportunity to change this Marketplace by providing information and insights to our consumers to help them make the right decision and you know and make this journey home search Journey easy and enjoyable so you know with that mindset with that goal what our Founders set and forth we we continue my goal was to continue this and provide amazing experiences to our consumers and we are investing a lot in our personalization Big Data machine learning platforms to help consumers like me to you know find their perfect home in much more efficient and better way than I did you know years back so that's in nutshell what role is and I'm happy to dive into more details about those uh Technologies what I'm talking about uh why don't we start with um you talking a little bit about the the data products and what are from a consumer experience perspective how do how is you know machine learning and Ai and the various data products that you create on your team how is that surfac to the truly a user yes so I think you know it's first of all data driven product companies you know Sam there is a big mind Shi needs to happen and I'm just you know three years back when I joined trulam my philosophy was always being to transform and use this data more on their offensive strategy rather than defensive strategy and I'm going to going to answer your question but I just wanted to give you a little bit more details because you know when you think about the data driven companies there are two aspects of the data driven company one is the data driven decision- making and other is the data driven product and the decision making is more your product analytics you know where you launch a feature and then you oh is this feature working or this feature is not working so my one goal is to transform this datadriven decision making more from a defensive to offensive by saying is this feature going to work so that's the the one component and the second component which is the discussion we are having today is around the data driven Product Company and this is where the way it surfaces to our consumers our average consumer have no idea when they come to Trulia when they engage with Trulia Via mobile web app or our browser or desktop applications they don't know that we use uh the data it is basically pretty much embedded in their user experience it is embedded when they explore when they start their search Journey it is our responsibility to understand their behavior what they're looking into and what we have done Sam is we have built an underlying personalization platform first and so think about that as our foundation and you know this is where we have our consumers unique preferences search CR area and you know what they're looking into like deep is looking into quiet neighborhood good school district and Mission District of San Francisco that's a personalization platform on top of it we have our machine learning pillars and there are many pillars what we have invested in machine learning the first one is our computer vision and deep learning the second one is our recommender engine the third one is our user engagement uh models and the fourth one is our uh natural language processing or the natural language generation so these are the machine learning pillars what we have and the you know and then we use all those pieces in tandem like machine learning pillars personalization platform all together to give that experience to our consumers when if when you come to our site and you look into you know the photos when you look into the when you receive an email from Trulia when you receive a push notification from Trulia all this is part of our machine learning Technologies which the goal is to engage our consumers and give them much more relevance experience during their stay with Trulia and I'm I will go definitely this conversation I will give you more details around each and every component what we're going to talk but if you have any question I'm happy to ask address that first okay uh there's there's just so much in there to to dig into um I like the I like the distinction between the decision-making versus the products and you mentioned specifically uh also this kind of dichotomy between offensive and defensive can you elaborate on that a bit more yeah definitely you know I I think is being in the Silicon Valley for close to two decades now and and I have seen startup companies coming up and you know their focus is mostly around building the products there there are companies like Google and Facebooks are definitely you know those who are more data driven but I have seen early on when the company start building the products their focus has never been the data side they have couple of analytics who are staying behind the scene oh should I change my pricing should I make my pricing for this consumer do this it's always and after five six years if they have to you know raise more funds or they are about to go public or when they go public now the mindset changes oh what is my differentiator how I'm going to differentiate my product how I'm going to bring my consumers back how I'm going to engage my consumers and this is where the offensive strategy comes into a picture that why not we have the companies start thinking about the data from the GetGo what data you collecting how do you build and I think that's a struggle and that struggle brings it to the point where companies have to go back and reinvest their resources their millions of dollars to build rebuild their architecture because if you think about the datas Sam at the end of the day this conversation what you and me are having is so much we are talking about in uh you know artificial intelligence machine learning we are collecting this data and we are compacting into a podcast but these are the signals right so there is a quality of the data Integrity of the data how do we take all those things and bundle up so that the organizations are thinking about changing the direction from the get-go rather than after the fact and after years thinking about so that's the way you know I think and I think I wanted to differentiate between the two aspects which I talked earlier and I just want to make sure that both you and me on the same page one is the decision making and other is the product building both of those facets requires the data decision making is our amazing analytics teams rather than them working on the data and saying is it working I want to transform that to is it going to work that's the big difference iation and the product what we talked about you know that's a SE second facet of it does it make sense so far yeah no I it does make sense and um you know I think the the the transformation that you described is kind of going you know maybe it's a different cut at that defensive versus offensive and in a lot in another way or put another way you're trying to get teams to stop building you you know rear viw mirror analytics and start you know building analytics that predicts what's you know going to be happening in front of the windshield here you go I think you nailed it better than me right it's it's the right way to explain it right yeah and I think that that is you know that transformation is something that's happening very broadly in you know industry not just in technology companies but also in Enterprises and it's you know that need to look out the the windshield and not be stuck you know reading reports that took weeks to create that reflect the previous quarter and aren't really even relevant anymore I think that's why you know Enterprises are kind of grasping onto you know machine learning and AI based Solutions as a way to kind of give them that forward-looking view yep and I I will add one more things Sam here so also you know when you think about any Enterprise any startup any technology company at the end of the day there you know all the work is done by the people and you have the limited resources and you know you are building a product rather build a product which is going to work in the marketplace like no one has this magic want to say this is going to work but this this uh front you know this uh offensive strategy or you know whatever way we want to say it helps us to align our resources in the right direction too so that we can change the direction of our ship going in the true north rather than you know go in the South Direction and then bring it back right right before we dive into the the platform yeah that you built you know one thing does strike me is that you know perhaps more than some other companies truly is you know truly as product this offering is data right and I'm making some assumptions um but I'm assuming that you're you know sourcing you know a bunch of different feeds and you even describe some of these you know you know your MLS listings your county you know data feeds maybe pulling in from good schools and other sites that are producing aggregate data on you know schools and crime and all these things like You're fundamental data is so fundamental to the thing the things that you do before you even get to how do you kind of what have you built and what have you learned about aggregating all of this data uh and and a little bit of a a little bit of context for this [Music] um I I often uh hear or you know I recently uh produced an event called the future of data Summit and we had speakers talking about you know different aspects of AI and several of them got up and said you know well in order to do you know machine learning in AI you have to have the data um and and I think that's true but you know it it kind of glosses over the fact that sometimes you have to get the data not just like it's not just sitting there waiting to be exploded you have to go and find it and it seems like a lot of what you did is is go and find it and so you know how did you you know to what extent does your team get involved in that and what's the platform for enabling that yeah I'm so glad talking to you right because you are nailing down the exactly the points what I'm passionate about so when you when you think about there are two pieces to the data and I'm going to make it very simple first to start with like when someone goes to Google and they search on a Google the first thing what they're doing is they are giving the search engine their intent what I'm searching for and then Google has this content which is they went ahead by crawlers and all those things by building the relevancy and all those thing so consumer gives the intent Google has this massive databases of the content and then the Magic in the middle which takes the intent and content and matches up which we call as a relevancy and give it to the consumer where consumer feels happy about it right now in the same context Trulia also have the two parts of the data one is the consumer uh for whom we are building this product and then the content where we get this content from and I think you know this is the listings as you talked about this is the public records which you talked about now schools data the crime data you know the commute data I think it is that's the difference between 1999 and 2017 where we have the Technologies like realtime messaging systems like CFA we has strong topologies or the streaming systems we have those Hadoop or spark Technologies where we can make it easier to ingest those data into our system so we have and this data is pretty open right I have written on my blog roughly we have 2.5 million active for sale listings on our system so across us you have agents those who are working with consumers to sell their home they enter this information into mlss how this data comes in from mlss to our system then you know when you sell your home when you buy your home you pay your taxes you have these assessments and the taxes which are going into the counties how we get this data so I think my team involves at the end of the day whatever you see on trulia's side it is my team's responsibility to use Technologies to bring this data in the raw form first and then enrich this data because when you think about you know you are mls1 and you will come and say 1 2 3 4 First Street and you can spell First Street as f iir s or someone just come and write number one First Street so we we we have to have this magic in the middle to join all this data and say this data is for this property and then once we know that uh you know the geolocation when we had the address cleansing address normalization and then when we work on the enrichment piece enrichment piece is for this listing this is the historical information about the property this is when the property was sold last time this is the Texas information this listing is 2 minute away from the public transit system this is the school and then we go through this enrichment process once we had that enrichment process it goes into our indexes which is you know we use our solar technology and we have buil you know our API layers on top of it which can take up till 10 to 15,000 request per second to serve our front end Technologies like web apps and you know mobile web or whatever it is so what I explained it to you on Surface it looks like a big process which takes days and days and days interestingly enough when the listing hits the marketplace by the time it goes from one point with the enrichment to the front and it is less than 15 minutes where we show the data so this is all because of the Technologies what we have enables us to give this content to Consumers faster so I just talked about the content piece which is you know the all the data flowing around and I think most likely when you're ready we will jump into the intent piece which is the personalization and all those but does it make sense so far uh it does it does and I still have tons of questions on that content side um so thinking about the various ways that you likely get data I'm imagining uh maybe three and and there are probably many more but I'm imagining you know some data is coming you via feeds uh maybe this is like the listing some data is coming you coming to you via streams and some data is coming to you via uh in batches um like can you characterize like how much of the data is each and are there C is there a category that I'm missing in in this and um and then like where do you you know where do you land it how much of you know what you're doing is you know real time stream based kind of uh processing yeah so I think it's it's depends upon the set of the data like so when you think about the listings listings are majority of our listings are the stream based which are real time because you know listings hits the market but then when you think about the public record which is Deeds assessments texes information that is mostly the batch based what we have and then you have the school data which is you know not it's not changing on a daily basis that's a feed based then you have a crime data which is you know more the streaming thing so I think it is it depends upon the data set what we so we have the Technologies where we Define if the data needs to be refreshed more frequently we use the streaming Technologies and otherwise you know we use um batch based systems we have invested in building our some of the systems uses the Lambda uh Technologies so this is you know the real time plus the batch based on a nightly basis we run the full Lambda and then make sure that there is you know the accuracy on the quality of the data being implemented there are some places we also use Kaa I don't know if you heard about Kaa so kaapa is also you know the real time but the batch based so I think it's at the end of the day my team have built those pipelines some pipeline uses you know the Kafka messages to strong topologies the streaming Technologies some places we have the spark where we need the data to be processed much more faster so I think it is at the different data set had the different refresh uh SLA and based on those refreshing slas we tend to you know bring it to our systems and before we move on why don't you um give us a brief overview of Lambda architectures and you mentioned Kaa uh as well yeah yeah uh and you mentioned stom uh as well just for folks that haven't come across those sure so I think let's start with the streaming right so I think what's happening is the streaming the realtime streaming and the processing so you know when we have those messages so think about if there are 2.5 million active listings are there across you know uh us when they hit our system they're coming into our messaging layer and there are different messaging Technologies are there we are using CFA kesis mix of that from there um we move into the streaming and you know the streaming can be spark or it can be a strong topology there there is a place where we use strong topologies because when the listing hits remember early on I was telling you that we need to do uh Geo cleans address cleansing address normalization and then you know the enrichment so this is where uh the bolts that we have the spouts and the bolts of our strong topology where spouts are the when which is ingesting the data and then you know the bols are the one which are making the decision making you know let I have to perform step XYZ so strong topology helps us in the real time take the stream of the data perform the enrichment perform the cleansing and then go and persist it into our new messaging layer from the data can be uh sent over so that's the strong topology Lambda is you know Lambda is being there in the Marketplace for long and what Lambda means is you know look at the on a daily basis when we are getting millions and millions of messages and you know it comes into our system it is very important for us to maintain the accuracy and the quality of the data so on a nightly basis we rerun the whole data set what we have collected on this day just to make sure that if there are gaps gaps we fill those gaps using the Lambda architecture and which will give us the higher level of accuracy and the quality of the data coming into our system so the only difference is in cap uh in Lambda you have to write your code base differently to consume the batch based but when you move into the Capa architecture you don't need to write the separate code base you can have the similar code base which is used during the real time streaming and you can use the same code base for your batch based uh typologies also so Kapa enables you to do that so that's how we use um those three Technologies which I just talked about okay all right great that's uh we'll include some links to these in the show notes I've come across Lambda architecture before but Kaa architecture is uh new to meing pretty yeah it is coming pretty new in the marketplace last couple of years I see people using it more okay great uh so you've ingested all of this data um and uh you've used Technologies like Lambda and Kappa architectures Apache storm and other uh Technologies uh CFA qes and messaging and all these things to get all this data in to enrich it uh and where do you where do you land it so if your question on the landing is where do we persist it that's yeah so we we we persisted in our Solar index so solar is the search technology and when you think about you know we have this millions and millions of rows coming in do we need to if there are 100 attributes in a row do we need to persist everything in the Solar so we basically all the searchable things we store in our Solar and then the things which are not searchable which are just an augmentation of the data can go into any of the nosql databases or some places where we feel the data is much more structured and we don't need we uses the relational databases also where the volume is pretty small so we use MySQL and that's how we purist so we use you know solar hbase is Dynamo DB MySQL and I'm pretty sure you know AOS spike is another one which we recently started using the key value pair systems so we use a very wide variety of the databases here in Trulia and again it all boils down to the use cases where do we need to store what right right so you have individual teams that are working on you know given products that are surfaced through the site and they choose whatever data store makes the most sense for their use cases um is that right yes yes and no so right so for example you know if we know the latency is a big thing for us then storing that in hbas may not make sense so people may decide to use you know reddis or they may go with the Dynamo DB and so I think we we had the some guidelines around like the biggest thing for us is build the databases and see the latency right because we had the API we have abstracted all the data as an API layer on top of those systems so that when frontend team comes and says give me all the data for this listing then this API goes across the different systems or the databases to bring the data Stitch it together so latency plays a major role but to some extent what you were trying to to say yes the decentralization of teams definitely enable us to have teams pick the Technologies what they want to pick we don't put so many guidelines except for the latency as one of the prerequisite making sure that we pick the right Technologies okay okay all right so then all that in place uh let's jump into the personalization platform and the the stuff that you're doing on top of it here you go that's stuff on piece right it's a right I I really love that piece I think now we what we talked about in last few minutes was mostly around the content right so now we need to start thinking from an intent point of view so when consumers they come to trulum you know when they are interacting with our website or mobile app or mobile web or email at any given moment of time what we have seen our consumers are generating those signals and the signals are nothing but their intent you know mhm deep is looking into you know a listing uh in noi Valley of a San Francisco which is in quiet neighborhood that's a signal and then deep is looking into photos you know and what kind of the photos deep is looking into so we have this stream of data flowing into our system what signals and what we internally call those as an events and events are generated by consumer interacting with those product so we basically take those events and our personalization platform in you know collects those events the events are just think about you know if Sam goes to Trulia and you look into some site you know on an average like when you look into a specific property Sam is going to generate an average of 20 events you know within few minutes of your interaction so we have this again the realtime messaging layer which collects those signals and you know we have Trulia has millions of consumers uh which are active on a monthly basis so when they send those signals we bring it to our cfal and from the cfal layer we basically brings it again we use uh streaming Technologies like spark or St again for the intent site of the technology tack to and this is where either we have the realtime machine learning models in place or we have some aggregated systems where those signals are getting evaluated right okay we just see deep or we saw an anonymous consumer we take all those data and then we persist it into our caching layer where that caching layer which can be so hedge base is our persistence layer for all the personalization platform but then the caching layer is the RIS what we have okay so if Sam is pretty much active on our site then Sam moves from hbase to RIS that's how we make because of the latency so this at the end you know this personalization platform stores Sam's unique preferences Sam's search criteria is you know Sam is looking into your Sam owns this two-bedroom three bath in St Louis area in a quiet neighborhood Sam is looking into this I think that's the personalization platform is a very foundational aspect which drives rest of the other thing then on top of I'm going to move over to machine Learning Systems is that fine now sure sure yeah so now when you think about this personalization platform is in put into place which is like an engine which is working on a daily basis by itself our first machine learning platform is computer vision and the Deep learning and this is where we we've been leading this industry in the computer vision and the Deep learning for years where you know computer vision right it's it's a system which we have built where we have you know trained our systems machines to look into photos and they can see oh I'm looking into a photo of a you know the swimming pool or I'm looking into a photo of a kitchen which has a granite countertop so that's the computer vision what we have implemented and then what we do is all those unique attributes the data which comes out of the system Powers our homepage and in our homepage you will see what we call as collections the collections are nothing but that the group of properties which we bring it together so you may see collections like you know homes with swimming pools or home with remodel homes or homes with kitchen granite countertop so those are the collections which uh we Powers our home page and the more our consumers engage with these collections the more insight we get into our consumers so that's the one use case of our computer vision the second use case of our computer vision is you know I'm pretty sure you me and all the consumers when they start their home buying Journey the first thing they do is they come into site like Trulia and they search for neighborhood then they go to a listing and then they start looking into the photos of the home that's how the journey starts and if if those photos are not engaging and if those photos are not telling story consumers are going to lose their interest and they will keep moving into the second and third so what we have done is we using uh conventional neuron Network CNN models we have invested in understanding you know the scene types of the photos whether the photo is appropriate or not like some someone can just put a photo of a dog so we say great you know we can see it is not a photo of a home it is a photo of a dog and then the quality of a photo is this photo is blur is this photo is much more clear so these three things what we take out from our CNN models is the quality of a photo appropriateness and the scene type we score those things and then the highest performing photo what we call as our hero image so what we do is most attractive photo when you start your journey we put the most attractive photo for you first so that you're engagement becomes much more better with trulam so that's the second use case of our uh machine learning uh computer vision and what we have seen by investing in those Technologies you know there are double digit increase in uh inquiries for our listing so that's the one piece make sense so far and is the is that lift based on uh do you think um primarily just getting the right listing in front of the the right person who's likely to like it or is it um you know getting rid of the or kind of suppressing the listings that aren't you know good in general you know is there any one factor that drives the kind of results that you've seen yeah so I think our relevance is driven mostly by the consumer Behavior what consumers are interested into and so we basically just based on the consumer needs and this is where the personalization platform comes into a play to drive that computer vision on the serving side what to serve to the consumer mhm makes sense so yes so two different users um you know say my wife and I are kind of collaboratively shopping for a home as husbands and wives tend to do um you know she might when she goes to the site she might see pool pictures first and I might see kitchen pictures first or what have you depending on what our what our interest are and these aren't interests that we've explicitly shared with you they're interests that you've derived from the various you know signals from watching the way we interact with the site that's that's fair and that's how basically the more you engage the more we know about you because if you come for the first time we really don't know about you right it is basically we need to reach enough confidence level to serve you the right content but yes your assessment was pretty good mhm um it's funny I can't help but uh the thing that I as as a as someone who travels a lot uh and as a result uses Yelp a lot I am always complaining about just how dumb the Yelp app is and I I don't think I've ever done it on the the podcast before but um I wish they were doing more of what you're doing when I land in a city I pretty much like open up Yelp and like type in Tha or type in Indian every time try trying to find a place to eat and I always wonder like why doesn't it just show me what it knows that I'm going to be looking for what it should know that I'm going to be looking for uh so maybe I'll use that rant as a segue into like what are the challenges that you've seen or what do you think you know what's the barrier to you know more companies you know having technology that enables them to better know and and personalized to their customers so just to understand so your question is what is the biggest barrier to investing in this kind of Technologies yeah yeah I I I I think it is mostly around making sure remember early on we talked about the data driven product companies that how do you understand the strength or the data what you have with you and I think it's it's the first it needs to start from the top level the commitment from the top level that's the area we want to invest in and the second thing Sam is rather than boiling the ocean right oh let's solve all the problems in one go pick the small use cases to evangelize within organization so that you know product people and other stakeholders can bought into those Concepts because you know AI or the machine learning is still in very infancy stage you know we have not reached the point where everyone understand stand so my recommendation to the people is definitely you know bring an evangelist build the small use cases show the value prop back it up with the data build slowly and gradually build the tsunami and when this tsunami is going to hit then everyone is going to bought into this so that's the way what I look into yeah that's a great articulation of the process um so you guys have also done some writing on your your engineering blog about how you use uh natural language processing and in particular natural language generation can you talk a little bit about that use case sure so yeah so I think you know thinking about so it all starts again for us we don't start from thinking about machine learning first we always start from thinking about the consumer first that's the number one goal and what we started seeing there are thousands and thousands of cities or neighborhoods across us when consumers come to our site you know they're looking for more information about this site they're looking into more information about that neighborhood and we said great now we you know how can we use the data what we have to build the story and one way you can do is you know you can have the human beings as editors and let them write the stories about the cities and the neighborhoods but when you go into this kind of a massive scale there is no way that's going to work and that's where we said okay great now let's rely on the machine learning Technologies to solve that problem and this is where we leaned on our natural language generation system so what we do is we we look into a location and we have built this feature extractor the feature extractor look into you know what are the restaurants closed by what are the commute systems looks like is the price going up for that neighborhood is it going down so we extract the features and then once those features comes out what we have is a document planner which looks into the features we have a document planner um but before we go into a description generator we have built our content bank so think about the content bank is where we build the sentences based on the features that if we say this like this neighborhood is a Victorian style homes so our content bank is going to have a sentence which will say Victorian this neighborhood has Victorian homes so we have this content bank and that content bank is build based on some of the crowd sourcing which definitely we do but we use our data Mining and machine learning Technologies to look into the data to build those content bank so now think about the document planner coming out for a neighborhood or a city or a specific location which has all the feature sets we have a Content bank and this is where then we use the description generator so description generator take the document planner take the content B bank and use the nlg to generate um the content for the specific location so that's how we use nlg and you know it's been going great on that front so what I what I think I hear you saying and this can be instructive to folks that want to use this is that as opposed to um you know throwing a b you know trying to throw a bunch of data to some uh natural language generation system and kind of hoping for hoping for it to generate something that makes sense you guys have broken up the problem and structured it you know in such a way that you know first you're um you're identifying the you know you call them features of your of a given neighborhood you know maybe not features in the sense of a training and machine learning algorithm but they're you know just attributes of a neighborhood and you kind of structure your descriptions you know so that you know you will highlight one or more of these attributes and then the content Bank what I thought I heard was that you kind of have a set of templates or rough structures of the way you talk about different things so you kind of have a template for how you talk about you know a neighborhood composition in terms of its architecture maybe some templates for restaurants things like that and then you know all of that you know those attributes that you decided to highlight in a given description and the the set of templates are uh utilized by this description generator to create something that you know sounds more human and is more readable and usable than you know what you might get if you just threw all the data against a a neuronet of some sort yeah uh one clarification the feature extractor it uses the data mining Technologies to extract the attributes what you're talking about so yes it goes into a neighborhood it uses the data mining it generates attribute and your right spot on the content Bank in a simple form you can think about the templates or you can think about you know which defines the much more vocabulary which is easily understood by our consumers that's great that's great well I know you're bumping up against uh a time constraint here uh I think this is a great you know use case we spent a lot more time on the you know data engineering data acquisition side than we usually do on the podcast and uh uh you know I enjoyed geeking out a little bit on some of that stuff um but it sounds like you guys are doing really really awesome things and so thank you so much for being on the show and sharing them with us great thanks Sam all right thanks deep bye-bye bye bye all right everyone that's our show for today once again thanks so much for listening and for your continued support don't forget to leave your review or comment to enter our one-year anniversary listener appreciation contest the full details can be found at t.com birthday and of course you can leave your questions and comments over on the show notes page at twiml a.com talk 2525 where you'll find links to deep and the various resources we mentioned in the show thanks so much for listening and catch you next time