PySpark Tutorial: Defining A Problem

Want to learn more? Take the full course at https://learn.datacamp.com/courses/feature-engineering-with-pyspark at your own pace. More than a video, you'll learn hands-on coding & quickly apply skills to your daily work. --- What's the point of doing analysis if you aren't solving the right problem? In this video, we will define our problem and the context of our data. We are going to build a model to predict how much a house sells for. This question can be interpreted in multiple ways which is why it's important to take the time formally define it. Let's assume we are real-estate tycoon's looking for the next best investment opportunity. For a given house on the market, with a listed price and series of attributes describing the home, what is it likely to actually sell for, aka the SALESCLOSEPRICE? The dataset we have is a sample of homes that sold over the course of 2017. Using this sample we are to provide a quick proof of the concept of whether it's worth investing in more data for the 5.5 million homes sold in the US in 2017. To do this we need to understand some of the limitations of the data we have. First, we only have a small geographical area, so to apply our model to new areas, poses a serious risk! We know that we only have residential data, so we shouldn't expect to predict how much a business location is worth! Lastly, we only have one year's worth of data which will make it hard to draw strong conclusions about seasonality in this dataset. The original dataset has hundreds of attributes available but in order to start simple we've already worked with our client to identify around 50 attributes they think are likely to influence the price of a home. These attributes generally fall into these groups. For Dates we have date listed, and the year the home was built. For locational data, we have the city that the home is in, its school district and its actual postal address. We also have many different metrics to gauge the size of the home like num