PySpark Tutorial : Loading Data

Want to learn more? Take the full course at https://learn.datacamp.com/courses/machine-learning-with-apache-spark at your own pace. More than a video, you'll learn hands-on coding & quickly apply skills to your daily work. --- In this lesson you'll look at how to read data into Spark. Spark represents tabular data using the DataFrame class. The data are captured as rows (or "records"), each of which is broken down into one or more columns (or "fields"). Every column has a name and a specific data type. Some selected methods and attributes of the DataFrame class are listed here. The count() method gives the number of rows. The show() method will display a subset of rows. The printSchema() method and the dtypes attribute give different views on column types. This is really scratching the surface of what's possible with a DataFrame. You can find out more by consulting the extensive documentation. CSV is a common format for storing tabular data. For illustration we'll be using a CSV file with characteristics for a selection of motor vehicles. Each line in a CSV file is a new record and within each record, fields are separated by a delimiter character, which is normally a comma. The first line is an optional header record which gives column names. Our session object has a "read" attribute which, in turn, has a csv() method which reads data from a CSV file and returns a DataFrame. The csv() method has one mandatory argument, the path to the CSV file. There are a number of optional arguments. We'll take a quick look at some of the most important ones. The header argument specifies whether or not there is a header record. The sep argument gives the field separator, which is a comma by default. There are two arguments which pertain to column data types, schema and inferSchema. Finally, the nullValue argument gives the placeholder used to indicate missing data. Let's take a look at the data we've just loaded. Using the show() method we can take a look at a slic