15 Homework 5: Exploratory data analysis

Activity prompt: What are the first steps to take when you start a project or get ahold of a new data set?

Additional reading:

In a real data science project, you will typically have a question or idea brought to you: How are people interacting with the new version of my software? How is weather in St. Paul changing in the last decade? What type of people are most likely to enroll in Obamacare?

You will sometimes be given a dataset when asked one of these questions, or more often, a general description of the data you could get if you talked to the right people or interacted with the right software systems.

In this section, we will talk about Exploratory Data Analysis (EDA), a name given to the process of 1) “getting to know” a dataset, and 2) trying to identify any meaningful insights within it. Grolemund and Wickham visualize the place of this “Understand” process with a simple diagram:

The process of EDA, as described by Grolemund and Wickham.

Figure 15.1: The process of EDA, as described by Grolemund and Wickham.

We view the process similarly:

  1. Understand the basic data that is available to you.
  2. Visualize and describe the variables that seem most interesting or relevant.
  3. Formulate a research question.
  4. Analyze the data related to the research question, starting from simple analyses to more complex ones.
  5. Interpret your findings, refine your research question, and return to step 4.

15.1 Understand the Basic Data

Start by understanding the data that is available to you. If you have a codebook, you have struck gold! If not (the more common case), you’ll need to do some detective work that often involves talking to people. At this stage, ask yourself:

  • Where does my data come from? How was it collected?6
  • Is there a codebook? If not, how can I learn about it?
  • Are there people I can reach out to who have experience with this data?

Next, you need to load the data and clean it. Once the data is loaded, ask yourself about each table:

  • What is an observation?
  • How many observations are there?
  • What is the meaning of each variable?
  • What is the type of each variable (date, location, string, factor, number, boolean, etc.)?

Some great methods to start with are the functions

  • str() to learn about the numbers of variables and observations as well as the classes of variables
  • head() to view the top of the data table (can specify the number of rows with n= )
  • tail() to view the bottom of the data table

Finally, ask yourself about the relationships between tables:

  • What variables link the tables (i.e., which variables can you use in join commands)?

15.2 Visualize and Describe the Data

Once you have the data loaded and cleaned, it is usually helpful to do some univariate visualization; e.g., plotting histograms, densities, and box plots of different variables. You might ask questions such as:

  • What do you see that is interesting?
  • Which values are most common or unusual (outliers)?
  • Is there a lot of missing data?
  • What type of variation occurs within the individual variables?
  • What might be causing the interesting findings?
  • How could you figure out whether your ideas are correct?

Once you have done some univariate visualization, you might examine the covariation between different variables. One convenient way to do this is with a pairs plot.

15.3 Formulate a Research Question

You will often end up with a ton of data, and it can be easy to be overwhelmed. How should you get started? One easy idea is to brainstorm ideas for research questions, and pick one that seems promising. This process is much easier with more than one brain! You will often be working off of a broad question posed by your business, organization, or supervisor, and be thinking about how to narrow it down. To do so, you can again revisit questions like “What patterns do you see?” or “Why might they be occurring?”

15.4 Examples

Here are some exploratory data analysis examples I like:

15.5 Homework Assignment: Group Mini-Project

Choose one of the following three options:

Complete the following tasks:

  1. Identify team members.
  2. Formulate a broad research question.
  3. Perform some exploratory data analysis.
  4. Develop at least one visualization that tells a story about your more specific research question/hypothesis. Note: the story may very well be something along the lines of “we thought variable X would affect measure Y in way Z, but the evidence does not support that.”
  5. Test out some of the modeling and machine learning techniques we have examined on your data. How do the predictions on a test data set compare to a null model? What measures of accuracy are you using? Try some models that include a larger number of explanatory variables. Try to learn a model that provides some insights (not necessarily high prediction accuracy), and demonstrate these insights with a visualization.

15.6 Flight Data Example

Here are data about flight delays from Kaggle.

There are three tables of data:

airlines <- read.csv("https://www.macalester.edu/~dshuman1/data/112/flights/airlines.csv")
airports <- read.csv("https://www.macalester.edu/~dshuman1/data/112/flights/airports.csv")
flights <- read.csv("https://www.macalester.edu/~dshuman1/data/112/flights/flights_jan_jul_sample2.csv")

Note that the full set of flight data has more than 5.8 million flights. To start, I have given you a subset that includes all flights in the first 15 days of January 2015 and in the first 15 days of July 2015. If at the end you wish to try your analysis on the whole data set, you can download the original .csv file from the Kaggle page and substitute a link to it in the code chunk above.

There is a ton of data here, and it can be easy to be overwhelmed. We are going to focus our exploration a bit by considering the following broad research question: which flights are most likely to be delayed?

Exercise 15.1 (Data source) Where does this data come from? Who collected it?

Exercise 15.2 (Explore codebook) This data set comes with a codebook on the Kaggle site. Have a look at the codebook to understand which variables are contained in each of the three tables. For example, what are the levels of CANCELLATION_REASON and what do they mean? What is the unit of observation for each table?

Exercise 15.3 (Possible joins) What variables link the three tables? How could you join data from one table to another?

Exercise 15.4 (Visualize and describe the data) Use some univariate and bivariate visualizations to start to explore the questions mentioned above:

  • What do you see that is interesting?
  • Which values are most common or unusual (outliers)?
  • Is there a lot of missing data?
  • What type of variation occurs within the individual variables?
  • What might be causing the interesting findings?
  • How could you figure out whether your ideas are correct?

Exercise 15.5 (Formulate a specific research question) Based on your preliminary visualizations and exploration of the date, formulate a more specific research question/hypothesis within this broad area of understanding the causes of flight delays.

Exercise 15.6 (Prediction) Use one or more of the statistical modeling techniques we explored to make predictions on a test data set. How do your predictions compare to a null model? Does your learned model provide qualitative insights?

  1. Particularly important questions about how it was collected include whether it is a sample of a larger data set, and, if so, how the sampling was done? Randomly? All cases during a specific time frame? All data for a selected set of users? Answers to such questions strongly impact the conclusions you will be able to draw from the data.↩︎

  2. The graphics in this one are a bit more developed than you would really see in an exploratory analysis, but I think the progression of visualizations is interesting and follows an exploratory story.↩︎