1.5 Descriptive Statistics

Objective

Sometimes, you want an easy way to quantitatively summarize the characteristics of a collection of data, whether that information comes from an experiment, an archive, a survey, or some other kind of source. Descriptive statistics provide simple summaries about the sample you have collected, and complement charts and graphs that provide an additional representation of the data. With descriptive statistics, you are not attempting to draw any conclusions about the data: you are just presenting what the data shows, prior to any analysis.

Descriptive statistics are often supplemented by measures such as skewness and kurtosis, which describe the basic state of the distribution, or basic charts and graphs, such as histograms, boxplots, scatterplots, and contingency tables, which are covered in Section 2: Charts, Graphs, and Plots.

Data Acquisition

First, let's load some data from the National Oceanic and Atmospheric Administration (NOAA) severe weather data inventory that was originally downloaded from their archive at http://www.ncdc.noaa.gov/swdi in 2014. I stored the file on GitHub so it’s easy for you to access directly. If you want to store your own data on GitHub, click on the “Raw” button when you are viewing your file to get the URL that R can use for the import function:

The read_csv command makes a tidyverse-friendly data frame called a tibble. The tibble format makes it easier to quickly identify the contents of a data frame without word wrap or scrolling problems. Tibbles also tell you what data types are available (fct for factors, which are categorical variables; int for integers, which are quantitative variables; date for dates and times; and dbl for “doubles” – or numbers with double precision decimals).

Now we can take a look at what's in the file. Notice that negative values are emphasized:

# A tibble: 1,532 x 17
ZTIME LON LAT WSR_ID CELL_ID CELL_TYPE RANGE AZIMUTH
*<dbl>* *<dbl>* *<dbl>* *<chr>* *<chr>* *<chr>* *<int>*
*<int>*
1 2.01e13 -91.5 41.3 KDVN M7 TVS 46 250
2 2.01e13 -94.5 40.2 KMCI J1 TVS 46 13
3 2.01e13 -88.1 41.5 KORD W7 TVS 19 210
4 2.01e13 -94.8 40.0 KMCI C1 TVS 33 356
5 2.01e13 -88.0 41.6 KORD W7 TVS 14 207
6 2.01e13 -91.1 41.2 KDVN S7 TVS 33 229
7 2.01e13 -102. 38.1 KGLD Z3 TVS 83 202
8 2.01e13 -80.3 26.5 KMIA Z4 TVS 44 10
9 2.01e13 -88.0 41.6 KORD W9 TVS 14 199
10 2.01e13 -94.8 40.0 KMCI U1 TVS 32 359

# ... with 1,522 more rows, and 9 more variables: AVGDV <int>,

# LLDV <int>, MXDV <int>, MXDV_HEIGHT <int>, DEPTH <int>,

# BASE <int>, TOP <int>, MAX_SHEAR <int>, MAX_SHEAR_HEIGHT <int>

What is this data about? It looks like the first column contains a timestamp that’s been corrupted (they’re all the same), and the second two contain the longitude and latitude of the observed TVS (a place where Doppler radar detected a likely tornado). WSR_ID is the name of the radar station that reported the observation. CELL_ID provides a way for us to distinguish between tornadoes associated with different thunderstorms. CELL_TYPE indicates that we are looking at TVS observations, so this value should be the same for all rows. RANGE and AZIMUTH tell us where the TVS was observed, relative to the radar, and the remaining values tell us physical parameters about the tornado signature. The information we will be interested in gathering descriptive statistics for are the DEPTH of the tornado signature (in thousands of feet), the TOP of the tornado signature (also in thousands of feet), and the MAX_SHEAR (or maximum wind shear, in meters per second). Since we are only interested in a subset of the data, let's construct a smaller data frame containing only those values that are of interest to us. This creates a new tibble, sub.tvs, which consists of just these three named columns:

What we have done is started with the original data set tvs, and then “piped it” to the select command using the pipe operator %>%. To read this statement in English from left to right, we would say “Create a new tibble called sub.tvs by starting with tvs and pushing it through the select function so that only DEPTH, TOP, and MAX_SHEAR are kept.”

Descriptive Statistics with summary

Once we have our data in the proper format, we can generate descriptive statistics using the summary command:

Min. : 5.00 Min. : 5.00 Min. : 10.00

1st Qu.: 7.00 1st Qu.: 9.00 1st Qu.: 24.00

Median :10.00 Median :13.00 Median : 30.00

Mean :12.34 Mean :15.57 Mean : 38.03

3rd Qu.:17.00 3rd Qu.:21.00 3rd Qu.: 42.00

Max. :63.00 Max. :69.00 Max. :283.00

The descriptive statistics that are displayed are:

  • The minimum value observed for this variable in the dataset

  • The first quantile (also called Q1); 25% of the observations are BELOW this value and 75% of the observations are ABOVE this value

  • The median; an actual observation within the dataset indicating that 50% of the observations are BELOW this value and 50% are ABOVE this value

  • The mean; a number characterizing the center of the distribution that is obtained by finding the average value across all the observations

  • The third quantile (also called Q3); 75% of the observations are BELOW this value and 25% of the observations are ABOVE this value

  • The maximum value observed for this variable in the dataset

Descriptive statistics can give us a quick indication of some of the characteristics of the distribution: for example, if the mean is far greater than the median (to the right of it on the probability distribution), we know that the distribution must be skewed to the right. Similarly, if the mean is far less than the median (to the left of it on the probability distribution), we know that the distribution must be skewed to the left. One limitation of the summary command in the base R package is that it doesn't give you really useful and critical descriptive statistics like the standard deviation, variance, and mode (the most frequently observed value in the set of observations). There are two ways to do it:

Base R Tidyverse (Using dplyr)

> sd(sub.tvs$DEPTH)

[1] 7.19

> var(sub.tvs$DEPTH)

[1] 51.7

> sub.tvs %>% summarise(sd=sd(DEPTH))

> sub.tvs %>% summarise(var=var(DEPTH))

To obtain the mode, first cut and paste the following base R code to load the mode function:

Alternatively, you can source it from GitHub directly:

>

Now you can use that function on your data:

Some of these limitations can be overcome by using an alternative approach: the stat.desc function which is part of the pastecs package.

Descriptive Statistics with stat.desc from the pastecs Package

If you don't have it already, first install.packages("pastecs")and call it into memory using library(pastecs). Then, the following commands will work:

Since stat.desc likes to use scientific notation, the first two commands allow you to set the number of significant digits you want to see. In addition to the basic descriptive statistics provided by summary, stat.desc shows you:

  • The number of observations as nbr.val

  • The number of null observations as nbr.null

  • The number of "not available" (NA) observations as nbr.na

  • The range of values (that is, how far it is to get from the minimum observed value to the maximum observed value) as range

  • The sum of all values as sum

  • The standard error of the mean (SE.mean) indicates how precisely you can know the true mean of the population, given only what you have in your sample. It accounts for the sample size and the scatteredness (or dispersion) of the dataset.

  • The width of the 95% confidence interval as CI.mean.0.95 (meaning that the mean minus this value indicates the lower bound of your confidence interval, and the mean plus this value indicates the upper bound of the confidence interval)

  • The variance of all the values, which gives you an indication of how scattered or dispersed they are, as var

  • The standard deviation, which is the square root of the variance and indicates pretty much the same thing as the variance, as std.dev

  • The coefficient of variance, which is the standard deviation divided by the mean, and also gives a sense of how scattered or dispersed the observations are, as coef.var

Here is a summary of the arguments that you can use to generate descriptive statistics:

Argument to stat.desc What it does
basic=TRUE Include number of observations, number of nulls, number of NAs, min, max, range and sum
desc=TRUE Include median, mean, standard error of the mean, width of the confidence interval, variance, standard deviation, and coefficient of variation
norm=TRUE Include measures for skewness and kurtosis
p=0.99 Specify the width of the desired confidence interval (e.g. 99%)
options(digits=3) Specify the number of significant digits to display (e.g. 3)
options(scipen=999) Specify a "penalty" to use when determining whether scientific notation (e.g. 1E+03 or 1000); higher values incur greater penalty, and a 999 will prevent use of scientific notation almost entirely

Descriptive Statistics with the xda Package

If you don't have it already, first install.packages("xda") and then when it's available on your machine, call it into memory using library(xda). You can try numSummary(sub.tvs) and charSummary(tvs) commands to extract descriptive statistics about your quantitative and categorical data, respectively. (There is also a plot function, but it doesn’t perform reliably.) I’ve included a couple examples on the R-Bloggers site at http://www.r-bloggers.com/using-xda-with-googlesheets-in-r/.

Other Resources

Here are some other resources to help you understand the concepts in this chapter and explore the severe weather datasets that were obtained: