Preface

Years ago, I got really tired of reading statistics textbooks that felt like textbooks. Why couldnt someone just write a book that made me feel like I had a real live person sitting next to me, who cares about me, who wants me to understand this stuff? Why can't someone step me through some examples in R (my favorite statistical software package) that actually include all of the steps I need to follow instead of leaving out one or two critical steps that leave me tearfully troubleshooting for hours? Why can't someone write a cookbook that provides just enough statistical theory and formulas so I can understand how the analytical solutions match up with the solutions provided by the statistical software?

That's the kind of book I wanted to recommend for my students. But after years of searching, and a couple more years of trying out books that didn't quite fit, I came to realize that the ideal book I was looking for just didn't exist: if I wanted my students to have a friendly, accessible, affordable, non-threatening textbook... I would have to write it for them. Finally, I started hitting LaTeX and the word processor. I'll do it, I committed. It took about three years to get to this point, but late is better than never (I hope).

SO... the purpose of this book is to help you analyze real data – quickly and easily, no fuss no muss – using the R statistical software. Along the way, you’ll learn some of the most important fundamentals of applied statistics, which will help you think more critically about the story your data is telling. My style is to tell you only what I think you need to know to quickly become productive. I’ll provide you with some background, some examples, and an explanation of what each of the commands in R does (and the options you can provide to those commands). I want to give you just enough theory so you know exactly what's going on under the surface, and just enough practice so that you know how to gain insights from your own data.

As a result, this book is NOT intended to be a substitute for a full-length text or course in statistics! You won't learn how to solve homeworky-style problems. However, I've often found that it's easier to get interested in how and why these methods work - and under what conditions - after you've had some success using them, and maybe even working with your own data.

Most importantly, each chapter has been written so that you don't need to be a math or statistics or programming ninja to be able to complete the exercises! Rather than trying to impress you with my slick coding skills in R, I have purposefully chosen less elegant but more instructive coding strategies, and I've attempted to show you ALL the lines of code for EVERYTHING that is produced in this book. I have also been repetitive, on PURPOSE, so you don't have to remember every single detail from previous chapters to get the job done in a later chapter! I have piloted all the chapters in one or more of my classes, and students in majors ranging from anthropology to business to science to technology have completed the work and provided the feedback that's been used to clarify each chapter.

I wrote this book as the primary text for the undergraduate sophomores and juniors in my introductory applied statistics courses. I was strongly motivated to do this for two reasons: 1) as silent activism against textbook publishers who charge $200+ a pop for very beautiful, glossy statistics textbooks that all of my students are required to purchase, yet none of my students actually read, and 2) to give my students just the essential information I think they need to become confident and capable data analysts using the most essential statistics. (At this point, the statistics textbooks tend to become a lot more interesting to everyone.)

To be clear, I have nothing against traditional textbooks themselves – or the authors who have labored to create clear, engaging texts with tons of pretty pictures and armies of practice problems. Actually, I admire the authors who spent so much time constructing their vision of a clear, cogent presentation of a subject. (It's not hard for me to admire and appreciate these contributions, since I myself have actually been a part of these teams of authors who have written reasonably awesome, yet expensive, textbooks -- which publishers are now making unconscionable bank with.) I just think traditional textbooks are kind of outmoded and passé. I can find lots of pretty pictures and examples on the web, and I’m not going to flatter myself by thinking that any more than one or two students work practice problems beyond what I cover in class.

I thought it was a sin that my students should pay $200 for a paperweight with a lifetime of just a semester and a resale value of less than 50% of the initial investment. I decided the book that I would create would not be heavy with beautiful and elegant typesetting, or stock photos of cheerful people working on statistical problems, or discussions that took the standard tone of "I am the great textbook, here to confer to you my power and wisdom." Textbooks never admit that some concepts are difficult, or why they're difficult. Textbooks aren't conversational, and don't really provide moral support. Textbooks have to be politically correct, which can preclude talking about really intense examples that students usually remember... like whether smoking too much pot is related to lower test scores.

I figured that a textbook that cost only 15-20% of the "standard", and that actually had some useful recipes that students could leverage for semester projects in other classes, might have a little more lasting value and maybe even find a permanent home on my students’ bookshelves. That’s my dream, at least.

Who Is This Book For?

Although originally designed for my undergraduate statistics students, I think this book will be useful to you if you are any of the following types of people:

  • Undergraduate college students in the first semester of applied statistics who are looking for a textbook that does not spend too much time on fluffy explanations of concepts, but rather, presents information in a more direct and less mathematical-ese way

  • Professors who want a relaxed, informal book for their intro stats classes that won't cost students $150-$250

  • Graduate students who need to use statistics to plan and complete a thesis or dissertation, but don't have that much background in it, and need help NOW

  • Early to mid-career data scientists who don't have a PhD in statistics and don't want one, but do want to get more familiar with R, with foundational concepts in statistics, and/or learning how to tell better and more compelling data-driven stories

  • Smart, business-savvy people who want to do more data analysis and business analytics, but don’t know where to start and don’t want to invest hundreds or thousands of dollars on statistical software!

  • People who are studying for their Six Sigma Black Belt exam. As a Black Belt myself, a lot of the understanding I've gained has come from the discipline of process improvement, so my treatment of topics naturally tends to reflect what's important for that exam.

  • Middle school or high school students who want to collect some original data and do cool science fair projects.

  • High school students who are taking regular or AP Statistics classes, and need a little extra help or information (or maybe you want to supplement the in-class instruction you're getting with some understanding of how to analyze real data).

I do not assume that you are a programmer. I also do not assume that you are super smart with either computers or statistics, only that you have the motivation to get some data analysis done with R. More about R in the next section. (I won't assume you know things like how important it is to coerce an object to another form if a certain algorithm won't work on the object you have. That would be cruel.)

I do assume that you already have R installed on your machine, and that you are impatient and just want to figure out how to do some useful stuff so you can start impressing your boss, your teachers, or yourself.

I do assume that you have a positive attitude, and that you'd like to learn some statistics and data analysis techniques that you can start using immediately. If you don't have a positive attitude, you should probably not be reading this book. In fact, it's likely that you shouldn't be doing statistics at all. You would be better served to go off and find some topic or some activity that does spin up a positive attitude in you. Why? Because you will have more fun. I wouldn't want you to do stats if it wasn't a little fun.

This Book Uses R

All of the examples in this book use the R statistical software. You need to download and install the software onto your own machine to be able to use it. Go to the R Project web site at http://www.r-project.org and select the link to "Download R". It will ask you to pick a "CRAN Mirror" so find a site that's geographically close to where you're sitting when you want to install R. For example, I live in Virginia so I might choose a Maryland or Ohio site rather than one in Argentina or Belgium. Data has to travel over geographic distances too, so I want all those mysterious bits and bytes to have to travel the shortest possible distance between where they live and MY laptop. That way, they will arrive as quickly as possible.

This book does not provide help installing R. Why? Simply because every time I have attempted to install R it has been really, really easy, and I don't consider myself an expert at installing anything on my computer. If you can't figure it out yourself, ask a friend (or if you're older, a friend's computer-savvy kid).

Or, you can try to decipher what's in the official R Installation and Administration manual, which can be found at http://cran.r-project.org/doc/manuals/R-admin.html.

How This Book Is Organized

It seems like every non-fiction book has a section in the front that talks about how the book is organized, so I thought I'd be a conformist and do the same. I mean, sure, you could look through the rest of the book to see how it's organized, but why do that when I can describe it right here so you don't have to flip through the pages?

  • Section 1 talks about basic concepts (like variable types, and loading data)

  • Section 2 explains how to create charts, graphs, and plots that can help you tell the story of your data through visualization

  • Section 3 provides the framework for how to do a research project using basic applied statistics, including fundamental aspects of theory, and also shows how to structure the story of your data and your results (featuring "Dr. R’s 7 Steps")

  • Section 4 summarizes how to create confidence intervals for several common scenarios, and recommends best approaches to handle the most challenging cases

  • Section 5 contains step-by-step recipes for conducting statistical inference tests, drawing conclusions, and interpreting your results

  • Section 6 contains recipes for regression (which is just a fancy word for finding relationships between variables), and

  • The Appendices provide some helpful reference information.

Inside each section, there's lots of useful stuff, and lots of me rambling about it.

Conventions Used in This Book

Joy in Repetition

Although I have aimed to keep the text as simple and concise as possible, you may notice that there is substantial repetition between sections, especially throughout Section 5: Statistical Inference. That’s because I also plan to use this text as a reference for myself. When I am doing an inference test, or even a confidence interval, I don't want to have to flip back and forth between chapters to do things like remember what all those assumptions are that I need to check. Or how to load the data. When I am doing a Pareto chart, I want to quickly and easily reference all of the most important ways to change colors, line styles, and common options. When I create pie charts, I want to be able to easily and quickly reference all the most important options to the pie command. I don’t want to have to flip back and forth between sections just because something was covered earlier in the book.

I don't want to pretend like I'm going to remember even the simplest syntaxes, because that's not easy (and gets less and less easy as your age and number of children increase).

My R Code

Code that you can type directly into R is written in 9-point Courier New, and indented half an inch from the left margin, like this:

defect.counts <- c(12,29,18,3,34,4)
names(defect.counts) <- c("Weather","Overslept", "Alarm
Failure", "Time Change","Traffic","Other")
df.defects <- data.frame(defect.counts)

The code above creates a vector of numbers in the first line, establishes names for what each of those numbers means in the next two lines, and creates a special object called a data frame in the last line. I had to use a smaller font size for code that you can type into R so that most of my text and output would show up looking decent on the printed page. Design purists, I give you my apologies up front.

Code that I typed into R and the output that code produced is also recorded in 9-point Courier New, but is not indented. This is important because it always has a leading caret (that’s the “>” at the beginning of each line). This caret is the R prompt, which you will see if you are using the R Console that comes with non-enterprise installations of the software. DO NOT TRY TO TYPE THE CARET AS PART OF YOUR CODE OR YOU WILL GET ERROR MESSAGES.

Here is an example:

df.defects
defect.counts
Weather 12
Overslept 29
Alarm Failure 18
Time Change 3
Traffic 34
Other 4

This means that I typed in the command df.defects to see the contents of the data frame that I named “df.defects” in R. The rest is what R responded back to me. If I typed in "> df.defects" I would get an error message that looked like this:

Error: unexpected '>' in ">"

(And for experienced R programmers: YES, I know that some of the R code in here is not optimized, and I know some of my simulations use loops instead of apply, and how could I ever possibly do something like that because that's not the best thing to do. You're right. I don't claim to provide elegant code in this book... just readable and/or explained code.)

My R Functions

Although I don’t rely on them as heavily in the 3rd Edition, I keep many of the utility functions I use on GitHub at https://github.com/NicoleRadziwill/R-Functions. Even though the text of these functions is included in this book, you might just want to load the function into your R console without having to cut and paste.

You can use the source command in R to load functions that are stored on Github:

source("https://raw.githubusercontent.com/NicoleRadziwill/R-Functions/master/ztest.R")

Then, just use the function as you would any other function:

z.test(80,112)
$estimate
[1] 0.7142857
$ts.z
NULL
$p.val
NULL
$cint
[1] 0.6306212 0.7979502
*My Data*

I keep the data I used in the examples in this book on GitHub. You can see a list of it all at https://github.com/NicoleRadziwill/Data-for-R-Examples. For data in CSV format, load library(tidyverse) and the URL for the raw data format on GitHub:

mnms <-
read_csv("https://raw.githubusercontent.com/NicoleRadziwill/Data-for-R-Examples/master/Fall%202016%20M%26Ms.csv")

For data in plain text format, use read.table and the URL that takes you to the raw data on GitHub, like this:

books <-
read.table("https://raw.githubusercontent.com/NicoleRadziwill/Data-for-R-Examples/master/anova-textbooks.txt",header=TRUE)

When lines of code are long and wrap to the next line like this, you should be able to copy the entire statement (without the leading caret), paste it into R, and get it to run. Be careful! Sometimes, lines of code depend on earlier lines of code – if you get red error messages, you may have to re-enter previous lines of code to get your current line to execute.

Changes Since the 2nd Edition

With thousands of contributing developers, R changes all the time. Fortunately, these changes usually mean that tasks get easier as time goes on, and new capabilities are added. Within days of the release of the 1st Edition, for example, it became much, much easier to source functions from GitHub: a seven-step process became a one-step, one line of code task. Since the release of the 2nd Edition, tidyverse and ggplot have become part of the collective consciousness, and it’s silly now to not learn it as a beginner so the 3rd Edition is the tidyversion. Here are the changes introduced in the 2nd and 3rd Editions:

  • Took out references to DataCamp (see, e.g., https://juliasilge.com/blog/datacamp-misconduct/)

  • Chapter 1.6 (“Getting Data Into R”) and all of the Chapters in Section 2 have been substantially revised; all chapters have been somewhat revised.

  • Dr. R’s 12 Steps have been consolidated into Dr. R’s 7 Steps. The role of the short 7-step solution process is now more fully explained in Chapter 3.3. Tests on regression slopes and intercepts have been adapted to the 7 Step method.

  • External datasets are easier to access, and no intermediate downloads to your local machine are required. I no longer use RCurl. It’s much easier to load functions and data from GitHub or any other location on the web than it used to be.

  • I use ggplot and geom_area instead of shadenorm and polygon although a few examples use both methods just to show you how much easier life is now.

  • There are headers on each page so you can more easily flip through the book and see what section and chapter you are in, and all equations have been re-rendered.

  • I still haven’t written an R package for this book because I don’t want a package to mask important tasks like acquiring data and acquiring functions from the web.

  • We are now tidyverse and ggplot compliant and examples have been done in RStudio. The 2nd Edition will still be available for instructors who want to use base R (I know there are still a lot of you; thanks for your emails).

How to Contact Me

If I have the time, I am more than happy to address comments and questions about this book, its examples, or problems that you are having working through your own data. I am more likely to have the time to respond to you in between semesters (which means most of December, and May through early August). If you don’t hear back from me, don’t despair: it just means that your message fell through the cracks, and I’m often either busy or distracted (or just plain scatterbrained), so I might have missed it. Try again. If I don’t have the time to answer or explore your query, I’ll let you know. I also invite you to follow me on Twitter - my ID is @nicoleradziwill - but be advised, you will also be getting a lot of tweet spam about severe weather, tornadoes, solar flares, coronal mass ejections, and Burning Man. (You may like this.) Also, if there is ever an interesting planetary configuration or an asteroid that is about to impact Earth, you will be among the first to know.

Disclaimer

The purpose of this book is not to be mathematically elegant or comprehensive, but to give you enough of what you need to know to be productive - quickly - without leaving you statistically naive. I've included links to longer, more extensive proofs in many places if you want to know more about the mathiness underlying everything in this book. I am not a professional statistician or mathematical purist, but a realist who analyzes data almost daily; I want you to be able to analyze your data too.

Acknowledgements

I would be remiss if I did not mention the people who influenced me to become a person who really likes to use, and teach, statistics. First of all, I am eternally indebted to all the students that test drove my explanations of concepts, and my early drafts of this book. They helped me recognize what can be confusing to learners, and helped me illuminate the things that were actually confusing to me. I'm also thankful to my teaching assistants Andy Duong, Meghan Mooney, Aimee Cunningham, Cassidy Moellers, and Chris Miller who helped me in the initial stages of piloting the new approaches and creating new course materials.

Dan Teague of the North Carolina School of Science and Math in Durham, NC was my first statistics teacher and deserves a lot of credit. He was brave enough to explain to the group of 14- to 17- year olds in my class nearly two decades ago that statistics is just a game of exploring variation in n-dimensional vector space. I was smitten. Bob Ryan of the 1990’s statistics department at Penn State deserves some credit for keeping me hooked. He was one of the executives of Minitab, a company he founded with his wife at the time, totally down to earth, and told cheesy (yet "good" to me!) jokes in class. The jokes made me want to keep doing applied statistics. (Yes, I can be that easily influenced.) Plus, he made all of us buy Lyman Ott’s applied statistics book, which I still have, and still use. Mike Hayden of Indiana State University also deserves a mention and much gratitude. His recipe for executing and documenting a statistical hypothesis test (which he calls “Hayden’s 16 Steps”) influenced me greatly, and certainly snuck into my own recipes and approach, even forming the basis for "Dr. R’s 7 Steps" which are a theme within this book. I am also eternally grateful to him for helping me really understand what setting your alpha level is all about… balancing cost and risk. And ethical considerations... but I added that last part.

I am also thankful for H. M. Schey's book Div, Grad, Curl, and All That: An Informal Text on Vector Calculus. Had I not possessed that book as an undergrad, I never would have passed my vector calculus course, which was essential for my major (meteorology). The book you're reading now is in many ways a tribute to this author's style… especially the subtitle, "an informal text on applied statistics and data science."

I also appreciate the support and feedback from the Department of Integrated Science and Technology (ISAT) at James Madison University (JMU) in Harrisonburg, Virginia, especially my colleagues Anne Henriksen, Rebecca Simmons, and Morgan C. Benton. Anne and I have had many discussions about how to teach introductory statistics in an integrated way (that embraces thinking about social context) for the past few years, and she helped me identify some really important typos in some of the early versions of my chapters. Rebecca (a fellow quality and lean enthusiast) was the first brave soul to pick up the curriculum, use it, and refine it from the third-party perspective. My partner and co-conspirator Morgan provided extensive technical, moral, and emotional support throughout the entire process of preparing this book, including extremely geeky conversations exploring things like the possible relationship between P-Values and the observer effect in quantum physics... and pointers to all of the XKCD cartoons. And Netflix. And many hours of fantastically blissful and enjoyable research, for which I have much gratitude.