class: center, middle, inverse, title-slide .title[ # MACS 30500 LECTURE 7 ] .author[ ### Topics: Tidying data with tidyr ] --- class: inverse, middle # Agenda * HW3: in-class demo (from last lecture in-class materials) * Tidy data with `tidyr`: theory and practice --- class: inverse, middle ## HW3: in-class demo Download last lecture in-class materials from the website (`scotus-lecture.Rmd`). **Goals: review lecture 6 concepts (importing data, relational data), and example of manipulating HW3 data** --- class: inverse, middle ## Tidy data with `tidyr`: theory and practice --- ## Tidy data: definition **"Tidy data" is a system or a particular way to organize data.** "Tidy data" is not the only way of organizing data, but useful because it is: * standardized (every tidy dataset must follow rules) * consistent (data formatted in a tidy way can be used without further data manipulation by various packages in the `tidyverse`) * clear (data formatted in a tidy way are easier to understand and use) -- If you get some data and you know you want to work within the `tidyverse` (e.g., `ggplot`, `dplyr`, etc), put them in a tidy format first! --- ## Tidy data: three key principles <img src="tidydata_1.jpg" alt="Stylized text providing an overview of Tidy Data. The top reads 'Tidy data is a standard way of mapping the meaning of a dataset to its structure. - Hadley Wickham.' On the left reads 'In tidy data: each variable forms a column; each observation forms a row; each cell is a single measurement.' There is an example table on the lower right with columns ‘id’, ‘name’ and ‘color’ with observations for different cats, illustrating tidy data structure." width="70%" style="display: block; margin: auto;" /> .footnote[Illustrations from the blog [*Tidy Data for reproducibility, efficiency, and collaboration*](https://www.openscapes.org/blog/2020/10/12/tidy-data/) by Julia Lowndes and Allison Horst] <!-- tidy data is a very SPECIFICY way of standardizing info in a dataframe but it is not the only way; we are going to see some examples the opposite of tidy would be messy data or untidy the reason why tidy data is popular is because provides a STANDARDIZED form; all packages we have learned so far ggplot, dplyr work with tidy data which means you can simply load the dataset and start working on it without reshaping it or cleaning it up (if tidy) point here: as soon as you get data and you know u want to work on them within the tidyverse (ggplot, dplyr etc) get them in a tidy format first, then focus with the analyses or anything else u want to do! --> --- ## Tidy datasets are all alike <img src="tidydata_2.jpg" alt="There are two sets of anthropomorphized data tables. The top group of three tables are all rectangular and smiling, with a shared speech bubble reading 'our columns are variables and our rows are observations!'. Text to the left of that group reads 'The standard structure of tidy data means that 'tidy datasets are all alike…' The lower group of four tables are all different shapes, look ragged and concerned, and have different speech bubbles reading (from left to right) 'my column are values and my rows are variables', 'I have variables in columns AND in rows', 'I have multiple variables in a single column', and 'I don’t even KNOW what my deal is.' Next to the frazzled data tables is text '...but every messy dataset is messy in its own way. -Hadley Wickham.'" width="70%" style="display: block; margin: auto;" /> .footnote[Illustrations from the [Openscapes](https://www.openscapes.org/) blog [*Tidy Data for reproducibility, efficiency, and collaboration*](https://www.openscapes.org/blog/2020/10/12/tidy-data/) by Julia Lowndes and Allison Horst] <!-- pause think you are starting a research project and you have data as top or bottom; which one you want to be and why? --> --- ## Tidy datasets are all alike <img src="tidydata_3.jpg" alt="On the left is a happy cute fuzzy monster holding a rectangular data frame with a tool that fits the data frame shape. On the workbench behind the monster are other data frames of similar rectangular shape, and neatly arranged tools that also look like they would fit those data frames. The workbench looks uncluttered and tidy. The text above the tidy workbench reads 'When working with tidy data, we can use the same tools in similar ways for different datasets…' On the right is a cute monster looking very frustrated, using duct tape and other tools to haphazardly tie data tables together, each in a different way. The monster is in front of a messy, cluttered workbench. The text above the frustrated monster reads '...but working with untidy data often means reinventing the wheel with one-time approaches that are hard to iterate or reuse.'" width="70%" style="display: block; margin: auto;" /> .footnote[Illustrations from the blog [*Tidy Data for reproducibility, efficiency, and collaboration*](https://www.openscapes.org/blog/2020/10/12/tidy-data/) by Julia Lowndes and Allison Horst] --- ## Tidy untidy data It might sound counterintutive, but **most data you will encounter in real life is stored in an untidy format.** Why so? -- Our book suggests two main reasons: * Data is often organized *not* to facilitate data wrangling and analysis. For example, it’s common for data to be structured to make data entry, not analysis, easy. * Not everyone working with data is familiar with the principles of tidy data, and it’s hard to derive them yourself unless you spend a lot of time working with data. --- ## Tidy untidy data **OK, you loaded your data into R and they are untidy. What should you do to tidy them up?** First, stare at the data: * identify the observations and the variables * identify what's untidy about them Then: * place each observation in a separate row * place each variable in a separate column * ensure that each value has its own cell -- **Typical issues:** 1. A variable is spread across multiple columns or an observation across multiple rows. 2. Variable values are column names, or variable names are in rows. --- ## Tidying tasks * **Pivoting** * Longer: makes the dataset longer by increasing rows * Wider: makes the dataset wider by increasing columns * **Separating** * **Uniting** Remember to **check the `tidyr` documentation: https://tidyr.tidyverse.org/** NB: every function in this package behaves differently (e.g., functions do not have the same arguments!) --- ## In-class Example: our goals We are going to illustrate these tasks using one of the datasets from the readings (Chapter 5 https://r4ds.hadley.nz/data-tidy) This is a longitudinal dataset with four variables: `country`, `year`, `cases` (number of tubercolosis cases), and `population`. We analyze: * several ways to organize the same dataset * how to use `tidyr` functions to tidy each of these datasets: pivoting (longer and wider), separate, and unite functions <!-- no "wrong" but some are messy or untidiy --> --- ## In-class Example: the data ```r library(tidyverse) table1 ``` ``` ## # A tibble: 6 × 4 ## country year cases population ## <chr> <dbl> <dbl> <dbl> ## 1 Afghanistan 1999 745 19987071 ## 2 Afghanistan 2000 2666 20595360 ## 3 Brazil 1999 37737 172006362 ## 4 Brazil 2000 80488 174504898 ## 5 China 1999 212258 1272915272 ## 6 China 2000 213766 1280428583 ``` --- ## Pivot longer Why is this dataset untidy? How can we tidy it? ```r library(tidyverse) table4a ``` ``` ## # A tibble: 3 × 3 ## country `1999` `2000` ## <chr> <dbl> <dbl> ## 1 Afghanistan 745 2666 ## 2 Brazil 37737 80488 ## 3 China 212258 213766 ``` -- "Each variable must have its own column": column names must be names of variables, not variables' values (1999 and 2000 are values of the `year` variable) "Each observation must have its own row": we have one row for every country, but this is panel data, so we should have the country-year pair to define one observation, not just country. --- ## Pivot longer https://tidyr.tidyverse.org/reference/pivot_longer.html .pull-left[ ```r table4a ``` ``` ## # A tibble: 3 x 3 ## country `1999` `2000` ## * <chr> <int> <int> ## 1 Afghanistan 745 2666 ## 2 Brazil 37737 80488 ## 3 China 212258 213766 ``` ] .pull-right[ ```r pivot_longer( data = table4a, cols = c(`1999`, `2000`), names_to = "year", values_to = "cases" ) ``` ``` ## # A tibble: 6 x 3 ## country year cases ## <chr> <chr> <int> ## 1 Afghanistan 1999 745 ## 2 Afghanistan 2000 2666 ## 3 Brazil 1999 37737 ## 4 Brazil 2000 80488 ## 5 China 1999 212258 ## 6 China 2000 213766 ``` ] <!-- We can reshape and tidy this data using `pivot_longer` form tidyr, which takes four main arguments: - data: data we are reshaping -- notice we go from a 3by3 to a 6by3 dataframe - cols: name of the columns in the original data that we want to change -- aka columns to use to make this pivot; note the use of back ticks! and we need to use two columns, thus we use the vector syntax with c() for concatenate - names_to column: new column we wish to create from column names - values_to column: new column we wish to create and fill with values --> -- --- ## Pivot wider Why is this dataset untidy? How can we tidy it? .pull-left[ ```r table2 ``` ``` ## # A tibble: 12 × 4 ## country year type count ## <chr> <dbl> <chr> <dbl> ## 1 Afghanistan 1999 cases 745 ## 2 Afghanistan 1999 population 19987071 ## 3 Afghanistan 2000 cases 2666 ## 4 Afghanistan 2000 population 20595360 ## 5 Brazil 1999 cases 37737 ## 6 Brazil 1999 population 172006362 ## 7 Brazil 2000 cases 80488 ## 8 Brazil 2000 population 174504898 ## 9 China 1999 cases 212258 ## 10 China 1999 population 1272915272 ## 11 China 2000 cases 213766 ## 12 China 2000 population 1280428583 ``` ] -- .pull-right[ "Each variable must have its own column". The current values of the `type` column are not values, but are variables names. "Each observation must have its own row". Here an observation is a country in a year, which is correct, but each observation is spread across two rows. ] --- ## Pivot wider https://tidyr.tidyverse.org/reference/pivot_wider.html .pull-left[ ```r table2 ``` ``` ## # A tibble: 12 x 4 ## country year type count ## <chr> <int> <chr> <int> ## 1 Afghanistan 1999 cases 745 ## 2 Afghanistan 1999 population 19987071 ## 3 Afghanistan 2000 cases 2666 ## 4 Afghanistan 2000 population 20595360 ## 5 Brazil 1999 cases 37737 ## 6 Brazil 1999 population 172006362 ## 7 Brazil 2000 cases 80488 ## 8 Brazil 2000 population 174504898 ## 9 China 1999 cases 212258 ## 10 China 1999 population 1272915272 ## 11 China 2000 cases 213766 ## 12 China 2000 population 1280428583 ``` ] -- .pull-right[ ```r pivot_wider( data = table2, names_from = type, values_from = count ) ``` ``` ## # A tibble: 6 x 4 ## country year cases population ## <chr> <int> <int> <int> ## 1 Afghanistan 1999 745 19987071 ## 2 Afghanistan 2000 2666 20595360 ## 3 Brazil 1999 37737 172006362 ## 4 Brazil 2000 80488 174504898 ## 5 China 1999 212258 1272915272 ## 6 China 2000 213766 1280428583 ``` ] --- ## Separating Why is this dataset untidy? How can we tidy it? ```r table3 ``` ``` ## # A tibble: 6 × 3 ## country year rate ## <chr> <dbl> <chr> ## 1 Afghanistan 1999 745/19987071 ## 2 Afghanistan 2000 2666/20595360 ## 3 Brazil 1999 37737/172006362 ## 4 Brazil 2000 80488/174504898 ## 5 China 1999 212258/1272915272 ## 6 China 2000 213766/1280428583 ``` --- ## Separating https://tidyr.tidyverse.org/reference/separate.html .pull-left[ ```r table3 ``` ``` ## # A tibble: 6 x 3 ## country year rate ## * <chr> <int> <chr> ## 1 Afghanistan 1999 745/19987071 ## 2 Afghanistan 2000 2666/20595360 ## 3 Brazil 1999 37737/172006362 ## 4 Brazil 2000 80488/174504898 ## 5 China 1999 212258/1272915272 ## 6 China 2000 213766/1280428583 ``` ] -- .pull-right[ ```r separate( data = table3, col = rate, into = c( "cases", "population" ), convert = TRUE ) ``` ``` ## # A tibble: 6 x 4 ## country year cases population ## <chr> <int> <int> <int> ## 1 Afghanistan 1999 745 19987071 ## 2 Afghanistan 2000 2666 20595360 ## 3 Brazil 1999 37737 172006362 ## 4 Brazil 2000 80488 174504898 ## 5 China 1999 212258 1272915272 ## 6 China 2000 213766 1280428583 ``` ] <!-- convert = TRUE bcs the default is FALSE the original data are coded as chr but we want them as integer; if you do not convert it will leave them as chr --> --- ## Uniting Why is this dataset untidy? How can we tidy it? ```r table5 ``` ``` ## # A tibble: 6 × 4 ## country century year rate ## <chr> <chr> <chr> <chr> ## 1 Afghanistan 19 99 745/19987071 ## 2 Afghanistan 20 00 2666/20595360 ## 3 Brazil 19 99 37737/172006362 ## 4 Brazil 20 00 80488/174504898 ## 5 China 19 99 212258/1272915272 ## 6 China 20 00 213766/1280428583 ``` --- ## Uniting https://tidyr.tidyverse.org/reference/unite.html .pull-left[ ```r table5 ``` ``` ## # A tibble: 6 x 4 ## country century year rate ## * <chr> <chr> <chr> <chr> ## 1 Afghanistan 19 99 745/19987071 ## 2 Afghanistan 20 00 2666/20595360 ## 3 Brazil 19 99 37737/172006362 ## 4 Brazil 20 00 80488/174504898 ## 5 China 19 99 212258/1272915272 ## 6 China 20 00 213766/1280428583 ``` ] -- .pull-right[ ```r unite( data = table5, col = "year", century, year ) ``` ``` ## # A tibble: 6 × 3 ## country year rate ## <chr> <chr> <chr> ## 1 Afghanistan 19_99 745/19987071 ## 2 Afghanistan 20_00 2666/20595360 ## 3 Brazil 19_99 37737/172006362 ## 4 Brazil 20_00 80488/174504898 ## 5 China 19_99 212258/1272915272 ## 6 China 20_00 213766/1280428583 ``` ] <!-- Unite multiple columns into one by pasting strings together --> --- ## Uniting .pull-left[ ```r table5 ``` ``` ## # A tibble: 6 x 4 ## country century year rate ## * <chr> <chr> <chr> <chr> ## 1 Afghanistan 19 99 745/19987071 ## 2 Afghanistan 20 00 2666/20595360 ## 3 Brazil 19 99 37737/172006362 ## 4 Brazil 20 00 80488/174504898 ## 5 China 19 99 212258/1272915272 ## 6 China 20 00 213766/1280428583 ``` ] .pull-right[ ```r unite( data = table5, col = "year", century, year, # remove underscore sep = "" ) ``` ``` ## # A tibble: 6 × 3 ## country year rate ## <chr> <chr> <chr> ## 1 Afghanistan 1999 745/19987071 ## 2 Afghanistan 2000 2666/20595360 ## 3 Brazil 1999 37737/172006362 ## 4 Brazil 2000 80488/174504898 ## 5 China 1999 212258/1272915272 ## 6 China 2000 213766/1280428583 ``` ] --- ## Uniting .pull-left[ ```r table5 ``` ``` ## # A tibble: 6 × 4 ## country century year rate ## <chr> <chr> <chr> <chr> ## 1 Afghanistan 19 99 745/19987071 ## 2 Afghanistan 20 00 2666/20595360 ## 3 Brazil 19 99 37737/172006362 ## 4 Brazil 20 00 80488/174504898 ## 5 China 19 99 212258/1272915272 ## 6 China 20 00 213766/1280428583 ``` ] .pull-right[ ```r unite( data = table5, col = "year", century, year, # remove underscore sep = "" ) %>% # store as numeric mutate(year = as.integer(year)) ``` ``` ## # A tibble: 6 × 3 ## country year rate ## <chr> <int> <chr> ## 1 Afghanistan 1999 745/19987071 ## 2 Afghanistan 2000 2666/20595360 ## 3 Brazil 1999 37737/172006362 ## 4 Brazil 2000 80488/174504898 ## 5 China 1999 212258/1272915272 ## 6 China 2000 213766/1280428583 ``` ] <!-- # Let's get messy! <img src="https://media.giphy.com/media/fCUCbWXe9JONVsJSUd/giphy.gif" width="50%" style="display: block; margin: auto;" /> --> --- class: inverse, middle # Practice tidying data Download today's in-class exercises from the website. <!-- ## Acknowledgments The content of these slides is derived in part from Benjamin Soltoff’s “Computing for the Social Sciences” course materials, licensed under the CC BY NC 4.0 Creative Commons License. Any errors or oversights are mine alone. <img src="index_files/figure-html/compare-speed-small-plot-1.png" width="80%" style="display: block; margin: auto;" /> ## `readr` vs. base R <img src="index_files/figure-html/compare-speed-large-plot-1.png" width="80%" style="display: block; margin: auto;" /> -->