Tools to help to create tidy data, where each column is a variable, each row is an observation, and each cell contains a single value. 'tidyr' contains tools for changing the shape (pivoting) and hierarchy (nesting and 'unnesting') of a dataset, turning deeply nested lists into rectangular data frames ('rectangling'), and extracting values out of string columns. It also includes tools for working with missing values (both implicit and explicit).
The goal of tidyr is to help you create tidy data. Tidy data is data where:
Tidy data describes a standard way of storing data that is used wherever possible throughout the tidyverse. If you ensure that your data is tidy, you’ll spend less time fighting with the tools and more time working on your analysis.
# The easiest way to get tidyr is to install the whole tidyverse:install.packages("tidyverse")# Alternatively, install just tidyr:install.packages("tidyr")# Or the development version from GitHub:# install.packages("devtools")devtools::install_github("tidyverse/tidyr")
There are two fundamental verbs of data tidying:
gather() takes multiple columns, and gathers them into key-value
pairs: it makes “wide” data longer.
spread() takes two columns (key & value), and spreads into
multiple columns: it makes “long” data wider.
tidyr also provides
extract() functions which makes
it easier to pull apart a column that represents multiple variables. The
To get started, read the tidy data vignette (
and check out the demos (
demo(package = "tidyr")).
tidyr replaces reshape2 (2010-2014) and reshape (2005-2010). Somewhat counterintuitively each iteration of the package has done less. tidyr is designed specifically for tidying data, not general reshaping (reshape2), or the general aggregation (reshape).
If you’d like to read more about data reshaping from a CS perspective, I’d recommend the following three papers:
An interactive framework for data cleaning (Potter’s wheel)
To guide your reading, here’s a translation between the terminology used in different places:
Please note that the tidyr project is released with a Contributor Code of Conduct. By contributing to this project, you agree to abide by its terms.
crossing() preserves factor levels (#410), now works with list-columns
(#446, @SamanthaToet). (These also help
expand() which is built on top
nest() is compatible with dplyr 0.8.0.
spread() works when the id variable has names (#525).
unnest() preserves column being unnested when input is zero-length (#483),
list_of() attribute to correctly restore columns, where possible.
unnest() will run with named and unnamed list-columns of same length
separate() now accepts
NA as a column name in the
into argument to
denote columns which are omitted from the result. (@markdly, #397).
Minor updates to ensure compatibility with dependencies.
unnest()weakens test of "atomicity" to restore previous behaviour when unnesting factors and dates (#407).
There are no deliberate breaking changes in this release. However, a number of packages are failing with errors related to numbers of elements in columns, and row names. It is possible that these are accidental API changes or new bugs. If you see such an error in your package, I would sincerely appreciate a minimal reprex.
separate() now correctly uses -1 to refer to the far right position,
instead of -2. If you depended on this behaviour, you'll need to switch
packageVersion("tidyr") > "0.7.2"
Increased test coverage from 84% to 99%.
uncount() performs the inverse operation of
complete(data) now returns
data rather than throwing an error (#390).
complete() with zero-length completions returns original input (#331).
expand() with empty input gives empty data frame instead of
complete() now complete empty factors instead
of dropping them (#270, #285)
extract() has a better error message if
regex does not contain the
expected number of groups (#313).
drop_na() no longer drops columns (@jennybryan, #245), and works with
list-cols (#280). Equivalent of
NA in a list column is any empty
(length 0) data structure.
nest() is now faster, especially when a long data frame is collapsed into
a nested data frame with few rows.
nest() on a zero-row data frame works as expected (#320).
replace_na() no longer complains if you try and replace missing values in
variables not present in the data (#356).
replace_na() now also works with vectors (#342, @flying-sheep), and
NULL in list-columns. It throws a better error message if
you attempt to replace with something other than length 1.
separate() no longer checks that
... is empty, allowing methods to make
use of it. This check was added in tidyr 0.4.0 (2016-02-02) to deprecate
previous behaviour where
... was passed to
extract() now insert columns in correct position when
drop = TRUE (#394).
separate() now works correctly counts from RHS when using negative
sep values (@markdly, #315).
separate() gets improved warning message when pieces aren't as expected
separate_rows() supports list columns (#321), and works with empty tibbles.
spread() now consistently returns 0 row outputs for 0 row inputs (#269).
spread() now works when
key column includes
spread() no longer returns tibbles with row names (#322).
extract() (#255), and
gather() (#347) now
replace existing variables rather than creating an invalid data frame with
duplicated variable names (matching the semantics of mutate).
unite() now works (as documented) if you don't supply any variables (#355).
preserve argument which allows you to preserve list
columns without unnesting them (#328).
unnest() can unnested list-columns contains lists of lists (#278).
unnest(df) now works if
df contains no list-cols (#344)
The SE variants
treat non-syntactic names in the same way as pre tidy eval versions
of tidyr (#361).
Fix tidyr bug revealed by R-devel.
This is a hotfix release to account for some tidyselect changes in the unit tests.
Note that the upcoming version of tidyselect backtracks on some of the
changes announced for 0.7.0. The special evaluation semantics for
selection have been changed back to the old behaviour because the new
rules were causing too much trouble and confusion. From now on data
expressions (symbols and calls to
c()) can refer to both
registered variables and to objects from the context.
However the semantics for context expressions (any calls other than to
c()) remain the same. Those expressions are evaluated in the
context only and cannot refer to registered variables. If you're
writing functions and refer to contextual objects, it is still a good
idea to avoid data expressions by following the advice of the 0.7.0
This release includes important changes to tidyr internals. Tidyr now supports the new tidy evaluation framework for quoting (NSE) functions. It also uses the new tidyselect package as selecting backend.
If you see error messages about objects or functions not found, it
is likely because the selecting functions are now stricter in their
arguments An example of selecting function is
gather() and its
... argument. This change makes the code more robust by
disallowing ambiguous scoping. Consider the following code:
x <- 3 df <- tibble(w = 1, x = 2, y = 3) gather(df, "variable", "value", 1:x)
Does it select the first three columns (using the
x defined in the
global environment), or does it select the first two columns (using
the column named
To solve this ambiguity, we now make a strict distinction between
data and context expressions. A data expression is either a bare
name or an expression like
c(x, y). In a data expression,
you can only refer to columns from the data frame. Everything else
is a context expression in which you can only refer to objects that
you have defined with
In practice this means that you can no longer refer to contextual objects like this:
mtcars %>% gather(var, value, 1:ncol(mtcars)) x <- 3 mtcars %>% gather(var, value, 1:x) mtcars %>% gather(var, value, -(1:x))
You now have to be explicit about where to find objects. To do so,
you can use the quasiquotation operator
!! which will evaluate its
argument early and inline the result:
mtcars %>% gather(var, value, !! 1:ncol(mtcars))mtcars %>% gather(var, value, !! 1:x)mtcars %>% gather(var, value, !! -(1:x))
An alternative is to turn your data expression into a context
expression by using
seq_len() instead of
:. See the
section on tidyselect for more information about these semantics.
Following the switch to tidy evaluation, you might see warnings
about the "variable context not set". This is most likely caused by
supplyng helpers like
everything() to underscored versions of
tidyr verbs. Helpers should be always be evaluated lazily. To fix
this, just quote the helper with a formula:
The selecting functions are now stricter when you supply integer positions. If you see an error along the lines of
`-0.949999999999999`, `-0.940000000000001`, ... must resolve to integer column positions, not a double vector
please round the positions before supplying them to tidyr. Double vectors are fine as long as they are rounded.
tidyr is now a tidy evaluation grammar. See the programming vignette in dplyr for practical information about tidy evaluation.
The tidyr port is a bit special. While the philosophy of tidy
evaluation is that R code should refer to real objects (from the data
frame or from the context), we had to make some exceptions to this
rule for tidyr. The reason is that several functions accept bare
symbols to specify the names of new columns to create (
being a prime example). This is not tidy because the symbol do not
represent any actual object. Our workaround is to capture these
rlang::quo_name() (so they still support
quasiquotation and you can unquote symbols or strings). This type of
NSE is now discouraged in the tidyverse: symbols in R code should
represent real objects.
Following the switch to tidy eval the underscored variants are softly deprecated. However they will remain around for some time and without warning for backward compatibility.
The selecting backend of dplyr has been extracted in a standalone
package tidyselect which tidyr now uses for selecting variables. It is
used for selecting multiple variables (in
drop_na()) as well as
single variables (the
col argument of
value arguments of
spread()). This implies the
The arguments for selecting a single variable now support all
dplyr::pull(). You can supply a name or a position,
including negative positions.
Multiple variables are now selected a bit differently. We now make a
strict distinction between data and context expressions. A data
expression is either a bare name of an expression like
c(x, y). In a data expression, you can only refer to columns from
the data frame. Everything else is a context expression in which you
can only refer to objects that you have defined with
You can still refer to contextual objects in a data expression by
being explicit. One way of being explicit is to unquote a variable
from the environment with the tidy eval operator
x <- 2drop_na(df, 2) # Works finedrop_na(df, x) # Object 'x' not founddrop_na(df, !! x) # Works as if you had supplied 2
On the other hand, select helpers like
start_with() are context
expressions. It is therefore easy to refer to objects and they will
never be ambiguous with data columns:
x <- "d"drop_na(df, starts_with(x))
While these special rules is in contrast to most dplyr and tidyr verbs (where both the data and the context are in scope) they make sense for selecting functions and should provide more robust and helpful semantics.
Register C functions
Added package docs
Patch tests to be compatible with dev dplyr.
Patch test to be compatible with dev tibble
Changed deprecation message of
extract_numeric() to point to
readr::parse_number() rather than
drop_na() removes observations which have
NA in the given variables. If no
variables are given, all variables are considered (#194, @janschulz).
extract_numeric() has been deprecated (#213).
table4b to make their
connection more clear. The
value variables in
been renamed to
nesting() now silently drop zero-length
nesting_() are versions of
that take a list as input.
full_seq() works correctly for dates and date/times.
getS3method(envir = )(#205, @krlmlr).
separate_rows()separates observations with multiple delimited values into separate rows (#69, @aaronwolen).
complete() preserves grouping created by dplyr (#168).
expand() (and hence
complete()) preserves the ordered attribute of
full_seq() preserve attributes for dates and date/times (#156),
and sequences no longer need to start at 0.
gather() can now gather together list columns (#175), and
gather_.data.frame(na.rm = TRUE) now only removes missing values
if they're actually present (#173).
nest() returns correct output if every variable is nested (#186).
separate() fills from right-to-left (not left-to-right!) when fill = "left"
unite() now automatically drop removed variables from
grouping (#159, #177).
spread() gains a
sep argument. If not-null, this will name columns
as "keyvalue". Additionally, if sep is
NULL missing values will be
spread() works in the presence of list-columns (#199)
unnest() works with non-syntactic names (#190).
unnest() gains a
sep argument. If non-null, this will rename the
columns of nested data frames to include both the original column name,
and the nested column name, separated by
.id argument that works the same way as
This is useful if you have a named list of data frames or vectors (#125).
Moved in useful sample datasets from the DSR package.
Made compatible with both dplyr 0.4 and 0.5.
tidyr functions that create new columns are more aggresive about re-encoding the column names as UTF-8.
nest()where nested data was ending up in the wrong row (#158).
unnest() have been overhauled to support a useful way of structuring data frames: the nested data frame. In a grouped data frame, you have one row per observation, and additional metadata define the groups. In a nested data frame, you have one row per group, and the individual observations are stored in a column that is a list of data frames. This is a useful structure when you have lists of other objects (like models) with one element per group.
nest() now produces a single list of data frames called "data" rather
than a list column for each variable. Nesting variables are not included
in nested data frames. It also works with grouped data frames made
dplyr::group_by(). You can override the default column name with
unnest() gains a
.drop argument which controls what happens to
other list columns. By default, they're kept if the output doesn't require
row duplication; otherwise they're dropped.
unnest() now has
mutate() semantics for
... - this allows you to
unnest transformed columns more easily. (Previously it used select semantics).
expand() once again allows you to evaluate arbitrary expressions like
full_seq(year). If you were previously using
c() to created nested
combinations, you'll now need to use
nesting() (#85, #121).
crossing() allow you to create nested and crossed data
frames from individual vectors.
crossing() is similar to
full_seq(x, period) creates the full sequence of values from
fill() fills in
NULLs in list-columns.
fill() gains a direction argument so that it can fill either upwards or
gather() now stores the key column as character, by default. To revert to
the previous behaviour of using a factor (which allows you to preserve the
ordering of the columns), use
key_factor = TRUE (#96).
All tidyr verbs do the right thing for grouped data frames created by
group_by() (#122, #129, #81).
seq_range() has been removed. It was never used or announced.
spread() once again creates columns of mixed type when
convert = TRUE
drop = FALSE handles zero-length
spread()ing a data frame with only key and value columns
creates a one row output (#41).
unite() now removes old columns before adding new (#89, @krlmlr).
separate() now warns if defunct ... argument is used (#151, @krlmlr).
complete() provides a wrapper around
replace_na() for a common task: completing a data frame with missing
combinations of variables.
fill() fills in missing values in a column with the last non-missing
replace_na() makes it easy to replace missing values with something
meaningful for your data.
nest() is the complement of
unnest() can now work with multiple list-columns at the same time.
If you don't supply any columns names, it will unlist all
unnest() can also handle columns that are
lists of data frames (#58).
tidyr no longer depends on reshape2. This should fix issues if you also try to load reshape (#88).
%>% is re-exported from magrittr.
expand() now supports nesting and crossing (see examples for details).
This comes at the expense of creating new variables inline (#46).
expand_ does SE evaluation correctly so you can pass it a character vector
of columns names (or list of formulas etc) (#70).
extract() is 10x faster because it now uses stringi instead of
base R regular expressions. It also returns NA instead of throwing
an error if the regular expression doesn't match (#72).
separate() preserve character vectors when
convert is TRUE (#99).
The internals of
spread() have been rewritten, and now preserve all
attributes of the input
value column. This means that you can now
spread date (#62) and factor (#35) inputs.
spread() gives a more informative error message if
exist in the input data (#36).
separate() only displays the first 20 failures (#50). It has
finer control over what happens if there are two few matches:
you can fill with missing values on either the "left" or the "right" (#49).
separate() no longer throws an error if the number of pieces aren't
as expected - instead it uses drops extra values and fills on the right
and gives a warning.
If the input is NA
extract() both return silently
return NA outputs, rather than throwing an error. (#77)
unnest() method for lists has been removed.
expand() function (#21).
unnest() function for converting named lists into
data frames. (#3, #22)
extract_numeric() preserves negative signs (#20).
gather() has better defaults if
value are not supplied.
... is ommitted,
gather() selects all columns (#28). Performance
is now comparable to
extra argument which lets you control what happens
to extra pieces. The default is to throw an "error", but you can also
"merge" or "drop".
drop argument, which allows you to preserve missing
factor levels (#25). It converts factor value variables to character vectors,
instead of embedding a matrix inside the data frame (#35).