A 'dplyr' Interface for Crunch

In order to facilitate analysis of datasets hosted on the Crunch data platform < http://crunch.io/>, the 'crplyr' package implements 'dplyr' methods on top of the Crunch backend. The usual methods 'select', 'filter', 'group_by', 'summarize', and 'collect' are implemented in such a way as to perform as much computation on the server and pull as little data locally as possible.


Build Status codecov Build status cran

dplyr defines "a grammar of data manipulation" popular among R users. In order to facilitate analysis of datasets hosted by Crunch, this package implements 'dplyr' methods on top of the Crunch backend. The usual methods "select", "filter", "mutate", "group_by", and "summarize" are implemented in such a way as to perform as much computation on the server and pull as little data locally as possible.

With a local data.frame, you might chain together a series of manipulations and create a table, such as: > data(mtcars) > mtcars %>% filter(vs == 1) %>% group_by(gear) %>% summarize(horses=mean(hp), sd_horses=sd(hp), count=n())

## # A tibble: 3 × 4
##    gear horses sd_horses count
##   <dbl>  <dbl>     <dbl> <int>
## 1     3  104.0  6.557439     3
## 2     4   85.4 26.596575    10
## 3     5  113.0        NA     1

With crplyr, you can do the same operations, except that the dataset you're working with sits in the Crunch platform, and Crunch is doing the aggregations in the cloud:

> library(crplyr)
> login()
[crunch] > mtcars <- loadDataset("mtcars from R")
[crunch] > mtcars %>%
    filter(vs == 1) %>%
    group_by(gear) %>%
    summarize(horses=mean(hp), sd_horses=sd(hp), count=n())

## # A tibble: 3 × 4
##    gear horses sd_horses count
##  <fctr>  <dbl>     <dbl> <dbl>
## 1     3  104.0  6.557439     3
## 2     4   85.4 26.596575    10
## 3     5  113.0        NA     1

Obviously, the fact that the calculations in crplyr are happening remotely doesn't matter as much when working with a tiny dataset like "mtcars", but Crunch allows you to work with datasets larger than can fit in memory on your machine, and it enables you to collaborate naturally with others on the same dataset.

Installing

Install the CRAN release of crplyr with

install.packages("crplyr")

The pre-release version of the package can be pulled from GitHub using the devtools package:

# install.packages("devtools")
devtools::install_github("Crunch-io/crplyr")

For developers

The repository includes a Makefile to facilitate some common tasks.

Running tests

$ make test. Requires the httptest package. You can also specify a specific test file or files to run by adding a "file=" argument, like $ make test file=select. test_package will do a regular-expression pattern match within the file names. See its documentation in the testthat package.

Updating documentation

$ make doc. Requires the roxygen2 package.

News

crplyr 0.2.0

  • Added collect method, which pulls the requested columns of data from the server.
  • summarize() and as_tibble.CrunchCube() methods now better handle array and multiple-response data and include metadata on which dimension values should be interpreted as missing.
  • unweighted_n() aggregation method for summarize(), which returns the unweighted counts even when the dataset has a weight applied.

crplyr 0.1.4

  • Update tests to match crunch 1.18.0.

crplyr 0.1.2

  • Update tests to use new setup provided in crunch 1.17.0.

crplyr 0.1.0

Initial implementation of dplyr interface for Crunch, including

  • select(), filter(), group_by(), and summarize() methods for crunch::CrunchDataset() objects
  • as_tibble() method for crunch::CrunchCube() objects

Reference manual

It appears you don't have a PDF plugin for this browser. You can click here to download the reference manual.

install.packages("crplyr")

0.3.1 by Mike Malecki, 4 months ago


https://crunch.io/r/crplyr/, https://github.com/Crunch-io/crplyr


Report a bug at https://github.com/Crunch-io/crplyr/issues


Browse source code at https://github.com/cran/crplyr


Authors: Neal Richardson [aut] , Jonathan Keane [aut] , Gordon Shotwell [aut] , Mike Malecki [aut, cre]


Documentation:   PDF Manual  


Task views:


LGPL (>= 3) license


Imports methods, ggplot2, lazyeval, purrr, tibble, rlang, scales, stringr, viridisLite

Depends on crunch, dplyr

Suggests covr, httptest, magrittr, spelling, vdiffr, knitr, testthat, rmarkdown


See at CRAN