Fast and Memory-Efficient Base R Table Manipulation

A tabular data manipulation, exploration and validation toolkit with a base R-style interface (subset, transform, aggregate, merge, split) and no external computation dependency. Grouping, joins, ordering, filtering, reshaping and delimited-file reading run in a bundled 'C++' engine that uses multiple threads for the heavier operations. Grouped reducers accumulate in compiled code without materialising intermediate columns, so grouped aggregation and counting allocate close to nothing. Results are returned as an ordinary data frame with a light 'basetable' class.

basetable

basetable is a fast in-memory data-manipulation package for R with a base-R interface and no dependencies. You write subset(), transform(), aggregate(), merge(), split(), and the work runs on the package's own C++ engine. There is no data.table, no dplyr, no Arrow underneath, and nothing in Imports beyond the base and recommended packages (parallel, stats, utils).

Every verb returns a basetable: an ordinary data.frame with one extra class so it prints compactly and [ keeps the class. as.data.frame() strips it back to a plain frame.

This is a deliberately focused tool. It is aimed at

• people teaching or learning base R who want speed without the cognitive load of tidy evaluation or [i, j, by], and
• codebases already built on subset() / merge() / aggregate() that want a faster engine without a rewrite.

Design

• Base-style naming and semantics. Functions read like subset(), transform(), aggregate(), merge(), split().
• A native C++ engine. Projection, filtering, ordering, distinct, grouping, all join kinds, row-bind and subset() predicate evaluation run in compiled .Call kernels. There is no third-party compute backend.
• Explicit, standard-evaluation interfaces. Column names are strings, not captured symbols (with the marked exceptions subset() and transform() inherit from base R).
• Zero hard dependencies. data.table and dplyr appear only in Suggests, and only as competitors in the benchmark vignette.

Installation

Install the released version from CRAN:

install.packages("basetable")

Install the development version from GitHub:

# install.packages("pak")
pak::pak("ielbadisy/basetable")

Minimal examples

library(basetable)

# nested
describe(
  transform(
    subset(mtcars, cyl == 6, select = c("mpg", "hp", "wt", "cyl")),
    power = hp / wt
  )
)

# pipe
mtcars |>
  pick(c("mpg", "hp", "wt", "cyl")) |>
  transform(power = hp / wt) |>
  aggregate(by = "cyl", value = c("mpg", "power"), fun = mean)

# table-1 style summary
summarytab(
  transform(mtcars, am = factor(am, labels = c("Automatic", "Manual"))),
  vars = c("mpg", "hp"), by = "am", p_value = TRUE
)

Performance

Timing and memory below come from the bench package at 1,000,000 rows on one Linux machine (inst/benchmarks/make-readme-figures.R regenerates the figures; the Benchmarks vignette has the full reproducible report). basetable is compared with data.table and dplyr.

Speed

Memory

(equi join pins data.table to sort = FALSE, matching basetable::merge(), which returns rows in input order.)

basetable is faster than data.table on filter, distinct, grouped count, sd by group and semi join, and is level with it on equi join. Against dplyr it is faster on every operation here, by more than 30x on high-cardinality count. The one operation it loses is string sort.

Memory, ranked by advantage

basetable allocates the least (or tied least) on every operation measured. The size of the edge splits in two: overwhelming on grouped reductions, where the result is tiny and nothing intermediate is materialised in R; modest on operations that return a full table, where the output frame itself sets a floor.

These are R-level allocations as reported by bench. The C++ engine also uses malloc'd scratch buffers (radix keys, per-thread row-position vectors) that bench does not count, so peak process memory during a sort or filter is higher than the figure above; data.table does the same.

The one gap is sorting: orderrows() is a stable parallel radix, ~20x faster than base order(), but still ~1.3x of data.table, whose hand-tuned parallel radix is the one operation basetable does not match.

Positioning

data.table is faster on some workloads (notably sorting) and has a far larger ecosystem; dplyr is the tidyverse standard. basetable is a good fit when you want:

• base-R syntax and semantics, not [i, j, by], tidy evaluation, or a method-chained frame object;
• no dependencies to install, pin, or reason about;
• a package small enough to read end to end, teach from, and hand to a language model as a stable target;
• competitive speed and best-in-class memory on the everyday operations (filter, group, join, distinct) without changing how you write code.

Grouping is a by argument on the verb that needs it (aggregate(), count(), summaries(), transform(), subset(), samplerows(), firstby(), ...), not a stateful group_by(). The group is named at the call and never persists, so there is no ungroup() to forget.

Using basetable alongside dplyr and data.table

basetable reuses base-R verb names (subset(), merge(), transform(), split(), aggregate()) on purpose. It does not ship the dplyr-coined verbs (filter(), select(), mutate(), arrange(), summarise(), distinct(), glimpse(), ...), so it can be attached next to dplyr without shadowing its grammar. The two names it shares with dplyr are count() and pick(), kept because they read as base-style verbs; with both packages attached, whichever was attached last wins for those (and for the base-R names data.table also defines). Two fixes:

• call it explicitly: basetable::transform(...);
• or conflicted::conflict_prefer("transform", "basetable") once per session.

Operation dictionary

btread() memory-maps the file and, with lazy = TRUE, returns columns as ALTREP vectors parsed on first access. aggregate(), count(), uniquerows() and freq() accept a single file path as their first argument and fuse the parse with the grouping, so unused columns are never materialised.

Status

Every exported function has direct test coverage. Vignettes cover getting started, data manipulation, exploration, a complete function reference, and benchmarks. CI checks release R on Linux, macOS and Windows plus oldrel and devel.