Thread-Speed Async and Parallel R over Shared Memory

Parallel computation and data exchange between R processes on the same machine: lock-free channels, work-stealing task pools, and a parallel map over 'POSIX' shared memory (Linux, macOS) or 'Win32' file mappings (Windows), with the hot path entirely in user space. Communication is cheap enough that work can be divided as finely as with threads. Vectors and data frames transfer zero-copy, received as 'ALTREP' views over the shared pages, and 'Apache Arrow' tables and record batches travel as a single frame with zero-copy conversion in both directions. Receives and task results integrate with 'later' and 'promises' for non-blocking, event-driven applications such as 'shiny'. Channels, pools, and maps interoperate with 'pymizu', so a peer or worker can equally be a Python process. It complements 'mirai', and its payloads interoperate with 'mori' shared objects.

mizu

Thread-speed async and parallel R over shared memory.

A task round trip drops from ~100 µs over sockets to < 1 µs, so you can divide work as finely as you would with threads (see Benchmarks). Common data structures like vectors, matrices, data frames, and Arrow tables can be shared between R and Python as zero-copy views.

Channels and work-stealing task pools run over POSIX shared memory (Linux, macOS) or Win32 file mappings (Windows): lock-free single-producer single-consumer rings with batched publication, spin-then-park waiting that stays out of the kernel on the hot path, and OS notification when a peer dies.

mizu is built on libmizu, a C library for lock-free shared-memory IPC. pymizu binds the same core for Python, so R and Python can talk to each other.

• Zero-copy vectors and data frames — received as ALTREP views over the shared pages
• Zero-copy Arrow interchange — arrow tables and record batches travel as one frame and convert back without copying
• Event-driven async — messages stream into a later event loop without polling, and channels, tasks, and maps plug into promises pipelines, ready for Shiny
• Reproducible parallel random numbers (.seed) — identical results for any worker count or steal order
• Nested parallel maps — a task can map over its own pool, deadlock-free

Lifecycle: experimental. The API is not yet stable and we may make changes in response to feedback.

Installation

Install from CRAN:

install.packages("mizu")

Or the development version from GitHub:

pak::pak("shikokuchuo/mizu")

Requires R 4.3 or later.

Quickstart

library(mizu)

p <- mizu_pool(n_workers = 4L)

t <- mizu_submit(p, sum(x) + y, x = 1:10, y = 100)
mizu_collect(t)
#> [1] 155

mizu_pool_stop(p)

Benchmarks

Communication overhead against mirai on one machine (M4 Pro, from dev/bench/mizu-mirai-bench.R):

Note: mirai also serves workers on remote machines and HPC clusters, which shared memory cannot reach.

Parallel map

mizu_map() maps a function over a vector or list on the pool and returns a map handle immediately, just as mizu_submit() returns a task. mizu_collect() on the handle waits and returns the results in order. The function and data cross into shared memory once. Workers divide the elements among themselves, claiming ranges from a shared cursor.

A map also nests: the mapped function can map over its own pool with mizu_map(mizu_current_pool(), ...), deadlock-free.

p <- mizu_pool(n_workers = 4L)

m <- mizu_map(p, 1:5, \(i) i * 2L, .template = integer(1))
mizu_collect(m)
#> [1]  2  4  6  8 10

parts <- split(as.numeric(1:100000), rep(1:4, each = 25000))
m <- mizu_map(p, parts, \(part) {
  nested <- mizu_map(mizu_current_pool(), part, sqrt, .template = numeric(1))
  sum(mizu_collect(nested))
}, .template = numeric(1))
mizu_collect(m)
#>       1       2       3       4 
#> 2635310 4818361 6239529 7388808

mizu_pool_stop(p)

Python interop

Pool workers can be Python processes that run pymizu, the Python binding of the same core. mizu_py_launcher() spawns them and mizu_call() specifies the task. Here, R’s built-in mtcars is summarized by polars in Python:

p <- mizu_pool(n_workers = 4L, launcher = mizu_py_launcher())

t <- mizu_submit_call(
  p,
  mizu_call(
    .source = r"(import polars as pl
pl.DataFrame(x).group_by('cyl').agg(pl.col('mpg').mean()).sort('cyl'))",
    x = mtcars
  )
)
mizu_collect(t)
#>   cyl      mpg
#> 1   4 26.66364
#> 2   6 19.74286
#> 3   8 15.10000

mizu_pool_stop(p)

• A numeric vector arrives in Python as a NumPy array, and a data frame as a pymizu.Frame
• A NumPy array, or a polars, pyarrow, or pandas frame, arrives back as a vector or data frame
• Arrow data (arrow tables and record batches, or any struct-typed nanoarrow_array_stream) crosses the same way

The pymizu README shows the reverse direction: a Python host that drives R workers with pymizu.r_launcher().

Vignettes

vignette("mizu", package = "mizu") is the main overview. Topic vignettes cover benchmarks, channels, task pools, the parallel map, Python interop, and operations (Linux memory tuning and crash semantics). The pkgdown site has the function reference and the rendered vignettes.

Please note that this project is released with a Contributor Code of Conduct. By participating in this project you agree to abide by its terms.