A 'Future' API for Parallel Processing using 'mirai'

Implementation of the 'Future' API on top of the 'mirai' package . By using this package, you get to take advantage of the benefits of 'mirai' plus everything else that 'future' and the 'Futureverse' adds on top of it. It allows you to process futures, as defined by the 'future' package, in parallel out of the box, on your local machine or across remote machines. Contrary to back-ends relying on the 'parallel' package (e.g. 'multisession') and socket connections, 'mirai_cluster' and 'mirai_multisession', provided here, can run more than 125 parallel R processes. As a reminder, regardless which future backend is used by the user, the code does not have to change, it gives identical results, and behaves exactly the same.

future.mirai: A 'Future' API for Parallel Processing using 'mirai'

Introduction

The future package provides a generic API for using futures in R. A future is a simple yet powerful mechanism to evaluate an R expression and retrieve its value at some point in time. Futures can be resolved in many different ways depending on which strategy is used. There are various types of synchronous and asynchronous futures to choose from in the future package.

This package, future.mirai, provides a type of futures that utilizes the mirai package.

For example,

> library(future)
> plan(future.mirai::mirai_multisession)
>
> x %<-% { Sys.sleep(5); 3.14 }
> y %<-% { Sys.sleep(5); 2.71 }
> x + y
[1] 5.85

This is obviously a toy example to illustrate what futures look like and how to work with them. For further examples on how to use futures, see the vignettes of the future package as well as those of future.apply, furrr, and doFuture.

Using the future.mirai backend

The future.mirai package implements a future backend wrapper for mirai.

Advantages of mirai futures

The mirai package provides a low-level future-like mechanism for evaluating R expression in separate R processes running on the local machine or on one or more remote machines. Central to mirai is its highly-optimized queueing mechanism, which is used to orchestrate communication between the main R process and parallel workers. A mirai cluster of workers can be configured to communicate securely via the well-established Transport Layer Security (TLS) protocol.

Another advantage with mirai_* futures, compared to multisession and cluster futures, is that we can use more than 125 parallel workers. The current limit of 125 workers for multisession and cluster futures stems from how the underlying parallel package uses one R connection per parallel worker and R has a limit of 125 R connections per session. In R (>= 4.4.0), we can increase this limit when we launch R, e.g. R --max-connections=200. For R (< 4.4.0), R has to be rebuilt from source after adjusting the source code. The mirai package does not rely on R connections for parallel workers and therefore does not suffer from this limit.

Demos

The future package provides a demo using futures for calculating a set of Mandelbrot planes. The demo does not assume anything about what type of futures are used. The user has full control of how futures are evaluated. For instance, to use mirai_multisession futures, run the demo as:

library(future)
plan(future.mirai::mirai_multisession)

demo("mandelbrot", package = "future", ask = FALSE)

and

library(future)
mirai::daemons(2)
plan(future.mirai::mirai_cluster)

demo("mandelbrot", package = "future", ask = FALSE)

Installation

R package future.mirai is available on CRAN and can be installed in R as:

install.packages("future.mirai")

Pre-release version

To install the pre-release version that is available in Git branch develop on GitHub, use:

remotes::install_github("futureverse/future.mirai", ref="develop")

This will install the package from source.