Submit R Calculations to a Slurm Cluster

Functions that simplify submitting R scripts to a Slurm workload manager, in part by automating the division of embarrassingly parallel calculations across cluster nodes.


Pre-release: Build Status

CRAN checks: rslurm results

About

Development of this R package was supported by the National Socio-Environmental Synthesis Center (SESYNC) under funding received from the National Science Foundation DBI-1052875.

The package was developed by Philippe Marchand, with Ian Carroll (current maintainer) and Mike Smorul contributing.

Installation

Install the package from R with install.packages('rslurm'). Note that job submission is only possible on a system with access to a Slurm workload manager (i.e. a system where the command line utilities squeue or sinfo return information from a Slurm head node).

Documentation

Package documentation is accessible from the R console through package?rslurm and online.

News

rslurm 0.3.1

2016-06-18

  • Minor bug fix: specify full path of 'Rscript' when running batch scripts.

rslurm 0.3.0

2016-05-27

First version on CRAN

Major update to the package interface and implementation:

  • Added a submit argument to slurm_apply and slurm_call. If submit = FALSE, the submission scripts are created but not run. This is useful if the files need to be transferred from a local machine to the cluster and run at a later time.

  • Added new optional arguments to slurm_apply and slurm_call, allowing users to give informative names to SLURM jobs (jobname) and set any options understood by sbatch (slurm_options).

  • The data_file arugment to slurm_apply and slurm_call is replaced with add_objects, which accepts a vector of R object names from the active workspace and automatically saves them in a .RData file to be loaded on each node.

  • slurm_apply and slurm_call now generate R and Bash scripts through whisker templates. Advanced users may want to edit those templates in the templates folder of the installed R package (e.g. to set default SBATCH options in submit.sh).

  • Files generated by the package (scripts, data files and output) are now saved in a subfolder named _rslurm_[jobname] in the current working directory.

  • Minor updates, including reformatting the output of print_job_status and removing this package's dependency on stringr.

rslurm 0.2.0

2015-11-23

  • Changed the slurm_apply function to use parallel::mcMap instead of mcmapply, which fixes a bug where list outputs (i.e. each function call returns a list) would be collapsed in a single list (rather than returned as a list of lists).

  • Changed the interface so that the output type (table or raw) is now an argument of get_slurm_out rather than of slurm_apply, and defaults to raw.

  • Added cpus_per_node argument to slurm_apply, indicating the number of parallel processes to be run on each node.

rslurm 0.1.3

2015-07-13

  • Added the slurm_call function, which submits a single function evaluation on the cluster, with syntax similar to the base function do.call.

  • get_slurm_out can now process the output even if some filese are missing, in which case it issues a warning.

rslurm 0.1.2

2015-06-29

  • Added the optional argument pkgs to slurm_apply, indicating which packages should be loaded on each node (by default, all packages currently attached to the user's R session).

rslurm 0.1.1

2015-06-24

  • Added the optional argument output to slurm_apply, which can take the value table (each function evaluation returns a row, output is a data frame) or raw (each function evaluation returns an arbitrary R object, output is a list).

  • Fixed a bug in the chunk size calculation for slurm_apply.

rslurm 0.1.0

2015-06-16

  • First version of the package released on Github.

Reference manual

It appears you don't have a PDF plugin for this browser. You can click here to download the reference manual.

install.packages("rslurm")

0.4.0 by Ian Carroll, 7 months ago


https://github.com/SESYNC-ci/rslurm


Report a bug at https://github.com/SESYNC-ci/rslurm/issues


Browse source code at https://github.com/cran/rslurm


Authors: Philippe Marchand [aut], Mike Smorul [ctb], Ian Carroll [ctb, cre]


Documentation:   PDF Manual  


Task views: High-Performance and Parallel Computing with R


GPL-3 license


Imports whisker

Suggests parallel, testthat, knitr, rmarkdown


See at CRAN