Simple Key Value Stores

Creates and manages simple key-value stores. These can use a variety of approaches for storing the data. This package implements the base methods and support for file system, in-memory and DBI-based database stores.

Simple object cacher for R. storr acts as a very simple key-value store (supporting get/set/del for arbitrary R objects as data). The actual storage can be transient or persistent, local or distributed without changing the interface. To allow for distributed access, data is returned by content rather than simply by key (with a key/content lookup step) so that if another process changes the data, storr will retrieve the current version.

Cached in-memory copies that might be faster to retrieve than on-disk/database copies
Content-addressable storage, storing and retrieving potentially fewer copies of identical data (useful if lookup is slow or over a network) and to make the system somewhat robust in the face of multiple accessing processes
Fetch from an external source (e.g. website) if a key is not found locally
Pluggable storage backends - currently
- environment (memory)
- rds (disk)
- DBI though which you can use:
  - SQLite (via RSQLite)
  - Postgres (via RPostgres)
- Redis (provided by redux)
- rlite (via rrlite, interface function in redux)
- An on disk memory-mapped file interface is implemented in thor

storr always goes back to the common storage (database, filesystem, whatever) for the current object to hash mapping, ensuring consistency when using multiple processes. However, when retrieving or writing the data given a hash we can often avoid accessing the underlying storage. This means that repeated lookups happen quickly while still being able to reflect changes elsewhere; time savings can be substantial where large objects are being stored.

Installation

From CRAN

install.packages("storr")

or install the development version with

remotes::install_github("richfitz/[email protected]", upgrade = FALSE)

Documentation

storr comes with two vignettes:

storr vignette("storr") outlines basic use and core implementation details.
external vignette("external") shows how to use storr to cache external resources such as files, web resources, etc, using the storr_external object.

News

storr 1.2.1 (2018-??-??)

Avoid a race condition in writing to rds storrs in parallel (#80, reported by @wlandau)

storr 1.2.0 (2018-05-31)

New index function (#22, requested by @wlandau)
Support for creating storrs with different storage for keys and data (driver_multistorr and storr_multistorr, #67)
New repair methods ($check and $repair) for detecting some problems with storr repositories. This is a work in progress and is currently only supported for the rds driver.
Support infrastructure for "remote storr" databases, which can be used to create a storr of rds objects on a remote location (#61). This does not implement any useful remote storrs, but only provides support for doing so.

storr 1.1.3 (2017-12-15)

Only use version 2 serialisation, avoiding breakage when version 3 is released in 3.5.0 (#62, reported by Tomas Kalibera)
Quote sql more safely (#60, reported by @wlandau)
Support for duplicating ($duplicate) and filling ($fill) keys with identical values, without duplicating or reserialising data (#56, requested by @wlandau)
Moved to a shiny new pkgdown website - https://richfitz.github.com/storr

storr 1.1.2 (2017-09-08)

Use the full path so storrs still work when the the working directory changes (#50, #51, @wlandau)
Avoid DBI disconnection warnings when running tests
Avoid running Redis tests in inappropriate settings (#52, #53, thanks @wlandau)
Move Redis code entirely into redux

storr 1.1.1 (2017-06-02)

Fix bug padding of base64 encoded filenames (exposed only when using vectorised interface

storr 1.1.0 (2017-05-05)

Support for using relational databases via DBI (SQLite, Postgres) #23 - MySQL and Microsoft SQL Server support is not implemented yet
Support for selecting hash algorithm at the driver level, so rather than being limited to md5, more robust algorithms can be used. This is implemented for all storr drivers, and for the rds driver will safely work on storr databases from before this version
mget/mset/mset_by_value added for vectorised get/set/set_by_value operations that reduce the number of roundtrips on some drivers. del and exists become vectorised at the same time. This requires extensive change to drivers but is backward compatible for users.
Large object support for rds is slightly improved (#25) but still very slow
Storr now supports exporting multiple namespaces at once (possibly all namespaces) making it easier migrate large amounts of data between storrs. This does cause a small breaking change where $import() now returns a matrix with columns name, namespace indicating what was imported (previously this was just a vector)
Storr import/export more gracefully handles missing keys
RDS storrs no longer pad base64 filenames with trailing with = which reduces the number of possily problematic characters. Existing storrs are unaffected (#43)
RDS storrs are not left in an inconsistent state if RDS file writing fails (#40, thanks @krlmlr)