storr
Simple Key Value Stores
Creates and manages simple key-value stores. These can use a variety of approaches for storing the data. This package implements the base methods and support for file system, in-memory and DBI-based database stores.
Simple object cacher for R. storr acts as a very simple key-value store (supporting get/set/del for arbitrary R objects as data). The actual storage can be transient or persistent, local or distributed without changing the interface. To allow for distributed access, data is returned by content rather than simply by key (with a key/content lookup step) so that if another process changes the data, storr will retrieve the current version.
- Cached in-memory copies that might be faster to retrieve than on-disk/database copies
- Content-addressable storage, storing and retrieving potentially fewer copies of identical data (useful if lookup is slow or over a network) and to make the system somewhat robust in the face of multiple accessing processes
- Fetch from an external source (e.g. website) if a key is not found locally
- Pluggable storage backends - currently
storr always goes back to the common storage (database, filesystem, whatever) for the current object to hash mapping, ensuring consistency when using multiple processes. However, when retrieving or writing the data given a hash we can often avoid accessing the underlying storage. This means that repeated lookups happen quickly while still being able to reflect changes elsewhere; time savings can be substantial where large objects are being stored.
Installation
From CRAN
install.packages("storr")or install the development version with
remotes::install_github("richfitz/[email protected]", upgrade = FALSE)
Documentation
storr comes with two vignettes:
News
storr 1.2.1 (2018-??-??)
- Avoid a race condition in writing to rds storrs in parallel (#80, reported by @wlandau)
storr 1.2.0 (2018-05-31)
- New index function (#22, requested by @wlandau)
- Support for creating storrs with different storage for keys and data (
driver_multistorrandstorr_multistorr, #67) - New repair methods (
$checkand$repair) for detecting some problems with storr repositories. This is a work in progress and is currently only supported for the rds driver. - Support infrastructure for "remote storr" databases, which can be used to create a storr of rds objects on a remote location (#61). This does not implement any useful remote storrs, but only provides support for doing so.
storr 1.1.3 (2017-12-15)
- Only use version 2 serialisation, avoiding breakage when version 3 is released in 3.5.0 (#62, reported by Tomas Kalibera)
- Quote sql more safely (#60, reported by @wlandau)
- Support for duplicating (
$duplicate) and filling ($fill) keys with identical values, without duplicating or reserialising data (#56, requested by @wlandau) - Moved to a shiny new pkgdown website - https://richfitz.github.com/storr
storr 1.1.2 (2017-09-08)
- Use the full path so
storrs still work when the the working directory changes (#50, #51, @wlandau) - Avoid DBI disconnection warnings when running tests
- Avoid running Redis tests in inappropriate settings (#52, #53, thanks @wlandau)
- Move Redis code entirely into redux
storr 1.1.1 (2017-06-02)
- Fix bug padding of base64 encoded filenames (exposed only when using vectorised interface
storr 1.1.0 (2017-05-05)
- Support for using relational databases via DBI (SQLite, Postgres) #23 - MySQL and Microsoft SQL Server support is not implemented yet
- Support for selecting hash algorithm at the driver level, so rather than being limited to md5, more robust algorithms can be used. This is implemented for all storr drivers, and for the rds driver will safely work on storr databases from before this version
mget/mset/mset_by_valueadded for vectorisedget/set/set_by_valueoperations that reduce the number of roundtrips on some drivers.delandexistsbecome vectorised at the same time. This requires extensive change to drivers but is backward compatible for users.- Large object support for rds is slightly improved (#25) but still very slow
- Storr now supports exporting multiple namespaces at once (possibly all namespaces) making it easier migrate large amounts of data between storrs. This does cause a small breaking change where
$import()now returns a matrix with columnsname,namespaceindicating what was imported (previously this was just a vector) - Storr import/export more gracefully handles missing keys
- RDS storrs no longer pad base64 filenames with trailing with
=which reduces the number of possily problematic characters. Existing storrs are unaffected (#43) - RDS storrs are not left in an inconsistent state if RDS file writing fails (#40, thanks @krlmlr)
storr 1.0.1 (2016-05-10)
- Updated to work with recent testthat (>= 1.0.0)
- Limited support for writing out very large serialised objects (rds)
storr 1.0.0 (2016-01-19)
- Initial CRAN release