Optimal Distribution Preserving Down-Sampling of Bio-Medical Data

An optimized method for distribution-preserving class-proportional down-sampling of bio-medical data .

opdisDownsampling: Distribution-preserving class-proportional downsampling for biomedical data

opdisDownsampling is an R package for optimal, distribution-preserving, class-proportional down-sampling of biomedical data. It reduces dataset size while preserving class proportions and the statistical structure of the original data.

This repository contains the package source and documentation.

Features

• Distribution-preserving downsampling: Selects a subset of samples whose statistical distribution closely matches the original dataset.

Figure adapted from: Lötsch J, Malkusch S, Ultsch A (2021). Optimal distribution-preserving downsampling of large biomedical data sets (opdisDownsampling). PLOS ONE. https://doi.org/10.1371/journal.pone.0255838 (see "Reference" paragraph below)

• Class-proportional selection: Maintains the proportions of different classes within the down-sampled data.
• Parallel computing support: Can exploit multiple CPU cores for efficient processing.
• Flexible test statistics: Supports several options for distribution comparison.

Installation

You can install opdisDownsampling directly from source:

From CRAN:

install.packages("opdisDownsampling")

From this GitHub repository:

remotes::install_github("JornLotsch/opdisDownsampling")

Or manually by cloning the repository and running:

devtools::install("path/to/opdisDownsampling")

Usage

The main function is opdisDownsampling().

Basic example

library(opdisDownsampling)

data(iris)
Iris50percent <- opdisDownsampling(Data = iris[,1:4], Cls = as.integer(iris$Species),
  Size = 50, Seed = 42, MaxCores = 1)

Memory-efficient processing for large datasets

set.seed(42)

# Small synthetic dataset for the first example
large_dataset <- data.frame(
  class = sample(c("A", "B"), 2000, replace = TRUE),
  x1 = rnorm(2000),
  x2 = runif(2000),
  x3 = rpois(2000, lambda = 3)
)

# Smaller synthetic dataset for the second example
my_data <- data.frame(
  class = sample(c("A", "B"), 300, replace = TRUE),
  x1 = rnorm(300),
  x2 = runif(300)
)

# Automatic memory optimisation for large datasets (for demonstration purposes, a relatively small 'large'  dataset is generated).
LargeDataSample <- opdisDownsampling(
  Data = large_dataset[,2:ncol(large_dataset)],
  Size = 0.1,
  Seed = 42,
  nTrials = 5000,
  JobSize = NULL,
  verbose = TRUE
)

# Custom chunk size for fine-tuned memory control
CustomSample <- opdisDownsampling(
  Data = my_data[,2:ncol(my_data)],
  Size = 100,
  Seed = 42,
  nTrials = 2000,
  JobSize = 500
)

Arguments

Available TestStat options

Variable Selection Method

The package offers PCA based variable selection approaches:

PCA-based Selection (PCAimportance = TRUE)

• Identifies variables with high loadings in principal components
• Focuses on variables that capture the most variance in the data
• Useful for dimensionality reduction while preserving data structure

Memory Optimization

The package automatically optimizes memory usage through intelligent chunking:

• Automatic chunk sizing: Considers data dimensions, available system memory, and number of processor cores
• Memory-constrained processing: Prevents memory exhaustion on large datasets or high trial counts
• Adaptive strategy: Uses larger chunks for small datasets (efficiency) and smaller chunks for large datasets (memory safety)
• Diagnostic output: Enable verbose = TRUE to understand memory usage patterns

Memory optimization is particularly beneficial for:

• Large datasets (>100MB)
• High trial counts (>1000 trials)
• Memory-constrained systems
• Datasets with many variables or observations

Output

Returns a list containing:

• ReducedData: Down-sampled data frame
• RemovedData: Data not included in the sample
• ReducedInstances: Row names of the reduced data
• RemovedInstances: Row names of the removed data

Handling missing values

opdisDownsampling() also works with data containing missing values.

library(opdisDownsampling)

set.seed(42)
iris_data <- data.frame(iris[, 1:4])

n_na <- round(0.05 * nrow(iris_data) * ncol(iris_data))
na_pos <- sample(nrow(iris_data) * ncol(iris_data), n_na)

x <- as.matrix(iris_data)
x[na_pos] <- NA
iris_with_missing <- as.data.frame(x)
iris_with_missing

downsampled_missing <- opdisDownsampling(
  Data = iris_with_missing,
  Cls = iris$Species,
  Size = 0.8,
  Seed = 42
)

downsampled_missing

Performance Tips

For Large Datasets

• Use JobSize = NULL to enable automatic memory-aware chunk-size calculation.
• Use verbose = TRUE to monitor memory usage and chunking diagnostics.
• Consider using fewer trials initially to estimate processing time.

For Memory-Constrained Systems

• Use JobSize = NULL for automatic chunk-size calculation, or manually set smaller values such as JobSize = 10 or JobSize = 25.
• Monitor system memory usage during processing.
• Use a smaller MaxCores value to reduce parallel memory overhead.

For Small Datasets

• The default JobSize = 0 processes all trials in a single batch.
• Manual chunk-size specification is usually not needed for small datasets.
• Higher trial counts can usually be used without memory concerns.

Documentation

See the CRAN package page for full documentation and the reference manual.

• Original article describing the method:
  "Optimal distribution preserving down‐sampling of bio‐medical data"
  PLoS ONE 16(8): e0255838

Authors and license

• Jorn Lotsch (author, maintainer)
• Sebastian Malkusch (author)
• Alfred Ultsch (author)
• License: GPL-3

Reference

Citing opdisDownsampling

If you use this package, please cite the CRAN package and the original paper:

Lötsch J, Malkusch S, Ultsch A. Optimal distribution-preserving downsampling of large biomedical data sets (opdisDownsampling). PLoS One. 2021 Aug 5;16(8):e0255838. doi: 10.1371/journal.pone.0255838. PMID: 34352006; PMCID: PMC8341664.

Related links

• CRAN package page (check and compare versions)
• Original publication (PLoS ONE)