Task view: Missing Data

Last updated on 2020-01-12 by Julie Josse, Nicholas Tierney and Nathalie Vialaneix (r-miss-tastic team)

Missing data are very frequently found in datasets. Base R provides a few options to handle them using computations that involve only observed data (na.rm = TRUE in functions mean, var, ... or use = complete.obs|na.or.complete|pairwise.complete.obs in functions cov, cor, ...). The base package stats also contains the generic function na.action that extracts information of the NA action used to create an object.

These basic options are complemented by many packages on CRAN, which we structure into main topics:

In addition to the present task view, this reference website on missing data might also be helpful.

If you think that we missed some important packages in this list, please contact the maintainer.

Exploration of missing data

  • Manipulation of missing data is implemented in the packages sjmisc and sjlabelled. memisc also provides defineable missing values, along with infrastruture for the management of survey data and variable labels.
  • Missing data patterns can be identified and explored using the packages mi, dlookr, wrangle, DescTools, and naniar.
  • Graphics that describe distributions and patterns of missing data are implemented in VIM (which has a Graphical User Interface, VIMGUI) and naniar (which abides by tidyverse principles).
  • Tests of the MAR assumption (versus the MCAR assumption) are implemented in the function LittleMCAR from BaylorEdPsych (Little's test) and from MissMech (a non parametric test).
  • Evaluation with simulations can be performed using the function ampute of mice.

Likelihood based approaches

  • Methods based on the Expectation Maximization (EM) algorithm are implemented in norm (using the function em.norm for multivariate Gaussian data), in cat (function em.cat for multivariate categorical data), in mix (function em.mix for multivariate mixed categorical and continuous data). These packages also implement Bayesian approaches (with Imputation and Posterior steps) for the same models (functions da.XXX for norm, cat and mix) and can be used to obtain imputed complete datasets or multiple imputations (functions imp.XXX for norm, cat and mix), once the model parameters have been estimated. In addition, TestDataImputation implements imputation based on EM estimation (and other simpler imputation methods) that are well suited for for dichotomous and polytomous test with item responses.
  • Full Information Maximum Likelihood (also known as "direct maximum likelihood" or "raw maximum likelihood") is available in lavaan, OpenMx and rsem, for handling missing data in structural equation modeling.
  • Bayesian approaches for handling missing values in model based clustering with variable selection is available in VarSelLCM. The package also provides imputation using the posterior mean.
  • Missing values in mixed-effect models and generalized linear models are supported in the packages mdmb, icdGLM and JointAI, the last one being based on a Bayesian approach. brlrmr also handles MNAR values in response variable for logistic regression using an EM approach.
  • Missing data in item response models is implemented in TAM, mirt and ltm.
  • Variable selection under ignorable and non ignorable missing data mechanisms is implemented in TVsMiss.
  • Robust covariance estimation is implemented in the package GSE.

Single imputation

  • The simplest method for missing data imputation is imputation by mean (or median, mode, ...). This approach is available in many packages among which ForImp, Hmisc, and dlookr that contain various proposals for imputing the same value for all missing data of a variable. This method and other simple imputation methods are also available in tidyimpute that works after the tidyverse approach.
  • k-nearest neighbors is a popular method for missing data imputation that is available in many packages including DMwR, impute, VIM, GenForImp and yaImpute (with many different methods for kNN imputation, including a CCA based imputation). wNNSel implements a kNN based method for imputation in large dimensional datasets.
  • hot-deck imputation is implemented in hot.deck, HotDeckImputation, FHDI and VIM (function hotdeck). StatMatch uses hot deck imputation to impute surveys from an external dataset.
  • Other regression based imputations are implemented in VIM (linear regression based imputation in the function regressionImp). In addition, simputation that is a general package for imputation by any prediction method that can be combined with various regression methods, and works well with the tidyverse. WaverR imputes data using a weighted average of several regressions.
  • Based on random forest in missForest.
  • Based on copula in CoImp and in sbgcop (semi-parametric Bayesian copula imputation). The last one supports multiple imputation.
  • PCA/Singular Value Decomposition/matrix completion is implemented in the package missMDA for numerical, categorical and mixed data. Heterogeneous missingness in a high-dimensional PCA is also addressed in primePCA. softImpute contains several methods for iterative matrix completion, as well as filling and denoiseR for numerical variables, or mimi that uses low rank assumption to impute mixed datasets. The package pcaMethods offers some Bayesian implementation of PCA with missing data. NIPALS (based on SVD computation) is implemented in the packages mixOmics (for PCA and PLS), ade4, nipals and plsRglm (for generalized model PLS). ddsPLS implements a multi-block imputation method based on PLS in a supervise framework. ROptSpace and CMF proposes a matrix completion method under low-rank assumption and collective matrix factorization for imputation using Bayesian matrix completion for groups of variables (binary, quantitative, poisson). Imputation for groups is also avalaible in the missMDA in the function imputeMFA.
  • Imputation for non-parametric regression by wavelet shrinkage is implemented in CVThresh using solely maximization of the h-likelihood.
  • mi and VIM also provide diagnostic plots to evaluate the quality of imputation.

Multiple imputation

Some of the above mentionned packages can also handle multiple imputations.

  • Amelia implements Bootstrap multiple imputation using EM to estimate the parameters, for quantitative data it imputes assuming a Multivariate Gaussian distribution. In addition, AmeliaView is a GUI for Amelia, available from the Amelia web page.
  • mi, mice and smcfcs implement multiple imputation by Chained Equations. smcfcs extends the models covered by the two previous packages. miceFast provides an alternative implementation of mice imputation methods using object oriented style programming and c++. miceMNAR imputes MNAR responses under Heckman selection model for use with mice.
  • missMDA implements multiple imputation based on SVD methods.
  • hot.deck implements hot deck based multiple imputation.
  • Multilevel imputation: Multilevel multiple imputation is implemented in hmi, jomo, mice, miceadds, micemd, mitml and pan.
  • Qtools implements multiple imputation based on quantile regression.
  • BaBooN implements a Bayesian bootstrap approach for discrete data imputation that is based on Predictive Mean Matching (PMM).
  • accelmissingmultiple imputation with the zero-inflated Poisson lognormal model for missing count values in accelerometer data.

In addition, mitools provide a generic approach to handle multiple imputation in combination with any imputation method.

Weighting methods

  • Computation of weights for observed data to account for data unobserved by Inverse Probability Weighting (IPW) is implemented in ipw.
  • Doubly Robust Inverse Probability Weighted Augmented GEE Estimator with missing outcome is implemented in CRTgeeDR.

Specific types of data

  • Longitudinal data / time series and censored data: Imputation for time series is implemented in imputeTS and imputePSF. Other packages, such as forecast, spacetime, timeSeries, xts, prophet, stlplus or zoo, are dedicated to time series but also contain some (often basic) methods to handle missing data (see also TimeSeries). To help fill down missing values for time series, the padr and tsibble packages provides methods for imputing implicit missing values. Imputation of time series based on Dynamic Time Warping is implemented in DTWBI for univariate time series and in DTWUMI for multivariate ones. naniar also imputed data below the range for exploratory graphical analysis with the function impute_below. TAR implements an estimation of the autoregressive threshold models with Gaussian noise and of positive-valued time series with a Bayesian approach in the presence of missing data. swgee implements a probability weighted generalized estimating equations method for longitudinal data with missing observations and measurement error in covariates based on SIMEX. icenReg performs imputation for censored responses for interval data. imputeTestbench proposes tools to benchmark missing data imputation in univariate time series.
  • Spatial data: Imputation for spatial data is implemented in phylin using interpolation with spatial distance weights or kriging. gapfill is dedicated to satellite data and geostatistical interpolation of data with irregular spatial support is implemented in rtop
  • Spatio-temporal data: Imputation for spatio-temporal data is implemented in the package cutoffR using different methods as knn and SVD. Similarly, reddPrec imputes missing values in daily precipitation time series accross different locations and sptemExp imputes missing data air polluant concentrations.
  • Graphs/networks: Imputation for graphs/networks is implemented in the package dils to impute missing edges. PST provides a framework for analyzing Probabilistic Suffix Trees, including functions for learning and optimizing VLMC (variable length Markov chains) models from sets of individual sequences possibly containing missing values.
  • Imputation for contingency table is implemented in lori that can also be used for the analysis of contingency tables with missing data.
  • Imputation for compositional data (CODA) is implemented in robCompositions (based on kNN or EM approaches) and in zCompositions (various imputation methods for zeros, left-censored and missing data).
  • Imputation for diffusion processes is implemented in DiffusionRimp by imputing missing sample paths with Brownian bridges.
  • experiment handles missing values in experimental design such as randomized experiments with missing covariate and outcome data, matched-pairs design with missing outcome.
  • cdparcoord handles missing values in parallel coordinates settings.

Specific application fields

  • Genetics: SNPassoc provides function to visualize missing data in the case of SNP studies (genetics). Analyses of Case-Parent Triad and/or Case-Control Data with SNP haplotypes is implemented in Haplin, where missing genotypic data are handled with an EM algorithm. FamEvent and snpStats implement imputation of missing genotypes, respectively with an EM algorithm and a nearest neighbor approach. Imputation for genotype and haplotype is implemented in alleHap using solely deterministic techniques on pedigree databases and imputation of missing genotypes are also implemented in QTLRel that contains tools for QTL analyses. Tools for Hardy-Weinberg equilibrium for bi- and multi-allelic genetic marker data are implemented in HardyWeinberg, where genotypes are imputed with a multinomial logit model. StAMPP computes genomic relationship when SNP genotype datasets contain missing data and PSIMEX computes inbreeding depression or heritability on pedigree structures affected by missing paternities with a variant of the SIMEX algorithm.
  • Genomics: Imputation for dropout events (i.e., under-sampling of mRNA molecules) in single-cell RNA-Sequencing data is implemented in DrImpute and Rmagic. RNAseqNet uses hot-deck imputation to improve RNA-seq network inference with an auxiliary dataset.
  • Epidemiology: powerlmm implements power calculation for time x treatment effects in the presence of dropouts and missing data in mixed linear models and pseval evaluates principal surrogates in a single clinical trial in the presence of missing counterfactual surrogate responses. idem provides missing data imputation with a sensitivity analysis strategy to handle the unobserved functional outcomes not due to death.
  • Causal inference: Causal inference with interactive fixed-effect models is available in gsynth with missing values handled by matrix completion. MatchThem matches multiply imputed datasets using several matching methods, and provides users with the tools to estimate causal effects in each imputed datasets. Sensitivity analysis to help diagnose missing data and imputation is implemented in TippingPoint.
  • Scoring: Basic methods (mean, median, mode, ...) for imputing missing data in scoring datasets are proposed in scorecardModelUtils.
  • Preference models: Missing data in preference models are handled with a Composite Link approach that allows for MCAR and MNAR patterns to be taken into account in prefmod.
  • Administrative records: fastLink provides a Fellegi-Sunter probabilistic record linkage that allows for missing data and the inclusion of auxiliary information.
  • Regression and classification eigenmodel handles missing values in regression models for symmetric relational data. randomForest and StratifiedRF handles missing values in predictors for random forest like methods.
  • robustrao computes the Rao-Stirling diversity index (a well-established bibliometric indicator to measure the interdisciplinarity of scientific publications) with data containing uncategorized references.


accelmissing — 1.4

Missing Value Imputation for Accelerometer Data

ade4 — 1.7-13

Analysis of Ecological Data: Exploratory and Euclidean Methods in Environmental Sciences

alleHap — 0.9.9

Allele Imputation and Haplotype Reconstruction from Pedigree Databases

Amelia — 1.7.6

A Program for Missing Data

BaBooN — 0.2-0

Bayesian Bootstrap Predictive Mean Matching - Multiple and Single Imputation for Discrete Data

brlrmr — 0.1.7

Bias Reduction with Missing Binary Response

BaylorEdPsych — 0.5

R Package for Baylor University Educational Psychology Quantitative Courses

cat — 0.0-6.5

Analysis of categorical-variable datasets with missing values

cdparcoord — 1.0.1

Top Frequency-Based Parallel Coordinates

CMF — 1.0

Collective matrix factorization

CoImp — 1.0

Copula Based Imputation Method

CRTgeeDR — 2.0

Doubly Robust Inverse Probability Weighted Augmented GEE Estimator

CVThresh — 1.1.1

Level-Dependent Cross-Validation Thresholding

cutoffR — 1.0

CUTOFF: A Spatio-temporal Imputation Method

ddsPLS — 1.1.1

Data-Driven Sparse Partial Least Squares Robust to Missing Samples for Mono and Multi-Block Data Sets

DMwR — 0.4.1

Functions and data for "Data Mining with R"

denoiseR — 1.0

Regularized Low Rank Matrix Estimation

DescTools — 0.99.32

Tools for Descriptive Statistics

DiffusionRimp — 0.1.2

Inference and Analysis for Diffusion Processes via Data Imputation and Method of Lines

dils — 0.8.1

Data-Informed Link Strength. Combine multiple-relationship networks into a single weighted network. Impute (fill-in) missing network links.

dlookr — 0.3.13

Tools for Data Diagnosis, Exploration, Transformation

DrImpute — 1.0

Imputing Dropout Events in Single-Cell RNA-Sequencing Data

DTWBI — 1.1

Imputation of Time Series Based on Dynamic Time Warping

DTWUMI — 1.0

Imputation of Multivariate Time Series Based on Dynamic Time Warping

eigenmodel — 1.11

Semiparametric Factor and Regression Models for Symmetric Relational Data

experiment — 1.2.0

R Package for Designing and Analyzing Randomized Experiments

FamEvent — 2.0

Family Age-at-Onset Data Simulation and Penetrance Estimation

fastLink — 0.5.0

Fast Probabilistic Record Linkage with Missing Data

FHDI — 1.3.2

Fractional Hot Deck and Fully Efficient Fractional Imputation

filling — 0.2.0

Matrix Completion, Imputation, and Inpainting Methods

forecast — 8.10

Forecasting Functions for Time Series and Linear Models

ForImp — 1.0.3

Imputation of Missing Values Through a Forward Imputation Algorithm

gapfill — 0.9.6

Fill Missing Values in Satellite Data

GenForImp — 1.0

The Forward Imputation: A Sequential Distance-Based Approach for Imputing Missing Data

GSE — 4.2

Robust Estimation in the Presence of Cellwise and Casewise Contamination and Missing Data

gsynth — 1.0.9

Generalized Synthetic Control Method

Haplin — 7.2.2

Analyzing Case-Parent Triad and/or Case-Control Data with SNP Haplotypes

HardyWeinberg — 1.6.3

Statistical Tests and Graphics for Hardy-Weinberg Equilibrium

hmi — 0.9.17

Hierarchical Multiple Imputation

Hmisc — 4.3-0

Harrell Miscellaneous

hot.deck — 1.1

Multiple Hot-Deck Imputation

HotDeckImputation — 1.1.0

Hot Deck Imputation Methods for Missing Data

icdGLM — 1.0.0

EM by the Method of Weights for Incomplete Categorical Data in Generlized Linear Models

icenReg — 2.0.13

Regression Models for Interval Censored Data

idem — 4.0

Inference in Randomized Controlled Trials with Death and Missingness

imputePSF — 0.1.0

Impute Missing Data in Time Series Data with PSF Based Method

imputeTestbench — 3.0.3

Test Bench for the Comparison of Imputation Methods

imputeTS — 3.0

Time Series Missing Value Imputation

ipw — 1.0-11

Estimate Inverse Probability Weights

JointAI — 0.6.0

Joint Analysis and Imputation of Incomplete Data

jomo — 2.6-10

Multilevel Joint Modelling Multiple Imputation

lavaan — 0.6-5

Latent Variable Analysis

lori — 2.2.0

Imputation of Count Data using Side Information

ltm — 1.1-1

Latent Trait Models under IRT

MatchThem — 0.9.1

Matching and Weighting Multiply Imputed Datasets

mdmb — 1.3-18

Model Based Treatment of Missing Data

memisc — 0.99.21

Management of Survey Data and Presentation of Analysis Results

mi — 1.0

Missing Data Imputation and Model Checking

mice — 3.7.0

Multivariate Imputation by Chained Equations

miceadds — 3.7-6

Some Additional Multiple Imputation Functions, Especially for 'mice'

miceFast — 0.5.1

Fast Imputations Using 'Rcpp' and 'Armadillo'

micemd — 1.6.0

Multiple Imputation by Chained Equations with Multilevel Data

miceMNAR — 1.0.2

Missing not at Random Imputation Models for Multiple Imputation by Chained Equation

mimi — 0.2.0

Main Effects and Interactions in Mixed and Incomplete Data

missForest — 1.4

Nonparametric Missing Value Imputation using Random Forest

missMDA — 1.16

Handling Missing Values with Multivariate Data Analysis

MissMech — 1.0.2

Testing Homoscedasticity, Multivariate Normality, and Missing Completely at Random

mirt — 1.31

Multidimensional Item Response Theory

mitml — 0.3-7

Tools for Multiple Imputation in Multilevel Modeling

mitools — 2.4

Tools for Multiple Imputation of Missing Data

mix — 1.0-10

Estimation/Multiple Imputation for Mixed Categorical and Continuous Data

naniar — 0.4.2

Data Structures, Summaries, and Visualisations for Missing Data

nipals — 0.7

Principal Components Analysis using NIPALS or Weighted EMPCA, with Gram-Schmidt Orthogonalization

norm — 1.0-9.5

Analysis of multivariate normal datasets with missing values

OpenMx — 2.15.5

Extended Structural Equation Modelling

padr — 0.5.0

Quickly Get Datetime Data Ready for Analysis

pan — 1.6

Multiple Imputation for Multivariate Panel or Clustered Data

phylin — 2.0.2

Spatial Interpolation of Genetic Data

plsRglm — 1.2.5

Partial Least Squares Regression for Generalized Linear Models

powerlmm — 0.4.0

Power Analysis for Longitudinal Multilevel Models

prefmod — 0.8-34

Utilities to Fit Paired Comparison Models for Preferences

primePCA — 1.0

Projected Refinement for Imputation of Missing Entries in PCA

prophet — 0.5

Automatic Forecasting Procedure

pseval — 1.3.1

Methods for Evaluating Principal Surrogates of Treatment Response

PSIMEX — 1.1

SIMEX Algorithm on Pedigree Structures

PST — 0.94

Probabilistic Suffix Trees and Variable Length Markov Chains

QTLRel — 1.4

Tools for Mapping of Quantitative Traits of Genetically Related Individuals and Calculating Identity Coefficients from Pedigrees

Qtools — 1.5.1

Utilities for Quantiles

randomForest — 4.6-14

Breiman and Cutler's Random Forests for Classification and Regression

reddPrec — 0.4.0

Reconstruction of Daily Data - Precipitation

Rmagic — 2.0.3

MAGIC - Markov Affinity-Based Graph Imputation of Cells

ROptSpace — 0.2.1

Matrix Reconstruction from a Few Entries

RNAseqNet — 0.1.2

Log-Linear Poisson Graphical Model with Hot-Deck Multiple Imputation

robCompositions — 2.2.0

Compositional Data Analysis

robustrao — 1.0-5

An Extended Rao-Stirling Diversity Index to Handle Missing Data

rsem — 0.4.6

Robust Structural Equation Modeling with Missing Data and Auxiliary Variables

rtop — 0.5-14

Interpolation of Data with Variable Spatial Support

sbgcop — 0.980

Semiparametric Bayesian Gaussian Copula Estimation and Imputation

scorecardModelUtils —

Credit Scorecard Modelling Utils

simputation — 0.2.3

Simple Imputation

sjlabelled — 1.1.2

Labelled Data Utility Functions

sjmisc — 2.8.3

Data and Variable Transformation Functions

smcfcs — 1.4.0

Multiple Imputation of Covariates by Substantive Model Compatible Fully Conditional Specification

SNPassoc — 1.9-2

SNPs-based whole genome association studies

softImpute — 1.4

Matrix Completion via Iterative Soft-Thresholded SVD

spacetime — 1.2-3

Classes and Methods for Spatio-Temporal Data

sptemExp — 0.1.4

Constrained Spatiotemporal Mixed Models for Exposure Estimation

StAMPP — 1.5.1

Statistical Analysis of Mixed Ploidy Populations

StatMatch — 1.3.0

Statistical Matching or Data Fusion

stlplus — 0.5.1

Enhanced Seasonal Decomposition of Time Series by Loess

StratifiedRF — 0.2.2

Builds Trees by Sampling Variables in Groups

swgee — 1.4

Simulation Extrapolation Inverse Probability Weighted Generalized Estimating Equations

TAM — 3.3-10

Test Analysis Modules

TAR — 1.0

Bayesian Modeling of Autoregressive Threshold Time Series Models

TestDataImputation — 1.1

Missing Item Responses Imputation for Test and Assessment Data

tidyimpute — 0.1.0

Imputation the Tidyverse Way

timeSeries — 3062.100

Financial Time Series Objects (Rmetrics)

TippingPoint — 1.1.0

Enhanced Tipping Point Displays the Results of Sensitivity Analyses for Missing Data

tsibble — 0.8.5

Tidy Temporal Data Frames and Tools

TVsMiss — 0.1.1

Variable Selection for Missing Data

VarSelLCM — 2.1.3

Variable Selection for Model-Based Clustering of Mixed-Type Data Set with Missing Values

VIM — 4.8.0

Visualization and Imputation of Missing Values

VIMGUI — 0.10.0

Visualization and Imputation of Missing Values - Graphical User Interface

WaverR — 1.0

Data Estimation using Weighted Averages of Multiple Regressions

wNNSel — 0.1

Weighted Nearest Neighbor Imputation of Missing Values using Selected Variables

wrangle — 0.5.2

A Systematic Data Wrangling Idiom

xts — 0.12-0

eXtensible Time Series

yaImpute — 1.0-31

Nearest Neighbor Observation Imputation and Evaluation Tools

zCompositions — 1.3.3-1

Treatment of Zeros, Left-Censored and Missing Values in Compositional Data Sets

zoo — 1.8-7

S3 Infrastructure for Regular and Irregular Time Series (Z's Ordered Observations)

Task view list