Enables automated visualization of variable distribution and changes over time for predictive model building. It efficiently computes summary statistics aggregated by time for large datasets, and create plots for variable level monitoring.
An important part of model building is the "proc eyeball" sanity check. It can
also be a painful part of the process, when you are the data scientist tasked
with creating and checking 10,000 or more near-identical plots. The
package is designed to streamline this process.
an R package which takes a csv file as input and provides a pdf of VLM plots
and csv files of summary statistics as output, optionally ordered so
that any severely abnormal time series will be at the top of the pdf. The only
strict requirement of the data scientist is to specify which column of the input
data file contains the date variable.
otvPlots is efficiently implemented using
ggplot2 packages in R.
Plots are automatically labeled if a variable dictionary is provided. Important
variables can be given a highlighted label. A custom fuzzy matching algorithm
can be provided by the user.
Discrete and numeric variables are handled automatically and given separate treatment. All binary variables are treated as categorical.
For each numerical variable, the output plots include
For each categorical variable (including a numerical variable with no more than 2 unique levels not including NA), the output plots include
The order of variables in the CSV files is the same as in the PDF file.
category == 'NA'corresponds to missing. Categories among the same variable are ordered by global prevalence in a descending order.
Open an R (or RStudio) console and install the package from CRAN
Alternatively, if you prefer to install from GitHub:
devtoolspackage if not yet. You only need to do this once, so feel free to skip this step if the
devtoolsis already installed. You will be asked to select a CRAN mirror.
You can also build the package yourself by cloning the repo, setting your
working directory to the otvPlots folder and running
in R, after installing the
Note that otvPlots does depend on R and several R packages to run. You can see a complete and up to date list of dependencies in the Imports field in the DESCRIPTION file.
Open an R console (or RStudio). Load the
otvPlots pacakge first (all its
dependent packages should be loaded automatically).
The main function of the package is
vlm. Before execute this function,
input data need to be prepared using the
Please check out the help files to see all options and many usage examples
bankData and its labels
bankLables are built-in datasets in the
After running the following code, a pdf file named "bank.pdf" and two csv files named "bank_numerical_summary.csv" and "bank_categorical_summary.csv" will be generated in the current working directory.
## Load the datasets data(bankData) data(bankLabels) ## Prepare data and labels bankData <- PrepData(bankData, dateNm = "date", dateGp = "months", dateGpBp = "quarters") bankLabels <- PrepLabels(bankLabels) ## Generate a pdf file of vlm plots, and csv files of summary statistics vlm(dataFl = bankData, dateNm = "date", labelFl = bankLabels, sortFn = "OrderByR2", dateGp = "months", dateGpBp = "quarters", outFl = "bank")
PrepData function only needs to be run once on a dataset. After that
can be run directly with the argument
dataNeedPrep = FALSE (the default).
genCSV = FALSE.
vlm(dataFl = bankData, dateNm = "date", labelFl = bankLabels, genCSV = FALSE, sortFn = "OrderByR2", dateGp = "months", dateGpBp = "quarters", outFl = "bank2")
bankData[, weight := rnorm(.N, 1, .1)] bankData[, weight := weight / mean(weight)] vlm(dataFl = bankData, dateNm = "date", labelFl = bankLabels, dateGp = "months", dateGpBp = "quarters", weightNm = "weight", outFl = "bank3")
sortVars, but the
"date"column must be excluded from
sortVars <- sort(bankLabels[varCol!="date", varCol]) vlm(dataFl = bankData, dateNm = "date", labelFl = bankLabels, dateGp = "months", dateGpBp = "quarters", outFl = "bank4", sortVars = sortVars)
vlm(dataFl = bankData, dateNm = "date", labelFl = bankLabels, dateGp = "months", dateGpBp = "quarters", outFl = "bank5", varNms = "age", sortVars = NULL)
All examples for this package come from the Bank Marketing dataset available at the UCI Machine Learning Repository. The UCI repository maintains a free collection of datasets for researchers at its website.
Moro et al., S. Moro, P. Cortez, and P. Rita (2014). A Data-Driven Approach to Predict the Success of Bank Telemarketing. Decision Support Systems, Elsevier, 62:22-31, June 2014
Lichman, M. (2013). UCI Machine Learning Repository [http://archive.ics.uci.edu/ml]. Irvine, CA: University of California, School of Information and Computer Science.
Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at
Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.
Contributors: We welcome your interest in Capital One’s Open Source Projects (the “Project”).
Any Contributor to the project must accept and sign a CLA indicating agreement to the license terms. Except for the license granted in this CLA to Capital One and to recipients of software distributed by Capital One, you reserve all right, title, and interest in and to your contributions; this CLA does not impact your rights to use your own contributions for any other purpose.
This project adheres to the Open Source Code of Conduct. By participating, you are expected to honor this code.