Automated Exploratory Data Analysis and Dataset Profiling

Profiles a data frame with minimal input: column type inference, missing-value analysis, distributional summary statistics (including skewness and kurtosis), normality tests, outlier detection, correlation and categorical-association analysis, date-column profiling, grouped comparisons and an overall data-quality score, alongside a set of 'ggplot2' visualisations. A single entry point, profile_data(), returns a structured S3 object holding metadata, statistics, diagnostics and plots, with print(), summary() and plot() methods, and report() renders the whole profile to a self-contained HTML file. Statistical methods include the Shapiro-Wilk normality test as implemented by Royston (1995) and the Anderson-Darling test following Stephens (1974) , with power comparisons of these tests in Yap and Sim (2011) , and the categorical association measure of Cramer (1946, ISBN:9780691080048).

dataProfilerR

Automated exploratory data analysis for R. Point it at a data frame and it returns a structured profile — column types, missingness, distributional statistics, normality tests, outliers, correlations, a data-quality score, and ggplot2 figures — through a single function, profile_data().

The aim is to cover the first hour of EDA that you'd otherwise write by hand for every new dataset, while keeping the result a plain, inspectable object you can build on.

Installation

# install.packages("remotes")
remotes::install_github("mqfarooqi1/dataProfilerR")

Depends on ggplot2. The Anderson–Darling normality test additionally uses the suggested nortest package; if it isn't installed, only Shapiro–Wilk is run.

Quick start

library(dataProfilerR)

p <- profile_data(iris)
p                                  # concise overview + quality score
summary(p)                         # numeric summary, missingness, normality, outliers, correlations
plot(p, which = "correlation")     # retrieve a figure
plot(p, which = "distribution", column = "Sepal.Length")

# components are just list elements
p$metadata$column_types
p$diagnostics$quality$score
p$statistics$numeric

# grouped comparison + a self-contained HTML report (needs pandoc)
p <- profile_data(iris, group_by = "Species")
p$diagnostics$groups$numeric_by_group
report(p, "iris_report.html")

See the vignette (vignette("dataProfilerR")) for a full walkthrough on a messy dataset.

Architecture and design decisions

The package is organised as a pipeline of independent, individually-callable functions, with one orchestrator on top:

                       profile_data()                 <- orchestrator
        ┌───────────────────┼───────────────────────────────┐
   profiling            statistics                      visualization
   ─────────            ──────────                       ────────────
   infer_column_types   normality_tests                 plot_missing
   analyze_missing      detect_outliers / outlier_summary plot_distribution
   summarize_columns    correlation_analysis            plot_correlation
   data_quality_score                                   plot_boxplots
                                                        plot_pairs
                          │
                          ▼
                  data_profile (S3 object)  ──  print() / summary() / plot()

Design choices worth calling out:

• S3, not S4. A profiling result is data, not behaviour. Modelling it as a plain list with a class keeps it transparent (str(p) just works), serialisable, and easy to extend with new elements without redefining a formal class. S4's validity and dispatch machinery would be overhead with no payoff here. The methods provided are print, summary, and plot.
• Each stage stands alone. infer_column_types(), detect_outliers(), plot_correlation() etc. all work directly on a data frame or vector, so the package is useful piecemeal, not only through the orchestrator.
• Type inference drives the rest. Columns are classified once (numeric/integer/date/logical/categorical/text) and that classification routes which statistics and plots apply.
• Fail early on bad input. A shared validator rejects non-data-frames, empty frames, and duplicate/blank column names with clear messages rather than letting them surface as cryptic downstream errors.
• Minimal dependencies. Only ggplot2 beyond base/recommended packages. Skewness and kurtosis are implemented directly rather than pulling in moments; Anderson–Darling degrades gracefully when nortest is absent.

Function reference

Profiling

Statistics

Visualization (ggplot2)

Pipeline, reporting & object

The data_profile object

profile_data() returns an S3 list with four parts plus the call:

• metadata — dataset name, dimensions, per-column types, type counts, timestamp.
• statistics — numeric summary, categorical summary, correlation matrices, and the categorical association matrix.
• diagnostics — missingness, normality, outliers, date-column profile, the grouped comparison (when group_by is set), and the quality score.
• plots — the ggplot2 objects (empty if build_plots = FALSE; the per-column distribution plots are also skipped when distributions = FALSE).

Folder structure

dataProfilerR/
├── DESCRIPTION
├── NAMESPACE                # generated by roxygen2
├── LICENSE
├── NEWS.md
├── R/
│   ├── dataProfilerR-package.R
│   ├── utils.R              # validation + skewness/kurtosis
│   ├── profiling.R          # types, missingness, summaries, quality score
│   ├── statistics.R         # normality, outliers, correlation
│   ├── association.R        # Cramer's V for categoricals
│   ├── dates.R              # date/datetime profiling
│   ├── groups.R             # grouped comparison
│   ├── visualization.R      # ggplot2 functions
│   ├── report.R             # HTML report (rmarkdown)
│   ├── profile_data.R       # orchestrator + S3 constructor
│   └── methods.R            # print / summary / plot
├── man/                     # generated by roxygen2
├── tests/testthat/          # unit + edge-case tests
└── vignettes/dataProfilerR.Rmd

Testing

testthat (edition 3) covers each function plus edge cases — empty frames, wrong types, all-NA columns, single-column frames, missing-column plot requests, and output-shape consistency. Run with devtools::test().

Limitations and future improvements

Added in 0.2.0: report() (HTML), categorical_association() (Cramer's V), analyze_dates(), compare_groups(), and a distributions = FALSE switch to avoid eager per-column plots on wide data. See NEWS.md.

Still open / honest gaps:

• Numeric-vs-categorical effect sizes (e.g. eta-squared, group-mean differences with tests) aren't here yet; compare_groups() reports descriptive summaries only, not significance.
• Date analysis is shallow — range and gaps, but no seasonality/trend.
• Distribution plots are still eager unless you opt out with distributions = FALSE; a fully lazy, build-on-demand path would be cleaner.
• report() requires pandoc (the usual R Markdown dependency); there is no pandoc-free fallback.
• Text columns are detected but not analysed beyond cardinality.

License

MIT © Muhammad Farooqi