Transforming Data Between Statistical Classifications

Provides support for transformations of numeric aggregates between statistical classifications (e.g. occupation or industry categorisations) using the 'Crossmaps' framework. Implements classes for representing transformations between a source and target classification as graph structures, and methods for validating and applying crossmaps to transform data collected under the source classification into data indexed using the target classification codes. Documentation about the 'Crossmaps' framework is provided in the included vignettes and in Huang (2024, ).

xmap

The {xmap} package provides support for transformations of numeric aggregates between statistical classifications (e.g. occupation or industry categorisations) using the Crossmaps framework. Implements classes for representing transformations between a source and target classification as graph structures (i.e. Crossmaps), and methods for validating and applying crossmaps to transform data collected under the source classification into data indexed using the target classification codes.

Overview

Crossmaps encodes instructions for transforming data between statistical classifications. It is a graph structure with links (to, from, and weight_by) that associate source and target classification codes with weights for redistributing numeric mass attached to each code in a source classification. For example, a given link could specify encode that 10% of people in a source classification are employed in a particular target occupation. A collection of links between two classifications forms a crossmap graph structure, which when represented as an table of links (i.e. an edge list table) can be easily verified against conditions that are required for a valid transformation of data between the specified classifications.

Using a valid crossmap guarantees that the total mass before and after the transformation remains the same. For example, if we reclassify counts of workers by occupation, we would expect that the total number of workers across all occupation categories remains unchanged after reclassification. However, comparing totals is not always sufficient to identify mistakes in data transformation as there can be multiple ways to redistribute mass between source and target classifications while maintaining the same total.

Validation Functions

This package allows you to create, validate and apply xmap_tbl objects to perform valid and mass-preserving transformations of numeric aggregates between statistical classifications. The crossmaps workflow saves users from having to manually check code lines for implementation errors by verifying crossmaps satisfy mathematically sufficient conditions for valid transformation. It provides ‘guardrails’ for transforming data between classifications.

validate_as_xmap() and diagnose_as_xmap_tbl() both check that:

• every link has a source, target, and weight (no missing values)
• there are no duplicated links (repeated source-target pairs)
• weights from a given .from key sum to one. This ensures that totals before and after transformation are the same.

validate_as_xmap() returns a single TRUE/FALSE — use it when you only need a pass/fail answer, e.g. filtering many groups. diagnose_as_xmap_tbl() returns the offending rows for any failing check — use it to find out why a crossmap is invalid.

diagnose_apply_xmap() checks that:

• all source categories in .data have matching instructions in .xmap. This ensures that data isn’t silently dropped in the transformation due to a missing category.
• there are no missing values in .data that could cause addition errors.

Citation and Related Papers

For more details the task abstraction underpinning the crossmap framework and some visualisations of crossmaps, see Visualising category recoding and numeric redistributions, and for more on how transformation guarantees arise from graph properities of crossmaps, see Crossmaps: A Unified Statistical And Computational Framework For Ex-Post Harmonisation Of Aggregate Statistics.

To cite this package use:

citation("xmap")

## Warning in citation("xmap"): could not determine year for 'xmap' from package
## DESCRIPTION file

## To cite package 'xmap' in publications use:
## 
##   Huang C, Puzzello L (????). _xmap: Transforming Data Between
##   Statistical Classifications_. R package version 0.1.0.9006,
##   <https://github.com/cynthiahqy/xmap>.
## 
## A BibTeX entry for LaTeX users is
## 
##   @Manual{,
##     title = {xmap: Transforming Data Between Statistical Classifications},
##     author = {Cynthia A. Huang and Laura Puzzello},
##     note = {R package version 0.1.0.9006},
##     url = {https://github.com/cynthiahqy/xmap},
##   }

Installation

To install the latest CRAN release of xmap:

install.packages("xmap")

To install the latest development version of xmap:

remotes::install_github("cynthiahqy/xmap")

Specify and Validate Crossmaps

• {dplyr} verbs
• as_xmap_tbl()
• validate_as_xmap()
• diagnose_as_xmap_tbl()

Apply Transformations to Data

• apply_xmap()
• diagnose_apply_xmap()

Usage

Creating crossmaps

The easiest way to create a crossmap is to coerce a dataframe (e.g. xmap::demo$abc_links) containing source codes, target codes and weights between them:

library(xmap)
demo$abc_links |>
  as_xmap_tbl(from = "lower", to = "upper", weight_by = "share")

## # A crossmap tibble: 6 × 3
## # with unique keys:  [4] lower -> [5] upper
##   .from$lower .to$upper .weight_by$share
##   <chr>       <chr>                <dbl>
## 1 a           AA                     1  
## 2 b           BB                     1  
## 3 c           BB                     1  
## 4 d           CC                     0.3
## 5 d           DD                     0.6
## 6 d           EE                     0.1

If the coercion fails, you can use diagnose_as_xmap() to identify issues:

bad_links <- demo$abc_links
bad_links[4, "share"] <- 5

diagnose_as_xmap_tbl(bad_links, from = "lower", to = "upper", weight_by = "share")

## ✖ xmap is invalid

## ✔ No duplicate `.from`-`.to` pairs

## ✔ No missing values in `.from`

## ✔ No missing values in `.to`

## ✔ No missing values in `.weight_by`

## ✔ All `.weight_by` values are positive

## ✖ Sum of `.weight_by` by `.from` are not near enough to one (1 row)

## # A tibble: 1 × 2
##   .from$lower .sum.weight_by
##   <chr>                <dbl>
## 1 d                      5.7

Applying crossmaps

When using a crossmap to transform data, you want to make sure that the crossmap covers all the codes present in your data. For example, if your data contained a count for the category “teacher”, but your crossmap doesn’t have any links with “teacher”, then you risk silently losing data in the transformation. Even if you wanted to remove the count for “teacher”, this should be done in the original dataset explicitly (e.g. via filtering and removing rows), rather than implictly in the transformation.

To use a suitable crossmap to transform data, you can use apply_xmap():

abc_xmap <- demo$abc_links |>
  as_xmap_tbl(from = "lower", to = "upper", weight_by = "share")
abc_data <- tibble::tibble(
  lower = unique(demo$abc_links$lower),
  count = runif(length(unique(demo$abc_links$lower)), min = 100, max = 500)
)
transformed_data <- apply_xmap(
  .data = abc_data,
  .xmap = abc_xmap,
  values_from = count
)

## Matching keys in `.data$lower` with `.xmap$.from$lower`
## ℹ To silence, set `keys_from = lower`

## totals still match!
sum(abc_data$count) == sum(transformed_data$count)

## [1] TRUE