Audit Statistical Fidelity of AI-Mediated Official Statistics

Provides deterministic tools for auditing whether artificial intelligence systems preserve the numerical, semantic, contextual, temporal, geographic, unit, provenance, revision, transformation, and uncertainty properties of official statistics. Structured reference statistics and machine-generated claims can be compared with non-compensatory critical-error rules, weakest-link and geometric fidelity summaries, provenance graphs, and portable SHA-256 proof bundles. The package also provides bounded connectors for official Eurostat, World Bank, OECD, United Nations SDG, United Kingdom Office for National Statistics, and United States Bureau of Labor Statistics application programming interfaces, plus an extensible HTTPS JSON API registry with session-only API-key support. Prompt perturbation, statistical red-team generation, minimal-pair tests, and starter benchmark data support reproducible evaluation of generative, retrieval-augmented, and agentic statistical systems. An embedded alignment layer maps claim-level controls to relevant activities of the Generic Statistical Business Process Model (GSBPM) 5.2, including Analyse, Disseminate, Evaluate, Quality Management, and Metadata Management. No specific model provider is required.

statfidelity

statfidelity audits whether AI-mediated answers preserve the statistical meaning of official data. It is designed for generative AI, retrieval-augmented generation (RAG), statistical agents, and model-context-protocol style tool workflows without depending on a particular model vendor.

Authors and maintainer

Authors, in package citation order:

1. Hossein Hassani
2. Steve MacFeely
3. Leila Marvian Mashhad

Maintainer: Leila Marvian Mashhad.

Relationship to GSBPM 5.2

statfidelity is designed to complement, not replace, the Generic Statistical Business Process Model (GSBPM) 5.2. GSBPM is a process-level reference model for the production of official statistics, covering activities from specifying needs and design through collection, processing, analysis, dissemination, and evaluation. statfidelity addresses a narrower executable problem at the statistical-claim level: whether the multidimensional meaning of an official statistical object is preserved when it is retrieved, transformed, summarised, or communicated through generative AI, retrieval-augmented generation (RAG), or statistical agents.

The closest operational alignment is with GSBPM activities related to Analyse, Disseminate, Evaluate, and overarching Quality Management. The package does not implement the GSBPM production process. Instead, it provides claim-level controls that can sit within or alongside those activities:

• canonical official references and AI-mediated claims;
• ten-dimensional fidelity auditing;
• geometric and weakest-link summaries;
• non-compensatory critical-error rules;
• PASS / WARN / FAIL decisions;
• provenance graphs and revision-aware evidence;
• tamper-evident SHA-256 proof bundles; and
• batch, red-team, minimal-pair, and stability evaluation utilities.

A useful distinction is: GSBPM provides the process-level quality architecture; statfidelity provides an executable AI claim-level fidelity-control layer.

Official GSBPM 5.2 information: https://unece.org/statistics/gsbpm-v5.2

This relationship is implemented directly in the package:

# Embedded offline mapping of relevant GSBPM 5.2 activities
gsbpm52_map()

# Where a fidelity dimension or package function fits
gsbpm_align("geographic")
gsbpm_align("audit_stat_ai")

# After an audit, create a GSBPM-oriented quality evidence report
# report <- gsbpm_quality_report(aud)
# report

The mapping deliberately uses the terms direct and supporting, not "compliant" or "certified". GSBPM 5.2 is a reference model for statistical business processes; statfidelity supplies executable evidence that can support selected GSBPM activities.

What statfidelity 0.1.0 includes

• bounded, user-initiated connectors for Eurostat, World Bank, OECD, UNSD SDG, UK ONS, and U.S. BLS APIs;
• cross-provider metadata search with search_official_stats();
• an extensible HTTPS JSON REST registry for additional NSOs and official providers, with session-only API-key handling;
• a common stat_provider_data representation and as_stat_reference() conversion;
• contextual fidelity for dataset, population, adjustment, frequency, and methodology;
• non-compensatory critical errors plus geometric and weakest-link fidelity;
• batch auditing for benchmark-scale model evaluation;
• portable Proof-Carrying Statistical Answers using canonical JSON and SHA-256;
• proof write/read/verification and tamper detection;
• revision propagation lag with right-censoring;
• adversarial red-team prompts, statistical minimal pairs, and an offline OfficialStatAI-Bench starter set.

Core audit

library(statfidelity)

ref <- stat_reference(
  value = 10.4,
  indicator = "unemployment rate",
  geo = "Spain",
  time = "2025",
  unit = "percent",
  source = "Eurostat",
  dataset = "example_lfs",
  population = "labour force"
)

claim <- stat_claim(
  value = 10.4,
  indicator = "unemployment rate",
  geo = "Spain",
  time = "2025",
  unit = "%",
  source = "Eurostat",
  dataset = "example_lfs",
  population = "labour force"
)

aud <- audit_stat_ai(ref, claim)
aud
proof <- certify_claim(aud)
verify_proof(proof)

Official-data connectors

The package builds requests against the current official provider endpoints documented by Eurostat, the World Bank and OECD. Network access happens only when a fetch_*() function is explicitly called. Session-local caching is enabled by default; it can be disabled with cache = FALSE or redirected with cache_dir.

# Eurostat URL only (offline)
eurostat_url("demo_pjan", list(geo = "ES", time = "2025", sex = "T"))

# World Bank URL only (offline)
worldbank_url("SP.POP.TOTL", country = "FRA", start = 2020, end = 2025)

# OECD URL only (offline)
oecd_url("OECD.SDD.STES", "DSD_STES@DF_CLI", start_period = "2025")

Live retrieval examples:

# Requires internet access
# wb <- fetch_worldbank("SP.POP.TOTL", country = "FRA", start = 2025, end = 2025)
# ref <- as_stat_reference(wb, row = 1, unit = "persons")

Search across official catalogues

# Requires internet access
# hits <- search_official_stats(
#   "unemployment",
#   providers = c("worldbank", "ons", "unsd_sdg")
# )
# hits

Extend statfidelity to another official JSON API

# Session-local registration; no secret is written to disk.
register_stat_api(
  "my_nso",
  base_url = "https://api.example.gov/v1",
  search_path = "search",
  query_param = "q",
  api_key_name = "api_key",
  provider_label = "Example National Statistical Office"
)

# Optional key for the current R process only:
# set_stat_api_key("my_nso", "YOUR_KEY")
# search_stat_api("my_nso", "population")

Additional built-in endpoints:

# UNSD SDG metadata search and series data
# search_unsd_sdg("social protection")
# sdg <- fetch_unsd_sdg("SI_COV_SOCINS", page_size = 100)

# UK ONS full-text search
# search_ons("unemployment", content_type = "dataset")

# BLS public single-series retrieval
# bls <- fetch_bls("LNS14000000", latest = TRUE)

# IMF SDMX URL builder
imf_api_url("dataflow/IMF/all/latest", version = "3.0")

Provider documentation used by this package:

• Eurostat API Statistics: https://ec.europa.eu/eurostat/web/user-guides/data-browser/api-data-access/api-detailed-guidelines/api-statistics
• World Bank Indicators API: https://datahelpdesk.worldbank.org/knowledgebase/articles/889392-about-the-indicators-api-documentation
• OECD Data Explorer API: https://www.oecd.org/en/data/insights/data-explainers/2024/09/api.html
• UNSD SDG API: https://unstats.un.org/sdgapi/swagger/
• UK ONS Developer Hub: https://developer.ons.gov.uk/
• U.S. BLS Public Data API: https://www.bls.gov/bls/api_features.htm
• IMF Data APIs: https://data.imf.org/en/Resource-Pages/IMF-API

Transformation fidelity

ref <- stat_reference(
  -0.7, "unemployment-rate change", "Exampleland", "2024-2025",
  "percentage points", "Official provider",
  transformation = list(
    operation = "percentage point change",
    inputs = c(11.1, 10.4), result = -0.7
  )
)

claim <- stat_claim(
  -6.31, "unemployment-rate change", "Exampleland", "2024-2025",
  "percent", "Official provider",
  transformation = list(
    operation = "percent change",
    inputs = c(11.1, 10.4), result = -6.31
  )
)

audit_stat_ai(ref, claim)

Offline benchmark tools

head(officialstat_benchmark())
benchmark_from_reference(ref)

CRAN-oriented design

Live API calls are not required by tests or runnable examples. Parser tests use local fixtures. Provider requests are bounded, cached in the session by default, and only occur after an explicit user call. Before CRAN submission, build the source tarball with the current R release or R-patched and run R CMD check --as-cran on that built tarball.