Unicode and Punycode Domain Name Processing

High-performance Unicode and Punycode processing for internationalized domain names. The 'puny_encode()' / 'puny_decode()' helpers are a low-level, RFC 3492 compliant Punycode codec for domain labels (the 'xn--' ASCII-Compatible Encoding of RFC 5890/5891); they perform the raw transform plus letter-digit-hyphen checks and do not apply Unicode IDNA normalization. 'host_normalize()' is the Unicode Technical Standard #46 host-normalization entry point, mapping a host name to a canonical lowercase ASCII comparison form (non-transitional profile, pinned default Unicode version, selectable per call from the set the build ships). Aimed at host normalization and data analysis workflows. Used as the Punycode and IDNA engine by the 'pslr' and 'rurl' packages.

punycoder

High-performance Unicode and Punycode encoding/decoding for internationalized domain names (IDNs) in R.

Overview

The punycoder package provides fast, standards-based conversion between Unicode and ASCII representations of domain names, across two distinct surfaces:

• a low-level Punycode codec — puny_encode() / puny_decode() — the raw RFC 3492 transform with xn-- A-label framing (RFC 5890/5891) and letter-digit-hyphen checks, not an IDNA normalization API (no Unicode NFC, UTS #46 mapping, or case folding);
• an IDNA/UTS-46 host-normalization surface — host_normalize() — mapping a host name to its canonical lowercase ASCII comparison form under a pinned UTS #46 non-transitional profile.

host_normalize() is a UTS #46 profile, not IDNA2008 conformance — UTS #46 is compatibility processing and deliberately accepts labels IDNA2008 would reject (e.g. ☕.example → xn--53h.example). See ?host_normalize and normalization_profile_info() for the normative profile and full standards references (RFC 3492/5890/5891/5892/5893, UTS #46, UAX #15/#44, STD 3, RFC 8753).

Normalization runs against vendored Unicode data, and a build ships a set of Unicode versions with one pinned as the default (currently 17.0.0, with 16.0.0 also shipped). unicode_versions() reports what the installed build carries, and host_normalize(x, unicode_version = "16.0.0") selects another one for a single call — which normalization_profile_info() reflects by appending +unicode-<version> to the reported profile token.

Dependencies

punycoder has a small dependency footprint:

• Runtime dependencies: R (>= 3.5.0), Rcpp
• Optional system dependency: libidn2 (detected at compile time)
• Optional build helper: pkg-config (used by configure to detect libidn2)
• Development dependencies: testthat, knitr, rmarkdown

Installation

Install the released version of punycoder from CRAN with:

install.packages("punycoder")

Or install the development version from GitLab with:

# install.packages("remotes")
remotes::install_gitlab("bart-turczynski/punycoder")

Optional native backend (libidn2)

punycoder works without extra system libraries. If libidn2 is available at build time, the package enables a native backend automatically; otherwise it uses the built-in C++ fallback backend.

To install the recommended optional dependency:

• macOS (Homebrew):
  • brew install libidn2 pkg-config
• Debian/Ubuntu:
  • sudo apt-get install libidn2-0-dev pkg-config
• Fedora/RHEL/CentOS:
  • sudo dnf install libidn2-devel pkgconf-pkg-config
• Arch Linux:
  • sudo pacman -S libidn2 pkgconf

Verify the library is visible before installing punycoder from source:

system("pkg-config --modversion libidn2")

Then install/reinstall punycoder:

remotes::install_gitlab("bart-turczynski/punycoder")

Example

library(punycoder)

# Basic encoding
puny_encode("café.com")
#> [1] "xn--caf-dma.com"

# Check if domain is punycode
is_punycode("xn--example")
#> [1] TRUE

# Validate domains
validate_domain("test.com")
#> Punycoder Domain Validation Results
#> ==================================
#> 
#> 1 domain: 1 valid, 0 invalid (strict = TRUE)
#> 
#> Domain: test.com 
#> Valid:  TRUE

Command-line use

punycoder is a library, not a CLI, but a one-line Rscript wrapper covers the usual shell need: turning an IDN into the ASCII form that dig, nslookup, or curl will actually accept. No extra install, no bin/ script. The snippets below are POSIX sh/bash — keep the -e argument in single quotes so the shell leaves the R code alone.

Unicode host → ASCII, via the UTS #46 canonical form:

Rscript -e 'cat(punycoder::host_normalize(commandArgs(TRUE)), sep = "\n")' münchen.de
#> xn--mnchen-3ya.de

Use puny_encode() instead when you want the raw RFC 3492 transform with no UTS #46 mapping, and puny_decode() to go back the other way:

Rscript -e 'cat(punycoder::puny_encode(commandArgs(TRUE)), sep = "\n")' münchen.de
#> xn--mnchen-3ya.de

Rscript -e 'cat(punycoder::puny_decode(commandArgs(TRUE)), sep = "\n")' xn--mnchen-3ya.de
#> münchen.de

Feeding a resolver is then just command substitution (drop sep so nothing but the host is printed):

dig +short "$(Rscript -e 'cat(punycoder::host_normalize(commandArgs(TRUE)))' münchen.de)"
#> 194.246.166.100

nslookup "$(Rscript -e 'cat(punycoder::host_normalize(commandArgs(TRUE)))' münchen.de)"

All three functions are vectorized, so several hosts can go through one R startup and out to xargs:

Rscript -e 'cat(punycoder::host_normalize(commandArgs(TRUE)), sep = "\n")' münchen.de москва.рф |
  xargs -n 1 dig +short

Failed conversions

host_normalize() never aborts: an input it cannot normalize comes back as NA, which prints as a literal NA line rather than silently disappearing from the pipeline.

Rscript -e 'cat(punycoder::host_normalize(commandArgs(TRUE)), sep = "\n")' münchen.de 'bad..domain'
#> xn--mnchen-3ya.de
#> NA

puny_encode() and puny_decode() behave differently: they are strict by default, so a bad input stops the script with a non-zero exit status.

Rscript -e 'cat(punycoder::puny_encode(commandArgs(TRUE)), sep = "\n")' 'bad..domain'
#> Error: Error encoding domain: Domain contains empty label
#> Execution halted

Pass strict = FALSE to get the per-element NA behavior instead, so one bad host does not take down a batch:

Rscript -e 'cat(punycoder::puny_encode(commandArgs(TRUE), strict = FALSE), sep = "\n")' münchen.de 'bad..domain'
#> xn--mnchen-3ya.de
#> NA

Since NA is printed, not swallowed, drop it before piping to a resolver — otherwise dig dutifully looks up a host named NA:

Rscript -e 'h <- punycoder::host_normalize(commandArgs(TRUE)); cat(h[!is.na(h)], sep = "\n")' münchen.de 'bad..domain' |
  xargs -n 1 dig +short

Key Features

• Reliable Encoding/Decoding: RFC 3492 compliant punycode conversion
• High Performance: Vectorized operations for processing large datasets
• Comprehensive Validation: Robust error handling with informative messages
• Flexible Backend: Automatically uses libidn2 when available, with a built-in fallback backend

Use Cases

Data Analysis

Clean and standardize URL datasets:

# Identify international domains
is_idn(c("café.com", "example.com", "москва.рф"))

# Validate domain names
validate_domain(c("valid.com", "invalid..domain"))

Current State

punycoder currently provides:

• Low-level Punycode codec: puny_encode(), puny_decode()
• IDNA/UTS-46 host normalization: host_normalize(), normalization_profile_info(), unicode_versions()
• Domain validation utilities: is_punycode(), is_idn(), validate_domain()
• Vectorized operations and strict/non-strict handling for malformed input
• Build-time backend selection (libidn2 when present, built-in fallback otherwise)
• Best-effort structured host extraction where invalid inputs are returned as missing components

Non-goals

punycoder is a standards primitive for Punycode and host normalization. It is deliberately agnostic about resolvability and safety; the following are not part of its acceptance criteria:

• No spoof / homograph / mixed-script / display-safety detection. host_normalize() is not a safety gate — a successful result says the host is valid and normalized under the pinned UTS #46 profile, nothing about whether it is visually safe or non-deceptive. Confusable and restriction-level checks (UTS #39 / UTR #36, which UTS #46 itself recommends only as application/UI-layer steps) belong upstack.
• No URL parsing or canonicalization. RFC 3986 / WHATWG URL parsing and canonicalization belong upstack; see the rurl package, which consumes punycoder’s host functions.
• No DNS resolvability or registrability / PSL classification.
• No address parsing. There is no email-to-ASCII helper; splitting an address and IDNA-encoding its domain part is an addressing concern for an upstack consumer, not a Punycode primitive.
• No per-TLD repertoire / allowed-character validation. host_normalize() validates against the pinned UTS #46 profile, not against registry-specific IDN tables (which evolve independently of Unicode). TLD policy belongs upstack.

These opinions belong in higher layers that consume punycoder’s host functions.

Prior art and comparison

Punycode/IDN libraries exist in most ecosystems. punycoder is most directly a maintained, IDNA2008-era successor to the libidn-based R tooling — its public API (puny_encode() / puny_decode() / is_punycode()) descends from hrbrmstr/punycode. The table below situates it against representative libraries.

The most consequential row is IDNA standard. IDNA2003 (GNU libidn, nameprep) and IDNA2008 / UTS #46 disagree on real domains: the deviation characters ß, ς, and the joiners ZWJ/ZWNJ. Under IDNA2003 faß.de is mapped to fass.de — a different host — whereas punycoder’s pinned UTS #46 non-transitional profile preserves it as xn--fa-hia.de. A libidn-era pipeline therefore silently rewrites some hosts rather than erroring, which is the class of bug punycoder exists to remove.

Comparisons reflect each project’s public documentation as of this writing and describe documented behavior, not an independent audit.

Observed behavior on the comparable R packages

Running the same inputs through the comparable R packages surfaces concrete behavioral differences (observed against punycode 0.2.5, urltools 1.7.3.1, and the author’s own upstack toolkit rurl 1.4.0). The raw RFC 3492 codec output agrees byte-for-byte across the codecs once direction is aligned — the divergences are in multi-label handling, idempotency, validity philosophy, and input scope. rurl is a URL parser/normalizer rather than a Punycode codec; it is included to show where the URL-shaped inputs punycoder deliberately rejects are actually handled (it delegates IDNA host conversion to punycoder), so — below means “out of scope for that layer,” not a defect:

punycode names its functions opposite to the usual convention: punycode::puny_encode() maps xn-- → Unicode and punycode::puny_decode() maps Unicode → xn--. The rows above align by transform direction, not by function name.

punycoder + rurl (+ pslr for the public-suffix/TLD truth) are designed to compose: rurl parses the URL and hands the host to punycoder for IDNA canonicalization, each package owning a single concern.

Acknowledgments

These packages build on data, libraries, and prior work from many others. See ACKNOWLEDGMENTS.md for the full list of thanks.

Related packages

punycoder is part of a small ecosystem of R packages by the same author:

• pslr — Public Suffix List engine that uses punycoder for IDNA canonicalization. Use it for eTLD and registrable-domain queries.
• rurl — Full URL parsing, normalization, and joining toolkit built on top of both punycoder and pslr.

Citation

If you use punycoder in your work, please cite it. Run citation("punycoder") for the current citation, or see CITATION.cff.

Each release is archived on Zenodo. Cite the concept DOI 10.5281/zenodo.20973629 to refer to the software in general (it always resolves to the latest version), or the version-specific DOI shown on the Zenodo record for a particular release.

Contributing

We welcome contributions. See CONTRIBUTING.md for the current development workflow, ARCHITECTURE.md for how the package is structured, and DECISIONS.md for the design-decision log.

Code of Conduct

Please note that this package is released with a Contributor Code of Conduct. By contributing to this project, you agree to abide by its terms.

License

MIT