Tidy Tools for Chinese, Japanese and Korean Text

A tidy toolkit for text that is written in Chinese, Japanese or Korean. Most text tooling in R assumes that words are separated by whitespace, which CJK writing does not use, so ordinary summaries of a text column either treat a sentence as one undifferentiated blob or split it into isolated characters. Word segmentation is a pluggable engine, with dictionary-based segmentation for Chinese and Japanese supplied through 'ICU' and further engines registrable by the caller, because where a word ends is a fact about a language and not about Unicode. 'tidycjk' classifies characters by Unicode block, reports which script and which language a text is written in, measures how much of a text is CJK, and turns those measurements into tibbles that slot straight into a 'tidyverse' workflow. It also measures display width in terminal columns, pads, truncates and wraps to a width rather than to a character count, and normalises fullwidth and halfwidth forms surgically -- including composing halfwidth katakana voiced marks into single code points -- while offering the Unicode normalisation forms separately for when a full 'NFKC' fold is what is wanted. Romanisation, conversion between simplified and traditional Han, conversion between the two kana syllabaries, and decomposition of Hangul syllables into jamo are provided on the same vectorised contract. Sentence segmentation, character n-grams, punctuation removal by Unicode category and locale-aware ordering cover the preprocessing that generic tooling gets wrong on CJK, where the sentence terminators are ideographic, a code point sort is not an alphabetical one, and the POSIX punctuation class is resolved by the C library rather than by Unicode. Language detection deliberately returns NA rather than guessing when a text is written in Han characters only, because Japanese written without kana cannot be distinguished from Chinese by script alone. Everything is derived from the Unicode specification; the package makes no network requests and needs no compiled code of its own.

tidycjk

Chinese, Japanese and Korean text in R: measured in columns, split into words, and kept in tibbles.

Padding by character count leaves a CJK column ragged; padding by display width aligns it.

📦 Install

install.packages("tidycjk")
library(tidycjk)
cjk_width("中文") prints 4: two characters, four terminal columns.

✨ What you get

Task	Verbs
Detect	`has_cjk()` · `cjk_script()` · `cjk_detect_language()` · `cjk_ratio()`
Lay out	`cjk_width()` · `cjk_pad()` · `cjk_truncate()` · `cjk_wrap()`
Segment	`cjk_segment()` · `cjk_tokens()` · `cjk_sentences()` · `cjk_ngrams()`
Clean	`to_halfwidth()` · `to_fullwidth()` · `cjk_normalize()` · `cjk_strip_punct()`
Convert	`cjk_romanize()` · `cjk_simplify()` · `cjk_traditionalize()` · `to_hiragana()` · `to_katakana()` · `cjk_jamo()` · `cjk_compose_jamo()`
Summarise, sort	`cjk_summary()` · `cjk_char_counts()` · `cjk_sort()` · `cjk_order()` · `cjk_blocks()`

Every verb is vectorised and keeps NA; the three that take (data, column) return a tibble for a dplyr pipeline.

🔍 Examples

cjk_detect_language(c("こんにちは", "안녕하세요", "東京都"))
#> [1] "japanese" "korean"   NA
cjk_segment("我今天很開心", engine = "icu")
#> [[1]]
#> [1] "我"   "今天" "很"   "開心"
cjk_pad(c("中文", "abcd"), 6)
#> [1] "中文  " "abcd  "
cjk_sort(c("張", "王", "李"), locale = "zh")
#> [1] "李" "王" "張"
cjk_summary(data.frame(text = c("我今天很開心", "hello 中文", "no CJK")), text)
#> # A tibble: 1 × 4
#>   n_docs n_with_cjk prop_with_cjk mean_ratio
#>    <int>      <int>         <dbl>      <dbl>
#> 1      3          2         0.667      0.417

東京都 is Tokyo: Japanese written only in Han characters, which no script test can tell from Chinese, so the answer is NA rather than a guess.

⚠️ Gotchas

Trap	What to do
Han-only text has no detectable language	pass `han_only = "chinese"` if you know the corpus
`engine` has no default	`"icu"` for words, `"character"` for characters
`cjk_romanize()` reads each Han character alone, as Chinese	treat it as a sort key; use gibasa for Japanese readings
`sort()` on Han depends on `LC_COLLATE`	`cjk_sort(x, locale = "zh")`
Plain NFC rewrites 460 of the 472 compatibility ideographs	keep the original column if the distinction matters
A file in GBK, Big5 or Shift_JIS	name it when reading: `read.csv(f, fileEncoding = "GBK")`

📚 Learn more

Four vignettes (tidycjk, segmentation, width-and-layout, transliteration) and a visual tour. For more than ICU gives: gibasa (MeCab, with part of speech), udpipe (models for all three languages), jiebaR (archived from CRAN, install from GitHub) and OpenCC (regional vocabulary).