It appears you don't have a PDF plugin for this browser. You can click here to download the reference manual.
0.1.3 by Jan Wijffels, a year ago
https://github.com/bnosac/tokenizers.bpe
Browse source code at https://github.com/cran/tokenizers.bpe
Authors: Jan Wijffels [aut, cre, cph] (R wrapper) , BNOSAC [cph] (R wrapper) , VK.com [cph] , Gregory Popovitch [ctb, cph] (Files at src/parallel_hashmap (Apache License , Version 2.0) , The Abseil Authors [ctb, cph] (Files at src/parallel_hashmap (Apache License , Version 2.0) , Ivan Belonogov [ctb, cph] (Files at src/youtokentome (MIT License))
Documentation: PDF Manual
MPL-2.0 license
Imports Rcpp
Linking to Rcpp
Suggested by doc2vec, sentencepiece, textrecipes.
See at CRAN