Reference manual

It appears you don't have a PDF plugin for this browser. You can click here to download the reference manual.

install.packages("tokenizers.bpe")

0.1.3 by Jan Wijffels, a year ago


https://github.com/bnosac/tokenizers.bpe


Browse source code at https://github.com/cran/tokenizers.bpe


Authors: Jan Wijffels [aut, cre, cph] (R wrapper) , BNOSAC [cph] (R wrapper) , VK.com [cph] , Gregory Popovitch [ctb, cph] (Files at src/parallel_hashmap (Apache License , Version 2.0) , The Abseil Authors [ctb, cph] (Files at src/parallel_hashmap (Apache License , Version 2.0) , Ivan Belonogov [ctb, cph] (Files at src/youtokentome (MIT License))


Documentation:   PDF Manual  


MPL-2.0 license


Imports Rcpp

Linking to Rcpp


Suggested by doc2vec, sentencepiece, textrecipes.


See at CRAN