0.2.1 by Jan Wijffels, 6 months ago
https://github.com/bnosac/sentencepiece
Browse source code at https://github.com/cran/sentencepiece
Authors: Jan Wijffels [aut, cre, cph] (R wrapper) , BNOSAC [cph] (R wrapper) , Google Inc. [ctb, cph] (Files at src/sentencepiece/src (Apache License , Version 2.0) , The Abseil Authors [ctb, cph] (Files at src/third_party/absl (Apache License , Version 2.0) , Google Inc. [ctb, cph] (Files at src/third_party/protobuf-lite (BSD-3 License)) , Kenton Varda (Google Inc.) [ctb, cph] (Files at src/third_party/protobuf-lite: coded_stream.cc , extension_set.cc , generated_message_util.cc , generated_message_util.cc , message_lite.cc , repeated_field.cc , wire_format_lite.cc , zero_copy_stream.cc , zero_copy_stream_impl_lite.cc , google/protobuf/extension_set.h , google/protobuf/generated_message_util.h , google/protobuf/wire_format_lite.h , google/protobuf/wire_format_lite_inl.h , google/protobuf/message_lite.h , google/protobuf/repeated_field.h , google/protobuf/io/coded_stream.h , google/protobuf/io/zero_copy_stream_impl_lite.h , google/protobuf/io/zero_copy_stream.h , google/protobuf/stubs/common.h , google/protobuf/stubs/hash.h , google/protobuf/stubs/once.h , google/protobuf/stubs/once.h.org (BSD-3 License)) , Sanjay Ghemawat (Google Inc.) [ctb, cph] (Design of files at src/third_party/protobuf-lite: coded_stream.cc , extension_set.cc , generated_message_util.cc , generated_message_util.cc , message_lite.cc , repeated_field.cc , wire_format_lite.cc , zero_copy_stream.cc , zero_copy_stream_impl_lite.cc , google/protobuf/extension_set.h , google/protobuf/generated_message_util.h , google/protobuf/wire_format_lite.h , google/protobuf/wire_format_lite_inl.h , google/protobuf/message_lite.h , google/protobuf/repeated_field.h , google/protobuf/io/coded_stream.h , google/protobuf/io/zero_copy_stream_impl_lite.h , google/protobuf/io/zero_copy_stream.h (BSD-3 License)) , Jeff Dean (Google Inc.) [ctb, cph] (Design of files at src/third_party/protobuf-lite: coded_stream.cc , extension_set.cc , generated_message_util.cc , generated_message_util.cc , message_lite.cc , repeated_field.cc , wire_format_lite.cc , zero_copy_stream.cc , zero_copy_stream_impl_lite.cc , google/protobuf/extension_set.h , google/protobuf/generated_message_util.h , google/protobuf/wire_format_lite.h , google/protobuf/wire_format_lite_inl.h , google/protobuf/message_lite.h , google/protobuf/repeated_field.h , google/protobuf/io/coded_stream.h , google/protobuf/io/zero_copy_stream_impl_lite.h , google/protobuf/io/zero_copy_stream.h (BSD-3 License)) , Laszlo Csomor (Google Inc.) [ctb, cph] (Files at src/third_party/protobuf-lite: io_win32.cc , google/protobuf/stubs/io_win32.h (BSD-3 License)) , Wink Saville (Google Inc.) [ctb, cph] (Files at src/third_party/protobuf-lite: message_lite.cc , google/protobuf/wire_format_lite.h , google/protobuf/wire_format_lite_inl.h , google/protobuf/message_lite.h (BSD-3 License)) , Jim Meehan (Google Inc.) [ctb, cph] (Files at src/third_party/protobuf-lite: structurally_valid.cc (BSD-3 License)) , Chris Atenasio (Google Inc.) [ctb, cph] (Files at src/third_party/protobuf-lite: google/protobuf/wire_format_lite.h (BSD-3 License)) , Jason Hsueh (Google Inc.) [ctb, cph] (Files at src/third_party/protobuf-lite: google/protobuf/io/coded_stream_inl.h (BSD-3 License)) , Anton Carver (Google Inc.) [ctb, cph] (Files at src/third_party/protobuf-lite: google/protobuf/stubs/map_util.h (BSD-3 License)) , Maxim Lifantsev (Google Inc.) [ctb, cph] (Files at src/third_party/protobuf-lite: google/protobuf/stubs/mathlimits.h (BSD-3 License)) , Susumu Yata [ctb, cph] (Files at src/third_party/darts_clone (BSD-3 License) , Daisuke Okanohara [ctb, cph] (File src/third_party/esaxx/esa.hxx (MIT License)) , Yuta Mori [ctb, cph] (File src/third_party/esaxx/sais.hxx (MIT License)) , Benjamin Heinzerling [ctb, cph] (Files data/models/nl.wiki.bpe.vs1000.d25.w2v.txt , data/models/nl.wiki.bpe.vs1000.d25.w2v.bin and data/models/nl.wiki.bpe.vs1000.model (MIT License))
Documentation:
PDF Manual
MPL-2.0 license
Suggests tokenizers.bpe, word2vec
Linking to Rcpp
Suggested by textrecipes.