Modern Text Mining Framework for R

Fast and memory-friendly tools for text vectorization, topic modeling (LDA, LSA), word embeddings (GloVe), similarities. This package provides a source-agnostic streaming API, which allows researchers to perform analysis of collections of documents which are larger than available RAM. All core functions are parallelized to benefit from multicore machines.

You've just discovered text2vec!

text2vec is an R package which provides an efficient framework with a concise API for text analysis and natural language processing (NLP).

Goals which we aimed to achieve as a result of development of text2vec:

Concise - expose as few functions as possible
Consistent - expose unified interfaces, no need to explore new interface for each task
Flexible - allow to easily solve complex tasks
Fast - maximize efficiency per single thread, transparently scale to multiple threads on multicore machines
Memory efficient - use streams and iterators, not keep data in RAM if possible

To learn how to use this package, see text2vec.org and the package vignettes. See also the text2vec articles on my blog.

Features

The core functionality at the moment includes

Fast text vectorization on arbitrary n-grams, using vocabulary or feature hashing.
GloVe word embeddings.
Topic modeling with:

Latent Dirichlet Allocation
Latent Sematic Analysis

Similarities/distances between 2 matrices

Cosine
Jaccard
Relaxed Word Mover's Distance
Euclidean

Performance

Author of the package is a little bit obsessed about efficiency.

This package is efficient because it is carefully written in C++, which also means that text2vec is memory friendly. Some parts, such as training GloVe word embeddings, are fully parallelized using the excellent RcppParallel package. This means that the word embeddings are computed in parallel on OS X, Linux, Windows, and Solaris (x86) without any additional tuning or tricks. Other emrassingly parallel tasks such as vectorization can use any parallel backend wich supports foreach package. So they can achieve near-linear scalability with number of available cores. Finally, a streaming API means that users do not have to load all the data into RAM.

Contributing

The package has issue tracker on GitHub where I'm filing feature requests and notes for future work. Any ideas are appreciated.

Contributors are welcome. You can help by:

testing and leaving feedback on the GitHub issuer tracker (preferably) or directly by e-mail
forking and contributing (chech code style guide). Vignettes, docs, tests, and use cases are very welcome
by giving me a star on project page :-)

License

GPL (>= 2)

News

text2vec 0.4.0

2016-10-03. See 0.4 milestone tags.

Now under GPL (>= 2) Licence
"immutable" iterators - no need to reinitialize them
unified models interface
New models: LSA, LDA, GloVe with L1 regularization
Fast similarity and distances calculation: Cosine, Jaccard, Relaxed Word Mover's Distance, Euclidean
Better hadnling UTF-8 strings, thanks to @qinwf
iterators and models rely on R6 package

text2vec 0.3.0

2016-01-13 fix for #46, thanks to @buhrmann for reporting
2016-01-16 format of vocabulary changed.
- do not keep doc_proportions. see #52.
- add stop_words argument to prune_vocabulary. signature also was changed.
2016-01-17 fix for #51. if iterator over tokens returns list with names, these names will be:
- stored as attr(corpus, 'ids')
- rownames in dtm
- names for dtm list in lda_c format
2016-02-02 high level function for corpus and vocabulary construction.
- construction of vocabulary from list of itoken.
- construction of dtm from list of itoken.
2016-02-10 rename transformers
- now all transformers starts with transform_* - more intuitive + simpler usage with autocompletion
2016-03-29 (accumulated since 2016-02-10)
- rename vocabulary to create_vocabulary.
- new functions create_dtm, create_tcm.
- All core functions are able to benefit from multicore machines (user have to register parallel backend themselves)
- Fix for progress bars. Now they are able to reach 100% and ticks increased after computation.
- ids argument to itoken. Simplifies assignement of ids to rows of DTM
- create_vocabulary now can handle stopwords
- see all updates here
2016-03-30 more robust split_into() util.

text2vec 0.2.0 (2016-01-10)

First CRAN release of text2vec.

Fast text vectorization with stable streaming API on arbitrary n-grams.
- Functions for vocabulary extraction and management
- Hash vectorizer (based on digest murmurhash3)
- Vocabulary vectorizer
GloVe algorithm word embeddings.
- Fast term-cooccurence matrix factorization via parallel async AdaGrad.
All core functions written in C++.