An Integrated Framework for Textual Sentiment Time Series Aggregation and Prediction

Optimized prediction based on textual sentiment, accounting for the intrinsic challenge that sentiment can be computed and pooled across texts and time in various ways. See Ardia et al. (2018) .

Introduction

The sentometrics package is designed to perform time series analysis based on textual sentiment. Put differently, it is an integrated framework for textual sentiment time series aggregation and prediction. It accounts for the intrinsic challenge that, for a given text, sentiment can be computed in many different ways, as well as the large number of possibilities to pool sentiment across texts and time. This additional layer of manipulation does not exist in standard text mining and time series analysis packages. As a final outcome, the package provides an automated means to econometrically model the impact of sentiment in texts on a given variable, by first computing a wide range of textual sentiment time series and then selecting those that are most informative. The package therefore integrates the fast qualification of sentiment from texts, the aggregation into different sentiment measures and the optimized prediction based on these measures.

The package implements the main methodology developed in the paper "Questioning the news about economic growth: Sparse forecasting using thousands of news-based sentiment values" (Ardia, Bluteau and Boudt, 2017). See the project page and the vignette for respectively a brief and an extensive introduction to the package.

Installation

To install the package from CRAN, simply do:

install.packages("sentometrics")

The latest development version of sentometrics is available at https://github.com/sborms/sentometrics. To install this version (which may contain bugs!), execute:

devtools::install_github("sborms/sentometrics")

References

Please cite sentometrics in publications. Use citation("sentometrics").

Acknowledgements

This software package originates from a Google Summer of Code 2017 project.

News

sentometrics 0.5.1

minor modifications to resolve few CRAN issues
set default value of nCore argument from the compute_sentiment() and ctr_agg() functions to 1
classed the output of the compute_sentiment.sentocorpus() function as a sentiment object, and modified the aggregate() function to aggregate.sentiment()

sentometrics 0.5.0

new functions: weights_beta(), get_dates(), get_dimensions(), get_measures(), and get_loss_data()
renamed following functions: to_global() to measures_global(), perform_agg() to aggregate(), almons() to weights_almon(), exponentials() to weights_exponential(), setup_lexicons() to sento_lexicons(), retrieve_attributions() to attributions(), plot_attributions() to plot.attributions()
defunct the ctr_merge() function, so that all merge parameters have to be passed on directly to the measures_merge() function
expanded the use of the center and scale arguments in the scale() function
added the dateBefore and dateAfter arguments to the measures_fill() function, and dropped NA option of its fill argument
added a "beta" time aggregation option (see associated weights_beta() function)
corrected update of "attribWeights" element of output sentomeasures object in required measures_xyz() functions
added a new attribution dimension ("lags") to the attributions() function, and corrected some edge cases
made a slight correction to the information criterion estimators
added a lambdas argument to the ctr_model() function, directly passed on to the glmnet::glmnet() function if used
omitted do.combine argument in measures_delete() and measures_select() functions to simplify
expanded set of unit tests, included a coverage badge, and added covr to Suggests
reimplementation (and improved documentation) of the sentiment calculation in the compute_sentiment() function, by writing part of the code in Rcpp relying on RcppParallel (added to Imports); there are now three approaches to computing sentiment (unigrams, bigrams and clusters)
replaced the dfm argument from the compute_sentiment() and ctr_agg() functions by a tokens argument, and altered the input and behaviour of the nCore argument in these same two functions
switched from the quanteda package to the stringi package for more direct tokenisation
trimmed the list_lexicons and list_valence_shifters built-in word lists by keeping only unigrams, and included same trimming procedure in the sento_lexicons() function
added a type column "t" to the list_valence_shifters built-in word list, and reset values of the "y" column from 2 to 1.8 and from 0.5 to 0.2
updated the epu built-in dataset with the newest available series, up to July 2018
corrected the word 'sparesly' to 'sparsely' in list_valence_shifters[["en"]]
further shortened project page to the bare essence
omitted statement printed ('Compute sentiment... Done.') in the compute_sentiment() function
slightly modified print() generic for a sentomeasures object
dropped the "tf-idf" option for within-document aggregation in the ctr_agg() function
the sento_lexicons() function outputs a sentolexicons object, which the compute_sentiment() function specifically requires as an input; a sentolexicons object also includes a "[" class-preserving extractor function
the attributions() function outputs an attributions object; the plot_attribtutions() function is therefore replaced by the plot() generic
defunct the perform_MCS() function, but the output of the get_loss_data() function can easily be used as an input to the MCSprocedure() function from the MCS package (discarded from Imports)
moved the parallel and doParallel packages to Suggests, as only needed (if enacted) in the sento_model() function
sligthly modified appearance of plotting functions, to drop ggthemes from Imports

sentometrics 0.4.0

new functions: measures_delete(), nmeasures(), nobs(), and to_sentocorpus()
renamed following functions: any xyz_measures() to measures_xyz(), extract_peakdocs() to peakdocs()
dropped do.normalizeAlm argument in the ctr_agg() function, but kept in the almons() function
inverted order of rows in output of the almons() function to be consistent with Ardia et al. (2017) paper
renamed lexicons to list_lexicons, and valence to list_valence_shifters
the stats element of a sentomeasures object is now also updated in measures_fill()
changed "_eng" to "_en"' in list_lexicons and list_valence_shifters objects, to be in accordance with two-letter ISO language naming
changed "valence_language" naming to "language" in list_valence_shifters object
the compute_sentiment() function now also accepts a quanteda corpus object and a character vector
the add_features() function now also accepts a quanteda corpus object
added an nCore argument to the compute_sentiment(), ctr_agg(), and ctr_model() functions to allow for (more straightforward) parallelized computations, and omitted the do.parallel argument in the ctr_model() function
added a do.difference argument to the ctr_model() function and expanded the use of the already existing oos argument
brought ggplot2 and foreach to Imports

sentometrics 0.3.5

faster to_global()
set tolower = FALSE of quanteda::dfm() constructor in compute_sentiment()
changed intercept argument in ctr_model() to do.intercept for consistency
proper checks on values of feature columns in sento_corpus() and add_features()