Item Response Theory Modeling and Scoring for Judgment Data

Psychometric analysis and scoring of judgment data using polytomous Item-Response Theory (IRT) models, as described in Myszkowski and Storme (2019) . A convenience function is used to automatically compare and select models, as well as to present a variety of model-based statistics. Plotting functions are used to present category curves, as well as information, reliability and standard error functions.


The goal of jrt is to provide tools to use Item-Response Theory (IRT) models on judgment data, especially in the context of the Consensual Assessment Technique, as presented in Myszkowski & Storme (2019).

  • Myszkowski, N., & Storme, M. (2019). Judge response theory? A call to upgrade our psychometrical account of creativity judgments. Psychology of Aesthetics, Creativity, and the Arts, 13(2), 167-175. http://dx.doi.org/10.1037/aca0000225

Installation

You can install the released version of jrt from CRAN with:

install.packages("jrt")

Example use

  • Load the library
library(jrt)
  • Load example dataset
data <- jrt::ratings
  • To automatically select models
fit <- jrt(data, progress.bar = F)
#> The possible responses detected are: 1-2-3-4-5
#> 
#> -== Model Selection (6 judges) ==-
#> AICc for Rating Scale Model: 4414.924 | Model weight: 0.000
#> AICc for Generalized Rating Scale Model: 4370.699 | Model weight: 0.000
#> AICc for Partial Credit Model: 4027.701 | Model weight: 0.000
#> AICc for Generalized Partial Credit Model: 4021.567 | Model weight: 0.000
#> AICc for Constrained Graded Rating Scale Model: 4400.553 | Model weight: 0.000
#> AICc for Graded Rating Scale Model: 4310.307 | Model weight: 0.000
#> AICc for Constrained Graded Response Model: 4003.993 | Model weight: 0.859
#> AICc for Graded Response Model: 4007.604 | Model weight: 0.141
#>  -> The best fitting model is the Constrained Graded Response Model.
#> 
#>  -== General Summary ==-
#> - 6 Judges
#> - 300 Products
#> - 5 response categories (1-2-3-4-5)
#> - Mean judgment = 2.977 | SD = 0.862
#> 
#> -== IRT Summary ==-
#> - Model: Constrained (equal slopes) Graded Response Model (Samejima, 1969) | doi: 10.1007/BF03372160
#> - Estimation package: mirt (Chalmers, 2012) | doi: 10.18637/jss.v048.i06
#> - Estimation algorithm: Expectation-Maximization (EM; Bock & Atkin, 1981) | doi: 10.1007/BF02293801
#> - Method of factor scoring: Expected A Posteriori (EAP)
#> - AIC = 3999.249 | AICc = 4003.993 | BIC = 4091.843 | SABIC = 3999.249
#> 
#> -== Model-based reliability ==-
#> - Empirical reliability | Average in the sample: .893
#> - Expected reliability | Assumes a Normal(0,1) prior density: .894

  • To select models a priori
fit <- jrt(data, irt.model = "PCM")
#> The possible responses detected are: 1-2-3-4-5
#> 
#>  -== General Summary ==-
#> - 6 Judges
#> - 300 Products
#> - 5 response categories (1-2-3-4-5)
#> - Mean judgment = 2.977 | SD = 0.862
#> 
#> -== IRT Summary ==-
#> - Model: Partial Credit Model (Masters, 1982) | doi: 10.1007/BF02296272
#> - Estimation package: mirt (Chalmers, 2012) | doi: 10.18637/jss.v048.i06
#> - Estimation algorithm: Expectation-Maximization (EM; Bock & Atkin, 1981) | doi: 10.1007/BF02293801
#> - Method of factor scoring: Expected A Posteriori (EAP)
#> - AIC = 4022.957 | AICc = 4027.701 | BIC = 4115.551 | SABIC = 4022.957
#> 
#> -== Model-based reliability ==-
#> - Empirical reliability | Average in the sample: .889
#> - Expected reliability | Assumes a Normal(0,1) prior density: .759

  • To plot all category curves
jcc.plot(fit)
  • To plot on judge's category curves
jcc.plot(fit, judge = 1)
  • Graphical options
jcc.plot(fit, judge = 1, overlay.reliability = T, greyscale = T, theme = "classic")
  • To plot total information
info.plot(fit)
  • To plot judge information
info.plot(fit, judge = 1)
  • Other options for information plots
info.plot(fit, type = "Reliability",
          y.line = .70,
          y.limits = c(0,1),
          theta.span = 4,
          theme = "classic")

News

jrt 1.0.1

  • Simulated dataset is now directly included as jrt::ratings.
  • Model weights (often called Akaike weights) are now presented in the model selection output (per reviewer suggestion).
  • Calls to mirt can now be showed with show.calls = TRUE argument in the jrt()function.
  • References were updated.

jrt 1.0.0

  • Initial release with functions jrt(), jcc.plot() and info.plot()

Reference manual

It appears you don't have a PDF plugin for this browser. You can click here to download the reference manual.

install.packages("jrt")

1.0.1 by Nils Myszkowski, 4 months ago


Browse source code at https://github.com/cran/jrt


Authors: Nils Myszkowski [aut, cre]


Documentation:   PDF Manual  


GPL-3 license


Imports mirt, psych, irr, methods, utils, stats, dplyr, tidyr, ggplot2, ggsci

Depends on directlabels

Suggests knitr, rmarkdown


See at CRAN