Automatic Generation of Exams in R

Automatic generation of exams based on exercises in Markdown or LaTeX format, possibly including R code for dynamic generation of exercise elements. Exercise types include single-choice and multiple-choice questions, arithmetic problems, string questions, and combinations thereof (cloze). Output formats include standalone files (PDF, HTML, Docx, ODT, ...), Moodle XML, QTI 1.2, QTI 2.1, Blackboard, Canvas, OpenOLAT, ARSnova, and TCExam. In addition to fully customizable PDF exams, a standardized PDF format (NOPS) is provided that can be printed, scanned, and automatically evaluated.


Changes in Version 2.3-2

o Added new interface exams2ilias() for the open-source ILIAS learning management system ( This is essentially a convenience wrapper to exams2qti12(), tweaking a few defaults and employing a somewhat modified XML template.

o nops_scan() gained a new argument trim=0.3 that controls how much of check boxes is trimmed in order to shave the borders prior to determining the average gray level. In versions up to 2.3-1 this implicitly had a value of 0.25 hard-coded. Now the default was increased to 0.3 in order to shave box borders more reliably.

o nops_scan() tries to process registration numbers more reliably. In case one of the registration columns contains more than one potential mark, the heuristics of determining the intended mark have been improved.

o New exercise confint3.Rmd/Rnw that illustrates how to use the "verbatim" clozetype for Moodle. The exercises yields 100% of the points for the correct solution based on t quantiles but still 50% for a partially correct solution based on normal quantiles (contributed by Ulrike Groemping).

o nops_eval() gained a new argument labels=NULL that can be used to give labels for the marks that differ from the default (length(mark) + 1):1.

o In all LaTeX templates that use Helvetica (phv) as the font for the main text, this is also used now in math mode by \usepackage[helvet]{sfmath} (rather than \usepackage{sfmath}, as employed previously). In particular, this also affects exams2nops() and tex2image() output.

o In exams2nops() the "header" argument can also be specified simply as header = "\mycommand{value}" rather than header = list(mycommand = "value"). The former is more flexible, e.g., for passing additional options or more complex commands. Internally, the former is turned into an unnamed list which is then processed correspondingly by exams2pdf().

o Various bug fixes in exams2qti21() pertaining to points and correct/incorrect computations for schoice/mchoice exercises.

o New language support in exams2nops: Russian (ru, contributed by Boris Demeshev) and Serbian (sr, contributed by Tatjana Kecojevic). Croatian (hr) was streamlined along with Serbian (by Tatjana Kecojevic).

o In include_supplement() an argument target=NULL was added to optionally include the supplement with a different file name than the original file name.

o Bug fix in the pandoc interface which previously erroneously produced unbalanced

tags in certain situations.

o Rather than fully importing the basic dependencies "stats", "graphics", "grDevices", "tools", and "utils", the required functions are imported selectively only. (Issue raised by Florian Oswald.)

Changes in Version 2.3-1

o Added new interface exams2openolat() for the open-source OpenOLAT learning management system ( This is only a convenience wrapper to exams2qti12() or exams2qti21() with some dedicated tweaks for optimizing MathJax output for OpenOLAT.

o When using exams2html(..., mathjax = TRUE) for testing purposes, is used now rather than which is currently redirecting and will eventually be shut down completely.

o Bug fixes and improvements in HTML transformers: - Only ="file.ext" (with =") is replaced now by Base 64 version. - href="file.ext" is replaced by href="file.ext" download="file.ext" prior to Base 64 replacement. - alt="file.ext" and download="file.ext" are preserved without Base 64 encoding.

o In exams2html() and other interfaces based on make_exercise_transform_html() base64 = TRUE now uses Base 64 encoding for all file extensions (known to the package) whereas base64 = NULL only encodes image files (previous default behavior).

o After setting a random seed exams2nops() and exams2pdf() now yield the same random versions of the exercises. Previously, this was not the case because exams2nops() internally generates a single random trial exam first for a couple of sanity checks. Now, the .GlobalEnv$.Random.seed is restored after generating the trial exam.

o Fixed the support for 'nsamp' argument in exams2nops(). Furthermore, current limitations of exams2nops() are pointed out more clearly and error messages and edge cases caught.

o Include further file URIs for Base 64 supplements, in particular .sav for SPSS data files.

o New language support in exams2nops: Croatian (hr, contributed by Krunoslav Juraic), Danish (da, contributed by Tue Vissing Jensen & Jakob Messner), Slovak (sk, contributed by Peter Fabsic), Swiss German (gsw, contributed by Reto Stauffer), Turkish (tr, contributed by Emrah Er). Furthermore, Portuguese has been distinguished into pt-PT (Portuguese Portuguese) vs. pt-BR (Brazilian Portuguese) with pt defaulting to the former (contributed by Thomas Dellinger).

o include_supplement(..., dir = "foo") now also works if "foo" is sub-directory to the exercise directory (edir).

o Allow varying points within a certain exercise in nops_eval().

o exams2blackboard(..., base64 = FALSE, ...) was erroneously ignored. No matter how base64 was specified essentially base64 = TRUE was used, it is honored again now.

o Fixed a bug in stresstest_exercise() where the "rank" (previously called "order") of the correct solution was computed incorrectly. Enhancement in plots and labels.

o Fixed a bug for tex2image(..., tikz = TRUE) where erroneously \usetikzlibrary{TRUE} was included. Also tex2image(..., Sweave = TRUE) (the default) did not run properly on Windows, fixed now.

o New function include_tikz() that facilitates compiling standalone tikz figures and including it in an exercise (especially for non-LaTeX-based output).

o Added support for \tightlist (as produced by pandoc) in all current LaTeX templates as well as exams2nops().

o \exshuffle{} can now also be used for schoice exercises with more than one TRUE answer. In a first step only one of the TRUE answers is selected and then -1 items from the FALSE answers.

o Better warnings if \exshuffle{} could not be honored due to a lack of sufficiently many (suitable) answer alternatives.

o Enable passing of 'envir' argument from exams2html() to xweave() in case of .Rmd exercises.

o Bug fix in CSV export of exams2arsnova(). Recent ARSnova versions use "mc" (rather than "MC") and "abcd" (rather than "SC") for multiple-choice and single-choice questions, respectively.

Changes in Version 2.3-0

o A new web page accompanying the package is now available at This already contains a first overview of the package, some tutorials, and a gallery of all exercise templates provided in the package. More tutorials will follow in the form of blog articles.

o A few exercise templates were slightly changed or improved for the new web page. Hence slightly different formulations or formatting may be used in some places.

o tex2image() has been rewritten to use the LaTeX class {standalone} instead of manually handling boxes and cropping of the PDF output. For converting to raster graphics like PNG (the default), R package "magick" is now used rather than system("convert ...") calls to ImageMagick on the system. For conversion to SVG it is still necessary to call pdfcrop and pdf2svg on via system().

o More fixups in LaTeX code prior to processing with pandoc_convert(), especially \url{...} is replaced by \href{...}{\texttt{...}} now to ensure typesetting in typewriter font.

o When including R input code chunks in .Rmd exercises, code decoration is switched off by default now (knitr option highlight=FALSE) so that the output can be combined with standard LaTeX templates. The default can be changed either in the individual .Rmd code chunks, e.g., ```{r, highlight=TRUE}, or by setting highlight=TRUE in the call to xweave(). For the latter case a new LaTeX template plain-highlight.tex is also provided, e.g., to be used via: exams2pdf("lm.Rmd", highlight = TRUE, template = "plain-highlight")

o exams2pdf() gained an argument texdir=NULL. By default a temporary directory is created for running LaTeX (and cleaned up subsequently). But using texdir="/path/to/dir" an output directory can be specified so that the LaTeX files can be inspected, e.g., for debugging.

o In exams2pdf() the "control" argument was extended to encompass a new "cloze.collapse" option to fine tune the display of mchoice/schoice alternatives within a cloze subexercise. By default the separator is " / " for collapsing the choice alternatives. Alternatively, for example, it could be set to "\\" to add line breaks between the alternatives. Finally, control = list(cloze.collapse = "enumerate") uses a nested enumerate environment.

o nops_scan() now uses the 'dir' argument not only for the output ZIP file but also for the input PDF/PNG image files.

o exams2pdf() and hence also exams2nops() now try to iconv() the header information to the specified 'encoding' (if any) rather than relying on header information (such as institution name etc.) are specified in LaTeX.

Changes in Version 2.2-1

o Added new interface exams2pandoc() which can generate a wide variety of output formats via pandoc. The main purpose is to generate .docx output files (or .odt). The output format can be controlled through a template in either LaTeX, HTML, or Markdown format.

o New function stresstest_exercise() for "stress testing" of exercises, i.e., rerunning the exercise many times and capturing how the outcome depends on the simulated input parameters.

o New (and still somewhat experimental) function match_exams_call() that can be employed to query which exams2xyz() function was called to create a certain exercise/exam. Only works from R 3.2.0 onwards.

o Bug fix in read_exercise() for the case when exshuffle = integer. This was not correctly reflected in metainfo$length.

o Avoid

formatting in questionlist/solutionlist when using converter = "pandoc" in the conversion to HTML.

o Formatting of exercises without "solution" section improved in exams2html() and exams2moodle().

o In exams2qti12() the case of no answer was not handled correctly if "partial = FALSE, negative = ..." was used. No answer was handled as incorrect (= negative points) rather than neutral (= zero points) but has been fixed now.

Changes in Version 2.2-0

o Added first release version of exams2blackboard(). This is essentially based on exams2qti12() but incorporates Blackboard's own flavor of QTI 1.2. It is still likely to change due to improvements in future versions. After uploading a test into Blackboard the material will appear under "Course Tools": The test will be available in "Tests" and each pool within the test will also appear in "Pools".

o Added a new convenience function include_supplement() which facilitates including static supplement files (e.g., graphics or datasets). This essentially just calls file.copy() but facilitates finding supplement files that are available along with the .Rnw/.Rmd exercise files.

o xweave() gained an extra argument svg=FALSE to optionally produce scalable vector graphics (SVG) instead of PNG for later rendering into HTML. All HTML-based exams2xyz() functions gained an svg=FALSE argument that can be set to TRUE to produce SVG instead of PNG. Base64 encoding is also used for SVG graphics by default.

o xweave() gained an argument engine=NULL that can optionally be set to engine="knitr" to enable processing Rnw exercises by knitr::knit() instead of the default utils::Sweave().

o exams2nops() now supports registration IDs with more than 7 digits. Up to now, 7 digits were hard-coded but now values of 8, 9, or 10 are also possible by setting the argument "reglength" in exams2nops() accordingly.

o In exams2nops() the argument 'blank' can now be a vector of length 2 with the number of blank pages before and after the extra "pages".

o Added arguments samepage=FALSE and twocolumn=FALSE in exams2nops(). If set to TRUE, samepage adds a {samepage} environment around the question and solution lists in order to prevent page breaks within these lists. And the twocolumn option, if set to TRUE, is added in the \documentclass so that the exam is displayed in two-column format.

o New language support in exams2nops: Hungarian (hu, contributed by Gergely Daroczi and Denes Toth), Finnish (fi, contributed by Klaus Nordhausen). Improvements in: Portugese (pt, contributed by Mauricio Calvao).

o Improvements and bug fixes in exams2tcexam(). Various HTML tags are now supported in the development version by adding a new [html] tag in TCExam. This has been integrated into UIBK's version of TCExam but is not in the official release yet.

o Fixed a bug in exams2html() with option solution = FALSE or question = FALSE. Now these correctly suppress the corresponding element of the exercise. In previous versions only the header had been set to NA erroneously.

o exams2moodle() can now optionally turn "string" exercises in "essay" quiz items rather than the default "shortanswer". Various customization options can be set the "exextra" metainformation. The most important options are illustrated in the new "essayreg" exercise.

o \exshuffle{} can now also be used in read_exercise() for subsampling rather than permutations only, e.g., via \exshuffle{3}. This first permutes the alternatives, always uses the first permuted TRUE (if any), the first permuted FALSE (if any), and then the first remaining alternatives. This facilitates static schoice/mchoice exercises which have many alternatives but should only use a limited number (say 5 for exams2nops) in each random replication.

o Bug fix in extract_extra() which prevented "exextra" metainformation from being read correctly in the previous version.

o Improvement in xexams() to better support using the same exercise template several times within the same exam (e.g., for generating a single PDF with multiple versions of the same exercise). The temporary file names employed are now run through make.unique() first.

o Improvement in exams2pdf() to assure that duplication in file names for graphics (across exercises within the same exam) are handled correctly.

o Within several exams2xyz() functions gsub(...) calls were changed to gsub(..., fixed = TRUE) for replacing placeholders that potentially have slashes.

o Updated exams2arsnova() to assure that ( ... ) instead of $ ... $ is used for inline math. Also the JSON output was augmented with (almost empty) "publicPool" information to keep current ARSnova versions happy.

o New exercises "deriv"/"deriv2" (both in .Rnw and .Rmd) that illustrate a simple question about derivatives to be solved by the product rule. The "deriv" exercise is in "num" format and "deriv2" is the same in "schoice" (suggested by Martin Obradovits).

o The tth/ttm-based HTML conversion now makes sure that empty question/solution/list elements remain empty and do not introduce line breaks.

o Added copies of num_to_schoice(), num_to_tol(), ... called num2schoice(), and num2tol(), ...

Changes in Version 2.1-0

o Exercises can now either be in .Rnw format (R/LaTeX, as in previous versions) or in .Rmd format (R/Markdown). A new function xweave() is provided for weaving either format by either calling utils::Sweave() for .Rnw and knitr::knit() for .Rmd. If .Rmd exercises are used, the weaved exercises require conversion by pandoc (see below).

o The vignettes have not yet been updated to incorporate the R/Markdown information or the new interfaces described below. For examples and some introductory comments, exams_skeleton() can generate demo scripts.

o Added new make_exercise_transform_pandoc() transfomer generator, interfacing rmarkdown::pandoc_convert(). This provides an additional option for generating HTML output and additionally enables Markdown output (e.g., for ARSnova, see below).

o Added new interface exams2nops() for generating PDF exams that can be printed, scanned, and automatically evaluated. Accompanying functions nops_scan() (using the png package) and nops_eval() are provided as well. The "nops" exams can be internationalized. So far there is language support for English (en), Dutch (nl, contributed by Niels Smits), French (fr, contributed by Arthur Allignol), German (de), Italian (it, contributed by Domenico Zambella), Portuguese (pt, contributed by Fabian Petutschnig), Romanian (ro, contributed by Cristian Gatu), Spanish (es, contributed by Maria Kogelnik).

o Added new interface exams2arsnova() for exporting exams/quizzes to the interactive audience response system ARSnova ( This can either create JSON output files (via RJSONIO::toJSON) that need to be manually imported into ARSnova - or the JSON string can directly be imported into an active ARSnova session (via RCurl).

o Added "abstention" option in schoice/mchoice questions in exams2moodle(). This can also be set directly through an \exabstention{} tag in the corresponding .Rnw exercise.

o Added various convenience functions for creating single- or multiple-choice questions from either a single numeric result (num_to_schoice) or from a matrix solution (matrix_to_schoice, matrix_to_mchoice, det_to_schoice).

o Added various convenience functions for formatting numbers within exercises: fmt() for conveniently displaying a certain number of digits, round2() for rounding away from zero, char_with_braces() for adding braces to negative numbers, and a toLatex() method for "matrix" objects.

o tex2image() now also supports format="svg" via pdfcrop followed by pdf2svg (instead of ImageMagick's convert).

o Fixed bug in file paths created for "media" supplements in QTI XML formats. File names had an unnecessary trailing slash which lead to unrecognized suffixes in some browsers.

Changes in Version 2.0-2

o Added an optional transform = NULL argument in exams2pdf().

o Avoid require(...) calls for suggested packages.

Changes in Version 2.0-1

o The default encoding in exams2moodle() is now ensured to be "UTF-8" rather than "utf8" for improved cross-platform compatibility.

o \exshuffle{TRUE} is now also used in read_exercise() so that schoice/mchoice exercises (or schoice/mchoice components in cloze exercises) are shuffled upon reading exercises. This facilitates static schoice/mchoice exercises.

o New cloze type "verbatim" which is not processed further (e.g., to include Moodle control structures).

o Empty questionlists for readily prepared Moodle cloze exercises can now be handled in exams2pdf().

o Functions from non-imported packages (e.g., only suggested) are now always called with package::function to make R-devel checks happy.

o Bug fix in processing of "duration" in exams2qti12().

Changes in Version 2.0-0

o New version accompanying publication of exams version 2 in the Journal of Statistical Software, 58(1), 1-36. See vignette("exams2", package = "exams") for more details.

o License extended from GPL-2 to GPL-2 | GPL-3.

o New exams2qti21() interface for generating QTI 2.1 (rather than QTI 1.2) XML. So far this was only tested with the ONYX Player within OpenOLAT. Currently, some switches do not appear to fully work, yet, at least within ONYX. The function still needs further thorough testing and might be further improved in future versions.

o Added verbose = FALSE option to xexams() and exams2xyz() interfaces. If set to TRUE, progress information on the exam generation is printed.

o The XML output in exams2qti12() sometimes did not yield fully correct QTI 1.2 output. This has been fixed now and test output was validated against the QTI 1.2 DTD.

o Added a new pluginfile = TRUE option in exams2moodle(). By default, supplementary files (graphics, data files, etc.) are now included using Moodle's Pluginfile mechanism. This may not work with older versions of Moodle (< 2.5) but in newer versions it is better and more robust than using data URIs.

o Added fix_num = TRUE option in exams2qti12() interface. Because OLAT does not display the correct answer of response_num items with tolerances (vargt/varlt), a workaround was added: In addition to the version with tolerances an exact equality is checked (varequal), adding and removing a 0.00001 points. Thus, evaluation of the exams is not affected but the display in OLAT is fixed.

o The {answerlist} in the {solution} environment can now also be omitted (in mchoice, schoice, cloze exercises). It is still recommended to include it but all exams2xyz() interfaces can process exercises correctly that do not provide it.

Changes in Version 1.9-6

o Changed tests/encodings.R to use 'encoding = "Latin-9"' (rather than "latin9" which does not work on Solaris, see ?iconv).

Changes in Version 1.9-5

o The new function exams_skeleton() helps users to create their first exam. It creates a directory with a demo.R script illustrating the use of the various exams2xyz() interfaces. Subdirectories with copies of all demonstration exercise .Rnw files and templates for different output formats (LaTeX, HTML, or XML) are also created.

o Encodings are now explicitly supported by all exams2xyz() interfaces. Essentially, the encoding argument is just passed on to Sweave() and the user has to make sure that the template (for LaTeX, HTML, or XML output) has the appropriate encoding. exams_skeleton() also supports this and especially the encoding UTF-8 is now fully integrated. The encodings ISO-8859-1 (aka latin1) or ISO-8859-15 (aka latin9) have also been tested while other encodings may need further touch-ups. Furthermore, the developers recommend to not rely on encodings but to code special characters with their LaTeX commands because these work in all locales on all platforms.

o The new function exams_eval() captures various types of evaluation policies for exams (with/without partial credits for multiple choice, with/without negative points). The resulting functions aid generation of e-learning exams or evaluation of an exam "by hand".

o If the "edir" argument is specified in exams2xyz(), then all files in edir and its subdirectories are searched recursively.

o Asymmetric tolerance intervals are not supported in the exams2xyz() interfaces anymore. Instead, vectors of solutions in "cloze" exercises can have vectors of tolerances.

o The original exam.tex/solution.tex templates have been renamed to oexam.tex/osolution.tex. The function exams() needs to be used with oexam/osolution while exam/solution are now optimized for exams2pdf(). The reason is that the questionnaire command has been changed. For a multiple-choice question with five answers and correct solutions 1 and 4, say, exams() produces \exmchoice{X}{ }{ }{X}{ } while exams2pdf now uses \exmchoice{X}[ ][ ][X][ ], i.e., with optional instead of required arguments. The latter solution is not restricted to multiple-choice questions with exactly five answers anymore.

Changes in Version 1.9-4

o Several functions now use find.package() rather than .find.package() internally (as the latter will be deprecated soon).

Changes in Version 1.9-3

o Enforced .tex suffix in LaTeX files used internally by exams2pdf(). The lack of .tex suffixes led to errors on certain Windows systems.

Changes in Version 1.9-2

o New demo exercises: - confint2.Rnw shows how a confidence interval can be embedded into a "cloze" question (for all version 2 interfaces). - dist.Rnw is a very short and simple example that is intelligible across a wide range of disciplines (not for statistics audiences only). - dist2.Rnw is an extension of dist.Rnw that is written as a "cloze".

o Questions in exams2moodle() were forced to be

now embedded in normal HTML (without 

Changes in Version 1.9-1

o Supplementary files had not been correctly inferred on Solaris systems. This has been fixed now so that the package behaves the same across different platforms.

o The code underlying the old "exams" vignette now avoids opening PDF viewers.

Changes in Version 1.9-0

o Major revision of the "exams" package, adding flexible support for generation of exams in different formats including PDF, HTML, Moodle XML, and QTI 1.2 (for OLAT). This new version of the exams package (called "version 2") is described in a new vignette: vignette("exams2", package = "exams")

Changes in Version 1.0-4

o Added an {answerlist} environment in the LaTeX templates (inst/tex) so that each \item in the {answerlist} is automatically labeled with (a), (b), (c), ...

o Replaced the old {itemize} environments with {answerlist} in anova.Rnw, boxplots.Rnw, relfreq.Rnw, scatterplot.Rnw, ttest.Rnw. (The resulting PDFs do not change.)

Changes in Version 1.0-3

o Small improvements in solution string formatting.

Changes in Version 1.0-2

o The argument "nsamp" was added to exams() in order to allow sampling more than one exercise from each element of the argument "file".

Changes in Version 1.0-1

o The man page of exams was modified to use suitable markup commands.

o The questions in the ANOVA example were slightly changed to more explicitly indicate which hypothesis is tested. A typo in the vignette was fixed.

Changes in Version 1.0-0

o New version accompanying publication in the Journal of Statistical Software. See vignette("exams", package = "exams") for more details.

o New CITATION file pointing to the JSS paper, see citation("exams").

Changes in Version 0.9-5

o The possible types of questions were extended to allow also text results. The new type is specified by "string".

o To specify the allowed tolerance for numeric solutions now different tolerances can be supplied for falling below or exceeding the true value via \extol{lower}{upper}.

o The "discussion" section in the vignette was extended to include several remarks and pointers on strategies for exercise generation.

Changes in Version 0.9-0

o First CRAN release version.

o New vignette "exams" that explains how to use the package.

o New set of exercises for illustation in examples and vignette.

Changes in Version 0.2-0

o First stable version for internal use at WU Wien.

Reference manual

It appears you don't have a PDF plugin for this browser. You can click here to download the reference manual.


2.3-6 by Achim Zeileis, a year ago

Report a bug at

Browse source code at

Authors: Achim Zeileis [aut, cre] , Bettina Gruen [aut] , Friedrich Leisch [aut] , Nikolaus Umlauf [aut] , Mirko Birbaumer [ctb] , Dominik Ernst [ctb] , Patrik Keller [ctb] , Niels Smits [ctb] , Reto Stauffer [ctb] , Kenji Sato [ctb]

Documentation:   PDF Manual  

Task views: Reproducible Research, Teaching Statistics

GPL-2 | GPL-3 license

Imports stats, graphics, grDevices, tools, utils

Suggests base64enc, knitr, magick, parallel, png, RCurl, RJSONIO, rmarkdown, tinytex, tth

Imported by exams.mylearn.

Depended on by exams2sakai, pafdR.

Suggested by Ryacas, Ryacas0.

See at CRAN