loo

Efficient approximate leave-one-out cross-validation for fitted Bayesian models

loo is an R package that allows users to compute efficient approximate
leave-one-out cross-validation for fitted Bayesian models, as well as model
weights that can be used to average predictive distributions.
The loo package package implements the fast and stable computations for
approximate LOO-CV and WAIC from

• Vehtari, A., Gelman, A., and Gabry, J. (2017). Practical Bayesian model
  evaluation using leave-one-out cross-validation and WAIC.
  Statistics and Computing. 27(5), 1413–1432.
  doi:10.1007/s11222-016-9696-4. Online,
  arXiv preprint arXiv:1507.04544.

and computes model weights as described in

• Yao, Y., Vehtari, A., Simpson, D., and Gelman, A. (2018). Using
  stacking to average Bayesian predictive distributions. In Bayesian
  Analysis, doi:10.1214/17-BA1091.
  Online,
  arXiv preprint arXiv:1704.02030.

From existing posterior simulation draws, we compute approximate LOO-CV using
Pareto smoothed importance sampling (PSIS), a new procedure for regularizing
importance weights. As a byproduct of our calculations, we also obtain
approximate standard errors for estimated predictive errors and for comparing
predictive errors between two models. We recommend PSIS-LOO-CV instead of WAIC,
because PSIS provides useful diagnostics and effective sample size and Monte
Carlo standard error estimates.

Resources

• mc-stan.org/loo (online documentation, vignettes)
• Ask a question (Stan Forums on Discourse)
• Open an issue (GitHub issues for bug reports, feature requests)

Installation

• Install the latest release from CRAN:

install.packages("loo")

• Install the latest development version from GitHub:

# install.packages("remotes")
remotes::install_github("stan-dev/loo")

We do not recommend setting build_vignettes=TRUE when installing from GitHub
because some of the vignettes take a long time to build and are always available
online at mc-stan.org/loo/articles/.

Python and Matlab/Octave Code

Corresponding Python and Matlab/Octave code can be found at the
avehtari/PSIS repository.

License

The code is distributed under the GPL 3 license. The documentation is distributed under the CC BY 4.0 license.