OpenOOD

Benchmarking Generalized Out-of-Distribution Detection

Jingkang50
871
113
Python

OpenOOD: Benchmarking Generalized OOD Detection

❗ When using OpenOOD in your research, it is vital to cite both the OpenOOD benchmark (versions 1 and 1.5) and the individual works that have contributed to your research. Accurate citation acknowledges the efforts and contributions of all researchers involved. For example, if your work involves the NINCO benchmark within OpenOOD, please include a citation for NINCO apart of OpenOOD.

paper
   
paper
   

paper
   
paper
   
paper

This repository reproduces representative methods within the Generalized Out-of-Distribution Detection Framework,
aiming to make a fair comparison across methods that were initially developed for anomaly detection, novelty detection, open set recognition, and out-of-distribution detection.
This codebase is still under construction.
Comments, issues, contributions, and collaborations are all welcomed!

timeline.jpg
Timeline of the methods that OpenOOD supports. More methods are included as OpenOOD iterates.

Updates

Contributing

We appreciate all contributions to improve OpenOOD. We sincerely welcome community users to participate in these projects.

  • For contributing to this repo, please refer to CONTRIBUTING.md for the guideline.
  • For adding your method to our leaderboard, simply open an issue where you will see the template that has detailed instructions.

FAQ

  • APS_mode means Automatic (hyper)Parameter Searching mode, which enables the model to validate all the hyperparameters in the sweep list based on the validation ID/OOD set. The default value is False. Check here for example.

Get Started

v1.5 (up-to-date)

Installation

OpenOOD now supports installation via pip.

pip install git+https://github.com/Jingkang50/OpenOOD
pip install libmr

# optional, if you want to use CLIP
# pip install git+https://github.com/openai/CLIP.git

Data

If you only use our evaluator, the benchmarks for evaluation will be automatically downloaded by the evaluator (again check out this tutorial). If you would like to also use OpenOOD for training, you can get all data with our downloading script. Note that ImageNet-1K training images should be downloaded from its official website.

Pre-trained checkpoints

OpenOOD v1.5 focuses on 4 ID datasets, and we release pre-trained models accordingly.

  • CIFAR-10 [Google Drive]: ResNet-18 classifiers trained with cross-entropy loss from 3 training runs.
  • CIFAR-100 [Google Drive]: ResNet-18 classifiers trained with cross-entropy loss from 3 training runs.
  • ImageNet-200 [Google Drive]: ResNet-18 classifiers trained with cross-entropy loss from 3 training runs.
  • ImageNet-1K [Google Drive]: ResNet-50 classifiers including 1) the one from torchvision, 2) the ones that are trained by us with specific methods such as MOS, CIDER, and 3) the official checkpoints of data augmentation methods such as AugMix, PixMix.

Again, these checkpoints can be downloaded with the downloading script here.

Our codebase accesses the datasets from ./data/ and pretrained models from ./results/checkpoints/ by default.

├── ...
├── data
│   ├── benchmark_imglist
│   ├── images_classic
│   └── images_largescale
├── openood
├── results
│   ├── checkpoints
│   └── ...
├── scripts
├── main.py
├── ...

Training and evaluation scripts

We provide training and evaluation scripts for all the methods we support in scripts folder.

Supported Benchmarks (10)

This part lists all the benchmarks we support. Feel free to include more.

Anomaly Detection (1)
Open Set Recognition (4)
Out-of-Distribution Detection (6)
  • [x] BIMCV (A COVID X-Ray Dataset)

    Near-OOD: CT-SCAN, X-Ray-Bone;

    Far-OOD: MNIST, CIFAR-10, Texture, Tiny-ImageNet;

  • [x] MNIST

    Near-OOD: NotMNIST, FashionMNIST;

    Far-OOD: Texture, CIFAR-10, TinyImageNet, Places365;

  • [x] CIFAR-10

    Near-OOD: CIFAR-100, TinyImageNet;

    Far-OOD: MNIST, SVHN, Texture, Places365;

  • [x] CIFAR-100

    Near-OOD: CIFAR-10, TinyImageNet;

    Far-OOD: MNIST, SVHN, Texture, Places365;

  • [x] ImageNet-200

    Near-OOD: SSB-hard, NINCO;

    Far-OOD: iNaturalist, Texture, OpenImage-O;

    Covariate-Shifted ID: ImageNet-C, ImageNet-R, ImageNet-v2;

  • [x] ImageNet-1K

    Near-OOD: SSB-hard, NINCO;

    Far-OOD: iNaturalist, Texture, OpenImage-O;

    Covariate-Shifted ID: ImageNet-C, ImageNet-R, ImageNet-v2, ImageNet-ES;

Note that OpenOOD v1.5 emphasizes and focuses on the last 4 benchmarks for OOD detection.

Supported Backbones (6)

This part lists all the backbones we will support in our codebase, including CNN-based and Transformer-based models. Backbones like ResNet-50 and Transformer have ImageNet-1K/22K pretrained models.

CNN-based Backbones (4)
Transformer-based Architectures (2)

Supported Methods (60+)

This part lists all the methods we include in this codebase. Up to v1.5, we totally support more than 50 popular methods for generalized OOD detection.

All the supported methodolgies can be placed in the following four categories.

density   reconstruction   classification   distance

We also note our supported methodolgies with the following tags if they have special designs in the corresponding steps, compared to the standard classifier training process.

preprocess   extradata   training   postprocess

Anomaly Detection (5)
  • [x] training postprocess
  • [x]
    training postprocess
  • [x]
    training postprocess
  • [x] training postprocess
  • [x] training postprocess
Open Set Recognition (3)

Post-Hoc Methods (2):

  • [x] postprocess
  • [x] postprocess

Training Methods (1):

  • [x] training postprocess
Out-of-Distribution Detection (41)

Post-Hoc Methods (23):

  • [x] msp
  • [x] odin    postprocess
  • [x] mds    postprocess
  • [x] mdsensemble    postprocess
  • [x] gram    postprocess
  • [x] ebo    postprocess
  • [x] rmds    postprocess
  • [x] gradnorm    postprocess
  • [x] react    postprocess
  • [x] mls    postprocess
  • [x] klm    postprocess
  • [x] sem    postprocess
  • [x] vim    postprocess
  • [x] knn    postprocess
  • [x] dice    postprocess
  • [x] rankfeat    postprocess
  • [x] ash    postprocess
  • [x] she    postprocess
  • [x] gen    postprocess
  • [x] nnguide    postprocess
  • [x] relation    postprocess
  • [x] scale    postprocess
  • [x] fdbd    postprocess

Training Methods (14):

  • [x] confbranch    preprocess   training
  • [x] rotpred    preprocess   training
  • [x] godin    training   postprocess
  • [x] csi    preprocess   training   postprocess
  • [x] ssd    training   postprocess
  • [x] mos    training
  • [x] vos    training   postprocess
  • [x] logitnorm    training   preprocess
  • [x] cider    training   postprocess
  • [x] npos    training   postprocess
  • [x] t2fnorm    training
  • [x] ish    training
  • [x] palm    training
  • [x] reweightood    training   postprocess

Training With Extra Data (4):

  • [x] oe    extradata   training
  • [x] mcd    extradata   training
  • [x] udg    extradata   training
  • [x] mixoe    extradata   training
Method Uncertainty (4)
  • [x] mcdropout    training   postprocess
  • [x] deepensemble    training
  • [x] tempscale    postprocess
  • [x] rts    training   postprocess
Data Augmentation (8)
  • [x] mixup    preprocess
  • [x] cutmix    preprocess
  • [x] styleaugment    preprocess
  • [x] randaugment    preprocess
  • [x] augmix    preprocess
  • [x] deepaugment    preprocess
  • [x] pixmix    preprocess
  • [x] regmixup    preprocess

Contributors

Citation

If you find our repository useful for your research, please consider citing these papers:

# v1.5 report
@article{zhang2023openood,
  title={OpenOOD v1.5: Enhanced Benchmark for Out-of-Distribution Detection},
  author={Zhang, Jingyang and Yang, Jingkang and Wang, Pengyun and Wang, Haoqi and Lin, Yueqian and Zhang, Haoran and Sun, Yiyou and Du, Xuefeng and Li, Yixuan and Liu, Ziwei and Chen, Yiran and Li, Hai},
  journal={arXiv preprint arXiv:2306.09301},
  year={2023}
}

# v1.0 report
@article{yang2022openood,
    author = {Yang, Jingkang and Wang, Pengyun and Zou, Dejian and Zhou, Zitang and Ding, Kunyuan and Peng, Wenxuan and Wang, Haoqi and Chen, Guangyao and Li, Bo and Sun, Yiyou and Du, Xuefeng and Zhou, Kaiyang and Zhang, Wayne and Hendrycks, Dan and Li, Yixuan and Liu, Ziwei},
    title = {OpenOOD: Benchmarking Generalized Out-of-Distribution Detection},
    year = {2022}
}

# full-spectrum OOD detection
@article{yang2022fsood,
    title = {Full-Spectrum Out-of-Distribution Detection},
    author = {Yang, Jingkang and Zhou, Kaiyang and Liu, Ziwei},
    journal={arXiv preprint arXiv:2204.05306},
    year = {2022}
}

# generalized OOD detection framework & survey
@article{yang2021oodsurvey,
    title={Generalized Out-of-Distribution Detection: A Survey},
    author={Yang, Jingkang and Zhou, Kaiyang and Li, Yixuan and Liu, Ziwei},
    journal={arXiv preprint arXiv:2110.11334},
    year={2021}
}

# OOD benchmarks
# NINCO
@inproceedings{bitterwolf2023ninco,
    title={In or Out? Fixing ImageNet Out-of-Distribution Detection Evaluation},
    author={Julian Bitterwolf and Maximilian Mueller and Matthias Hein},
    booktitle={ICML},
    year={2023},
    url={https://proceedings.mlr.press/v202/bitterwolf23a.html}
}

# SSB
@inproceedings{vaze2021open,
    title={Open-Set Recognition: A Good Closed-Set Classifier is All You Need},
    author={Vaze, Sagar and Han, Kai and Vedaldi, Andrea and Zisserman, Andrew},
    booktitle={ICLR},
    year={2022}
}