Home

Basic Content

Download

In the Download page, we provide the research community with:
A download link to get access to noisy labels in CIFAR-N;
A starter code (in Pytorch) to train CIFAR-N with CE loss.

Observations

In the Observations page, we share our major observations on CIFAR-N, the real-world human annotated noisy labels.

Leaderboard

We welcome researchers to share their method performances on CIFAR-N, and contribute to an abundant leaderboard.

Contributor

Researchers who contribute to CIFAR-N or the leaderboard.

Dataset Collection

CIFAR-10N and CIFAR-100N provide CIFAR-10 and CIFAR-100 train images with human annotated noisy labels obtained from Amazon Mechanical Turk.
3 independent workers for each train image of CIFAR-10, 1 worker for each train image of CIFAR-100.

CIFAR-10N construction
We provide 5 noisy label sets for CIFAR-10 train images, named as:

Aggregate: aggregation of three noisy labels by majority voting. If the submitted three labels are different for an image, the aggregated label will be randomly selected among the three labels;
Random i=1, 2, 3: the i-th submitted label for each image;
Worst: human label set with the highest noise rate. For each image, if there exist any wrongly annotated labels in three noisy labels, the worst label is randomly selected from wrong labels.

CIFAR-100N construction
Each image contains a coarse label and a fine label given by a human annotator, named as:

Coarse: the human annotated coarse label which belongs to one of 20 super-classes;
Fine: the human annotated fine label which belongs to one of 100 fine-classes.

Noise levels of CIFAR-N

Label Set	CIFAR-10N Aggregate	CIFAR-10N Random 1	CIFAR-10N Random 2	CIFAR-10N Random 3	CIFAR-10N Worst	CIFAR-100N Coarse	CIFAR-100N Fine
Noise Rate	9.03%	17.23%	18.12%	17.64%	40.21%	25.60%	40.20%

Reference

If you use this dataset of our reproduced results, please cite:

Learning with Noisy Labels Revisited: A Study Using Real-World Human Annotations.
Jiaheng Wei, Zhaowei Zhu, Hao Cheng, Tongliang Liu, Gang Niu, and Yang Liu. (*: equal contributions)

The BibTex infomation is detached as:
@inproceedings{
wei2022learning,
title={Learning with Noisy Labels Revisited: A Study Using Real-World Human Annotations},
author={Jiaheng Wei and Zhaowei Zhu and Hao Cheng and Tongliang Liu and Gang Niu and Yang Liu},
booktitle={International Conference on Learning Representations},
year={2022},
url={https://openreview.net/forum?id=TBWA6PLJZQm}
}

REsponsible & Accountable Learning (REAL)
@ University of California, Santa Cruz

CIFAR-N Dataset

Human annotations on CIFAR-10, CIFAR-100 train images