The CID22 full dataset download (7.2 GB, 250 references, ~22k compressed images) only includes CID22_validation_set.csv with annotations for the 49-reference validation subset (4,292 scored pairs).
The remaining ~17.7k pairs from the other 201 references have no published human scores. The CID22 labs page doesn't mention that the full set annotations are withheld — it describes the dataset as containing "22,000 subjective scores" but only ~4,300 are in the download.
Could the full set annotations (MCOS/RMOS/Elo scores for all 250 references) be published? This would make CID22 much more useful as a benchmark for independent metric development and comparison.
The CID22 full dataset download (7.2 GB, 250 references, ~22k compressed images) only includes
CID22_validation_set.csvwith annotations for the 49-reference validation subset (4,292 scored pairs).The remaining ~17.7k pairs from the other 201 references have no published human scores. The CID22 labs page doesn't mention that the full set annotations are withheld — it describes the dataset as containing "22,000 subjective scores" but only ~4,300 are in the download.
Could the full set annotations (MCOS/RMOS/Elo scores for all 250 references) be published? This would make CID22 much more useful as a benchmark for independent metric development and comparison.