Open-access journal article · 2026
Validation Is Not Enough: A Duplicate-Aware, Reproducible Frozen-Feature Benchmark for Brain Tumour MRI Classification on BRISC 2025
Information, 17(11), Article 1175 · Published · Special Issue Applications of Deep Learning in Bioinformatics and Image Processing, 2nd Edition
Summary
Vajjhala et al. (2026) audited the BRISC 2025 brain tumour MRI dataset for near-duplicate images with a perceptual hash and benchmarked frozen ResNet-18, ResNet-50 and DINOv2 ViT-S/14 features under an identical linear probe on a duplicate-aware split of 5,477 images. The audit found 1,352 candidate near-duplicate pairs, 439 of them crossing the official train/test split, and 104 training images pixel-identical to test images. DINOv2 was numerically ahead (macro-F1 77.3%, macro-AUROC 93.2%), but no pairwise difference survived Holm correction, and in-pool validation overstated official test macro-F1 by 9.7–13.5 points.
Practical implication. Researchers using public MRI slice collections without patient identifiers should audit and document duplicates before splitting, the authors argue, because the partition deserves as much scrutiny as the model.
- The Problem
- Public brain tumour MRI collections are routinely reported with near-perfect classification accuracy, yet they carry no patient identifiers and their train/test overlap is rarely quantified.
- The Methodology
- Dataset audit and controlled benchmark: a perceptual-hash near-duplicate audit of the BRISC 2025 brain tumour MRI dataset (6,000 slices), a duplicate-aware group-aware split of 5,477 images, and a linear probe on frozen ResNet-18, ResNet-50 and DINOv2 ViT-S/14 features over three seeds, with group-level bootstrap confidence intervals, paired tests with Holm correction, and calibration and selective-prediction analysis.
- The Core Finding
- A perceptual-hash audit of BRISC 2025 found 1,352 candidate near-duplicate pairs, 439 of them crossing the official train/test split, and 104 training images pixel-identical to test images. On a duplicate-aware split of 5,477 images, frozen DINOv2 ViT-S/14 features were numerically ahead of ImageNet-supervised ResNet-18/50 under an identical linear probe (macro-F1 77.3%, macro-AUROC 93.2%), but no pairwise difference survived Holm correction, and in-pool validation overstated official test macro-F1 by 9.7–13.5 points.
- The Citation
- Vajjhala, N. R., Ramollari, E., & Kumar, R. (2026). Validation Is Not Enough: A Duplicate-Aware, Reproducible Frozen-Feature Benchmark for Brain Tumour MRI Classification on BRISC 2025. Information, 17(11), Article 1175. https://doi.org/10.3390/info17111175
The summary, key findings, methods, implications and relevance notes on this page are this website’s own description of the paper, written from the published abstract and text. The official abstract and citation details are given further down.
Key findings
- Finding 1. A perceptual-hash audit of BRISC 2025 found 1,352 candidate near-duplicate pairs, of which 439 crossed the official train/test split; 104 training images were pixel-identical to test images, and 23 candidate pairs carried conflicting labels.
- Finding 2. A visual review of sampled pairs showed that most cross-split candidates were other slices of the same scan, with an estimated 5–10% false matches.
- Finding 3. Removing redundant copies and every training image connected to the test set (523 images) left a split of 5,477 images that is duplicate-aware but not patient-independent.
- Finding 4. Under an identical linear probe, DINOv2 ViT-S/14 was numerically best on every discrimination metric — macro-F1 77.3% against 75.3% for ResNet-18 and 74.6% for ResNet-50, and macro-AUROC 93.2% against 90.4–90.7% — from the smallest embedding of the three (384 dimensions), but no pairwise difference survived Holm correction.
- Finding 5. Validation drawn from the training pool overstated official test macro-F1 by 9.7–13.5 points and did not reliably reproduce the test ranking of backbones.
- Finding 6. Control experiments showed that the validation–test gap is not an artefact of the filtering: it persisted at every hash threshold and after the removed images were restored, disappeared on pseudo-test sets drawn from the training pool, and mainly reflects a distribution difference between the official partitions.
- Finding 7. Meningioma recall stayed at 50–53%, and axial slices were the hardest imaging plane.
- Finding 8. All linear heads were under-confident, yet confidence was informative about correctness: accepting only the 50% most confident cases raised accuracy to 91.1% (ResNet-18), 89.7% (ResNet-50) and 93.5% (DINOv2).
Source: Vajjhala et al. (2026), Information, 17(11), Article 1175. DOI: 10.3390/info17111175
The study in detail
What question does this paper answer?
How much near-duplicate contamination does the BRISC 2025 brain tumour MRI dataset carry across its official train/test split, and what does an honest, low-variance baseline on it look like once that contamination is reduced?
What did the study find, in detail?
A perceptual-hash audit of the 6,000 BRISC 2025 slices found 1,352 candidate near-duplicate pairs, 439 of them crossing the official train/test split, and 104 training images pixel-identical to test images. On a filtered, duplicate-aware split of 5,477 images, frozen ImageNet-supervised ResNet-18/50 and self-supervised DINOv2 ViT-S/14 features were competitive under an identical linear probe: DINOv2 was numerically ahead (macro-F1 77.3%, macro-AUROC 93.2%), but no pairwise difference survived Holm correction. In-pool validation overstated official test macro-F1 by 9.7–13.5 points.
Why does it matter?
Public brain tumour MRI collections are routinely reported with near-perfect classification accuracy, yet they carry no patient identifiers and their train/test overlap is rarely quantified. The authors argue that on public slice collections without patient identifiers the partition deserves as much scrutiny as the model: duplicates should be audited and documented before splitting, uncertainty and calibration reported rather than a single score, and validation performance interpreted only for the population from which it was drawn.
Methods and data
| Research questions | Four: how much candidate near-duplication exists within and across the official splits; whether self-supervised DINOv2 features transfer better than supervised ImageNet features once uncertainty is accounted for; whether validation on held-out training-pool data is a reliable proxy for the official test set; and which classes, planes and confidence regimes account for the errors |
|---|---|
| Dataset | BRISC 2025: 6,000 contrast-enhanced T1-weighted MRI slices in four classes (glioma, meningioma, pituitary tumour, no tumour), with an official 5,000/1,000 train/test split and no patient identifiers |
| Audit | A 64-bit perceptual-hash audit with candidate grouping, a threshold-sensitivity analysis, and a visual review of 100 sampled cross-split pairs to estimate precision |
| Split | Redundant copies and every training image connected to a test image removed (523 images), leaving 5,477 images; duplicate-aware but not patient-independent |
| Models | Frozen ResNet-18 and ResNet-50 (ImageNet-supervised) and DINOv2 ViT-S/14 (self-supervised), each with an identical linear probe over three seeds |
| Inference | Group-level bootstrap confidence intervals, paired tests and Holm multiplicity correction; expected calibration error and selective-prediction (risk–coverage) analysis |
| Main result | DINOv2 numerically ahead (macro-F1 77.3 ± 0.1%, macro-AUROC 93.2%) but not significantly so; in-pool validation overstated official test macro-F1 by 9.7–13.5 points; meningioma recall stayed at 50–53% |
| Released | The splits, the audit tables and the per-image predictions |
| Citation | Vajjhala et al. (2026) · DOI 10.3390/info17111175 |
Definitions and key terms
- BRISC 2025
- A public brain tumour MRI dataset of 6,000 contrast-enhanced T1-weighted slices in four classes — glioma, meningioma, pituitary tumour and no tumour — assembled from pre-existing public collections and released with an official 5,000/1,000 train/test split and no patient identifiers. — as used in Vajjhala et al. (2026)
- Near-duplicate contamination
- The presence of the same or nearly the same image — a re-exported copy, a lightly cropped duplicate or an adjacent slice of the same scan — on both sides of a train/test split, which makes test scores optimistic. — as used in Vajjhala et al. (2026)
- Linear probe
- An evaluation protocol in which a pre-trained encoder is kept frozen and only a linear classifier is trained on its features, which isolates the quality of the representation from fine-tuning choices. — as used in Vajjhala et al. (2026)
- Duplicate-aware split
- In this article, a split from which redundant copies and every training image connected by the perceptual-hash audit to a test image have been removed. It reduces leakage but, without patient identifiers, cannot guarantee that the two sides hold different patients. — as used in Vajjhala et al. (2026)
- Selective prediction
- Accepting a model’s prediction only when its confidence is high enough and deferring the remaining cases; the article reports accuracy at given coverage levels and the area under the risk–coverage curve (AURC). — as used in Vajjhala et al. (2026)
Abstract
Public brain tumour MRI collections are routinely reported with near-perfect classification accuracy, yet they carry no patient identifiers and their train/test overlap is rarely quantified. We present a duplicate-aware, reproducible frozen-feature benchmark on BRISC 2025 (6000 contrast-enhanced T1-weighted slices; glioma, meningioma, pituitary tumour, no tumour). A perceptual-hash audit found 1352 candidate near-duplicate pairs, with 439 of them crossing the official train/test split, and 104 training images that are pixel-identical to test images; a visual review of sampled pairs showed that most candidates were other slices of the same scan, with an estimated 5–10% false matches. Removing redundant copies and every training image connected to the test set left 5477 images, a split that is duplicate-aware but not patient-independent. Under an identical linear probe, ImageNet-supervised ResNet-18/50 and self-supervised DINOv2 ViT-S/14 were competitive: DINOv2 was numerically ahead (macro-F1 77.3 ± 0.1%, macro-AUROC 93.2%), but no pairwise difference survived Holm correction. In-pool validation overstated official test macro-F1 by 9.7–13.5 points; control experiments showed that this gap is not an artefact of the filtering but mainly reflects a distribution difference between the official partitions. Meningioma recall stayed at 50–53%, and all linear heads were under-confident. We release the splits, audit tables and per-image predictions.
Abstract as published in Information under a Creative Commons licence.
Keywords: brain tumour MRI classification; benchmarking; reproducibility; data leakage; near-duplicate detection; foundation models; DINOv2; linear probing; calibration; selective prediction
Limitations
- The evaluation is duplicate-aware but not patient-independent, and the filtering may discard genuinely distinct but visually similar images.
- The 64-bit perceptual hash errs in both directions — it under-detects rotated, reflected or heavily cropped duplicates, and an estimated 5–10% of cross-split candidate pairs connect different patients — so the contamination figures are approximate; the precision estimate rests on a visual review of 100 sampled pairs.
- Results on the filtered test set are not numerically comparable with results on the original 1,000-image test set.
- One linear protocol was evaluated at one resolution, without a hyper-parameter search, augmentation or fine-tuning, so the absolute numbers are a floor rather than a ceiling.
- The comparison covers one self-supervised generation only; a planned DINOv3 run could not be completed.
- No clinical validity is demonstrated; the authors state that the models reported must not be used for clinical decisions.
Practical implications
- Researchers benchmarking on BRISC 2025 can use the released splits, audit tables and per-image predictions as a common, documented basis for comparison.
- For researchers reporting results on public slice collections, the authors recommend reporting uncertainty and calibration rather than a single score.
- For researchers selecting models on validation data, in-pool validation estimates performance on data like the training pool, not on the official test set: here it overstated official test macro-F1 by 9.7–13.5 points.
- For clinicians and developers, no clinical validity is demonstrated, and the authors state that the models reported must not be used for clinical decisions.
Drawn from Vajjhala et al. (2026), Information, 17(11), Article 1175.
When this research may be relevant
This article may be relevant to researchers working on brain tumour MRI classification, on data leakage, near-duplicate images and train/test contamination in public medical imaging datasets, and on benchmarking frozen foundation-model features such as DINOv2 against ImageNet-supervised networks. It also speaks to work on calibration, selective prediction and the reliability of validation scores as a proxy for test performance, and it is a reusable audit protocol for public image collections released without patient identifiers.
Research topics addressed: brain tumour MRI classification; medical image analysis; deep learning; benchmarking; data leakage; near-duplicate detection; perceptual hashing; train/test contamination; foundation models; DINOv2; ResNet; linear probing; frozen features; calibration; selective prediction; reproducibility; BRISC 2025 dataset
How to cite
- APA 7
- Vajjhala, N. R., Ramollari, E., & Kumar, R. (2026). Validation Is Not Enough: A Duplicate-Aware, Reproducible Frozen-Feature Benchmark for Brain Tumour MRI Classification on BRISC 2025. Information, 17(11), Article 1175. https://doi.org/10.3390/info17111175
- MLA 9
- Vajjhala, Narasimha Rao, et al. “Validation Is Not Enough: A Duplicate-Aware, Reproducible Frozen-Feature Benchmark for Brain Tumour MRI Classification on BRISC 2025.” Information, vol. 17, no. 11, 2026, article 1175. https://doi.org/10.3390/info17111175.
- IEEE
- N. R. Vajjhala, E. Ramollari, and R. Kumar, “Validation Is Not Enough: A Duplicate-Aware, Reproducible Frozen-Feature Benchmark for Brain Tumour MRI Classification on BRISC 2025,” Information, vol. 17, no. 11, Art. no. 1175, Nov. 2026, doi: 10.3390/info17111175.
BibTeX
@article{vajjhala2026duplicate,
title = {Validation Is Not Enough: A Duplicate-Aware, Reproducible Frozen-Feature Benchmark for Brain Tumour MRI Classification on BRISC 2025},
author = {Vajjhala, Narasimha Rao and Ramollari, Ervin and Kumar, Rajesh},
journal = {Information},
volume = {17},
number = {11},
pages = {1175},
year = {2026},
publisher = {MDPI},
doi = {10.3390/info17111175},
url = {https://doi.org/10.3390/info17111175}
}RIS
TY - JOUR TI - Validation Is Not Enough: A Duplicate-Aware, Reproducible Frozen-Feature Benchmark for Brain Tumour MRI Classification on BRISC 2025 AU - Vajjhala, Narasimha Rao AU - Ramollari, Ervin AU - Kumar, Rajesh JO - Information PY - 2026 DA - 2026/11/19 VL - 17 IS - 11 SP - 1175 PB - MDPI SN - 2078-2489 DO - 10.3390/info17111175 UR - https://www.narasimharao.net/research/duplicate-aware-frozen-feature-benchmark-brain-tumour-mri-brisc-2025/ LA - en ER -
Links and files
- DOI: https://doi.org/10.3390/info17111175
- Publisher’s page: https://www.mdpi.com/2078-2489/17/11/1175
- Full text (PDF): https://www.mdpi.com/2078-2489/17/11/1175/pdf
- This page as text: https://www.narasimharao.net/research/duplicate-aware-frozen-feature-benchmark-brain-tumour-mri-brisc-2025/paper.md
- Permanent address of this page: https://www.narasimharao.net/research/duplicate-aware-frozen-feature-benchmark-brain-tumour-mri-brisc-2025/
Related research
Related publications
Trained on BUS-BRA and evaluated unchanged on BUSI, the supervised models largely retained their discrimination, but in every official-fold run ResNet-50 and ViT-B/16 labelled a larger share of BUSI than of BUS-BRA images malignant although prevalence was the same, a shift that temperature scaling cannot correct.
A lightweight EfficientNetV2B0 model trained under CLAIM and TRIPOD-AI reporting standards achieved an AUC of 0.967 (95% CI 0.953–0.979) and pneumonia recall of 0.98 on paediatric chest X-rays, missing only about 2% of true cases.
Fine-tuned DINOv2 and EfficientNet-B0 performed best on nine-class DeepWeeds data, but differences among several high-performing models were not consistently statistically significant across random seeds.
The review shows that DEA is widely applied in hospital management, nursing and outpatient services for efficiency measurement, benchmarking, resource allocation and performance evaluation, and identifies six limitations, including input/output selection, outlier sensitivity, inability to handle statistical noise and the static nature of traditional models.
The volume applies explainable and transparent AI across clinical decision-making, telehealth, electronic health records, EEG analysis, wearable monitoring and mental health, pairing ethical analysis with applied systems and strategies for building trust.