Open-access journal article · 2026
Breast Ultrasound Classification Under Cross-Dataset Shift: A Multi-Seed Benchmark of Convolutional Networks, a Vision Transformer, and a Self-Supervised Foundation Model
Algorithms, 19(11), Article 1011 · Published · Special Issue AI-Powered Biomedical Image Analysis
Summary
Ramollari et al. (2026) fine-tuned ResNet-50, EfficientNet-B0, a supervised Vision Transformer and self-supervised DINOv2 under one shared protocol on the BUS-BRA breast ultrasound dataset, over repeated folds and seeds, and evaluated every trained model unchanged on the external BUSI dataset. The supervised models largely retained their discrimination externally, but in every official-fold run ResNet-50 and ViT-B/16 labelled a larger share of BUSI than of BUS-BRA images malignant although prevalence was the same, a shift that temperature scaling cannot correct. Full-frame DINOv2 did not fit its training data in any run.
Practical implication. External evaluations of medical-imaging classifiers should report operating-point transport and class-wise calibration alongside discrimination, the authors argue.
- The Problem
- Breast ultrasound classifiers are usually evaluated on images from their training institution, and whether ranking ability, decision thresholds and probability calibration survive a change of institution has rarely been tested under leakage-controlled conditions.
- The Methodology
- Leakage-controlled benchmark: ResNet-50, EfficientNet-B0, a supervised Vision Transformer (ViT-B/16) and self-supervised DINOv2 fine-tuned under one shared protocol on the official BUS-BRA five-fold partitions with three seeds per fold and on three case-grouped hold-out splits, with every trained model evaluated unchanged on the external BUSI dataset; discrimination, operating-point transport and class-wise calibration analysed with clustered and paired inference.
- The Core Finding
- Trained on BUS-BRA and evaluated unchanged on BUSI, the supervised models largely retained their discrimination, but in every official-fold run ResNet-50 and ViT-B/16 labelled a larger share of BUSI than of BUS-BRA images malignant although prevalence was the same, a shift that temperature scaling cannot correct. Full-frame DINOv2 did not fit its training data in any run, although exploratory lesion cropping produced good discrimination.
- The Citation
- Ramollari, E., Vajjhala, N. R., Kumar, R., Ramchandani, K., Mahraj, R., & Hussain, K. (2026). Breast Ultrasound Classification Under Cross-Dataset Shift: A Multi-Seed Benchmark of Convolutional Networks, a Vision Transformer, and a Self-Supervised Foundation Model. Algorithms, 19(11), Article 1011. https://doi.org/10.3390/a19111011
The summary, key findings, methods, implications and relevance notes on this page are this website’s own description of the paper, written from the published abstract and text. The official abstract and citation details are given further down.
Key findings
- Finding 1. The supervised models largely retained their discrimination when evaluated unchanged on the external BUSI dataset.
- Finding 2. ResNet-50 had the highest mean area under the receiver operating characteristic curve under the official BUS-BRA folds, but no criterion separated it from ViT-B/16, whose small hold-out advantage at the decision threshold did not recur.
- Finding 3. In every official-fold run, both models labelled a larger share of BUSI than of BUS-BRA images malignant although prevalence was the same — a shift that temperature scaling cannot correct.
- Finding 4. Top-label calibration error understated malignant-class miscalibration.
- Finding 5. Full-frame DINOv2 did not fit its training data in any run under the shared configuration, whereas the similarly sized supervised Vision Transformer trained successfully; exploratory lesion cropping produced good discrimination.
- Finding 6. The authors conclude that large-scale self-supervised pretraining does not by itself guarantee successful fine-tuning, and that the input representation and adaptation of a foundation model must be evaluated and reported alongside it.
Source: Ramollari et al. (2026), Algorithms, 19(11), Article 1011. DOI: 10.3390/a19111011
The study in detail
What question does this paper answer?
Do ranking ability, decision thresholds and probability calibration of breast ultrasound classifiers survive a change of institution under leakage-controlled conditions, and does a self-supervised foundation model transfer better than supervised networks trained under the same protocol?
What did the study find, in detail?
Trained on BUS-BRA and evaluated unchanged on BUSI, the supervised models largely retained their discrimination externally, but ResNet-50 and ViT-B/16 could not be separated reliably. The most reproducible finding concerned the operating point: in every official-fold run both models labelled a larger share of BUSI than of BUS-BRA images malignant although prevalence was the same — a shift that temperature scaling cannot correct. Full-frame DINOv2 did not fit its training data in any run, although exploratory lesion cropping produced good discrimination.
Why does it matter?
Deep-learning classifiers for breast ultrasound are usually evaluated on images from their training institution, and self-supervised foundation models are increasingly proposed as more transferable than supervised networks. The study shows that a model can keep its ranking ability while its decision threshold and calibration drift, so the authors argue that external evaluations should report operating-point transport and class-wise calibration alongside discrimination. They describe the models as research prototypes that require patient-level, multicentre, prospective validation before any clinical use.
Methods and data
| Design | Leakage-controlled, multi-seed benchmark with a fixed and untouched external cohort |
|---|---|
| Development data | BUS-BRA, on its official five-fold partitions (three seeds per fold, 15 runs per model) and on three case-grouped hold-out splits |
| External data | The Breast Ultrasound Images (BUSI) dataset, evaluated at the image level with every trained model unchanged |
| Models | ResNet-50, EfficientNet-B0, a supervised Vision Transformer (ViT-B/16) and self-supervised DINOv2, fine-tuned under one shared protocol |
| Outcomes | Discrimination (AUROC), transport of the decision threshold, and probability calibration including class-wise calibration, with temperature scaling |
| Main result | Supervised models largely kept their discrimination externally; the operating point shifted in every official-fold run; full-frame DINOv2 did not fit its training data in any run |
| Released | Per-image predictions, analysis scripts and training histories |
| Citation | Ramollari et al. (2026) · DOI 10.3390/a19111011 |
Definitions and key terms
- Cross-dataset shift
- The change in data between the dataset a model was developed on and another dataset — here from BUS-BRA to BUSI — arising from differences in scanners, image dimensions, on-image annotations and acquisition practice. — as used in Ramollari et al. (2026)
- Operating-point transport
- Applying the decision threshold chosen on the development data unchanged to external data, and checking whether the share of cases labelled positive, and with it sensitivity and specificity, stays the same. — as used in Ramollari et al. (2026)
- Temperature scaling
- A post hoc calibration method that rescales a model’s outputs by a single fitted parameter; the article shows that it cannot correct the shift in operating point observed on the external dataset. — as used in Ramollari et al. (2026)
- Self-supervised foundation model
- A general-purpose encoder such as DINOv2, pre-trained on a large image collection without task labels and then adapted to a downstream task. — as used in Ramollari et al. (2026)
Abstract
Deep-learning classifiers for breast ultrasound are usually evaluated on images from their training institution, and self-supervised foundation models are increasingly proposed as more transferable than supervised networks; whether ranking ability, decision thresholds, and probability calibration survive a change in institution has rarely been tested under leakage-controlled conditions. We fine-tuned ResNet-50, EfficientNet-B0, a supervised Vision Transformer (ViT-B/16), and self-supervised DINOv2 under one shared protocol on the official BUS-BRA five-fold partitions, each fold with three seeds, and on three case-grouped hold-out splits, and evaluated every trained model unchanged on the Breast Ultrasound Images (BUSI) dataset. The supervised models largely retained their discrimination externally. ResNet-50 had the highest mean area under the receiver operating characteristic curve under the official folds, but no criterion separated it from ViT-B/16, whose small hold-out advantage at the decision threshold did not recur. In every official-fold run, both models labeled a larger share of BUSI than of BUS-BRA images malignant, although prevalence was the same—a shift that temperature scaling cannot correct. Top-label calibration error also understated malignant-class miscalibration. Full-frame DINOv2 did not fit its training data in any run, although exploratory lesion cropping produced good discrimination. External evaluations should report operating-point transport and class-wise calibration alongside discrimination.
Abstract as published in Algorithms under a Creative Commons licence.
Keywords: breast ultrasound; foundation models; self-supervised learning; DINOv2; vision transformer; convolutional neural network; external validation; dataset shift; probability calibration; decision threshold
Limitations
- Development used a single dataset and only one external cohort was available, evaluated at the image level because reliable BUSI patient identifiers are unavailable.
- The official folds share training data, so the spread over folds understates the uncertainty of the cross-validation estimate, and three seeds per fold estimate training-related variation only roughly.
- All architectures shared one training configuration without architecture-specific tuning; full-frame DINOv2 was not retrained with a lower backbone learning rate or at its native resolution, nor evaluated as a frozen-feature probe.
- The input ablation with lesion cropping was exploratory, limited to one seed and one model, and based on reference contours.
- No prospective workflow, radiologist comparison, subgroup analysis or clinical-utility analysis was performed; the models are research prototypes.
Practical implications
- For researchers validating classifiers on another institution’s data, retained discrimination does not mean a retained operating point: the decision threshold should be checked on the new site.
- For researchers adopting foundation models, large-scale self-supervised pretraining does not by itself guarantee successful fine-tuning, so the input representation and adaptation procedure should be evaluated and reported.
- For clinicians and developers, the models studied are research prototypes that require patient-level, multicentre, prospective validation before any clinical use.
Drawn from Ramollari et al. (2026), Algorithms, 19(11), Article 1011.
When this research may be relevant
This article may be relevant to researchers working on breast ultrasound classification and computer-aided diagnosis, on external validation and dataset shift in medical imaging, and on whether self-supervised foundation models such as DINOv2 transfer better than supervised convolutional networks and Vision Transformers. It also speaks to work on probability calibration, decision-threshold transport between institutions, and reproducible model comparison over repeated folds and seeds.
Research topics addressed: breast ultrasound classification; medical image analysis; deep learning; external validation; dataset shift; cross-institution generalization; ResNet-50; EfficientNet-B0; Vision Transformer; DINOv2; foundation models; probability calibration; temperature scaling; decision thresholds; reproducibility; BUS-BRA dataset; BUSI dataset
How to cite
- APA 7
- Ramollari, E., Vajjhala, N. R., Kumar, R., Ramchandani, K., Mahraj, R., & Hussain, K. (2026). Breast Ultrasound Classification Under Cross-Dataset Shift: A Multi-Seed Benchmark of Convolutional Networks, a Vision Transformer, and a Self-Supervised Foundation Model. Algorithms, 19(11), Article 1011. https://doi.org/10.3390/a19111011
- MLA 9
- Ramollari, Ervin, et al. “Breast Ultrasound Classification Under Cross-Dataset Shift: A Multi-Seed Benchmark of Convolutional Networks, a Vision Transformer, and a Self-Supervised Foundation Model.” Algorithms, vol. 19, no. 11, 2026, article 1011. https://doi.org/10.3390/a19111011.
- IEEE
- E. Ramollari, N. R. Vajjhala, R. Kumar, K. Ramchandani, R. Mahraj, and K. Hussain, “Breast Ultrasound Classification Under Cross-Dataset Shift: A Multi-Seed Benchmark of Convolutional Networks, a Vision Transformer, and a Self-Supervised Foundation Model,” Algorithms, vol. 19, no. 11, Art. no. 1011, Nov. 2026, doi: 10.3390/a19111011.
BibTeX
@article{ramollari2026breast,
title = {Breast Ultrasound Classification Under Cross-Dataset Shift: A Multi-Seed Benchmark of Convolutional Networks, a Vision Transformer, and a Self-Supervised Foundation Model},
author = {Ramollari, Ervin and Vajjhala, Narasimha Rao and Kumar, Rajesh and Ramchandani, Komal and Mahraj, Ranjeeta and Hussain, Khalid},
journal = {Algorithms},
volume = {19},
number = {11},
pages = {1011},
year = {2026},
publisher = {MDPI},
doi = {10.3390/a19111011},
url = {https://doi.org/10.3390/a19111011}
}RIS
TY - JOUR TI - Breast Ultrasound Classification Under Cross-Dataset Shift: A Multi-Seed Benchmark of Convolutional Networks, a Vision Transformer, and a Self-Supervised Foundation Model AU - Ramollari, Ervin AU - Vajjhala, Narasimha Rao AU - Kumar, Rajesh AU - Ramchandani, Komal AU - Mahraj, Ranjeeta AU - Hussain, Khalid JO - Algorithms PY - 2026 DA - 2026/11/18 VL - 19 IS - 11 SP - 1011 PB - MDPI SN - 1999-4893 DO - 10.3390/a19111011 UR - https://www.narasimharao.net/research/breast-ultrasound-classification-cross-dataset-shift-multi-seed-benchmark/ LA - en ER -
Links and files
- DOI: https://doi.org/10.3390/a19111011
- Publisher’s page: https://www.mdpi.com/1999-4893/19/11/1011
- Full text (PDF): https://www.mdpi.com/1999-4893/19/11/1011/pdf
- This page as text: https://www.narasimharao.net/research/breast-ultrasound-classification-cross-dataset-shift-multi-seed-benchmark/paper.md
- Permanent address of this page: https://www.narasimharao.net/research/breast-ultrasound-classification-cross-dataset-shift-multi-seed-benchmark/
Related research
Related publications
A perceptual-hash audit of BRISC 2025 found 1,352 candidate near-duplicate pairs, 439 of them crossing the official train/test split, and 104 training images pixel-identical to test images.
Fine-tuned DINOv2 and EfficientNet-B0 performed best on nine-class DeepWeeds data, but differences among several high-performing models were not consistently statistically significant across random seeds.
A lightweight EfficientNetV2B0 model trained under CLAIM and TRIPOD-AI reporting standards achieved an AUC of 0.967 (95% CI 0.953–0.979) and pneumonia recall of 0.98 on paediatric chest X-rays, missing only about 2% of true cases.
The volume applies explainable and transparent AI across clinical decision-making, telehealth, electronic health records, EEG analysis, wearable monitoring and mental health, pairing ethical analysis with applied systems and strategies for building trust.
The chapter shows that successful healthcare AI depends on transparency, accountability and collaboration among patients, providers, developers, regulators and insurers, and identifies liability, algorithmic opacity and unclear accountability for AI-influenced outcomes as the key barriers to adoption.