An Ensemble Learning Approach for Software Defect Prediction in Developing Quality Software Product

Summary

Saheed et al. (2021) compared seven models for software defect prediction (CatBoost, LightGBM, XGBoost, boosted CatBoost, bagged logistic regression, boosted LightGBM and boosted XGBoost) with a separate logistic regression base model on six software defect datasets (the author keywords name the NASA repository), judging performance on AUC, precision, recall, F-measure and Matthews correlation coefficient rather than accuracy alone. The ensemble CatBoost model gave outstanding performance on all three defect datasets reported, which the authors attribute to reduced overfitting and shorter training time. The abstract mentions six datasets but reports results for three and gives no numeric scores.

Practical implication. Software developers and engineers evaluating defect prediction models get a fuller picture from AUC, F-measure and Matthews correlation coefficient than from accuracy alone, especially on imbalanced defect datasets.

The Problem

Software defect prediction models are usually judged on accuracy alone, which is insufficient for the imbalanced data typical of defect datasets.

The Methodology

Experimental machine learning comparison: seven ensemble models (CatBoost, LightGBM, XGBoost and boosted/bagged variants) plus a logistic regression baseline evaluated on six software defect datasets (NASA repository) with AUC, precision, recall, F-measure and Matthews correlation coefficient.

The Core Finding

A seven-model ensemble of boosted and bagged learners, evaluated with AUC, precision, recall, F-measure and Matthews correlation coefficient on NASA defect datasets, outperformed a logistic regression baseline, and the ensemble CatBoost model gave outstanding performance on all three datasets reported.

The Citation

Saheed, Y. K., Longe, O., Baba, U. A., Rakshit, S., & Vajjhala, N. R. (2021). An Ensemble Learning Approach for Software Defect Prediction in Developing Quality Software Product. In M. Singh, V. Tyagi, P. K. Gupta, J. Flusser, T. Ören, & V. R. Sonawane (Eds.), Advances in Computing and Data Sciences: 5th International Conference, ICACDS 2021, Nashik, India, April 23–24, 2021, Revised Selected Papers, Part I (pp. 317–326). Springer International Publishing. https://doi.org/10.1007/978-3-030-81462-5_29 Copy APA

The summary, key findings, methods, implications and relevance notes on this page are this website’s own description of the paper, written from the published abstract and text. The official abstract and citation details are given further down.

The study in detail

What question does this paper answer?

Can an ensemble of boosted and bagged machine learning models predict software defects more reliably than single models, when performance is judged on more than accuracy alone?

What did the study find, in detail?

The authors propose a seven-model ensemble for software defect prediction — CatBoost, LightGBM, XGBoost, boosted CatBoost, bagged logistic regression, boosted LightGBM and boosted XGBoost — compared with a logistic regression base model on six datasets. Performance is measured with AUC, precision, recall, F-measure and Matthews correlation coefficient rather than accuracy alone. The ensemble CatBoost model gave outstanding performance on all three defect datasets reported, attributed to reduced overfitting and shorter training time.

Why does it matter?

Accurate defect prediction helps developers target testing and produce more reliable software. Many earlier studies judged models on accuracy only; using AUC, F-measure and MCC gives a fuller picture, especially on the imbalanced datasets typical of defect prediction.

Methods and data

Standards, datasets and methods named in this study

• CatBoost Method
  One of the boosting methods among the models compared.
  Prokhorenkova, L., Gusev, G., Vorobev, A., Dorogush, A. V., & Gulin, A. (2018). CatBoost: Unbiased boosting with categorical features. Advances in Neural Information Processing Systems 31.
• LightGBM Method
  One of the boosting methods among the models compared.
  Ke, G., Meng, Q., Finley, T., Wang, T., Chen, W., Ma, W., Ye, Q., & Liu, T.-Y. (2017). LightGBM: A highly efficient gradient boosting decision tree. Advances in Neural Information Processing Systems 30.
• XGBoost Method
  One of the boosting methods among the models compared.
  Chen, T., & Guestrin, C. (2016). XGBoost: A scalable tree boosting system. Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 785–794. https://doi.org/10.1145/2939672.2939785

Definitions and key terms

Software defect prediction (SDP)

Research field in the software development life cycle in which machine learning techniques are used to predict software defects, where accurate prediction would assist software developers and engineers in developing a reliable software product. — as used in Saheed et al. (2021)

All key terms across the publications →

Abstract

Software Defect Prediction (SDP) is a major research field in the software development life cycle. The accurate SDP would assist software developers and engineers in developing a reliable software product. Several machine learning techniques for SDP have been reported in the literature. Most of these studies suffered in terms of prediction accuracy and other performance metrics. Many of these studies focus only on accuracy and this is not enough in measuring the performance of SDP. In this research, we propose a seven-ensemble machine learning model for SDP. The Cat boost, Light Gradient Boosting Machine (LGBM), Extreme Gradient Boosting (XgBoost), boosted cat boost, bagged logistic regression, boosted LGBM, and boosted XgBoost were used for the experimental analysis. We also used the separate individual base model of logistic regression for the analysis on six datasets. This paper extends the performance metrics from only the accuracy, the Area Under Curve (AUC), precision, recall, F-measure, and Matthew Correlation Coefficient (MCC) were used as performance metrics. The results obtained showed that the proposed ensemble Cat boost model gave an outstanding performance for all the three defects datasets as a result of being able to decrease overfitting and reduce the training time.

Abstract as published in Advances in Computing and Data Sciences: 5th International Conference, ICACDS 2021, Nashik, India, April 23–24, 2021, Revised Selected Papers, Part I.

Keywords: Software defect prediction; Software product; Cat boost; Light gradient boosting machine; NASA repository; Extreme gradient boosting; Area under curve; F-measure

Limitations

• The abstract mentions six datasets but reports results for three defect datasets; numeric scores are not given in the abstract.

Practical implications

• The authors argue that judging software defect prediction models on accuracy alone, as many earlier studies did, is not enough to measure their performance.
• For software developers and engineers choosing a defect prediction model, the ensemble CatBoost model gave outstanding performance on all three defect datasets reported, which the authors attribute to decreased overfitting and reduced training time.
• Software developers can use accurate defect prediction to target testing and produce more reliable software.

Drawn from Saheed et al. (2021), Advances in Computing and Data Sciences: 5th International Conference, ICACDS 2021, Nashik, India, April 23–24, 2021, Revised Selected Papers, Part I, Communications in Computer and Information Science, vol. 1440, pp. 317–326, Springer International Publishing.

When this research may be relevant

This paper may be relevant to literature on software defect prediction, gradient boosting and ensemble methods (CatBoost, LightGBM, XGBoost) in software engineering, evaluation of classifiers on imbalanced NASA defect datasets, and the use of MCC and AUC instead of accuracy.

Research topics addressed: software defect prediction; ensemble learning; CatBoost; LightGBM; XGBoost; gradient boosting; bagging and boosting; software quality; NASA defect datasets; Matthews correlation coefficient; machine learning evaluation metrics; software engineering

How to cite

APA 7

Saheed, Y. K., Longe, O., Baba, U. A., Rakshit, S., & Vajjhala, N. R. (2021). An Ensemble Learning Approach for Software Defect Prediction in Developing Quality Software Product. In M. Singh, V. Tyagi, P. K. Gupta, J. Flusser, T. Ören, & V. R. Sonawane (Eds.), Advances in Computing and Data Sciences: 5th International Conference, ICACDS 2021, Nashik, India, April 23–24, 2021, Revised Selected Papers, Part I (pp. 317–326). Springer International Publishing. https://doi.org/10.1007/978-3-030-81462-5_29Copy APA citation

MLA 9

Saheed, Yakub Kayode, et al. “An Ensemble Learning Approach for Software Defect Prediction in Developing Quality Software Product.” Advances in Computing and Data Sciences: 5th International Conference, ICACDS 2021, Nashik, India, April 23–24, 2021, Revised Selected Papers, Part I, edited by Mayank Singh et al., Springer International Publishing, 2021, pp. 317–26. https://doi.org/10.1007/978-3-030-81462-5_29.Copy MLA citation

IEEE

Y. K. Saheed, O. Longe, U. A. Baba, S. Rakshit, and N. R. Vajjhala, “An Ensemble Learning Approach for Software Defect Prediction in Developing Quality Software Product,” in Advances in Computing and Data Sciences: 5th International Conference, ICACDS 2021, Nashik, India, April 23–24, 2021, Revised Selected Papers, Part I, M. Singh, V. Tyagi, P. K. Gupta, J. Flusser, T. Ören, and V. R. Sonawane, Eds. Springer International Publishing, 2021, pp. 317–326, doi: 10.1007/978-3-030-81462-5_29.Copy IEEE citation

@inproceedings{saheed2021ensemble,
  title = {An Ensemble Learning Approach for Software Defect Prediction in Developing Quality Software Product},
  author = {Saheed, Yakub Kayode and Longe, Olumide and Baba, Usman Ahmad and Rakshit, Sandip and Vajjhala, Narasimha Rao},
  booktitle = {Advances in Computing and Data Sciences: 5th International Conference, ICACDS 2021, Nashik, India, April 23–24, 2021, Revised Selected Papers, Part I},
  series = {Communications in Computer and Information Science},
  editor = {Singh, Mayank and Tyagi, Vipin and Gupta, P. K. and Flusser, Jan and Ören, Tuncer and Sonawane, V. R.},
  pages = {317--326},
  year = {2021},
  publisher = {Springer International Publishing},
  doi = {10.1007/978-3-030-81462-5_29},
  url = {https://doi.org/10.1007/978-3-030-81462-5_29}
}

TY  - CPAPER
TI  - An Ensemble Learning Approach for Software Defect Prediction in Developing Quality Software Product
AU  - Saheed, Yakub Kayode
AU  - Longe, Olumide
AU  - Baba, Usman Ahmad
AU  - Rakshit, Sandip
AU  - Vajjhala, Narasimha Rao
ED  - Singh, Mayank
ED  - Tyagi, Vipin
ED  - Gupta, P. K.
ED  - Flusser, Jan
ED  - Ören, Tuncer
ED  - Sonawane, V. R.
T2  - Advances in Computing and Data Sciences: 5th International Conference, ICACDS 2021, Nashik, India, April 23–24, 2021, Revised Selected Papers, Part I
T3  - Communications in Computer and Information Science
PY  - 2021
DA  - 2021/10/23
VL  - 1440
SP  - 317
EP  - 326
PB  - Springer International Publishing
SN  - 1865-0929
DO  - 10.1007/978-3-030-81462-5_29
UR  - https://www.narasimharao.net/research/ensemble-learning-software-defect-prediction/
LA  - en
ER  - 

Links and files

• DOI: https://doi.org/10.1007/978-3-030-81462-5_29
• Publisher’s page: https://link.springer.com/chapter/10.1007/978-3-030-81462-5_29
• This page as text: https://www.narasimharao.net/research/ensemble-learning-software-defect-prediction/paper.md
• Permanent address of this page: https://www.narasimharao.net/research/ensemble-learning-software-defect-prediction/