An Ensemble Learning Approach for Software Defect Prediction in Developing Quality Software Product

Research summary

The summary, key findings, methodology and relevance notes below are this website’s own description of the paper, written from the published abstract and text. The official abstract and citation details are given further down.

The Problem

Software defect prediction models are usually judged on accuracy alone, which is insufficient for the imbalanced data typical of defect datasets.

The Methodology

Experimental machine learning comparison: seven ensemble models (CatBoost, LightGBM, XGBoost and boosted/bagged variants) plus a logistic regression baseline evaluated on six software defect datasets (NASA repository) with AUC, precision, recall, F-measure and Matthews correlation coefficient.

The Core Finding

A seven-model ensemble of boosted and bagged learners, evaluated with AUC, precision, recall, F-measure and Matthews correlation coefficient on NASA defect datasets, outperformed a logistic regression baseline, and the ensemble CatBoost model gave outstanding performance on all three datasets reported.

The Citation

Saheed, Y. K., Longe, O., Baba, U. A., Rakshit, S., & Vajjhala, N. R. (2021). An Ensemble Learning Approach for Software Defect Prediction in Developing Quality Software Product. In Mayank Singh, Vipin Tyagi, P. K. Gupta, Jan Flusser, Tuncer Ören, V. R. Sonawane (Eds.), Advances in Computing and Data Sciences: 5th International Conference, ICACDS 2021, Nashik, India, April 23–24, 2021, Revised Selected Papers, Part I (pp. 317–326). Springer International Publishing. https://doi.org/10.1007/978-3-030-81462-5_29 Copy APA

What question does this paper answer?

Can an ensemble of boosted and bagged machine learning models predict software defects more reliably than single models, when performance is judged on more than accuracy alone?

What did the study find, in detail?

The authors propose a seven-model ensemble for software defect prediction — CatBoost, LightGBM, XGBoost, boosted CatBoost, bagged logistic regression, boosted LightGBM and boosted XGBoost — compared with a logistic regression base model on six datasets. Performance is measured with AUC, precision, recall, F-measure and Matthews correlation coefficient rather than accuracy alone. The ensemble CatBoost model gave outstanding performance on all three defect datasets reported, attributed to reduced overfitting and shorter training time.

Why does it matter?

Accurate defect prediction helps developers target testing and produce more reliable software. Many earlier studies judged models on accuracy only; using AUC, F-measure and MCC gives a fuller picture, especially on the imbalanced datasets typical of defect prediction.

Study at a glance

Abstract

Software Defect Prediction (SDP) is a major research field in the software development life cycle. The accurate SDP would assist software developers and engineers in developing a reliable software product. Several machine learning techniques for SDP have been reported in the literature. Most of these studies suffered in terms of prediction accuracy and other performance metrics. Many of these studies focus only on accuracy and this is not enough in measuring the performance of SDP. In this research, we propose a seven-ensemble machine learning model for SDP. The Cat boost, Light Gradient Boosting Machine (LGBM), Extreme Gradient Boosting (XgBoost), boosted cat boost, bagged logistic regression, boosted LGBM, and boosted XgBoost were used for the experimental analysis. We also used the separate individual base model of logistic regression for the analysis on six datasets. This paper extends the performance metrics from only the accuracy, the Area Under Curve (AUC), precision, recall, F-measure, and Matthew Correlation Coefficient (MCC) were used as performance metrics. The results obtained showed that the proposed ensemble Cat boost model gave an outstanding performance for all the three defects datasets as a result of being able to decrease overfitting and reduce the training time.

Abstract as published in Advances in Computing and Data Sciences: 5th International Conference, ICACDS 2021, Nashik, India, April 23–24, 2021, Revised Selected Papers, Part I.

Keywords: Software defect prediction; Software product; Cat boost; Light gradient boosting machine; NASA repository; Extreme gradient boosting; Area under curve; F-measure

Limitations

• The abstract mentions six datasets but reports results for three defect datasets; numeric scores are not given in the abstract.

When this research may be relevant

This paper may be relevant to literature on software defect prediction, gradient boosting and ensemble methods (CatBoost, LightGBM, XGBoost) in software engineering, evaluation of classifiers on imbalanced NASA defect datasets, and the use of MCC and AUC instead of accuracy.

Research topics addressed: software defect prediction; ensemble learning; CatBoost; LightGBM; XGBoost; gradient boosting; bagging and boosting; software quality; NASA defect datasets; Matthews correlation coefficient; machine learning evaluation metrics; software engineering

How to cite

Saheed, Y. K., Longe, O., Baba, U. A., Rakshit, S., & Vajjhala, N. R. (2021). An Ensemble Learning Approach for Software Defect Prediction in Developing Quality Software Product. In Mayank Singh, Vipin Tyagi, P. K. Gupta, Jan Flusser, Tuncer Ören, V. R. Sonawane (Eds.), Advances in Computing and Data Sciences: 5th International Conference, ICACDS 2021, Nashik, India, April 23–24, 2021, Revised Selected Papers, Part I (pp. 317–326). Springer International Publishing. https://doi.org/10.1007/978-3-030-81462-5_29

@inproceedings{saheed2021ensemble,
  title = {An Ensemble Learning Approach for Software Defect Prediction in Developing Quality Software Product},
  author = {Saheed, Yakub Kayode and Longe, Olumide and Baba, Usman Ahmad and Rakshit, Sandip and Vajjhala, Narasimha Rao},
  booktitle = {Advances in Computing and Data Sciences: 5th International Conference, ICACDS 2021, Nashik, India, April 23–24, 2021, Revised Selected Papers, Part I},
  series = {Communications in Computer and Information Science},
  editor = {Mayank Singh and Vipin Tyagi and P. K. Gupta and Jan Flusser and Tuncer Ören and V. R. Sonawane},
  pages = {317--326},
  year = {2021},
  publisher = {Springer International Publishing},
  doi = {10.1007/978-3-030-81462-5_29},
  url = {https://doi.org/10.1007/978-3-030-81462-5_29}
}