Mining Project Failure Indicators From Big Data Using Machine Learning Mixed Methods

Summary

Strang and Vajjhala (2023) analyzed big data on about 17,430 US government IT projects to identify the indicators most strongly associated with IT project failure, using random forest machine learning followed by post-hoc logistic regression in a pragmatic recursive mixed-methods design. The random forest identified seven failure indicators with 81% recall, an F1 score of 0.798 and a ROC area of 0.849. The logistic regression (McFadden r2 = 0.267) confirmed project manager experience, project budget and outsourced versus in-house project managers as significant predictors of failure.

Practical implication. For sponsors and decision makers, project manager experience was the most important predictor of IT project success in about 17,430 US government IT projects: more experience was associated with avoiding failure.

The Problem

Thousands of IT projects fail every year, yet the indicators most strongly associated with failure have not been mined systematically from large project datasets.

The Methodology

Pragmatic recursive mixed-methods design on U.S. Government Accountability Office big data (about 17,430 IT projects): structured data cleaning, random forest classification of project breach/failure, and post-hoc logistic regression in SPSS.

The Core Finding

Random forest analysis of about 17,430 US government IT projects identified seven failure indicators with 79.9% precision, 81% recall and a ROC area of 0.849. Post-hoc logistic regression confirmed project manager experience, project budget and outsourced versus in-house project managers as significant predictors of failure.

The Citation

Strang, K. D., & Vajjhala, N. R. (2023). Mining Project Failure Indicators From Big Data Using Machine Learning Mixed Methods. International Journal of Information Technology Project Management, 14(1), 1–24. https://doi.org/10.4018/IJITPM.317221 Copy APA

The summary, key findings, methods, implications and relevance notes on this page are this website’s own description of the paper, written from the published abstract and text. The official abstract and citation details are given further down.

The study in detail

What question does this paper answer?

Can machine learning explain why thousands of IT projects failed by mining hundreds of big data attributes, and which indicators are most likely associated with IT project failure?

What did the study find, in detail?

Applying random forest machine learning to big data on about 17,430 U.S. government IT-related projects, the authors identified seven failure indicators with 79.9% precision, 81% recall, an F1 score of 0.798 and a ROC area of 0.849. A post-hoc logistic regression on those seven features was significant (χ2 = 76.287, p < .001, McFadden r2 = 0.267), confirming project manager experience, project budget and outsourced versus in-house project managers as significant predictors.

Why does it matter?

Roughly half of IT-related projects fail and earlier studies of success factors left 88–98% of the variation unexplained. The study shows that a pragmatic mixed-methods sequence — machine learning first, followed by logistic regression — can pull useful failure indicators out of messy project big data, and the authors urge researchers to use actual organizational project metrics rather than surveys of opinion.

This study is part of the evidence overview What predicts IT project failure and success? Evidence on project managers, bias and team transparency from machine learning and experiments

Methods and data

Definitions and key terms

Random forest

An ensemble machine learning technique that builds many decision trees and combines their votes to classify cases, here IT projects as failed (breached) or successful. — as used in Strang & Vajjhala (2023)

Node purity

In random forest, the increase in node purity measures a feature's importance, which the authors liken to the change in r2 effect size when a factor is added to a regression model. — as used in Strang & Vajjhala (2023)

ROC area (ROCa)

The area under the Receiver Operating Characteristic curve, estimating how well the model's classifications separate the two outcome classes; also called AUC. — as used in Strang & Vajjhala (2023)

All key terms across the publications →

Abstract

The literature revealed approximately 50% of IT-related projects around the world fail, which must frustrate a sponsor or decision maker since their ability to forecast success is statistically about the same as guessing with a random coin toss. Nonetheless, some project success/failure factors have been identified, but often the effect sizes were statistically negligible. A pragmatic mixed methods recursive approach was applied, using structured programming, machine learning (ML), and statistical software to mine a large data source for probable project success/failure indicators. Seven feature indicators were detected from ML, producing an accuracy of 79.9%, a recall rate of 81%, an F1 score of 0.798, and a ROCa of 0.849. A post-hoc regression model confirmed three indicators were significant with a 27% effect size. The contributions made to the body of knowledge included: A conceptual model comparing ML methods by artificial intelligence capability and research decision making goal, a mixed methods recursive pragmatic research design, application of the random forest ML technique with post hoc statistical methods, and a preliminary list of IT project failure indicators analyzed from big data.

Abstract as published in International Journal of Information Technology Project Management under a Creative Commons licence.

Keywords: Big Data; Information Technology; Machine Learning; Model; Prediction; Project Failure; Project Management; Random Forest

Limitations and open questions

• The big data were messy, with alphanumeric codes, corrupted fields and missing or null values, requiring extensive cleaning with object-oriented Visual Basic before analysis.
• The very high share of male project managers (94%) would cause statistical skew of the model.
• The 'remote allowed' feature could not be tested in the logistic regression because it contained too many missing or null values, and comparative importance scores were not available from the regression due to data irregularity.
• Line of business was an internal defense industry code that could not be further interpreted.
• The study took place in the U.S.; the authors call for studies from other countries and for replication with other machine learning techniques.

Open questions

• Do the failure indicators replicate outside the U.S. and with other machine learning techniques, using project performance records rather than surveys?

Open questions from the evidence overview Project management and project success.

Practical implications

• For a sponsor or decision maker forecasting IT project success, more years of project manager experience were associated with avoiding project failure (beta = −1.441, z = −9.434, p < .001).
• For a sponsor or decision maker, higher-budget US government IT projects were more likely to end in a breach (failure) condition, and in-house rather than outsourced project managers were more likely to have breached projects.
• For a sponsor or decision maker, project manager salary/contract value (p = 0.278) and project manager certification (p = 0.591) had no significant impact on IT project breach versus success outcomes.
• The authors urge researchers to use actual organizational project metrics rather than surveys of opinion; the study shows that machine learning followed by logistic regression can pull useful failure indicators out of messy project big data.

Drawn from Strang & Vajjhala (2023), International Journal of Information Technology Project Management, 14(1), pp. 1–24.

When this research may be relevant

This article may be relevant to literature searches on IT project failure prediction, machine learning in project management, project manager experience and budget as success factors, government and defence IT programme performance, and mixed-methods designs that combine random forest feature importance with inferential statistics.

Research topics addressed: machine learning; random forest; project failure; IT project management; big data; mixed methods; logistic regression; project success factors; project manager experience; U.S. government projects; defense projects; data mining; predictive analytics

How to cite

APA 7

Strang, K. D., & Vajjhala, N. R. (2023). Mining Project Failure Indicators From Big Data Using Machine Learning Mixed Methods. International Journal of Information Technology Project Management, 14(1), 1–24. https://doi.org/10.4018/IJITPM.317221Copy APA citation

MLA 9

Strang, Kenneth David, and Narasimha Rao Vajjhala. “Mining Project Failure Indicators From Big Data Using Machine Learning Mixed Methods.” International Journal of Information Technology Project Management, vol. 14, no. 1, 2023, pp. 1–24. https://doi.org/10.4018/IJITPM.317221.Copy MLA citation

IEEE

K. D. Strang and N. R. Vajjhala, “Mining Project Failure Indicators From Big Data Using Machine Learning Mixed Methods,” International Journal of Information Technology Project Management, vol. 14, no. 1, pp. 1–24, Feb. 2023, doi: 10.4018/IJITPM.317221.Copy IEEE citation

@article{strang2023machine,
  title = {Mining Project Failure Indicators From Big Data Using Machine Learning Mixed Methods},
  author = {Strang, Kenneth David and Vajjhala, Narasimha Rao},
  journal = {International Journal of Information Technology Project Management},
  volume = {14},
  number = {1},
  pages = {1--24},
  year = {2023},
  publisher = {IGI Global},
  doi = {10.4018/IJITPM.317221},
  url = {https://doi.org/10.4018/IJITPM.317221}
}

TY  - JOUR
TI  - Mining Project Failure Indicators From Big Data Using Machine Learning Mixed Methods
AU  - Strang, Kenneth David
AU  - Vajjhala, Narasimha Rao
JO  - International Journal of Information Technology Project Management
PY  - 2023
DA  - 2023/02/03
VL  - 14
IS  - 1
SP  - 1
EP  - 24
PB  - IGI Global
DO  - 10.4018/IJITPM.317221
UR  - https://www.narasimharao.net/research/machine-learning-it-project-failure-indicators-big-data/
LA  - en
ER  - 

Links and files

• DOI: https://doi.org/10.4018/IJITPM.317221
• This page as text: https://www.narasimharao.net/research/machine-learning-it-project-failure-indicators-big-data/paper.md
• Permanent address of this page: https://www.narasimharao.net/research/machine-learning-it-project-failure-indicators-big-data/