# Mining Project Failure Indicators From Big Data Using Machine Learning Mixed Methods

**Authors:** Kenneth David Strang (RMIT, Australia & W3 Research, USA) — ORCID 0000-0002-4333-4399; Narasimha Rao Vajjhala (American University of Nigeria, Nigeria) — ORCID 0000-0002-8260-2392
**Type:** Journal article (open access)
**Source:** International Journal of Information Technology Project Management, 14(1), pp. 1–24
**Published:** 2023-02-03
**DOI:** https://doi.org/10.4018/IJITPM.317221
**Canonical page:** https://www.narasimharao.net/research/machine-learning-it-project-failure-indicators-big-data/
**Indexing:** Scopus · Web of Science · Q3
**Keywords:** Big Data; Information Technology; Machine Learning; Model; Prediction; Project Failure; Project Management; Random Forest

## Summary

**Question.** Can machine learning explain why thousands of IT projects failed by mining hundreds of big data attributes, and which indicators are most likely associated with IT project failure?

**Finding.** Applying random forest machine learning to big data on about 17,430 U.S. government IT-related projects, the authors identified seven failure indicators with 79.9% precision, 81% recall, an F1 score of 0.798 and a ROC area of 0.849. A post-hoc logistic regression on those seven features was significant (χ2 = 76.287, p < .001, McFadden r2 = 0.267), confirming project manager experience, project budget and outsourced versus in-house project managers as significant predictors.

**Why it matters.** Roughly half of IT-related projects fail and earlier studies of success factors left 88–98% of the variation unexplained. The study shows that a pragmatic mixed-methods sequence — machine learning first, followed by logistic regression — can pull useful failure indicators out of messy project big data, and the authors urge researchers to use actual organizational project metrics rather than surveys of opinion.

## Key findings

1. The study analyzed big data on an estimated 17,430 U.S. government IT-related projects (cutoff 31 December 2019) sourced via the U.S. Government Accountability Office, with a binary breach/failure outcome.
2. A random forest machine learning model with 94 trees and 4 predictors per split achieved an average precision of 0.799, recall of 0.810, F1 score of 0.798 and ROC area of 0.849 in classifying IT project failure.
3. Seven feature indicators exceeded the authors' node-purity cutoff of 0.02: PM experience (0.063), contract versus in-house PM (0.058), line of business (0.048), remote work allowed (0.032), PM contract/salary (0.031), project budget (0.025) and PM certification (0.024).
4. A post-hoc logistic regression on the seven machine-learning features was statistically significant (AIC = 67.068, BIC = 87.068, χ2 = 76.287, p < .001) with a McFadden r2 effect size of 0.267 (27%).
5. Project manager experience was the most important predictor of IT project success (beta = −1.441, z = −9.434, Wald = 88.996, p < .001): more years of experience were associated with avoiding project failure.
6. Projects with higher budgets were more likely to end in a breach (failure) condition (beta = 0.744, Wald = 21.651, p < .001), and in-house rather than outsourced project managers were more likely to have breached projects (beta = 0.869, Wald = 135.224, p < .001).
7. Project manager salary/contract value (p = 0.278) and project manager certification (p = 0.591) had no significant impact on IT project breach versus success outcomes.

## Study at a glance

| Item | Detail |
|---|---|
| Research question | Can machine learning explain why thousands of IT projects failed, and what are the most likely indicators of IT project failure? |
| Design | Pragmatic, recursive mixed-methods design: structured programming (object-oriented Visual Basic) to clean big data, random forest machine learning, then post-hoc logistic regression in SPSS v. 25. |
| Data | Big data from the U.S. Government Accountability Office covering close to a million records, filtered to IT-related (largely military) projects up to 31 December 2019; an estimated 17,430 projects. |
| Methods | Random forest (94 trees, 4 predictors per split) with a binary breach/failure outcome; feature importance by node purity; logistic regression on the top 7 features. |
| Main result | Random forest: precision 0.799, recall 0.810, F1 0.798, ROC area 0.849; logistic regression: McFadden r2 = 0.267, with PM experience, budget and outsourced/in-house significant. |
| Implication | More project manager experience is associated with avoiding failure; researchers should mine actual project performance data with machine learning rather than rely on surveys. |

## Abstract

The literature revealed approximately 50% of IT-related projects around the world fail, which must frustrate a sponsor or decision maker since their ability to forecast success is statistically about the same as guessing with a random coin toss. Nonetheless, some project success/failure factors have been identified, but often the effect sizes were statistically negligible. A pragmatic mixed methods recursive approach was applied, using structured programming, machine learning (ML), and statistical software to mine a large data source for probable project success/failure indicators. Seven feature indicators were detected from ML, producing an accuracy of 79.9%, a recall rate of 81%, an F1 score of 0.798, and a ROCa of 0.849. A post-hoc regression model confirmed three indicators were significant with a 27% effect size. The contributions made to the body of knowledge included: A conceptual model comparing ML methods by artificial intelligence capability and research decision making goal, a mixed methods recursive pragmatic research design, application of the random forest ML technique with post hoc statistical methods, and a preliminary list of IT project failure indicators analyzed from big data.

## Key terms

- **Random forest:** An ensemble machine learning technique that builds many decision trees and combines their votes to classify cases, here IT projects as failed (breached) or successful.
- **Node purity:** In random forest, the increase in node purity measures a feature's importance, which the authors liken to the change in r2 effect size when a factor is added to a regression model.
- **ROC area (ROCa):** The area under the Receiver Operating Characteristic curve, estimating how well the model's classifications separate the two outcome classes; also called AUC.

## Limitations

- The big data were messy, with alphanumeric codes, corrupted fields and missing or null values, requiring extensive cleaning with object-oriented Visual Basic before analysis.
- The very high share of male project managers (94%) would cause statistical skew of the model.
- The 'remote allowed' feature could not be tested in the logistic regression because it contained too many missing or null values, and comparative importance scores were not available from the regression due to data irregularity.
- Line of business was an internal defense industry code that could not be further interpreted.
- The study took place in the U.S.; the authors call for studies from other countries and for replication with other machine learning techniques.

## How to cite

Strang, K. D., & Vajjhala, N. R. (2023). Mining Project Failure Indicators From Big Data Using Machine Learning Mixed Methods. International Journal of Information Technology Project Management, 14(1), 1–24. https://doi.org/10.4018/IJITPM.317221

```bibtex
@article{strang2023machine,
  title = {Mining Project Failure Indicators From Big Data Using Machine Learning Mixed Methods},
  author = {Strang, Kenneth David and Vajjhala, Narasimha Rao},
  journal = {International Journal of Information Technology Project Management},
  volume = {14},
  number = {1},
  pages = {1--24},
  year = {2023},
  publisher = {IGI Global},
  doi = {10.4018/IJITPM.317221},
  url = {https://doi.org/10.4018/IJITPM.317221}
}
```