Evidence overview · ESG and sustainability measurement

Can ESG compliance be measured reliably? Evidence from ESG ratings, project-level instruments, machine learning and supply chain records

Across four studies, Strang and Vajjhala find that ESG compliance can be measured at the project level with a short validated survey (CFI = 0.99, RMSEA = 0.052, N = 2,231) (Strang & Vajjhala, 2026, Information), but that existing ESG ratings and organizational records are much weaker measures than they appear. For 14 asset managers with more than USD 1 trillion under management, ESG ratings were unrelated to misconduct and arbitration cases decided against the firms (Strang & Vajjhala, 2024, Sustainability); in 207 financial software project records, machine learning classifiers performed at or near chance (kNN AUC = 0.497) (Strang & Vajjhala, 2026, Systems); and in 1,988 supply chain decision records, labor and environmental risks were rated lowest (medians 0 and 1 on 0–5 scales) while a perfect in-sample classifier (AUC = 1.000) reflected near-duplicate items (r = 0.915) rather than predictive skill (Strang & Vajjhala, 2026, Sustainability). The common thread is that how ESG data are generated shapes what can be learned from them.

A summary of 4 peer-reviewed studies by Narasimha Rao Vajjhala and co-authors, set against the literature those studies cite. Each study links to its own page with the full abstract and DOI.

Key takeaways

  1. Among 14 asset managers with over USD 1 trillion under management, MSCI, Sustainalytics and CSRhub ESG ratings were not related to counts of misconduct and arbitration cases decided against the firms (Strang & Vajjhala, 2024, Sustainability).
  2. ESG compliance in manufacturing projects was captured by a seven-item, two-factor instrument (ESG planning; ESG monitoring and controlling) with CFI = 0.99 and RMSEA = 0.052 in a sample of 2,231 sponsors (Strang & Vajjhala, 2026, Information).
  3. In one financial firm's project records, social and governance ratings tracked the overall project score almost perfectly (r = +0.995 and +0.966), while machine learning classifiers performed at or near chance (Strang & Vajjhala, 2026, Systems).
  4. Supply chain decision-makers rated labor and environmental regulatory risks lowest (medians 0 and 1 on 0–5 scales) even during a period of record forced-labor enforcement (Strang & Vajjhala, 2026, Sustainability).
  5. Perfect in-sample prediction of supply chain engagement outcomes (AUC = 1.000) was traced to near-duplicate items and possible label leakage, not to a usable predictive model (Strang & Vajjhala, 2026, Sustainability).

Context: why ESG compliance measurement is contested

Environmental, Social, and Governance (ESG) frameworks operate at the level of the firm, whereas the UN Sustainable Development Goals (SDGs) are framed nationally, and sustainability commitments are ultimately carried out in projects and supply chains (Strang & Vajjhala, 2026, Information). The literature cited by these studies documents persistent problems with firm-level ESG ratings: Berg and colleagues attributed 56% of rating divergence across six agencies to measurement differences, 38% to scope and 6% to weights[1], Chatterji and colleagues linked divergent interpretations to a lack of standardized methodologies[2], and Hess reported that many firms disclose anti-corruption policies without detail on implementation[3]. Connecting SDGs to firm-level ESG factors has been argued to require a double materiality perspective[4], and firms strong on material sustainability issues were found to outperform financially, which underlines industry-specific materiality[5].

In project management, sustainability has broadened the definition of project success[6], yet a review of 770 publications concluded that empirical validation remained limited[7]. Strang and Vajjhala therefore address ESG compliance measurement at three levels: firm ratings in financial services (Strang & Vajjhala, 2024, Sustainability), individual projects (Strang & Vajjhala, 2026, Information) (Strang & Vajjhala, 2026, Systems), and supply chain engagements (Strang & Vajjhala, 2026, Sustainability).

What the studies found

ESG ratings and anti-corruption governance. Using AI-assisted searches of regulatory and arbitration records for 14 US asset management firms, measured against the GRI 206-1 disclosure on legal actions, the study found that none of the three ESG scores (MSCI, Sustainalytics, CSRhub) was related to misconduct or arbitration counts; all corresponding Bayes factors were below 1 (Strang & Vajjhala, 2024, Sustainability). Misconduct and arbitration counts were strongly correlated with each other (r = 0.897, BF+0 = 3005, 99% CI 0.617–0.965; Vovk-Sellke p-ratio 4411), and ratings diverged across providers: MSCI and CSRhub agreed (r = 0.891) while MSCI and Sustainalytics did not (r = −0.069) (Strang & Vajjhala, 2024, Sustainability). The sample was small, several firms lacked ratings, and when the two firms with the most cases were removed the misconduct–arbitration correlation disappeared, so the results describe a narrow, skewed population (Strang & Vajjhala, 2024, Sustainability).

A project-level ESG compliance instrument. A 30-item, six-dimension survey completed by 2,231 project sponsors and decision-makers in North American manufacturing (NAICS 31–33) reduced to seven items on two correlated factors (r = 0.50), ESG planning and ESG monitoring and controlling, with CFI = 0.99, TLI = 0.98, RMSEA = 0.052 and SRMR < 0.035 (Strang & Vajjhala, 2026, Information). The authors acknowledge that the E, S and G pillars are not separately distinguishable in the final model, that stakeholder and community items did not survive screening, and that predictive validity against independent ESG outcomes was not tested (Strang & Vajjhala, 2026, Information).

Machine learning on ESG project records. In 207 archival records from a financial software engineering firm, stakeholder-rated social and governance factors correlated almost perfectly with the overall project score (r = +0.995 and +0.966), while the environmental factor was unrelated to it (Strang & Vajjhala, 2026, Systems). Classifiers were weak (kNN AUC = 0.497; SVM accuracy 61.8%), and diagnostics could bound but not eliminate method-based explanations, so the correlations are read as a property of the firm's evaluation system rather than of distinct ESG constructs (Strang & Vajjhala, 2026, Systems).

ESG risk perception in digital supply chains. Six perceived regulatory-volatility risks in 1,988 decision-maker records did not form a single construct (KMO = 0.470; three single-indicator factors), and labor and environmental risks were rated lowest during a period of record forced-labor enforcement (Strang & Vajjhala, 2026, Sustainability). A logistic regression separated success from failure perfectly in-sample, but because perceptions and outcomes were recorded together at engagement close-out, the authors treat this as possible label leakage and report no individual predictor effects (Strang & Vajjhala, 2026, Sustainability).

How this fits the wider literature

The rating divergence found among asset managers is consistent with the divergence documented by Berg and colleagues and Chatterji and colleagues[1][2] (Strang & Vajjhala, 2024, Sustainability). To explain the extreme case counts, the study draws on Egan, Matvos and Seru, who reported that some firms have far more adviser misconduct relative to their size[8], and on Honigsberg and Jacob, who found that brokers with prior expungements were 3.3 times as likely to engage in new misconduct[9] (Strang & Vajjhala, 2024, Sustainability).

The two-factor project instrument is interpreted through lifecycle governance, which holds that sustainability requires governance mechanisms across the project lifecycle[10], and is consistent with evidence that sustainability management relates to project success[11] and that governance and leadership form key sustainability factor clusters[12] (Strang & Vajjhala, 2026, Information).

In supply chains, behavioral research shows that buyers' perceptions of disruption magnitude and probability drive decisions such as supplier switching[13], and that how risk is communicated inside an organization alters perceived risk[14]. The finding of compartmentalized, low-rated sustainability risks is linked to literature on forced labor and purchasing practices, which the authors use to describe a possible sustainability leakage, where surprise enforcement leads to supplier exit rather than remediation[15][16] (Strang & Vajjhala, 2026, Sustainability). The authors stress that their data show only the perception gap, not that leakage occurred (Strang & Vajjhala, 2026, Sustainability). The leakage diagnosis follows the data-mining literature on target leakage[17], which the authors argue has received little attention in supply chain analytics despite enthusiasm for AI-based risk prediction[18] (Strang & Vajjhala, 2026, Sustainability).

Methods used and their limits

The studies combine Bayesian correlation with bootstrapping on public legal records (Strang & Vajjhala, 2024, Sustainability), sequential PCA, EFA and SEM for scale validation (Strang & Vajjhala, 2026, Information), correlation, principal component and method-bias diagnostics with kNN and SVM classifiers (Strang & Vajjhala, 2026, Systems), and exploratory factor analysis with parallel analysis plus logistic regression (Strang & Vajjhala, 2026, Sustainability). Three of the four rely on self-reported or single-source ratings, so common method bias, as described by Podsakoff and colleagues, is a recurring concern[19] (Strang & Vajjhala, 2026, Information) (Strang & Vajjhala, 2026, Sustainability). Each dataset comes from one country, sector or firm, and the authors call for multi-firm, cross-industry and prospective replication (Strang & Vajjhala, 2026, Information) (Strang & Vajjhala, 2026, Sustainability) (Strang & Vajjhala, 2024, Sustainability).

Practical implications for ESG measurement

For investors and regulators, the asset-manager results suggest that governance evidence such as fines and arbitration decisions is not captured in the ratings studied, and the authors recommend that legal proceedings be disclosed in specific rather than sweeping terms (Strang & Vajjhala, 2024, Sustainability). For project offices, the seven-item instrument offers a process-based benchmark that can be administered at close-out (Strang & Vajjhala, 2026, Information). For anyone building AI-enabled ESG or supply chain risk tools, the studies recommend multidimensional rather than composite risk dashboards, capture-time provenance metadata and leakage diagnostics (Strang & Vajjhala, 2026, Sustainability), and caution that models trained on a firm's own evaluation records may learn its rating habits rather than ESG performance (Strang & Vajjhala, 2026, Systems).

Evidence table

StudySetting / dataMethodKey result
Strang & Vajjhala (2024)
Sustainability
14 US asset management firms with AUM of USD 1 trillion or more; MSCI, Sustainalytics and CSRhub ratings; FINRA/SEC legal recordsAI-assisted data collection; Pearson and Bayesian correlation with 1,000 bootstrapsESG ratings unrelated to misconduct or arbitration counts (all BF < 1); misconduct–arbitration r = 0.897, BF+0 = 3005; MSCI–Sustainalytics r = −0.069
Strang & Vajjhala (2026)
Information
2,231 project sponsors and decision-makers, North American manufacturing (NAICS 31–33)30-item survey; PCA, EFA, CFA/SEM (DWLS)Seven-item, two-factor model (ESG planning; ESG monitoring and controlling); CFI = 0.99, TLI = 0.98, RMSEA = 0.052, SRMR < 0.035
Strang & Vajjhala (2026)
Systems
207 anonymized project records from one financial software engineering firmSingle-case socio-technical analysis; correlation, PCA, method-bias diagnostics; kNN and SVMSocial r = +0.995 and governance r = +0.966 with overall score; environmental unrelated; kNN AUC = 0.497, SVM accuracy 61.8%
Strang & Vajjhala (2026)
Sustainability
1,988 anonymized decision-maker records from a multinational logistics firm's lessons-learned repositoryDescriptive statistics; EFA with parallel analysis; binary logistic regressionKMO = 0.470, no single construct; median labor severity 0 and environmental 1 (0–5); AUC = 1.000 in-sample due to near-duplicate items (r = 0.915)

Questions researchers ask

Do ESG ratings capture corruption and misconduct risk in financial firms?
In a sample of 14 large US asset managers, none of the three ESG scores studied was related to misconduct or arbitration counts, and the providers disagreed with each other for the same firms (Strang & Vajjhala, 2024, Sustainability). The sample was small and skewed by two firms with very high case counts, so the authors call for larger and international samples (Strang & Vajjhala, 2024, Sustainability).
Is there a validated instrument for measuring ESG compliance at the project level?
Strang and Vajjhala validated a seven-item, two-factor survey (ESG planning; ESG monitoring and controlling) with 2,231 manufacturing project sponsors, reporting CFI = 0.99, TLI = 0.98 and RMSEA = 0.052 (Strang & Vajjhala, 2026, Information). It measures ESG as a project management process rather than environmental, social or governance outcomes, and has not yet been tested against independent outcome measures (Strang & Vajjhala, 2026, Information).
Can machine learning measure ESG compliance from existing project records?
In one financial software firm's 207 project records, kNN performed at chance (AUC = 0.497) and SVM reached 61.8% accuracy, only modestly above baseline (Strang & Vajjhala, 2026, Systems). The authors attribute this partly to how the firm rates projects and partly to technical factors such as sample size and dichotomization of the target (Strang & Vajjhala, 2026, Systems).
Why can a perfect classifier on supply chain ESG risk records be a warning sign?
In 1,988 logistics records, a logistic regression classified every engagement correctly in-sample, but the separation was driven by two near-duplicate items (r = 0.915) recorded at the same time as the outcome (Strang & Vajjhala, 2026, Sustainability). The authors read this as possible label leakage, consistent with the data-mining literature on leakage, and recommend provenance-aware data governance[17] (Strang & Vajjhala, 2026, Sustainability).

Open questions

  • Do scores on the project-level ESG instrument predict independent ESG outcomes such as audit results, emissions reductions or corporate ESG ratings, and does the two-factor structure hold outside North American manufacturing (Strang & Vajjhala, 2026, Information)?
  • Can pillar-specific (E, S and G) project-level measures be built, given that the pillars collapsed into process-based factors (Strang & Vajjhala, 2026, Information)?
  • Does the perception gap around labor and environmental regulation actually lead to supplier exit rather than remediation, which would require supplier-level sourcing data linked to enforcement events (Strang & Vajjhala, 2026, Sustainability)?
  • Would ESG ratings reflect anti-corruption records in larger, non-US and non-financial samples, and why are misconduct and arbitration counts concentrated in a few firms (Strang & Vajjhala, 2024, Sustainability)?
  • Can prospective, time-stamped data capture and penalized estimation separate genuine predictive signal from leakage in organizational ESG and risk records (Strang & Vajjhala, 2026, Sustainability) (Strang & Vajjhala, 2026, Systems)?

References

Studies summarised

Other literature cited

  1. [1] Berg, F.; Kolbel, J.F.; Rigobon, R. Aggregate Confusion: The Divergence of ESG Ratings. Rev. Financ. 2022, 26, 1315–1344.
  2. [2] Chatterji, A.K.; Durand, R.; Levine, D.I.; Touboul, S. Do ratings of firms converge? Implications for managers, investors and strategy researchers. Strat. Manag. J. 2016, 37, 1597–1614.
  3. [3] Hess, D. The Transparency Trap: Non-Financial Disclosure and the Responsibility of Business to Respect Human Rights. Am. Bus. Law J. 2019, 56, 5–53.
  4. [4] Delgado-Ceballos, J.; Ortiz-De-Mandojana, N.; Antolin-Lopez, R.; Montiel, I. Connecting the Sustainable Development Goals to firm-level sustainability and ESG factors: The need for double materiality. BRQ Bus. Res. Q. 2023, 26, 2–10.
  5. [5] Khan, M.; Serafeim, G.; Yoon, A. Corporate Sustainability: First Evidence on Materiality. Account. Rev. 2016, 91, 1697–1724.
  6. [6] Silvius, A.; Schipper, R.P. Sustainability in project management: A literature review and impact analysis. Soc. Bus. 2014, 4, 63–96.
  7. [7] Sabini, L.; Muzio, D.; Alderman, N. 25 years of ‘sustainable projects’. What we know and what the literature says. Int. J. Proj. Manag. 2019, 37, 820–838.
  8. [8] Egan, M.; Matvos, G.; Seru, A. The Market for Financial Adviser Misconduct. J. Polit. Econ. 2019, 127, 233–295.
  9. [9] Honigsberg, C.; Jacob, M. Deleting misconduct: The expungement of BrokerCheck records. J. Financ. Econ. 2021, 139, 800–831.
  10. [10] Kivila, J.; Martinsuo, M.; Vuorinen, L. Sustainable project management through project control in infrastructure projects. Int. J. Proj. Manag. 2017, 35, 1167–1183.
  11. [11] Carvalho, M.M.; Rabechini, R. Can project sustainability management impact project success? An empirical study applying a contingent approach. Int. J. Proj. Manag. 2017, 35, 1120–1132.
  12. [12] Martens, M.L.; Carvalho, M.M. Key factors of sustainability in project management context: A survey exploring the project managers’ perspective. Int. J. Proj. Manag. 2017, 35, 1084–1102.
  13. [13] Ellis, S.C.; Henry, R.M.; Shockley, J. Buyer perceptions of supply disruption risk: A behavioral view and empirical assessment. J. Oper. Manag. 2010, 28, 34–46. https://doi.org/10.1016/j.jom.2009.07.002
  14. [14] DuHadway, S.; Carnovale, S.; Kannan, V.R. Organizational communication and individual behavior: Implications for supply chain risk management. J. Supply Chain Manag. 2018, 54, 3–19. https://doi.org/10.1111/jscm.12182
  15. [15] LeBaron, G. The role of supply chains in the global business of forced labour. J. Supply Chain Manag. 2021, 57, 29–42. https://doi.org/10.1111/jscm.12258
  16. [16] Anner, M. Squeezing workers’ rights in global supply chains: Purchasing practices in the Bangladesh garment export sector in comparative perspective. Rev. Int. Polit. Econ. 2020, 27, 320–347. https://doi.org/10.1080/09692290.2019.1625426
  17. [17] Kaufman, S.; Rosset, S.; Perlich, C.; Stitelman, O. Leakage in data mining: Formulation, detection, and avoidance. ACM Trans. Knowl. Discov. Data 2012, 6, 15.
  18. [18] Ivanov, D.; Dolgui, A.; Sokolov, B. The impact of digital technology and Industry 4.0 on the ripple effect and supply chain risk analytics. Int. J. Prod. Res. 2019, 57, 829–846. https://doi.org/10.1080/00207543.2018.1488086
  19. [19] Podsakoff, P.M.; MacKenzie, S.B.; Lee, J.-Y.; Podsakoff, N.P. Common method biases in behavioral research: A critical review of the literature and recommended remedies. J. Appl. Psychol. 2003, 88, 879–903.