DETECTING SHIFT IS NOT DETECTING HARM: A TEMPORAL AUDIT OF LABEL-FREE DRIFT MONITORING IN JUST-IN-TIME DEFECT PREDICTION

Abstract

Label-free drift monitoring is attractive for just-in-time software defect prediction (JIT-SDP) because defect labels arrive late, but detecting that a distribution has changed does not establish that a deployed model has become worse for the decision it supports. This study audits whether drift in the predicted-score distribution of a fixed, temporally deployed JIT-SDP model tracks subsequent degradation. Ten projects from the public jit-sdp-data repository were modelled with 13 change metrics and three frozen learners; each project was split into 12 chronological windows, with an early fixed reference window, and drift was measured as the Jensen-Shannon divergence between current and reference score distributions. A staged protocol separated frozen discovery on four projects, replication on six untouched projects of the same benchmark family and a descriptive post-confirmatory audit, with the project as the inferential unit. The original hypothesis, that score drift tracks loss of Recall@20%, failed for all three learners (median Spearman ρ −0.400 to −0.067). A redesigned hypothesis linking drift to PR-AUC loss, frozen on discovery data, met its replication rule for all three learners (median ρ 0.350 to 0.625). However, PR-AUC loss closely tracked changes in observed defect prevalence (median ρ 0.900 to 0.950), the association between drift and PR-AUC loss became negative once PR-AUC was normalised by prevalence (median ρ −0.458 to −0.250), and the association between drift and Recall@20% loss remained negative (median ρ −0.442 to −0.108). Because the design does not separate drift from elapsed time, and declining prevalence may partly reflect label maturation, the drift signal is best read as tracking a prevalence-sensitive performance summary rather than fixed-budget harm. Drift and harm are therefore different constructs, and drift alarms should be validated against the deployment utility that matters.