OPERATIONAL REVIEW-QUEUE CHURN UNDER TEMPORAL RETRAINING IN JUST-IN-TIME SOFTWARE DEFECT PREDICTION

Abstract

Just-in-time software defect prediction (JIT-SDP) models are periodically retrained as software projects evolve, and replacement models are usually judged by aggregate measures such as ROC-AUC. Teams that act on these models, however, review a ranked subset of incoming commits, so a retrained model can preserve its aggregate accuracy while selecting different commits. This operational consequence of temporal retraining has received limited empirical attention. We study it with a strictly chronological protocol on six open-source projects. Each project is split into eight time-ordered windows; at every step, an OLD model trained on the earlier history and a NEW model trained on one additional window score the same subsequent commits. We compare their probabilities, rankings, and fixed-budget review queues for Logistic Regression (LR), Random Forest (RF), and scikit-learn HistGradientBoostingClassifier (HGB), using retrospectively established defect labels. Across 108 OLD-vs-NEW comparisons, 62.96% showed top-20% review-queue Jaccard overlap below 0.75, and 37.04% combined this with an absolute ROC-AUC change of at most 0.02. In one JGroups case, ROC-AUC moved from 0.6878 to 0.6880 while the top-20% queues overlapped by only 0.427. The pattern appeared in the three discovery projects (44.44% stable-performance/high-churn) and in three independent replication projects (29.63%). Churn was strongly learner-dependent: 88.89% of HGB comparisons were high-churn, compared with 58.33% for RF and 41.67% for LR. Smaller review budgets were less stable, and the phenomenon persisted with six, eight and ten windows. Updates that incorporated two windows of new history were associated with lower queue overlap than one-window updates, and denser scores near the review cutoff were associated with lower overlap in most settings. Under the studied protocol, similar aggregate performance did not guarantee similar commit prioritisation, suggesting that model replacements should also be assessed for review-queue stability.