WHEN THE BEST MODEL DOES NOT STAY BEST IN SOFTWARE DEFECT PREDICTION
Abstract
Software defect prediction studies routinely compare several learners and deploy the one that performed best on historical validation data, implicitly assuming that this choice remains sound on the next release. We audit the temporal reliability of that model-selection decision. Using 11 longitudinal Java projects (41 releases) from an audited PROMISE repository snapshot, 20 common static-code metrics and five fixed, untuned learner families, each rolling transition trains candidates on releases up to Rt−2, selects among them on Rt−1 and evaluates the same fitted candidates on Rt. The primary outcome is next-release AUROC selection regret, the gap between the best deployment AUROC among the candidates and that of the historically selected learner. Projects, not transitions, are the inferential unit. Four projects formed a frozen discovery set and seven untouched projects from the same benchmark family were held out for replication. The original portability hypothesis failed all three frozen discovery criteria (median project-level Kendall τb = −0.05, median regret = 0.0598, median exact-winner retention = 0). One discovery-only redesign selected a fixed Random Forest policy, but it did not replicate on the held-out projects: the median paired regret reduction was 0.000 and Random Forest had lower regret in 3 of 7 projects, below the frozen requirement of 5. Post-confirmatory, descriptive analyses showed persistent ranking, winner and top-2 instability, only exploratory associations with release-to-release structural change, and no recovery of the Random Forest advantage after excluding exact repeated metric vectors. In this benchmark, historical model selection was temporally unreliable, and the simple fixed alternative was not a robust remedy. Model-selection portability therefore merits evaluation as a reliability property distinct from average predictive performance.
