LEXICAL SHORTCUT DEPENDENCE IN SELF-ADMITTED TECHNICAL DEBT DETECTION: A 20-PROJECT RELIABILITY STUDY

Abstract

Self-admitted technical debt (SATD) detectors trained on source-code comments routinely report strong aggregate performance, yet aggregate scores do not show which textual evidence a classifier relies on. This study audits whether conventional SATD text classifiers depend on shortcut-prone lexical cues, namely explicit debt-admission markers (TODO, FIXME, XXX, HACK, WORKAROUND, TEMP) and code-like identifiers. Using the 20-project MAT benchmark (118,316 comments, 5,807 of them SATD), audited at a fixed repository commit, TF-IDF logistic regression and linear support vector machine classifiers, with multinomial naive Bayes as an additional baseline, were evaluated under deterministic text transformations, duplicate-controlled splits, and leave-one-project-out (LOPO) evaluation. The design separated a five-project discovery stage, a fifteen-project independent replication stage with pre-specified decision rules, and a twenty-project robustness and mechanism stage. The broad hypothesis that cross-project performance is systematically lower than within-project performance did not replicate: only 5 of 15 replication projects met the frozen project rule (9 were required), and the median within-project minus LOPO SATD-F1 differences were 0.028 for logistic regression and −0.003 for the linear SVM (Holm-adjusted p = 1.000 for both). In contrast, removing explicit markers reduced LOPO SATD-F1 in all 15 replication projects (median reductions 0.297 and 0.316; Holm-adjusted p = 7.3 × 10⁻⁴) and in all 20 projects of the robustness stage (median reductions 0.285 and 0.291). Prediction churn after marker removal was concentrated in marker-containing comments (medians 0.355 and 0.366, against 0.068 and 0.045 in marker-free comments; higher in 20 of 20 projects for both models). The marker effect persisted in high- and low-prevalence project strata and was essentially unchanged between raw and duplicate-controlled data. Masking code-like identifiers also degraded performance, but among marker-free comments the effect was smaller and less uniform (median reductions 0.058 and 0.046; positive in 16 and 15 of 20 projects), which identifies it as a secondary mechanism. Removing project-specific vocabulary had a negligible effect. Strong SATD scores can therefore rest heavily on a small set of explicit lexical cues, and SATD evaluations should report marker-stratified performance alongside aggregate metrics.