Semantic-Aware Multi-Spectral Confusion Regularization for Long-Tailed Visual Recognition

Abstract

Long-tailed visual recognition remains dominated by a persistent failure
mode: even methods that raise overall accuracy leave worst-class and
tail-class performance far behind. The recently proposed ConfusionAware Spectral Regularizer (CAR) [1] reframes this problem through
a confusion-matrix lens, deriving a PAC-Bayesian generalization bound
in which the worst-class error is controlled by the spectral norm of a
frequency-weighted confusion matrix, and introducing a differentiable
surrogate with an exponential-moving-average (EMA) estimator to
optimize this quantity during training. While CAR yields substantial
gains over prior long-tailed baselines, its design rests on three simplifying assumptions that become increasingly strained as class count and
semantic granularity grow: (i) a single dominant confusion direction
(the leading singular value) is assumed to capture worst-class risk; (ii)
all off-diagonal confusions are weighted identically regardless of the
semantic relationship between the confused classes; and (iii) a single
global EMA momentum is applied uniformly across classes, even though
rare classes are observed far less frequently within a mini-batch. In this
work we propose Semantic-Aware Multi-Spectral CAR (SAMS-CAR), an
extension that (1) regularizes the Ky-Fan k-norm the sum of the top-k
singular values of the confusion matrix rather than only its spectral norm,
capturing multiple concurrent confusion clusters typical of large-K,
fine-grained taxonomies; (2) reweights confusion entries using a learned
semantic-prototype affinity matrix that up-weights confusions between
semantically distant classes, targeting the feature collapse confusions
most damaging to tail-class generalization