arXiv Analytics

Sign in

arXiv:2205.06265 [cs.LG]AbstractReferencesReviewsResources

ELODI: Ensemble Logit Difference Inhibition for Positive-Congruent Training

Yue Zhao, Yantao Shen, Yuanjun Xiong, Shuo Yang, Wei Xia, Zhuowen Tu, Bernt Shiele, Stefano Soatto

Published 2022-05-12Version 1

Negative flips are errors introduced in a classification system when a legacy model is replaced with a new one. Existing methods to reduce the negative flip rate (NFR) either do so at the expense of overall accuracy using model distillation, or use ensembles, which multiply inference cost prohibitively. We present a method to train a classification system that achieves paragon performance in both error rate and NFR, at the inference cost of a single model. Our method introduces a generalized distillation objective, Logit Difference Inhibition (LDI), that penalizes changes in the logits between the new and old model, without forcing them to coincide as in ordinary distillation. LDI affords the model flexibility to reduce error rate along with NFR. The method uses a homogeneous ensemble as the reference model for LDI, hence the name Ensemble LDI, or ELODI. The reference model can then be substituted with a single model at inference time. The method leverages the observation that negative flips are typically not close to the decision boundary, but often exhibit large deviations in the distance among their logits, which are reduced by ELODI.

Related articles: Most relevant | Search more
arXiv:1312.6117 [cs.LG] (Published 2013-12-19, updated 2014-11-13)
Comparison three methods of clustering: k-means, spectral clustering and hierarchical clustering
arXiv:2010.08360 [cs.LG] (Published 2020-10-16)
G-DARTS-A: Groups of Channel Parallel Sampling with Attention
arXiv:1505.05215 [cs.LG] (Published 2015-05-20)
Learning with a Drifting Target Concept