Correlations of cross-entropy loss in Machine Learning

Connor R., Dearle A., Claydon B., Vadicamo L.

Kullback–Leibler divergence  Astrophysics  Jensen-Shannon 12 divergence  Cross-entropy  Article  cross-entropy  Science  QA75  Jensen–Shannon divergence  Physics  F-divergence  Triangular divergence  softmax  Q  triangular divergence  QA75 Electronic computers. Computer science  Kullback-Liebler divergence  T-NDAS  Softmax  f-divergence  QB460-466  QC1-999 

Cross-entropy loss is crucial in training many deep neural networks. In this context, we show a number of novel and strong correlations among various related divergence functions. In particular, we demonstrate that, in some circumstances, (a) cross-entropy is almost perfectly correlated with the little-known triangular divergence, and (b) cross-entropy is strongly correlated with the Euclidean distance over the logits from which the softmax is derived. The consequences of these observations are as follows. First, triangular divergence may be used as a cheaper alternative to cross-entropy. Second, logits can be used as features in a Euclidean space which is strongly synergistic with the classification process. This justifies the use of Euclidean distance over logits as a measure of similarity, in cases where the network is trained using softmax and cross-entropy. We establish these correlations via empirical observation, supported by a mathematical explanation encompassing a number of strongly related divergence functions.

Source: ENTROPY, vol. 26 (issue 6)

Metrics