A neural network does not answer questions, it produces numbers. Ask it to recognise an animal and the last layer hands you something like $(2.0,\; 1.0,\; 0.1)$ for cat, dog, bird. That is not an answer, it is a mood. Two small functions turn that mood into a decision and, crucially, into a lesson : softmax converts the raw scores into honest probabilities, and cross-entropy measures how wrong those probabilities were. Together they are the last two boxes of almost every classifier ever trained, and they hide a small miracle : their combined gradient is simply predicted minus true. This post builds both from scratch, with numbers, and shows exactly where that miracle comes from.
1. The problem : scores are not probabilities
The final layer of a classifier outputs one number per class. These raw numbers are called logits. In our example the network is leaning towards cat :
\[\begin{equation} z = (z_{\text{cat}},\, z_{\text{dog}},\, z_{\text{bird}}) = (2.0,\; 1.0,\; 0.1). \label{eq:logits} \end{equation}\]We would like to say “cat with 66 % confidence”, but $\eqref{eq:logits}$ cannot be read as probabilities : the numbers do not sum to $1$, and nothing stops a logit from being negative. We need a converter that takes any list of real numbers and returns a proper probability distribution : all entries positive, summing to one, and preserving the ranking (whoever scored highest must stay the most likely).
The naive fix, dividing each score by the total, fails immediately on negative numbers : with $z = (-1, 2)$ the sum is $1$ and we would report a probability of $-100\,\%$ for the first class. We need something that makes every number positive before normalising.
2. Softmax : exponentiate, then normalise
The exponential is exactly the tool for that. It is always positive, and it is increasing, so it never reorders anything. Apply it to each logit and divide by the total :
In plain words : give every class a number of lottery tickets equal to $e^{z}$, then a class’s probability is its share of all the tickets in the urn. Because the ticket count grows exponentially with the score, a slightly better score buys a lot more tickets. On our example :
\[\begin{aligned} e^{2.0} &= 7.389, & e^{1.0} &= 2.718, & e^{0.1} &= 1.105, & \text{sum} &= 11.212, \\ p_{\text{cat}} &= \tfrac{7.389}{11.212} = 0.659, & p_{\text{dog}} &= \tfrac{2.718}{11.212} = 0.242, & p_{\text{bird}} &= \tfrac{1.105}{11.212} = 0.099. && \end{aligned}\]Two properties are worth keeping in your pocket. First, softmax depends only on the differences between logits : adding the same constant $c$ to every score changes nothing, since $e^{z_i+c} = e^{c}e^{z_i}$ and the $e^{c}$ cancels between numerator and denominator. Second, with only two classes it collapses to the familiar sigmoid of backpropagation :
\[\begin{equation} p_1 = \frac{e^{z_1}}{e^{z_1}+e^{z_2}} = \frac{1}{1 + e^{-(z_1 - z_2)}} = \sigma(z_1 - z_2). \label{eq:sigmoid} \end{equation}\]So the sigmoid was never a different animal, it is softmax with $K=2$ looking at a single score gap.
max function, it is a smooth version of argmax : a hard argmax would return $(1,0,0)$, softmax returns the blurred $(0.659, 0.242, 0.099)$. That blur is the whole point, because a hard argmax has a derivative of zero everywhere and would tell gradient descent absolutely nothing.
3. Measuring the mistake : surprise, then cross-entropy
The network has now given an opinion. Suppose the picture was actually a dog. How bad is that opinion ? We need a loss, and it must respect one intuition : being confidently wrong should hurt far more than being hesitantly wrong.
Information theory gives the natural measure. Shannon (Shannon, 1948) defines the surprise of an event you assigned probability $p$ as $-\log p$. Predict the truth with $p=1$ and you are not surprised at all ($-\log 1 = 0$) ; predict it with $p = 0.01$ and you are very surprised ($-\log 0.01 = 4.6$) ; predict it with $p \to 0$ and your surprise goes to infinity.
Now write the truth as a one-hot vector : $y = (0, 1, 0)$ for dog, meaning “the true distribution puts all its mass on dog”. The cross-entropy between the truth $y$ and the prediction $p$ averages the surprise over the true distribution :
In plain words : the loss only ever looks at the probability you assigned to the right answer, and asks how surprised you were to learn it was right. For our misclassified dog :
\[\begin{equation} L = -\log p_{\text{dog}} = -\log(0.242) = 1.417. \label{eq:loss} \end{equation}\]Had the network said $p_{\text{dog}} = 0.9$ the loss would have been $0.105$ ; had it said $0.01$, it would have been $4.6$. That steepness is what makes the network flee confident mistakes.
4. The miracle : the gradient is just $p - y$
So far, two reasonable functions. Here is why they are always used together. Chain them, $z \to p = \operatorname{softmax}(z) \to L = -\sum_i y_i\log p_i$, and ask what backpropagation needs : the gradient of the loss with respect to the logits.
No exponentials, no fractions, no special cases : predicted minus true. The derivation is two lines. Write $S = \sum_k e^{z_k}$, so $\log p_i = z_i - \log S$ and therefore
\[L = -\sum_i y_i\,(z_i - \log S) = -\sum_i y_i z_i + \log S .\]Differentiating with respect to one logit $z_k$, the first sum contributes $-y_k$, and the second gives $\frac{\partial \log S}{\partial z_k} = \frac{e^{z_k}}{S} = p_k$. Adding them :
\[\begin{equation} \frac{\partial L}{\partial z_k} = p_k - y_k . \label{eq:grad} \end{equation}\]On our example the gradient is $(0.659,\; 0.242 - 1,\; 0.099) = (0.659,\; -0.758,\; 0.099)$. Read it as an instruction : lower the cat score, raise the dog score, lower the bird score a little — each by an amount proportional to how badly it was misjudged. Gradient descent then does the obvious thing, $z \leftarrow z - \eta(p-y)$, and the network’s next opinion is a little less wrong.
This is not a coincidence. The logarithm in the loss is precisely the inverse of the exponential in the softmax, and the two annihilate. Pair softmax with the squared-error loss instead and the gradient picks up a factor $p_k(1-p_k)$, which is nearly zero whenever the network is confident — including when it is confidently wrong, exactly the case you most need to fix. That is the vanishing-gradient trap of backpropagation reappearing at the very last layer, and cross-entropy is what removes it.
5. Temperature : the same scores, more or less bold
Because softmax reads only the differences between logits, scaling them all by a constant changes the shape of the distribution without changing the ranking. Divide the logits by a temperature $T$ before the softmax :
\[\begin{equation} p_i(T) = \frac{e^{z_i/T}}{\sum_k e^{z_k/T}}. \label{eq:temp} \end{equation}\]A small $T$ magnifies the gaps and makes the model decisive ; a large $T$ shrinks them towards a uniform distribution and makes it hesitant. In the limit, $T \to 0$ gives a hard argmax and $T \to \infty$ gives pure uniform noise.
This is the temperature knob you set when sampling text from a Transformer : low temperature gives safe, repetitive prose, high temperature gives creative and occasionally deranged prose. The same rescaling appears inside attention as the division by $\sqrt{d_k}$, whose job is to keep the softmax away from the saturated regime where all the gradients die.
6. Making it survive a computer
There is one practical trap. Logits of a few hundred are common in a large model, and $e^{800}$ overflows to infinity in floating point, turning the whole distribution into NaN. The fix uses the shift invariance we noticed earlier : subtracting a constant from every logit changes nothing mathematically, so subtract the largest one.
Now the biggest exponent is $e^{0} = 1$ and every other is smaller, so nothing can overflow. This is the log-sum-exp trick, and it is why the loss is computed in log space directly :
\[L = -\log p_c = -(z_c - m) + \log\!\sum_k e^{z_k - m}.\]It is also why every framework offers a fused operation that takes the raw logits, never the probabilities : torch.nn.CrossEntropyLoss and tf.nn.softmax_cross_entropy_with_logits both apply the softmax internally. Computing softmax yourself and then feeding it to a separate log is the single most common beginner bug in this corner of deep learning : it throws away the numerical stability and the clean $p-y$ gradient of $\eqref{eq:grad}$.
import numpy as np
def softmax(z):
z = z - z.max() # subtract max : no overflow
e = np.exp(z)
return e / e.sum()
def cross_entropy(z, c): # c = index of the true class
m = z.max()
return -(z[c] - m) + np.log(np.exp(z - m).sum())
z = np.array([2.0, 1.0, 0.1])
p = softmax(z) # [0.659, 0.242, 0.099]
loss = cross_entropy(z, 1) # 1.417 (truth = dog)
grad = p - np.array([0., 1., 0.]) # [0.659, -0.758, 0.099]
Nine lines, and they are the last two boxes of nearly every classifier in production.
7. Conclusion
Softmax and cross-entropy are a matched pair, not two independent choices. Softmax answers “how do I turn arbitrary scores into a distribution ?” with the only construction that is positive, order-preserving and depends solely on score differences ; cross-entropy answers “how wrong is that distribution ?” with the information-theoretic surprise of the true answer, which is the same thing as the KL divergence to the truth. Chained together, the exponential and the logarithm cancel and leave the cleanest gradient in machine learning, $p - y$ : push down what you predicted, push up what was true. Hand that to backpropagation, let gradient descent take a small step, and repeat. 1
The name softmax itself dates back to Bridle (Bridle, 1990), who introduced it precisely so that a classifier’s outputs could be read as probabilities and trained by maximum likelihood — which, once you take the logarithm and flip the sign, is exactly the cross-entropy of this post. Goodfellow, Bengio and Courville (Goodfellow et al., 2016) give the full treatment.
References
- Bridle, J. S. (1990). Probabilistic Interpretation of Feedforward Classification Network Outputs, with Relationships to Statistical Pattern Recognition. Neurocomputing: Algorithms, Architectures and Applications, 68, 227–236.
@inproceedings{Bridle1990, author = {Bridle, John S.}, title = {Probabilistic Interpretation of Feedforward Classification Network Outputs, with Relationships to Statistical Pattern Recognition}, booktitle = {Neurocomputing: Algorithms, Architectures and Applications}, series = {NATO ASI Series}, volume = {68}, pages = {227--236}, year = {1990}, publisher = {Springer} } - Goodfellow, I., Bengio, Y., & Courville, A. (2016). Deep Learning. MIT Press.
@book{Goodfellow2016, author = {Goodfellow, Ian and Bengio, Yoshua and Courville, Aaron}, title = {Deep Learning}, year = {2016}, publisher = {MIT Press} } - Kullback, S., & Leibler, R. A. (1951). On Information and Sufficiency. The Annals of Mathematical Statistics, 22(1), 79–86.
@article{Kullback1951, author = {Kullback, Solomon and Leibler, Richard A.}, title = {On Information and Sufficiency}, journal = {The Annals of Mathematical Statistics}, volume = {22}, number = {1}, pages = {79--86}, year = {1951} } - Shannon, C. E. (1948). A Mathematical Theory of Communication. The Bell System Technical Journal, 27(3), 379–423.
@article{Shannon1948, author = {Shannon, Claude E.}, title = {A Mathematical Theory of Communication}, journal = {The Bell System Technical Journal}, volume = {27}, number = {3}, pages = {379--423}, year = {1948} }
-
The same pairing shows up under different names all over statistics and machine learning : with $K=2$ it is logistic regression trained by log-loss, which is also the default objective of the gradient boosting machinery, and in general it is nothing but maximum likelihood estimation for a categorical distribution. ↩