Why is the cross-entropy always more than the entropy?

We are relying heavily on the fact that the cross-entropy with a wrong model. It is fairly easy to show that this is so

Solution:

We are only interested in showing that this number is negative, not in its absolute value, so we can drop the irrelevant factor of $\log 2$and then use the fairly well-known fact that to substitute into 9.2. This gives

$\diamondsuit$

  

Figure 9.2: Plot of x-1 and log(x)