Ndtyjky

I'm following the TensorFlow basic classification example with the Keras API provided in the "Getting Started" docs. I get through the tutorial as-is just fine, but if I change the loss function from sparse_categorical_crossentropy to categorical_crossentropy, the code below:

model = keras.Sequential([

    keras.layers.Flatten(input_shape=(28, 28)),

    keras.layers.Dense(128, activation=tf.nn.relu),

    keras.layers.Dense(10, activation=tf.nn.softmax)

])



model.compile(optimizer=tf.train.AdamOptimizer(), 

          loss='categorical_crossentropy',

          metrics=['accuracy'])



model.fit(train_images, train_labels, epochs=5)

fails during the training/fitting step with the following error:

ValueError: Error when checking target: expected dense_1 to have shape (10,) but got array with shape (1,)

The documentation on the loss functions doesn't delve much into expected input and output. Obviously there is a dimensionality issue here, but if any experts can give a detailed explanation, what is it about this loss function or any other loss function that raises this ValueError?

sparse_categorical_crossentropy loss expects the provided labels to be integers like 0, 1, 2 and so on, where each integer indicates a particular class. For example class 0 might be dogs, class 1 might be cats and class 2 might be lions. On the other hand, categorical_crossentropy loss takes one-hot encoded labels such as [1,0,0], [0,1,0], [0,0,1] and they are interpreted such that the index of 1 indicates the class of the sample. For example [0,0,1] means this sample belongs to class 2 (i.e. lions). Further, in the context of classification models, since the output is usually a probability distribution produced by the output of softmax layer, this form of labels also corresponds to a probability distribution and match with the output of the model. Again, [0,0,1] means that with probability of one we know that this sample belongs to class two.

sparse_categorical_crossentropy is almost a convenient way to use categorical_crossentropy as the loss function where Keras (or its backend) would handle the integer labels internally and you don't need to manually convert labels to one-hot encoded form. However, if the labels you provide are one-hot encoded then you must use categorical_crossentropy as the loss function.

Also you might be interested to look at this answer as well, where I have explained briefly about the activation and loss functions and the format of labels used in the context of different kinds of classification tasks.