MNIST Using Keras and Convolutional Neural Networks

In a previous notebook, MNIST Using Keras, we built a fully-connected (dense) shallow neural network (one hidden layer with 512 nodes) to classify images of digits in the MNIST data set.

In this notebook, we will tackle the same problem but instead of using a wide and shallow network we will build a narrow and slightly deeper network (five hidden layers). This network should achieve better accuracy in classifying MNIST digits since it will use the structure present in images through the use of convolution layers.

In the shallow model we used in the other notebook, the 28x28 pixel images where flattened into vectors of 784 components. Thus any shape digits have is lost. In contrast, a convolutional neural network processes images as they are, retaining and extracting salient features like edges and curves from the patterns present in each image.

For a detailed explanation of convolutional neural networks I suggest reading the Convolutional Neural Networks (CNNs / ConvNets) section of Stanford’s CS231n Convolutional Neural Networks for Visual Recognition course.

So let us start by importing all the required libraries to load the MNIST data set, visualise some digits and finally build and evaluate our convolutional neural network model.

Importing Required Libraries

# required to visualise sample MNIST digits
import matplotlib.pyplot as plt

# seed random number generator to improve reproducibility
import numpy as np
np.random.seed(22)

# the MNIST data set
from keras.datasets import mnist
# feed-forward neural network
from keras.models import Sequential
# required for dense final layers
from keras.layers import Dense, Flatten
# required for convolutional layers and max pooling
from keras.layers.convolutional import Conv2D, MaxPooling2D
# required to vectorize digit labels into one-hot encoding
from keras.utils import to_categorical

Using TensorFlow backend.

Loading the MNIST Data Set

Each digit in the MNIST data set is a monochrome 28 by 28 pixels image. The training set consists of 60,000 images and the testing set of 10,000 images. Each image in the training and testing set has a corresponding label provided, indicating the true value of the digit in the image.

(train_images, train_labels), (test_images, test_labels) = mnist.load_data()

print('Training Data')
print('{0} images {1}x{2} pixels'.format(train_images.shape[0],
                                         train_images.shape[1],
                                         train_images.shape[2]))
print("First ten labels: {0} -> {1}".format(train_labels[:10], 
                                            type(train_labels[0])))

print('Testing Data')
print('{0} images {1}x{2} pixels'.format(test_images.shape[0],
                                         test_images.shape[1],
                                         test_images.shape[2]))
print("First ten labels: {0} -> {1}".format(test_labels[:10], 
                                            type(test_labels[0])))

Downloading data from https://s3.amazonaws.com/img-datasets/mnist.npz
11493376/11490434 [==============================] - 2s 0us/step
Training Data
60000 images 28x28 pixels
First ten labels: [5 0 4 1 9 2 1 3 1 4] -> <class 'numpy.uint8'>
Testing Data
10000 images 28x28 pixels
First ten labels: [7 2 1 0 4 1 4 9 5 9] -> <class 'numpy.uint8'>

Displaying Some Random Samples from Training Digits

num_plot_digits = 5
digits_to_plot = np.random.randint(0, 60000, num_plot_digits)
fig, axes = plt.subplots(1, 5, figsize=(12,2))

for i in range(num_plot_digits):
    axes[i].imshow(train_images[digits_to_plot[i]], cmap=plt.cm.binary)
    axes[i].set_title(train_labels[digits_to_plot[i]])
    axes[i].set_xticks([])
    axes[i].set_yticks([])

Convolutional Neural Network Architecture

Now that we have imported the required libraries and loaded and visualised the MNIST data set, we are ready to define the architecture of our convolutional neural network.

Convolutional neural networks typically follow this pattern. The input data, in this case 28x28 pixel monochrome MNIST digit images, are inputted as is to a first convolutional layer. The convolutional layer, whose activation is typically set to ReLU, i.e. max(0,x), is followed by a pooling layer, typically max pooling.

For both convolution layer and pooling layer we need to specify the kernel (patch) size and the stride with which they move across the image. In general, the stride for convolution layers is set to (1,1), i.e. move 1 pixel at a time both when moving horizontally and vertically. This is done so that we do not downsample too aggressively. Downsampling is then achieved through the pooling layer, whose stride is commonly set to (2,2).

One final parameter to specify for the convolution layer is how many filters you want to apply. All filters will have the same kernel size and stride, but each one of them will capture different patterns as they move across the input images. You might think of each filter as encoding some semantic quality. In general, as the spatial dimensions of the input image get smaller through pooling layers, we increase the number of filters to add semantic depth.

Finally, after one or more convolution and pooling pairs, we flatten the outputs to feed them into one or more fully-connected (dense) layers, like in a classical neural network. These dense layers are then connected to the output layer that typically uses a softmax activation function to assign probabilities to each categorical class.

Our Convolutional Neural Network

In this notebook, we will build a network consisting of five hidden layers, two convolution and pooling layer pairs, followed by one dense layer. Finally, a ten class softmax output layer to classify the input images. Before we do that however, let us prepare the input data and labels to be compatible with the network.

Reshaping and Normalizing the Input Images

Since we are using TensorFlow as a backend to Keras, the images need to be reshaped to a channels-last convention, i.e. from [samples][height][width] to [samples][height][width][color_depth] format. Furthermore, it always makes sense to normalize input data, since this makes learning faster.

train_images = train_images.reshape(train_images.shape[0], 28, 28, 1).astype('float32')
test_images = test_images.reshape(test_images.shape[0], 28, 28, 1).astype('float32')

train_images = train_images / 255
test_images = test_images / 255

One-hot Encoding the Image Labels

The neural network output layer consists of ten nodes using the softmax activation function. Therefore, to be able to compare the network output with our training and testing labels we need to convert integer values into one-hot encodings. So, for example, 0 becomes 1000000000, 1 becomes 0100000000, and so on all the way to 9 becoming 0000000001.

This will make it possible to use categorical_crossentropy as our loss (objective) function, which in turn will be optimized using adam.

train_labels = to_categorical(train_labels)
test_labels = to_categorical(test_labels)

Creating the Convolutional Neural Network

Now that we have prepared the data for training and testing and have a clear idea of the type of network we need, we can go ahead and define it using Keras models and layers.

model = Sequential()

# First convolutional layer
model.add(Conv2D(4, (5, 5), strides=(1, 1), 
                 activation='relu', 
                 input_shape=(28, 28, 1)))
model.add(MaxPooling2D(pool_size=(2, 2), strides=(2, 2)))

# Second convolutional layer
model.add(Conv2D(8, (3, 3), strides=(1, 1), 
                 activation='relu'))
model.add(MaxPooling2D(pool_size=(2, 2), strides=(2, 2)))

# Dense layer
model.add(Flatten())
model.add(Dense(50, activation='relu'))

# Output layer
model.add(Dense(10, activation='softmax'))

model.compile(optimizer='adam', 
              loss='categorical_crossentropy',  
              metrics=['accuracy'])

Fitting the Model

All that is left to do now is to fit the convolutional neural network model using the MNIST training data set.

# Apart from the training accuracy we want to see how the model
# performs against a validation set made up of the last 6000 
# images in the training data set.
history = model.fit(train_images, train_labels, 
                    epochs=10, 
                    batch_size=128, 
                    validation_split=0.1,
                    verbose=0)

Plot Loss and Accuracy on Both Training and Validation Sets

fig, axes = plt.subplots(1,2, figsize=(14,5))
axes[0].plot(history.history['loss'])
axes[0].plot(history.history['val_loss'])
axes[0].set_title('Model Loss')
axes[0].set_xlabel('Epoch')
axes[0].set_ylabel('Loss')
axes[0].legend(['Training', 'Validation'])
axes[1].plot(history.history['acc'])
axes[1].plot(history.history['val_acc'])
axes[1].set_title('Model Accuracy')
axes[1].set_xlabel('Epoch')
axes[1].set_ylabel('Accuracy')
axes[1].legend(['Training', 'Validation']);

From the above charts it seems like the model could learn some more, since the model accuracy is improving on both the training and validation data sets. This suggests that the model is not yet over-fitting the training data. For the time being let us evaluate how this model performs on the testing data set.

Evaluating Model Performance on MNIST Testing Data Set

scores = model.evaluate(test_images, test_labels, verbose=0)
print("Accuracy: {} Error: {}".format(scores[1], (1-scores[1])*100))

Accuracy: 0.9839 Error: 1.6100000000000003

The accuracy on the testing data set using this convolutional neural network model is 98.39%. A slightly improved score when compared to the 98.06% achieved by the dense shallow network we trained in the MNIST Using Keras notebook.

Searching to Find Better Hyperparameters

The network we trained so far was configured with sensible settings. However, to find better settings so as to increase the model’s classification accuracy, we need to try various models, altering one setting at a time. These settings are called hyperparameters and in this example some of these are:

• number of convolutional layers
• number of filters in each convolutional layer
• convolution kernel size
• convolution and pooling stride
• activation functions to use
• choice between max or average pooling
• number of dense layers
• number of nodes in each dense layer

As you can see, the number of combinations possible with a few options for each of these settings gets big really fast. Fitting and evaulating each model on a validation set is therefore a time consuming process.

In practice the various models can be tested in parallel, depending on the computing resources available. Another method to tune many hyperparameters is to randomly sample from each setting instead of trying all possible combinations exhaustively.

In this notebook, however, we will limit the hyperparameters we will tweak so that we can find improved hyperparameters in a reasonable time. However, the concepts we will go through in the next section are still valid whether you are fine tuning a few or many hyperparameters.

Grid Search Hyperparameters of a Keras Model

We will now optimize the following hyperparameters using an exhaustive grid search.

• number of convolution layers
• number of dense layers
• number of epochs
• batch size

To do this we will create a function that creates the desired network and use the KerasClassifier wrapper so that we can employ the GridSearchCV functionality of Scikit-learn.

from keras.wrappers.scikit_learn import KerasClassifier
from sklearn.model_selection import GridSearchCV

def create_mnist_cnn_model(conv_layers=[(4,5,2,2),(8,3,2,2)], 
                           dense_layers=[50]):
  '''
    conv_layers:  list of tuples - each tuple represents a convolution layer
                  (filters, kernel_size, pooling_size, pool_stride)
                  Kernel stride is always set to (1,1)
                 
    dense_layers: list of dense layers - each number represent nodes in a dense
                  layer, e.g., [50]
                  
    The activation function for both convolution and dense layers is always
    set to ReLU.
    
    The input shape is always set to 28,28,1 since this function is for 
    MNIST digits.
    
    The output layer is always set to a dense ten node layer using softmax 
    function, since this function is building a model to classify MNIST digits.
  '''
  model = Sequential()
  
  # add convolution layers
  
  conv_layers_count = 0
  
  for (filters, kernel_size, pooling_size, pool_stride) in conv_layers:
    conv_layers_count += 1
    if 1 == conv_layers_count:
      # First convolutional layer
      model.add(Conv2D(filters, 
                       (kernel_size, kernel_size), 
                       strides=(1, 1), 
                       activation='relu', 
                       input_shape=(28, 28, 1)))
      model.add(MaxPooling2D(pool_size=(pooling_size, pooling_size), 
                             strides=(pool_stride, pool_stride)))
    else:
      # Subsequent convolutional layer
      model.add(Conv2D(filters, 
                       (kernel_size, kernel_size), 
                       strides=(1, 1), 
                       activation='relu'))
      model.add(MaxPooling2D(pool_size=(pooling_size, pooling_size), 
                             strides=(pool_stride, pool_stride)))
  
  # add dense layers
  
  model.add(Flatten())
  
  for nodes in dense_layers:
    model.add(Dense(nodes, activation='relu'))

  # Output layer
  model.add(Dense(10, activation='softmax'))

  model.compile(optimizer='adam', 
                loss='categorical_crossentropy',  
                metrics=['accuracy'])
  
  return model

# Keras model is wrapped so that we can use it with Scikit-learn GridSearchCV
mnist_cnn_model = KerasClassifier(build_fn=create_mnist_cnn_model, verbose=1)

# Specify the hyperparameters we want to test
# 2x2x3 = 12 combinations
hyperparameters = {
  #'conv_layers': [[(4,5,2,2),(8,3,2,2)],
  #                [(8,5,2,2),(16,3,2,2)]],
  'dense_layers': [[50],
                   [50,20]],
  'epochs':[10, 20],
  'batch_size':[64, 128, 256]
}

grid = GridSearchCV(estimator=mnist_cnn_model, 
                    param_grid=hyperparameters,
                    cv=3)
grid_result = grid.fit(train_images, train_labels)

Epoch 1/10
40000/40000 [==============================] - 6s 138us/step - loss: 0.4305 - acc: 0.8707
Epoch 2/10
40000/40000 [==============================] - 5s 123us/step - loss: 0.1390 - acc: 0.9574
Epoch 3/10
40000/40000 [==============================] - 5s 120us/step - loss: 0.0999 - acc: 0.9687
Epoch 4/10
40000/40000 [==============================] - 5s 120us/step - loss: 0.0808 - acc: 0.9739
Epoch 5/10
40000/40000 [==============================] - 5s 121us/step - loss: 0.0695 - acc: 0.9764
Epoch 6/10
40000/40000 [==============================] - 5s 121us/step - loss: 0.0602 - acc: 0.9804
Epoch 7/10
40000/40000 [==============================] - 5s 120us/step - loss: 0.0541 - acc: 0.9821
Epoch 8/10
40000/40000 [==============================] - 5s 120us/step - loss: 0.0490 - acc: 0.9841
Epoch 9/10
40000/40000 [==============================] - 5s 120us/step - loss: 0.0449 - acc: 0.9852
Epoch 10/10
40000/40000 [==============================] - 5s 119us/step - loss: 0.0410 - acc: 0.9864
20000/20000 [==============================] - 1s 53us/step
40000/40000 [==============================] - 2s 43us/step
Epoch 1/10
40000/40000 [==============================] - 5s 134us/step - loss: 0.4537 - acc: 0.8638
Epoch 2/10
40000/40000 [==============================] - 5s 120us/step - loss: 0.1319 - acc: 0.9603
Epoch 3/10
40000/40000 [==============================] - 5s 121us/step - loss: 0.0929 - acc: 0.9711
Epoch 4/10
40000/40000 [==============================] - 5s 121us/step - loss: 0.0779 - acc: 0.9757
Epoch 5/10
40000/40000 [==============================] - 5s 121us/step - loss: 0.0661 - acc: 0.9791
Epoch 6/10
40000/40000 [==============================] - 5s 119us/step - loss: 0.0578 - acc: 0.9816
Epoch 7/10
40000/40000 [==============================] - 5s 120us/step - loss: 0.0508 - acc: 0.9836
Epoch 8/10
40000/40000 [==============================] - 5s 121us/step - loss: 0.0471 - acc: 0.9850
Epoch 9/10
40000/40000 [==============================] - 5s 120us/step - loss: 0.0413 - acc: 0.9864
Epoch 10/10
40000/40000 [==============================] - 5s 121us/step - loss: 0.0378 - acc: 0.9879
20000/20000 [==============================] - 1s 56us/step
40000/40000 [==============================] - 2s 43us/step
Epoch 1/10
40000/40000 [==============================] - 6s 138us/step - loss: 0.4842 - acc: 0.8539
Epoch 2/10
40000/40000 [==============================] - 5s 121us/step - loss: 0.1428 - acc: 0.9567
Epoch 3/10
40000/40000 [==============================] - 5s 121us/step - loss: 0.1052 - acc: 0.9676
Epoch 4/10
40000/40000 [==============================] - 5s 121us/step - loss: 0.0864 - acc: 0.9728
Epoch 5/10
40000/40000 [==============================] - 5s 120us/step - loss: 0.0757 - acc: 0.9765
Epoch 6/10
40000/40000 [==============================] - 5s 120us/step - loss: 0.0669 - acc: 0.9791
Epoch 7/10
40000/40000 [==============================] - 5s 120us/step - loss: 0.0599 - acc: 0.9812
Epoch 8/10
40000/40000 [==============================] - 5s 120us/step - loss: 0.0548 - acc: 0.9829
Epoch 9/10
40000/40000 [==============================] - 5s 120us/step - loss: 0.0496 - acc: 0.9845
Epoch 10/10
40000/40000 [==============================] - 5s 120us/step - loss: 0.0460 - acc: 0.9854
20000/20000 [==============================] - 1s 56us/step
40000/40000 [==============================] - 2s 44us/step
Epoch 1/20
40000/40000 [==============================] - 5s 137us/step - loss: 0.4658 - acc: 0.8611
Epoch 2/20
40000/40000 [==============================] - 5s 122us/step - loss: 0.1557 - acc: 0.9529
Epoch 3/20
40000/40000 [==============================] - 5s 121us/step - loss: 0.1140 - acc: 0.9661
Epoch 4/20
40000/40000 [==============================] - 5s 121us/step - loss: 0.0923 - acc: 0.9725
Epoch 5/20
40000/40000 [==============================] - 5s 121us/step - loss: 0.0803 - acc: 0.9752
Epoch 6/20
40000/40000 [==============================] - 5s 120us/step - loss: 0.0717 - acc: 0.9779
Epoch 7/20
40000/40000 [==============================] - 5s 119us/step - loss: 0.0648 - acc: 0.9795
Epoch 8/20
40000/40000 [==============================] - 5s 120us/step - loss: 0.0574 - acc: 0.9818
Epoch 9/20
40000/40000 [==============================] - 5s 119us/step - loss: 0.0541 - acc: 0.9836
Epoch 10/20
40000/40000 [==============================] - 5s 120us/step - loss: 0.0488 - acc: 0.9842
Epoch 11/20
40000/40000 [==============================] - 5s 121us/step - loss: 0.0451 - acc: 0.9851
Epoch 12/20
40000/40000 [==============================] - 5s 121us/step - loss: 0.0418 - acc: 0.9865
Epoch 13/20
40000/40000 [==============================] - 5s 120us/step - loss: 0.0393 - acc: 0.9872
Epoch 14/20
40000/40000 [==============================] - 5s 120us/step - loss: 0.0362 - acc: 0.9884
Epoch 15/20
40000/40000 [==============================] - 5s 120us/step - loss: 0.0346 - acc: 0.9887
Epoch 16/20
40000/40000 [==============================] - 5s 119us/step - loss: 0.0318 - acc: 0.9901
Epoch 17/20
40000/40000 [==============================] - 5s 118us/step - loss: 0.0290 - acc: 0.9909
Epoch 18/20
40000/40000 [==============================] - 5s 119us/step - loss: 0.0285 - acc: 0.9906
Epoch 19/20
40000/40000 [==============================] - 5s 124us/step - loss: 0.0267 - acc: 0.9911
Epoch 20/20
40000/40000 [==============================] - 5s 120us/step - loss: 0.0236 - acc: 0.9924
20000/20000 [==============================] - 1s 57us/step
40000/40000 [==============================] - 2s 44us/step
Epoch 1/20
40000/40000 [==============================] - 5s 136us/step - loss: 0.4247 - acc: 0.8761
Epoch 2/20
40000/40000 [==============================] - 5s 120us/step - loss: 0.1338 - acc: 0.9602
Epoch 3/20
40000/40000 [==============================] - 5s 121us/step - loss: 0.0970 - acc: 0.9707
Epoch 4/20
40000/40000 [==============================] - 5s 121us/step - loss: 0.0783 - acc: 0.9763
Epoch 5/20
40000/40000 [==============================] - 5s 120us/step - loss: 0.0667 - acc: 0.9791
Epoch 6/20
40000/40000 [==============================] - 5s 120us/step - loss: 0.0581 - acc: 0.9824
Epoch 7/20
40000/40000 [==============================] - 5s 120us/step - loss: 0.0531 - acc: 0.9838
Epoch 8/20
40000/40000 [==============================] - 5s 120us/step - loss: 0.0470 - acc: 0.9855
Epoch 9/20
40000/40000 [==============================] - 5s 120us/step - loss: 0.0428 - acc: 0.9865
Epoch 10/20
40000/40000 [==============================] - 5s 125us/step - loss: 0.0388 - acc: 0.9878
Epoch 11/20
40000/40000 [==============================] - 5s 125us/step - loss: 0.0362 - acc: 0.9888
Epoch 12/20
40000/40000 [==============================] - 5s 126us/step - loss: 0.0346 - acc: 0.9887
Epoch 13/20
40000/40000 [==============================] - 5s 123us/step - loss: 0.0312 - acc: 0.9896
Epoch 14/20
40000/40000 [==============================] - 5s 121us/step - loss: 0.0272 - acc: 0.9910
Epoch 15/20
40000/40000 [==============================] - 5s 122us/step - loss: 0.0262 - acc: 0.9913
Epoch 16/20
40000/40000 [==============================] - 5s 121us/step - loss: 0.0238 - acc: 0.9924
Epoch 17/20
40000/40000 [==============================] - 5s 121us/step - loss: 0.0219 - acc: 0.9926
Epoch 18/20
40000/40000 [==============================] - 5s 122us/step - loss: 0.0199 - acc: 0.9935
Epoch 19/20
40000/40000 [==============================] - 5s 122us/step - loss: 0.0196 - acc: 0.9935
Epoch 20/20
40000/40000 [==============================] - 5s 124us/step - loss: 0.0188 - acc: 0.9937
20000/20000 [==============================] - 1s 56us/step
40000/40000 [==============================] - 2s 45us/step
Epoch 1/20
40000/40000 [==============================] - 6s 142us/step - loss: 0.4400 - acc: 0.8668
Epoch 2/20
40000/40000 [==============================] - 5s 123us/step - loss: 0.1321 - acc: 0.9589
Epoch 3/20
40000/40000 [==============================] - 5s 122us/step - loss: 0.0991 - acc: 0.9687
Epoch 4/20
40000/40000 [==============================] - 5s 123us/step - loss: 0.0828 - acc: 0.9744
Epoch 5/20
40000/40000 [==============================] - 5s 122us/step - loss: 0.0735 - acc: 0.9764
Epoch 6/20
40000/40000 [==============================] - 5s 123us/step - loss: 0.0620 - acc: 0.9805
Epoch 7/20
40000/40000 [==============================] - 5s 123us/step - loss: 0.0550 - acc: 0.9828
Epoch 8/20
40000/40000 [==============================] - 5s 123us/step - loss: 0.0514 - acc: 0.9835
Epoch 9/20
40000/40000 [==============================] - 5s 124us/step - loss: 0.0448 - acc: 0.9856
Epoch 10/20
40000/40000 [==============================] - 5s 124us/step - loss: 0.0398 - acc: 0.9870
Epoch 11/20
40000/40000 [==============================] - 5s 124us/step - loss: 0.0359 - acc: 0.9887
Epoch 12/20
40000/40000 [==============================] - 5s 124us/step - loss: 0.0347 - acc: 0.9885
Epoch 13/20
40000/40000 [==============================] - 5s 123us/step - loss: 0.0315 - acc: 0.9896
Epoch 14/20
40000/40000 [==============================] - 5s 122us/step - loss: 0.0265 - acc: 0.9915
Epoch 15/20
40000/40000 [==============================] - 5s 123us/step - loss: 0.0266 - acc: 0.9908
Epoch 16/20
40000/40000 [==============================] - 5s 122us/step - loss: 0.0236 - acc: 0.9919
Epoch 17/20
40000/40000 [==============================] - 5s 121us/step - loss: 0.0216 - acc: 0.9927
Epoch 18/20
40000/40000 [==============================] - 5s 122us/step - loss: 0.0199 - acc: 0.9936
Epoch 19/20
40000/40000 [==============================] - 5s 124us/step - loss: 0.0184 - acc: 0.9938
Epoch 20/20
40000/40000 [==============================] - 5s 122us/step - loss: 0.0169 - acc: 0.9941
20000/20000 [==============================] - 1s 59us/step
40000/40000 [==============================] - 2s 44us/step
Epoch 1/10
40000/40000 [==============================] - 7s 163us/step - loss: 0.4931 - acc: 0.8461
Epoch 2/10
40000/40000 [==============================] - 5s 133us/step - loss: 0.1324 - acc: 0.9610
Epoch 3/10
40000/40000 [==============================] - 5s 132us/step - loss: 0.1009 - acc: 0.9690
Epoch 4/10
40000/40000 [==============================] - 5s 134us/step - loss: 0.0829 - acc: 0.9743
Epoch 5/10
40000/40000 [==============================] - 5s 133us/step - loss: 0.0725 - acc: 0.9774
Epoch 6/10
40000/40000 [==============================] - 5s 132us/step - loss: 0.0631 - acc: 0.9808
Epoch 7/10
40000/40000 [==============================] - 5s 133us/step - loss: 0.0567 - acc: 0.9824
Epoch 8/10
40000/40000 [==============================] - 5s 131us/step - loss: 0.0519 - acc: 0.9840
Epoch 9/10
40000/40000 [==============================] - 5s 132us/step - loss: 0.0476 - acc: 0.9851
Epoch 10/10
40000/40000 [==============================] - 5s 132us/step - loss: 0.0444 - acc: 0.9864
20000/20000 [==============================] - 1s 59us/step
40000/40000 [==============================] - 2s 45us/step
Epoch 1/10
40000/40000 [==============================] - 6s 155us/step - loss: 0.5471 - acc: 0.8319
Epoch 2/10
40000/40000 [==============================] - 5s 132us/step - loss: 0.1706 - acc: 0.9489
Epoch 3/10
40000/40000 [==============================] - 5s 133us/step - loss: 0.1204 - acc: 0.9632
Epoch 4/10
40000/40000 [==============================] - 5s 132us/step - loss: 0.0982 - acc: 0.9701
Epoch 5/10
40000/40000 [==============================] - 5s 132us/step - loss: 0.0851 - acc: 0.9741
Epoch 6/10
40000/40000 [==============================] - 5s 134us/step - loss: 0.0748 - acc: 0.9765
Epoch 7/10
40000/40000 [==============================] - 5s 134us/step - loss: 0.0662 - acc: 0.9797
Epoch 8/10
40000/40000 [==============================] - 5s 132us/step - loss: 0.0588 - acc: 0.9818
Epoch 9/10
40000/40000 [==============================] - 5s 133us/step - loss: 0.0522 - acc: 0.9833
Epoch 10/10
40000/40000 [==============================] - 5s 133us/step - loss: 0.0483 - acc: 0.9847
20000/20000 [==============================] - 1s 65us/step
40000/40000 [==============================] - 2s 45us/step
Epoch 1/10
40000/40000 [==============================] - 6s 159us/step - loss: 0.4408 - acc: 0.8639
Epoch 2/10
40000/40000 [==============================] - 5s 131us/step - loss: 0.1394 - acc: 0.9587
Epoch 3/10
40000/40000 [==============================] - 5s 131us/step - loss: 0.1037 - acc: 0.9688
Epoch 4/10
40000/40000 [==============================] - 5s 131us/step - loss: 0.0828 - acc: 0.9753
Epoch 5/10
40000/40000 [==============================] - 5s 131us/step - loss: 0.0717 - acc: 0.9776
Epoch 6/10
40000/40000 [==============================] - 5s 133us/step - loss: 0.0610 - acc: 0.9817
Epoch 7/10
40000/40000 [==============================] - 5s 135us/step - loss: 0.0576 - acc: 0.9814
Epoch 8/10
40000/40000 [==============================] - 6s 138us/step - loss: 0.0503 - acc: 0.9838
Epoch 9/10
40000/40000 [==============================] - 6s 139us/step - loss: 0.0457 - acc: 0.9855
Epoch 10/10
40000/40000 [==============================] - 5s 137us/step - loss: 0.0393 - acc: 0.9869
20000/20000 [==============================] - 1s 62us/step
40000/40000 [==============================] - 2s 46us/step
Epoch 1/20
40000/40000 [==============================] - 6s 159us/step - loss: 0.4794 - acc: 0.8472
Epoch 2/20
40000/40000 [==============================] - 5s 135us/step - loss: 0.1369 - acc: 0.9586
Epoch 3/20
40000/40000 [==============================] - 5s 132us/step - loss: 0.1011 - acc: 0.9686
Epoch 4/20
40000/40000 [==============================] - 5s 132us/step - loss: 0.0832 - acc: 0.9737
Epoch 5/20
40000/40000 [==============================] - 6s 138us/step - loss: 0.0742 - acc: 0.9767
Epoch 6/20
40000/40000 [==============================] - 5s 132us/step - loss: 0.0650 - acc: 0.9792
Epoch 7/20
40000/40000 [==============================] - 5s 133us/step - loss: 0.0587 - acc: 0.9819
Epoch 8/20
40000/40000 [==============================] - 5s 133us/step - loss: 0.0530 - acc: 0.9833
Epoch 9/20
40000/40000 [==============================] - 5s 133us/step - loss: 0.0494 - acc: 0.9841
Epoch 10/20
40000/40000 [==============================] - 5s 134us/step - loss: 0.0453 - acc: 0.9856
Epoch 11/20
40000/40000 [==============================] - 5s 134us/step - loss: 0.0443 - acc: 0.9860
Epoch 12/20
40000/40000 [==============================] - 5s 133us/step - loss: 0.0387 - acc: 0.9874
Epoch 13/20
40000/40000 [==============================] - 5s 133us/step - loss: 0.0352 - acc: 0.9887
Epoch 14/20
40000/40000 [==============================] - 5s 132us/step - loss: 0.0341 - acc: 0.9892
Epoch 15/20
40000/40000 [==============================] - 5s 133us/step - loss: 0.0333 - acc: 0.9890
Epoch 16/20
40000/40000 [==============================] - 5s 133us/step - loss: 0.0280 - acc: 0.9909
Epoch 17/20
40000/40000 [==============================] - 5s 133us/step - loss: 0.0270 - acc: 0.9910
Epoch 18/20
40000/40000 [==============================] - 5s 133us/step - loss: 0.0261 - acc: 0.9915
Epoch 19/20
40000/40000 [==============================] - 5s 133us/step - loss: 0.0232 - acc: 0.9921
Epoch 20/20
40000/40000 [==============================] - 5s 132us/step - loss: 0.0192 - acc: 0.9939
20000/20000 [==============================] - 1s 65us/step
40000/40000 [==============================] - 2s 46us/step
Epoch 1/20
40000/40000 [==============================] - 6s 159us/step - loss: 0.4505 - acc: 0.8602
Epoch 2/20
40000/40000 [==============================] - 5s 132us/step - loss: 0.1463 - acc: 0.9555
Epoch 3/20
40000/40000 [==============================] - 5s 132us/step - loss: 0.1103 - acc: 0.9664
Epoch 4/20
40000/40000 [==============================] - 5s 131us/step - loss: 0.0905 - acc: 0.9730
Epoch 5/20
40000/40000 [==============================] - 5s 132us/step - loss: 0.0775 - acc: 0.9766
Epoch 6/20
40000/40000 [==============================] - 5s 131us/step - loss: 0.0698 - acc: 0.9786
Epoch 7/20
40000/40000 [==============================] - 5s 132us/step - loss: 0.0624 - acc: 0.9804
Epoch 8/20
40000/40000 [==============================] - 5s 135us/step - loss: 0.0554 - acc: 0.9827
Epoch 9/20
40000/40000 [==============================] - 5s 132us/step - loss: 0.0489 - acc: 0.9850
Epoch 10/20
40000/40000 [==============================] - 5s 133us/step - loss: 0.0467 - acc: 0.9852
Epoch 11/20
40000/40000 [==============================] - 5s 132us/step - loss: 0.0404 - acc: 0.9874
Epoch 12/20
40000/40000 [==============================] - 5s 133us/step - loss: 0.0395 - acc: 0.9870
Epoch 13/20
40000/40000 [==============================] - 5s 133us/step - loss: 0.0360 - acc: 0.9885
Epoch 14/20
40000/40000 [==============================] - 5s 132us/step - loss: 0.0318 - acc: 0.9898
Epoch 15/20
40000/40000 [==============================] - 5s 134us/step - loss: 0.0289 - acc: 0.9906
Epoch 16/20
40000/40000 [==============================] - 5s 132us/step - loss: 0.0302 - acc: 0.9902
Epoch 17/20
40000/40000 [==============================] - 5s 133us/step - loss: 0.0272 - acc: 0.9910
Epoch 18/20
40000/40000 [==============================] - 5s 132us/step - loss: 0.0244 - acc: 0.9919
Epoch 19/20
40000/40000 [==============================] - 5s 134us/step - loss: 0.0235 - acc: 0.9924
Epoch 20/20
40000/40000 [==============================] - 5s 134us/step - loss: 0.0212 - acc: 0.9928
20000/20000 [==============================] - 1s 71us/step
40000/40000 [==============================] - 2s 47us/step
Epoch 1/20
40000/40000 [==============================] - 7s 164us/step - loss: 0.4984 - acc: 0.8454
Epoch 2/20
40000/40000 [==============================] - 5s 134us/step - loss: 0.1569 - acc: 0.9531
Epoch 3/20
40000/40000 [==============================] - 5s 134us/step - loss: 0.1140 - acc: 0.9648
Epoch 4/20
40000/40000 [==============================] - 5s 133us/step - loss: 0.0954 - acc: 0.9709
Epoch 5/20
40000/40000 [==============================] - 5s 134us/step - loss: 0.0807 - acc: 0.9749
Epoch 6/20
40000/40000 [==============================] - 5s 134us/step - loss: 0.0717 - acc: 0.9774
Epoch 7/20
40000/40000 [==============================] - 5s 133us/step - loss: 0.0617 - acc: 0.9807
Epoch 8/20
40000/40000 [==============================] - 5s 133us/step - loss: 0.0564 - acc: 0.9825
Epoch 9/20
40000/40000 [==============================] - 5s 133us/step - loss: 0.0499 - acc: 0.9835
Epoch 10/20
40000/40000 [==============================] - 5s 133us/step - loss: 0.0459 - acc: 0.9855
Epoch 11/20
40000/40000 [==============================] - 5s 134us/step - loss: 0.0410 - acc: 0.9872
Epoch 12/20
40000/40000 [==============================] - 5s 135us/step - loss: 0.0388 - acc: 0.9875
Epoch 13/20
40000/40000 [==============================] - 6s 139us/step - loss: 0.0360 - acc: 0.9887
Epoch 14/20
40000/40000 [==============================] - 6s 139us/step - loss: 0.0336 - acc: 0.9886
Epoch 15/20
40000/40000 [==============================] - 6s 138us/step - loss: 0.0290 - acc: 0.9903
Epoch 16/20
40000/40000 [==============================] - 5s 133us/step - loss: 0.0292 - acc: 0.9902
Epoch 17/20
40000/40000 [==============================] - 5s 133us/step - loss: 0.0271 - acc: 0.9909
Epoch 18/20
40000/40000 [==============================] - 5s 133us/step - loss: 0.0247 - acc: 0.9919
Epoch 19/20
40000/40000 [==============================] - 5s 132us/step - loss: 0.0235 - acc: 0.9919
Epoch 20/20
40000/40000 [==============================] - 5s 132us/step - loss: 0.0233 - acc: 0.9924
20000/20000 [==============================] - 1s 72us/step
40000/40000 [==============================] - 2s 47us/step
Epoch 1/10
40000/40000 [==============================] - 4s 97us/step - loss: 0.6058 - acc: 0.8296
Epoch 2/10
40000/40000 [==============================] - 3s 64us/step - loss: 0.1844 - acc: 0.9449
Epoch 3/10
40000/40000 [==============================] - 3s 64us/step - loss: 0.1313 - acc: 0.9602
Epoch 4/10
40000/40000 [==============================] - 3s 65us/step - loss: 0.1054 - acc: 0.9683
Epoch 5/10
40000/40000 [==============================] - 3s 65us/step - loss: 0.0905 - acc: 0.9725
Epoch 6/10
40000/40000 [==============================] - 3s 64us/step - loss: 0.0813 - acc: 0.9760
Epoch 7/10
40000/40000 [==============================] - 3s 65us/step - loss: 0.0722 - acc: 0.9782
Epoch 8/10
40000/40000 [==============================] - 3s 66us/step - loss: 0.0660 - acc: 0.9786
Epoch 9/10
40000/40000 [==============================] - 3s 65us/step - loss: 0.0619 - acc: 0.9808
Epoch 10/10
40000/40000 [==============================] - 3s 65us/step - loss: 0.0560 - acc: 0.9828
20000/20000 [==============================] - 1s 53us/step
40000/40000 [==============================] - 1s 26us/step
Epoch 1/10
40000/40000 [==============================] - 4s 100us/step - loss: 0.6336 - acc: 0.8181
Epoch 2/10
40000/40000 [==============================] - 3s 66us/step - loss: 0.1593 - acc: 0.9524
Epoch 3/10
40000/40000 [==============================] - 3s 66us/step - loss: 0.1219 - acc: 0.9633
Epoch 4/10
40000/40000 [==============================] - 3s 66us/step - loss: 0.1034 - acc: 0.9688
Epoch 5/10
40000/40000 [==============================] - 3s 66us/step - loss: 0.0911 - acc: 0.9719
Epoch 6/10
40000/40000 [==============================] - 3s 65us/step - loss: 0.0864 - acc: 0.9729
Epoch 7/10
40000/40000 [==============================] - 3s 65us/step - loss: 0.0767 - acc: 0.9763
Epoch 8/10
40000/40000 [==============================] - 3s 66us/step - loss: 0.0716 - acc: 0.9776
Epoch 9/10
40000/40000 [==============================] - 3s 66us/step - loss: 0.0646 - acc: 0.9800
Epoch 10/10
40000/40000 [==============================] - 3s 66us/step - loss: 0.0614 - acc: 0.9807
20000/20000 [==============================] - 1s 50us/step
40000/40000 [==============================] - 1s 26us/step
Epoch 1/10
40000/40000 [==============================] - 4s 102us/step - loss: 0.7437 - acc: 0.7769
Epoch 2/10
40000/40000 [==============================] - 3s 66us/step - loss: 0.2403 - acc: 0.9283
Epoch 3/10
40000/40000 [==============================] - 3s 65us/step - loss: 0.1638 - acc: 0.9520
Epoch 4/10
40000/40000 [==============================] - 3s 66us/step - loss: 0.1239 - acc: 0.9624
Epoch 5/10
40000/40000 [==============================] - 3s 66us/step - loss: 0.1004 - acc: 0.9700
Epoch 6/10
40000/40000 [==============================] - 3s 66us/step - loss: 0.0874 - acc: 0.9731
Epoch 7/10
40000/40000 [==============================] - 3s 65us/step - loss: 0.0756 - acc: 0.9765
Epoch 8/10
40000/40000 [==============================] - 3s 66us/step - loss: 0.0673 - acc: 0.9796
Epoch 9/10
40000/40000 [==============================] - 3s 65us/step - loss: 0.0621 - acc: 0.9811
Epoch 10/10
40000/40000 [==============================] - 3s 66us/step - loss: 0.0550 - acc: 0.9828
20000/20000 [==============================] - 1s 60us/step
40000/40000 [==============================] - 1s 27us/step
Epoch 1/20
40000/40000 [==============================] - 4s 104us/step - loss: 0.6523 - acc: 0.8054
Epoch 2/20
40000/40000 [==============================] - 3s 64us/step - loss: 0.1972 - acc: 0.9400
Epoch 3/20
40000/40000 [==============================] - 3s 64us/step - loss: 0.1375 - acc: 0.9578
Epoch 4/20
40000/40000 [==============================] - 3s 65us/step - loss: 0.1064 - acc: 0.9672
Epoch 5/20
40000/40000 [==============================] - 3s 64us/step - loss: 0.0889 - acc: 0.9719
Epoch 6/20
40000/40000 [==============================] - 3s 65us/step - loss: 0.0780 - acc: 0.9758
Epoch 7/20
40000/40000 [==============================] - 3s 65us/step - loss: 0.0701 - acc: 0.9779
Epoch 8/20
40000/40000 [==============================] - 3s 65us/step - loss: 0.0618 - acc: 0.9804
Epoch 9/20
40000/40000 [==============================] - 3s 64us/step - loss: 0.0570 - acc: 0.9819
Epoch 10/20
40000/40000 [==============================] - 3s 65us/step - loss: 0.0508 - acc: 0.9841
Epoch 11/20
40000/40000 [==============================] - 3s 66us/step - loss: 0.0477 - acc: 0.9850
Epoch 12/20
40000/40000 [==============================] - 3s 65us/step - loss: 0.0431 - acc: 0.9868
Epoch 13/20
40000/40000 [==============================] - 3s 65us/step - loss: 0.0412 - acc: 0.9871
Epoch 14/20
40000/40000 [==============================] - 3s 65us/step - loss: 0.0401 - acc: 0.9871
Epoch 15/20
40000/40000 [==============================] - 3s 64us/step - loss: 0.0378 - acc: 0.9878
Epoch 16/20
40000/40000 [==============================] - 3s 65us/step - loss: 0.0333 - acc: 0.9894
Epoch 17/20
40000/40000 [==============================] - 3s 65us/step - loss: 0.0321 - acc: 0.9899
Epoch 18/20
40000/40000 [==============================] - 3s 65us/step - loss: 0.0311 - acc: 0.9904
Epoch 19/20
40000/40000 [==============================] - 3s 66us/step - loss: 0.0281 - acc: 0.9915
Epoch 20/20
40000/40000 [==============================] - 3s 65us/step - loss: 0.0270 - acc: 0.9919
20000/20000 [==============================] - 1s 53us/step
40000/40000 [==============================] - 1s 26us/step
Epoch 1/20
40000/40000 [==============================] - 4s 102us/step - loss: 0.5667 - acc: 0.8395
Epoch 2/20
40000/40000 [==============================] - 3s 64us/step - loss: 0.1535 - acc: 0.9540
Epoch 3/20
40000/40000 [==============================] - 3s 65us/step - loss: 0.1162 - acc: 0.9639
Epoch 4/20
40000/40000 [==============================] - 3s 64us/step - loss: 0.0981 - acc: 0.9707
Epoch 5/20
40000/40000 [==============================] - 3s 65us/step - loss: 0.0863 - acc: 0.9734
Epoch 6/20
40000/40000 [==============================] - 3s 65us/step - loss: 0.0772 - acc: 0.9766
Epoch 7/20
40000/40000 [==============================] - 3s 65us/step - loss: 0.0701 - acc: 0.9775
Epoch 8/20
40000/40000 [==============================] - 3s 64us/step - loss: 0.0628 - acc: 0.9806
Epoch 9/20
40000/40000 [==============================] - 3s 65us/step - loss: 0.0585 - acc: 0.9821
Epoch 10/20
40000/40000 [==============================] - 3s 65us/step - loss: 0.0524 - acc: 0.9840
Epoch 11/20
40000/40000 [==============================] - 3s 65us/step - loss: 0.0492 - acc: 0.9850
Epoch 12/20
40000/40000 [==============================] - 3s 65us/step - loss: 0.0450 - acc: 0.9860
Epoch 13/20
40000/40000 [==============================] - 3s 64us/step - loss: 0.0425 - acc: 0.9868
Epoch 14/20
40000/40000 [==============================] - 3s 65us/step - loss: 0.0390 - acc: 0.9885
Epoch 15/20
40000/40000 [==============================] - 3s 65us/step - loss: 0.0386 - acc: 0.9880
Epoch 16/20
40000/40000 [==============================] - 3s 65us/step - loss: 0.0360 - acc: 0.9890
Epoch 17/20
40000/40000 [==============================] - 3s 65us/step - loss: 0.0333 - acc: 0.9893
Epoch 18/20
40000/40000 [==============================] - 3s 65us/step - loss: 0.0318 - acc: 0.9892
Epoch 19/20
40000/40000 [==============================] - 3s 65us/step - loss: 0.0303 - acc: 0.9904
Epoch 20/20
40000/40000 [==============================] - 3s 66us/step - loss: 0.0279 - acc: 0.9910
20000/20000 [==============================] - 1s 61us/step
40000/40000 [==============================] - 1s 27us/step
Epoch 1/20
40000/40000 [==============================] - 4s 107us/step - loss: 0.6108 - acc: 0.8172
Epoch 2/20
40000/40000 [==============================] - 3s 66us/step - loss: 0.1623 - acc: 0.9517
Epoch 3/20
40000/40000 [==============================] - 3s 66us/step - loss: 0.1179 - acc: 0.9651
Epoch 4/20
40000/40000 [==============================] - 3s 66us/step - loss: 0.0990 - acc: 0.9702
Epoch 5/20
40000/40000 [==============================] - 3s 66us/step - loss: 0.0848 - acc: 0.9742
Epoch 6/20
40000/40000 [==============================] - 3s 66us/step - loss: 0.0751 - acc: 0.9771
Epoch 7/20
40000/40000 [==============================] - 3s 67us/step - loss: 0.0660 - acc: 0.9796
Epoch 8/20
40000/40000 [==============================] - 3s 66us/step - loss: 0.0600 - acc: 0.9815
Epoch 9/20
40000/40000 [==============================] - 3s 66us/step - loss: 0.0547 - acc: 0.9829
Epoch 10/20
40000/40000 [==============================] - 3s 66us/step - loss: 0.0515 - acc: 0.9845
Epoch 11/20
40000/40000 [==============================] - 3s 67us/step - loss: 0.0477 - acc: 0.9849
Epoch 12/20
40000/40000 [==============================] - 3s 66us/step - loss: 0.0449 - acc: 0.9858
Epoch 13/20
40000/40000 [==============================] - 3s 66us/step - loss: 0.0420 - acc: 0.9866
Epoch 14/20
40000/40000 [==============================] - 3s 66us/step - loss: 0.0400 - acc: 0.9875
Epoch 15/20
40000/40000 [==============================] - 3s 66us/step - loss: 0.0359 - acc: 0.9893
Epoch 16/20
40000/40000 [==============================] - 3s 66us/step - loss: 0.0343 - acc: 0.9894
Epoch 17/20
40000/40000 [==============================] - 3s 66us/step - loss: 0.0313 - acc: 0.9903
Epoch 18/20
40000/40000 [==============================] - 3s 67us/step - loss: 0.0307 - acc: 0.9905
Epoch 19/20
40000/40000 [==============================] - 3s 67us/step - loss: 0.0280 - acc: 0.9912
Epoch 20/20
40000/40000 [==============================] - 3s 66us/step - loss: 0.0259 - acc: 0.9916
20000/20000 [==============================] - 1s 59us/step
40000/40000 [==============================] - 1s 27us/step
Epoch 1/10
40000/40000 [==============================] - 5s 117us/step - loss: 0.6844 - acc: 0.7820
Epoch 2/10
40000/40000 [==============================] - 3s 73us/step - loss: 0.1664 - acc: 0.9498
Epoch 3/10
40000/40000 [==============================] - 3s 73us/step - loss: 0.1176 - acc: 0.9636
Epoch 4/10
40000/40000 [==============================] - 3s 73us/step - loss: 0.0966 - acc: 0.9700
Epoch 5/10
40000/40000 [==============================] - 3s 74us/step - loss: 0.0839 - acc: 0.9742
Epoch 6/10
40000/40000 [==============================] - 3s 71us/step - loss: 0.0729 - acc: 0.9778
Epoch 7/10
40000/40000 [==============================] - 3s 70us/step - loss: 0.0661 - acc: 0.9802
Epoch 8/10
40000/40000 [==============================] - 3s 70us/step - loss: 0.0592 - acc: 0.9819
Epoch 9/10
40000/40000 [==============================] - 3s 70us/step - loss: 0.0558 - acc: 0.9831
Epoch 10/10
40000/40000 [==============================] - 3s 70us/step - loss: 0.0499 - acc: 0.9844
20000/20000 [==============================] - 1s 60us/step
40000/40000 [==============================] - 1s 27us/step
Epoch 1/10
40000/40000 [==============================] - 5s 117us/step - loss: 0.6511 - acc: 0.8034
Epoch 2/10
40000/40000 [==============================] - 3s 71us/step - loss: 0.1944 - acc: 0.9435
Epoch 3/10
40000/40000 [==============================] - 3s 71us/step - loss: 0.1462 - acc: 0.9568
Epoch 4/10
40000/40000 [==============================] - 3s 71us/step - loss: 0.1228 - acc: 0.9640
Epoch 5/10
40000/40000 [==============================] - 3s 72us/step - loss: 0.1063 - acc: 0.9677
Epoch 6/10
40000/40000 [==============================] - 3s 72us/step - loss: 0.0951 - acc: 0.9720
Epoch 7/10
40000/40000 [==============================] - 3s 71us/step - loss: 0.0870 - acc: 0.9731
Epoch 8/10
40000/40000 [==============================] - 3s 71us/step - loss: 0.0790 - acc: 0.9759
Epoch 9/10
40000/40000 [==============================] - 3s 72us/step - loss: 0.0700 - acc: 0.9781
Epoch 10/10
40000/40000 [==============================] - 3s 71us/step - loss: 0.0675 - acc: 0.9790
20000/20000 [==============================] - 1s 63us/step
40000/40000 [==============================] - 1s 27us/step
Epoch 1/10
40000/40000 [==============================] - 5s 118us/step - loss: 0.6614 - acc: 0.7967
Epoch 2/10
40000/40000 [==============================] - 3s 71us/step - loss: 0.1856 - acc: 0.9448
Epoch 3/10
40000/40000 [==============================] - 3s 71us/step - loss: 0.1322 - acc: 0.9606
Epoch 4/10
40000/40000 [==============================] - 3s 71us/step - loss: 0.1089 - acc: 0.9674
Epoch 5/10
40000/40000 [==============================] - 3s 72us/step - loss: 0.0932 - acc: 0.9715
Epoch 6/10
40000/40000 [==============================] - 3s 71us/step - loss: 0.0836 - acc: 0.9745
Epoch 7/10
40000/40000 [==============================] - 3s 71us/step - loss: 0.0751 - acc: 0.9770
Epoch 8/10
40000/40000 [==============================] - 3s 71us/step - loss: 0.0710 - acc: 0.9781
Epoch 9/10
40000/40000 [==============================] - 3s 71us/step - loss: 0.0647 - acc: 0.9804
Epoch 10/10
40000/40000 [==============================] - 3s 71us/step - loss: 0.0595 - acc: 0.9813
20000/20000 [==============================] - 1s 63us/step
40000/40000 [==============================] - 1s 27us/step
Epoch 1/20
40000/40000 [==============================] - 5s 121us/step - loss: 0.6866 - acc: 0.7825
Epoch 2/20
40000/40000 [==============================] - 3s 71us/step - loss: 0.2362 - acc: 0.9290
Epoch 3/20
40000/40000 [==============================] - 3s 72us/step - loss: 0.1746 - acc: 0.9483
Epoch 4/20
40000/40000 [==============================] - 3s 71us/step - loss: 0.1386 - acc: 0.9582
Epoch 5/20
40000/40000 [==============================] - 3s 71us/step - loss: 0.1204 - acc: 0.9637
Epoch 6/20
40000/40000 [==============================] - 3s 71us/step - loss: 0.1050 - acc: 0.9678
Epoch 7/20
40000/40000 [==============================] - 3s 71us/step - loss: 0.0942 - acc: 0.9715
Epoch 8/20
40000/40000 [==============================] - 3s 71us/step - loss: 0.0877 - acc: 0.9723
Epoch 9/20
40000/40000 [==============================] - 3s 71us/step - loss: 0.0783 - acc: 0.9760
Epoch 10/20
40000/40000 [==============================] - 3s 71us/step - loss: 0.0704 - acc: 0.9781
Epoch 11/20
40000/40000 [==============================] - 3s 71us/step - loss: 0.0664 - acc: 0.9793
Epoch 12/20
40000/40000 [==============================] - 3s 71us/step - loss: 0.0620 - acc: 0.9806
Epoch 13/20
40000/40000 [==============================] - 3s 71us/step - loss: 0.0587 - acc: 0.9813
Epoch 14/20
40000/40000 [==============================] - 3s 71us/step - loss: 0.0548 - acc: 0.9821
Epoch 15/20
40000/40000 [==============================] - 3s 71us/step - loss: 0.0494 - acc: 0.9847
Epoch 16/20
40000/40000 [==============================] - 3s 71us/step - loss: 0.0480 - acc: 0.9850
Epoch 17/20
40000/40000 [==============================] - 3s 71us/step - loss: 0.0464 - acc: 0.9851
Epoch 18/20
40000/40000 [==============================] - 3s 72us/step - loss: 0.0428 - acc: 0.9864
Epoch 19/20
40000/40000 [==============================] - 3s 71us/step - loss: 0.0402 - acc: 0.9874
Epoch 20/20
40000/40000 [==============================] - 3s 71us/step - loss: 0.0395 - acc: 0.9870
20000/20000 [==============================] - 1s 66us/step
40000/40000 [==============================] - 1s 28us/step
Epoch 1/20
40000/40000 [==============================] - 5s 122us/step - loss: 0.6985 - acc: 0.7679
Epoch 2/20
40000/40000 [==============================] - 3s 71us/step - loss: 0.2252 - acc: 0.9311
Epoch 3/20
40000/40000 [==============================] - 3s 72us/step - loss: 0.1654 - acc: 0.9498
Epoch 4/20
40000/40000 [==============================] - 3s 71us/step - loss: 0.1334 - acc: 0.9588
Epoch 5/20
40000/40000 [==============================] - 3s 72us/step - loss: 0.1078 - acc: 0.9667
Epoch 6/20
40000/40000 [==============================] - 3s 72us/step - loss: 0.0979 - acc: 0.9699
Epoch 7/20
40000/40000 [==============================] - 3s 72us/step - loss: 0.0862 - acc: 0.9735
Epoch 8/20
40000/40000 [==============================] - 3s 71us/step - loss: 0.0794 - acc: 0.9758
Epoch 9/20
40000/40000 [==============================] - 3s 71us/step - loss: 0.0762 - acc: 0.9761
Epoch 10/20
40000/40000 [==============================] - 3s 72us/step - loss: 0.0690 - acc: 0.9784
Epoch 11/20
40000/40000 [==============================] - 3s 71us/step - loss: 0.0644 - acc: 0.9797
Epoch 12/20
40000/40000 [==============================] - 3s 71us/step - loss: 0.0598 - acc: 0.9814
Epoch 13/20
40000/40000 [==============================] - 3s 71us/step - loss: 0.0571 - acc: 0.9820
Epoch 14/20
40000/40000 [==============================] - 3s 71us/step - loss: 0.0542 - acc: 0.9832
Epoch 15/20
40000/40000 [==============================] - 3s 71us/step - loss: 0.0516 - acc: 0.9838
Epoch 16/20
40000/40000 [==============================] - 3s 71us/step - loss: 0.0462 - acc: 0.9861
Epoch 17/20
40000/40000 [==============================] - 3s 71us/step - loss: 0.0465 - acc: 0.9853
Epoch 18/20
40000/40000 [==============================] - 3s 71us/step - loss: 0.0439 - acc: 0.9861
Epoch 19/20
40000/40000 [==============================] - 3s 71us/step - loss: 0.0418 - acc: 0.9868
Epoch 20/20
40000/40000 [==============================] - 3s 71us/step - loss: 0.0396 - acc: 0.9873
20000/20000 [==============================] - 1s 66us/step
40000/40000 [==============================] - 1s 27us/step
Epoch 1/20
40000/40000 [==============================] - 5s 119us/step - loss: 0.7215 - acc: 0.7713
Epoch 2/20
40000/40000 [==============================] - 3s 70us/step - loss: 0.1900 - acc: 0.9434
Epoch 3/20
40000/40000 [==============================] - 3s 70us/step - loss: 0.1305 - acc: 0.9600
Epoch 4/20
40000/40000 [==============================] - 3s 73us/step - loss: 0.1058 - acc: 0.9680
Epoch 5/20
40000/40000 [==============================] - 3s 75us/step - loss: 0.0921 - acc: 0.9717
Epoch 6/20
40000/40000 [==============================] - 3s 73us/step - loss: 0.0811 - acc: 0.9743
Epoch 7/20
40000/40000 [==============================] - 3s 70us/step - loss: 0.0728 - acc: 0.9770
Epoch 8/20
40000/40000 [==============================] - 3s 72us/step - loss: 0.0646 - acc: 0.9800
Epoch 9/20
40000/40000 [==============================] - 3s 71us/step - loss: 0.0588 - acc: 0.9811
Epoch 10/20
40000/40000 [==============================] - 3s 71us/step - loss: 0.0551 - acc: 0.9828
Epoch 11/20
40000/40000 [==============================] - 3s 71us/step - loss: 0.0511 - acc: 0.9840
Epoch 12/20
40000/40000 [==============================] - 3s 71us/step - loss: 0.0481 - acc: 0.9847
Epoch 13/20
40000/40000 [==============================] - 3s 71us/step - loss: 0.0437 - acc: 0.9863
Epoch 14/20
40000/40000 [==============================] - 3s 70us/step - loss: 0.0427 - acc: 0.9866
Epoch 15/20
40000/40000 [==============================] - 3s 71us/step - loss: 0.0391 - acc: 0.9878
Epoch 16/20
40000/40000 [==============================] - 3s 71us/step - loss: 0.0377 - acc: 0.9882
Epoch 17/20
40000/40000 [==============================] - 3s 70us/step - loss: 0.0367 - acc: 0.9886
Epoch 18/20
40000/40000 [==============================] - 3s 71us/step - loss: 0.0344 - acc: 0.9890
Epoch 19/20
40000/40000 [==============================] - 3s 71us/step - loss: 0.0315 - acc: 0.9905
Epoch 20/20
40000/40000 [==============================] - 3s 70us/step - loss: 0.0309 - acc: 0.9903
20000/20000 [==============================] - 1s 72us/step
40000/40000 [==============================] - 1s 28us/step
Epoch 1/10
40000/40000 [==============================] - 4s 92us/step - loss: 0.8380 - acc: 0.7603
Epoch 2/10
40000/40000 [==============================] - 2s 38us/step - loss: 0.2205 - acc: 0.9357
Epoch 3/10
40000/40000 [==============================] - 2s 38us/step - loss: 0.1564 - acc: 0.9537
Epoch 4/10
40000/40000 [==============================] - 2s 38us/step - loss: 0.1256 - acc: 0.9621
Epoch 5/10
40000/40000 [==============================] - 2s 38us/step - loss: 0.1089 - acc: 0.9669
Epoch 6/10
40000/40000 [==============================] - 2s 38us/step - loss: 0.0951 - acc: 0.9718
Epoch 7/10
40000/40000 [==============================] - 2s 38us/step - loss: 0.0858 - acc: 0.9746
Epoch 8/10
40000/40000 [==============================] - 2s 38us/step - loss: 0.0782 - acc: 0.9773
Epoch 9/10
40000/40000 [==============================] - 2s 38us/step - loss: 0.0724 - acc: 0.9784
Epoch 10/10
40000/40000 [==============================] - 1s 37us/step - loss: 0.0679 - acc: 0.9796
20000/20000 [==============================] - 1s 59us/step
40000/40000 [==============================] - 1s 16us/step
Epoch 1/10
40000/40000 [==============================] - 4s 94us/step - loss: 0.8356 - acc: 0.7716
Epoch 2/10
40000/40000 [==============================] - 2s 38us/step - loss: 0.2016 - acc: 0.9409
Epoch 3/10
40000/40000 [==============================] - 2s 39us/step - loss: 0.1408 - acc: 0.9586
Epoch 4/10
40000/40000 [==============================] - 2s 39us/step - loss: 0.1130 - acc: 0.9667
Epoch 5/10
40000/40000 [==============================] - 2s 39us/step - loss: 0.0977 - acc: 0.9710
Epoch 6/10
40000/40000 [==============================] - 2s 38us/step - loss: 0.0878 - acc: 0.9738
Epoch 7/10
40000/40000 [==============================] - 2s 39us/step - loss: 0.0798 - acc: 0.9767
Epoch 8/10
40000/40000 [==============================] - 2s 39us/step - loss: 0.0745 - acc: 0.9773
Epoch 9/10
40000/40000 [==============================] - 2s 39us/step - loss: 0.0695 - acc: 0.9788
Epoch 10/10
40000/40000 [==============================] - 2s 39us/step - loss: 0.0637 - acc: 0.9809
20000/20000 [==============================] - 1s 63us/step
40000/40000 [==============================] - 1s 17us/step
Epoch 1/10
40000/40000 [==============================] - 4s 95us/step - loss: 0.8928 - acc: 0.7352
Epoch 2/10
40000/40000 [==============================] - 1s 37us/step - loss: 0.2237 - acc: 0.9336
Epoch 3/10
40000/40000 [==============================] - 2s 38us/step - loss: 0.1582 - acc: 0.9526
Epoch 4/10
40000/40000 [==============================] - 2s 38us/step - loss: 0.1236 - acc: 0.9621
Epoch 5/10
40000/40000 [==============================] - 1s 37us/step - loss: 0.1016 - acc: 0.9692
Epoch 6/10
40000/40000 [==============================] - 2s 38us/step - loss: 0.0875 - acc: 0.9733
Epoch 7/10
40000/40000 [==============================] - 2s 38us/step - loss: 0.0782 - acc: 0.9761
Epoch 8/10
40000/40000 [==============================] - 2s 38us/step - loss: 0.0717 - acc: 0.9778
Epoch 9/10
40000/40000 [==============================] - 2s 38us/step - loss: 0.0675 - acc: 0.9791
Epoch 10/10
40000/40000 [==============================] - 2s 38us/step - loss: 0.0644 - acc: 0.9804
20000/20000 [==============================] - 1s 62us/step
40000/40000 [==============================] - 1s 16us/step
Epoch 1/20
40000/40000 [==============================] - 4s 98us/step - loss: 0.8769 - acc: 0.7500
Epoch 2/20
40000/40000 [==============================] - 2s 38us/step - loss: 0.2449 - acc: 0.9281
Epoch 3/20
40000/40000 [==============================] - 2s 38us/step - loss: 0.1582 - acc: 0.9548
Epoch 4/20
40000/40000 [==============================] - 2s 38us/step - loss: 0.1216 - acc: 0.9643
Epoch 5/20
40000/40000 [==============================] - 2s 38us/step - loss: 0.1037 - acc: 0.9689
Epoch 6/20
40000/40000 [==============================] - 2s 38us/step - loss: 0.0920 - acc: 0.9725
Epoch 7/20
40000/40000 [==============================] - 2s 38us/step - loss: 0.0829 - acc: 0.9751
Epoch 8/20
40000/40000 [==============================] - 2s 38us/step - loss: 0.0766 - acc: 0.9762
Epoch 9/20
40000/40000 [==============================] - 2s 39us/step - loss: 0.0704 - acc: 0.9780
Epoch 10/20
40000/40000 [==============================] - 2s 38us/step - loss: 0.0659 - acc: 0.9797
Epoch 11/20
40000/40000 [==============================] - 1s 37us/step - loss: 0.0606 - acc: 0.9814
Epoch 12/20
40000/40000 [==============================] - 2s 38us/step - loss: 0.0566 - acc: 0.9835
Epoch 13/20
40000/40000 [==============================] - 2s 38us/step - loss: 0.0552 - acc: 0.9833
Epoch 14/20
40000/40000 [==============================] - 2s 38us/step - loss: 0.0512 - acc: 0.9845
Epoch 15/20
40000/40000 [==============================] - 2s 38us/step - loss: 0.0488 - acc: 0.9855
Epoch 16/20
40000/40000 [==============================] - 1s 37us/step - loss: 0.0473 - acc: 0.9855
Epoch 17/20
40000/40000 [==============================] - 2s 38us/step - loss: 0.0445 - acc: 0.9862
Epoch 18/20
40000/40000 [==============================] - 2s 38us/step - loss: 0.0429 - acc: 0.9867
Epoch 19/20
40000/40000 [==============================] - 1s 37us/step - loss: 0.0407 - acc: 0.9875
Epoch 20/20
40000/40000 [==============================] - 2s 38us/step - loss: 0.0396 - acc: 0.9873
20000/20000 [==============================] - 1s 65us/step
40000/40000 [==============================] - 1s 17us/step
Epoch 1/20
40000/40000 [==============================] - 4s 99us/step - loss: 0.9242 - acc: 0.7323
Epoch 2/20
40000/40000 [==============================] - 2s 38us/step - loss: 0.2147 - acc: 0.9379
Epoch 3/20
40000/40000 [==============================] - 2s 38us/step - loss: 0.1408 - acc: 0.9575
Epoch 4/20
40000/40000 [==============================] - 2s 38us/step - loss: 0.1092 - acc: 0.9675
Epoch 5/20
40000/40000 [==============================] - 1s 37us/step - loss: 0.0924 - acc: 0.9727
Epoch 6/20
40000/40000 [==============================] - 2s 38us/step - loss: 0.0832 - acc: 0.9740
Epoch 7/20
40000/40000 [==============================] - 2s 38us/step - loss: 0.0751 - acc: 0.9776
Epoch 8/20
40000/40000 [==============================] - 2s 38us/step - loss: 0.0680 - acc: 0.9795
Epoch 9/20
40000/40000 [==============================] - 2s 38us/step - loss: 0.0635 - acc: 0.9806
Epoch 10/20
40000/40000 [==============================] - 2s 38us/step - loss: 0.0602 - acc: 0.9814
Epoch 11/20
40000/40000 [==============================] - 2s 38us/step - loss: 0.0566 - acc: 0.9829
Epoch 12/20
40000/40000 [==============================] - 1s 37us/step - loss: 0.0533 - acc: 0.9835
Epoch 13/20
40000/40000 [==============================] - 2s 38us/step - loss: 0.0521 - acc: 0.9847
Epoch 14/20
40000/40000 [==============================] - 2s 38us/step - loss: 0.0487 - acc: 0.9852
Epoch 15/20
40000/40000 [==============================] - 2s 38us/step - loss: 0.0470 - acc: 0.9856
Epoch 16/20
40000/40000 [==============================] - 2s 38us/step - loss: 0.0435 - acc: 0.9865
Epoch 17/20
40000/40000 [==============================] - 2s 38us/step - loss: 0.0435 - acc: 0.9867
Epoch 18/20
40000/40000 [==============================] - 2s 38us/step - loss: 0.0409 - acc: 0.9872
Epoch 19/20
40000/40000 [==============================] - 2s 38us/step - loss: 0.0384 - acc: 0.9882
Epoch 20/20
40000/40000 [==============================] - 1s 37us/step - loss: 0.0384 - acc: 0.9883
20000/20000 [==============================] - 1s 68us/step
40000/40000 [==============================] - 1s 16us/step
Epoch 1/20
40000/40000 [==============================] - 4s 101us/step - loss: 0.8428 - acc: 0.7671
Epoch 2/20
40000/40000 [==============================] - 2s 38us/step - loss: 0.2276 - acc: 0.9327
Epoch 3/20
40000/40000 [==============================] - 2s 38us/step - loss: 0.1594 - acc: 0.9518
Epoch 4/20
40000/40000 [==============================] - 2s 38us/step - loss: 0.1275 - acc: 0.9624
Epoch 5/20
40000/40000 [==============================] - 2s 38us/step - loss: 0.1082 - acc: 0.9666
Epoch 6/20
40000/40000 [==============================] - 1s 37us/step - loss: 0.0958 - acc: 0.9707
Epoch 7/20
40000/40000 [==============================] - 2s 38us/step - loss: 0.0858 - acc: 0.9736
Epoch 8/20
40000/40000 [==============================] - 2s 38us/step - loss: 0.0776 - acc: 0.9760
Epoch 9/20
40000/40000 [==============================] - 2s 38us/step - loss: 0.0735 - acc: 0.9770
Epoch 10/20
40000/40000 [==============================] - 2s 38us/step - loss: 0.0670 - acc: 0.9787
Epoch 11/20
40000/40000 [==============================] - 2s 38us/step - loss: 0.0631 - acc: 0.9807
Epoch 12/20
40000/40000 [==============================] - 2s 38us/step - loss: 0.0591 - acc: 0.9815
Epoch 13/20
40000/40000 [==============================] - 2s 39us/step - loss: 0.0569 - acc: 0.9822
Epoch 14/20
40000/40000 [==============================] - 2s 38us/step - loss: 0.0513 - acc: 0.9842
Epoch 15/20
40000/40000 [==============================] - 2s 38us/step - loss: 0.0506 - acc: 0.9836
Epoch 16/20
40000/40000 [==============================] - 2s 38us/step - loss: 0.0479 - acc: 0.9858
Epoch 17/20
40000/40000 [==============================] - 2s 38us/step - loss: 0.0455 - acc: 0.9854
Epoch 18/20
40000/40000 [==============================] - 2s 38us/step - loss: 0.0420 - acc: 0.9869
Epoch 19/20
40000/40000 [==============================] - 2s 38us/step - loss: 0.0403 - acc: 0.9872
Epoch 20/20
40000/40000 [==============================] - 2s 38us/step - loss: 0.0391 - acc: 0.9875
20000/20000 [==============================] - 1s 66us/step
40000/40000 [==============================] - 1s 17us/step
Epoch 1/10
40000/40000 [==============================] - 4s 105us/step - loss: 0.9641 - acc: 0.6952
Epoch 2/10
40000/40000 [==============================] - 2s 41us/step - loss: 0.2614 - acc: 0.9216
Epoch 3/10
40000/40000 [==============================] - 2s 40us/step - loss: 0.1912 - acc: 0.9419
Epoch 4/10
40000/40000 [==============================] - 2s 40us/step - loss: 0.1533 - acc: 0.9533
Epoch 5/10
40000/40000 [==============================] - 2s 41us/step - loss: 0.1300 - acc: 0.9603
Epoch 6/10
40000/40000 [==============================] - 2s 40us/step - loss: 0.1146 - acc: 0.9647
Epoch 7/10
40000/40000 [==============================] - 2s 40us/step - loss: 0.1041 - acc: 0.9683
Epoch 8/10
40000/40000 [==============================] - 2s 41us/step - loss: 0.0951 - acc: 0.9704
Epoch 9/10
40000/40000 [==============================] - 2s 40us/step - loss: 0.0876 - acc: 0.9721
Epoch 10/10
40000/40000 [==============================] - 2s 41us/step - loss: 0.0789 - acc: 0.9748
20000/20000 [==============================] - 1s 72us/step
40000/40000 [==============================] - 1s 17us/step
Epoch 1/10
40000/40000 [==============================] - 4s 109us/step - loss: 0.9541 - acc: 0.7025
Epoch 2/10
40000/40000 [==============================] - 2s 41us/step - loss: 0.2419 - acc: 0.9285
Epoch 3/10
40000/40000 [==============================] - 2s 40us/step - loss: 0.1646 - acc: 0.9518
Epoch 4/10
40000/40000 [==============================] - 2s 41us/step - loss: 0.1324 - acc: 0.9601
Epoch 5/10
40000/40000 [==============================] - 2s 41us/step - loss: 0.1135 - acc: 0.9653
Epoch 6/10
40000/40000 [==============================] - 2s 41us/step - loss: 0.0987 - acc: 0.9701
Epoch 7/10
40000/40000 [==============================] - 2s 41us/step - loss: 0.0896 - acc: 0.9721
Epoch 8/10
40000/40000 [==============================] - 2s 42us/step - loss: 0.0824 - acc: 0.9741
Epoch 9/10
40000/40000 [==============================] - 2s 40us/step - loss: 0.0742 - acc: 0.9768
Epoch 10/10
40000/40000 [==============================] - 2s 40us/step - loss: 0.0704 - acc: 0.9774
20000/20000 [==============================] - 1s 69us/step
40000/40000 [==============================] - 1s 17us/step
Epoch 1/10
40000/40000 [==============================] - 4s 110us/step - loss: 0.9033 - acc: 0.7302
Epoch 2/10
40000/40000 [==============================] - 2s 41us/step - loss: 0.2021 - acc: 0.9410
Epoch 3/10
40000/40000 [==============================] - 2s 40us/step - loss: 0.1412 - acc: 0.9575
Epoch 4/10
40000/40000 [==============================] - 2s 40us/step - loss: 0.1164 - acc: 0.9645
Epoch 5/10
40000/40000 [==============================] - 2s 40us/step - loss: 0.1015 - acc: 0.9694
Epoch 6/10
40000/40000 [==============================] - 2s 41us/step - loss: 0.0893 - acc: 0.9734
Epoch 7/10
40000/40000 [==============================] - 2s 41us/step - loss: 0.0820 - acc: 0.9750
Epoch 8/10
40000/40000 [==============================] - 2s 40us/step - loss: 0.0772 - acc: 0.9765
Epoch 9/10
40000/40000 [==============================] - 2s 40us/step - loss: 0.0705 - acc: 0.9782
Epoch 10/10
40000/40000 [==============================] - 2s 41us/step - loss: 0.0644 - acc: 0.9812
20000/20000 [==============================] - 1s 74us/step
40000/40000 [==============================] - 1s 17us/step
Epoch 1/20
40000/40000 [==============================] - 5s 113us/step - loss: 1.0899 - acc: 0.6421
Epoch 2/20
40000/40000 [==============================] - 2s 41us/step - loss: 0.3275 - acc: 0.8991
Epoch 3/20
40000/40000 [==============================] - 2s 40us/step - loss: 0.2198 - acc: 0.9344
Epoch 4/20
40000/40000 [==============================] - 2s 41us/step - loss: 0.1674 - acc: 0.9509
Epoch 5/20
40000/40000 [==============================] - 2s 41us/step - loss: 0.1397 - acc: 0.9582
Epoch 6/20
40000/40000 [==============================] - 2s 41us/step - loss: 0.1189 - acc: 0.9652
Epoch 7/20
40000/40000 [==============================] - 2s 41us/step - loss: 0.1057 - acc: 0.9690
Epoch 8/20
40000/40000 [==============================] - 2s 41us/step - loss: 0.0948 - acc: 0.9722
Epoch 9/20
40000/40000 [==============================] - 2s 41us/step - loss: 0.0865 - acc: 0.9747
Epoch 10/20
40000/40000 [==============================] - 2s 40us/step - loss: 0.0813 - acc: 0.9760
Epoch 11/20
40000/40000 [==============================] - 2s 41us/step - loss: 0.0755 - acc: 0.9772
Epoch 12/20
40000/40000 [==============================] - 2s 41us/step - loss: 0.0694 - acc: 0.9795
Epoch 13/20
40000/40000 [==============================] - 2s 41us/step - loss: 0.0685 - acc: 0.9782
Epoch 14/20
40000/40000 [==============================] - 2s 41us/step - loss: 0.0622 - acc: 0.9811
Epoch 15/20
40000/40000 [==============================] - 2s 40us/step - loss: 0.0596 - acc: 0.9822
Epoch 16/20
40000/40000 [==============================] - 2s 41us/step - loss: 0.0564 - acc: 0.9834
Epoch 17/20
40000/40000 [==============================] - 2s 41us/step - loss: 0.0563 - acc: 0.9820
Epoch 18/20
40000/40000 [==============================] - 2s 41us/step - loss: 0.0535 - acc: 0.9838
Epoch 19/20
40000/40000 [==============================] - 2s 40us/step - loss: 0.0509 - acc: 0.9839
Epoch 20/20
40000/40000 [==============================] - 2s 41us/step - loss: 0.0467 - acc: 0.9854
20000/20000 [==============================] - 2s 75us/step
40000/40000 [==============================] - 1s 16us/step
Epoch 1/20
40000/40000 [==============================] - 5s 116us/step - loss: 1.1492 - acc: 0.6313
Epoch 2/20
40000/40000 [==============================] - 2s 41us/step - loss: 0.3404 - acc: 0.8973
Epoch 3/20
40000/40000 [==============================] - 2s 40us/step - loss: 0.2344 - acc: 0.9301
Epoch 4/20
40000/40000 [==============================] - 2s 41us/step - loss: 0.1830 - acc: 0.9445
Epoch 5/20
40000/40000 [==============================] - 2s 41us/step - loss: 0.1450 - acc: 0.9556
Epoch 6/20
40000/40000 [==============================] - 2s 41us/step - loss: 0.1248 - acc: 0.9611
Epoch 7/20
40000/40000 [==============================] - 2s 41us/step - loss: 0.1097 - acc: 0.9667
Epoch 8/20
40000/40000 [==============================] - 2s 41us/step - loss: 0.0985 - acc: 0.9695
Epoch 9/20
40000/40000 [==============================] - 2s 41us/step - loss: 0.0909 - acc: 0.9722
Epoch 10/20
40000/40000 [==============================] - 2s 41us/step - loss: 0.0832 - acc: 0.9745
Epoch 11/20
40000/40000 [==============================] - 2s 40us/step - loss: 0.0791 - acc: 0.9752
Epoch 12/20
40000/40000 [==============================] - 2s 41us/step - loss: 0.0716 - acc: 0.9774
Epoch 13/20
40000/40000 [==============================] - 2s 41us/step - loss: 0.0684 - acc: 0.9780
Epoch 14/20
40000/40000 [==============================] - 2s 41us/step - loss: 0.0641 - acc: 0.9800
Epoch 15/20
40000/40000 [==============================] - 2s 40us/step - loss: 0.0636 - acc: 0.9799
Epoch 16/20
40000/40000 [==============================] - 2s 41us/step - loss: 0.0597 - acc: 0.9810
Epoch 17/20
40000/40000 [==============================] - 2s 41us/step - loss: 0.0555 - acc: 0.9827
Epoch 18/20
40000/40000 [==============================] - 2s 40us/step - loss: 0.0528 - acc: 0.9832
Epoch 19/20
40000/40000 [==============================] - 2s 41us/step - loss: 0.0533 - acc: 0.9828
Epoch 20/20
40000/40000 [==============================] - 2s 42us/step - loss: 0.0514 - acc: 0.9837
20000/20000 [==============================] - 2s 80us/step
40000/40000 [==============================] - 1s 17us/step
Epoch 1/20
40000/40000 [==============================] - 5s 118us/step - loss: 0.9539 - acc: 0.6981
Epoch 2/20
40000/40000 [==============================] - 2s 41us/step - loss: 0.2079 - acc: 0.9384
Epoch 3/20
40000/40000 [==============================] - 2s 41us/step - loss: 0.1430 - acc: 0.9579
Epoch 4/20
40000/40000 [==============================] - 2s 41us/step - loss: 0.1160 - acc: 0.9649
Epoch 5/20
40000/40000 [==============================] - 2s 41us/step - loss: 0.0990 - acc: 0.9706
Epoch 6/20
40000/40000 [==============================] - 2s 40us/step - loss: 0.0862 - acc: 0.9736
Epoch 7/20
40000/40000 [==============================] - 2s 40us/step - loss: 0.0757 - acc: 0.9771
Epoch 8/20
40000/40000 [==============================] - 2s 41us/step - loss: 0.0691 - acc: 0.9793
Epoch 9/20
40000/40000 [==============================] - 2s 41us/step - loss: 0.0647 - acc: 0.9804
Epoch 10/20
40000/40000 [==============================] - 2s 40us/step - loss: 0.0592 - acc: 0.9820
Epoch 11/20
40000/40000 [==============================] - 2s 42us/step - loss: 0.0546 - acc: 0.9838
Epoch 12/20
40000/40000 [==============================] - 2s 41us/step - loss: 0.0508 - acc: 0.9848
Epoch 13/20
40000/40000 [==============================] - 2s 41us/step - loss: 0.0469 - acc: 0.9861
Epoch 14/20
40000/40000 [==============================] - 2s 42us/step - loss: 0.0456 - acc: 0.9864
Epoch 15/20
40000/40000 [==============================] - 2s 42us/step - loss: 0.0433 - acc: 0.9871
Epoch 16/20
40000/40000 [==============================] - 2s 42us/step - loss: 0.0426 - acc: 0.9866
Epoch 17/20
40000/40000 [==============================] - 2s 42us/step - loss: 0.0395 - acc: 0.9877
Epoch 18/20
40000/40000 [==============================] - 2s 42us/step - loss: 0.0388 - acc: 0.9879
Epoch 19/20
40000/40000 [==============================] - 2s 42us/step - loss: 0.0359 - acc: 0.9888
Epoch 20/20
40000/40000 [==============================] - 2s 42us/step - loss: 0.0334 - acc: 0.9900
20000/20000 [==============================] - 2s 82us/step
40000/40000 [==============================] - 1s 18us/step
Epoch 1/20
60000/60000 [==============================] - 11s 190us/step - loss: 0.4142 - acc: 0.8725
Epoch 2/20
60000/60000 [==============================] - 8s 139us/step - loss: 0.1258 - acc: 0.9618
Epoch 3/20
60000/60000 [==============================] - 8s 139us/step - loss: 0.0919 - acc: 0.9721
Epoch 4/20
60000/60000 [==============================] - 9s 143us/step - loss: 0.0740 - acc: 0.9774
Epoch 5/20
60000/60000 [==============================] - 8s 139us/step - loss: 0.0655 - acc: 0.9800
Epoch 6/20
60000/60000 [==============================] - 8s 139us/step - loss: 0.0570 - acc: 0.9830
Epoch 7/20
60000/60000 [==============================] - 8s 138us/step - loss: 0.0506 - acc: 0.9854
Epoch 8/20
60000/60000 [==============================] - 8s 139us/step - loss: 0.0454 - acc: 0.9860
Epoch 9/20
60000/60000 [==============================] - 8s 139us/step - loss: 0.0440 - acc: 0.9863
Epoch 10/20
60000/60000 [==============================] - 8s 139us/step - loss: 0.0385 - acc: 0.9877
Epoch 11/20
60000/60000 [==============================] - 8s 138us/step - loss: 0.0361 - acc: 0.9886
Epoch 12/20
60000/60000 [==============================] - 8s 139us/step - loss: 0.0324 - acc: 0.9897
Epoch 13/20
60000/60000 [==============================] - 8s 140us/step - loss: 0.0307 - acc: 0.9903
Epoch 14/20
60000/60000 [==============================] - 8s 139us/step - loss: 0.0284 - acc: 0.9912
Epoch 15/20
60000/60000 [==============================] - 8s 139us/step - loss: 0.0257 - acc: 0.9917
Epoch 16/20
60000/60000 [==============================] - 8s 141us/step - loss: 0.0251 - acc: 0.9916
Epoch 17/20
60000/60000 [==============================] - 8s 142us/step - loss: 0.0234 - acc: 0.9923
Epoch 18/20
60000/60000 [==============================] - 8s 139us/step - loss: 0.0216 - acc: 0.9931
Epoch 19/20
60000/60000 [==============================] - 8s 139us/step - loss: 0.0200 - acc: 0.9936
Epoch 20/20
60000/60000 [==============================] - 8s 139us/step - loss: 0.0193 - acc: 0.9936

print('Best accuracy: {0:.4f}, Best Params: {1}'.format(grid_result.best_score_,
                                                        grid_result.best_params_))

Best accuracy: 0.9830, Best Params: {'batch_size': 64, 'dense_layers': [50, 20], 'epochs': 20}

Evaluating Model with Optimized Hyperparameters on Test Data Set

model = create_mnist_cnn_model(dense_layers=grid_result.best_params_['dense_layers'])
history = model.fit(train_images, train_labels,
                    epochs=grid_result.best_params_['epochs'],
                    batch_size=grid_result.best_params_['batch_size'],
                    verbose = 0)

fig, axes = plt.subplots(1,2, figsize=(14,5))
axes[0].plot(history.history['loss'])
axes[0].set_title('Model Loss')
axes[0].set_xlabel('Epoch')
axes[0].set_ylabel('Loss')
axes[0].legend(['Training', 'Validation'])
axes[1].plot(history.history['acc'])
axes[1].set_title('Model Accuracy')
axes[1].set_xlabel('Epoch')
axes[1].set_ylabel('Accuracy')
axes[1].legend(['Training', 'Validation']);

scores = model.evaluate(test_images, test_labels, verbose=0)
print("Accuracy: {0} Error: {1:.4f}".format(scores[1], (1-scores[1])*100))

Accuracy: 0.9858 Error: 1.4199

Conclusion

In this notebook, we attempted to improve the classification accuracy achieved on the MNIST digits data set. In another notebook, MNIST Using Keras, we built a dense shallow network that achieved 98.06% accuracy.

In this notebook, we used the structure present in images by implementing a convolutional neural network. Using some sensible choices and values for the network structure and hyperparameters respectively, we achieved an accuracy of 98.39%.

Finally, we optimized some of the hyperparameters using a grid search to find values which improve the accuracy further. To keep the grid search reasonable we only optimized three hyperparameters, namely, batch size, number of epochs and the number of dense layers. By fitting a convolutional neural network with these optimized hyperparameters on the full training set, we achieved an accuracy of 98.58% on the test set.

Further optimization is possible to achieve even higher accuracy, but keep in mind that improvements above the 98% level require lots of fine tuning. Refer to Yann LeCun’s MNIST page for details of other approaches and the test error rate achieved.

Irrespective of the final accuracy achieved by the model, in this notebook you learned how to build, train and evaluate convolutional neural networks using Keras and how to optimize hyperparameters using grid search.